2 The purpose of this document is to introduce the reader with vkernel debugging.
3 The vkernel architecture allows us to run DragonFly kernels in userland. These virtual
4 kernels can be paniced or otherwise abused, without affecting the host operating system.
6 To make things a bit more interesting, we will use a real life example.
9 ... I wrote a simple program that used the AIO interface. As it turned out we don't support
10 this feature, but at that point I didn't know.
12 [beket@sadness ~]$ gcc t_aio.c -o t_aio -Wall -ansi -pedantic
13 [beket@sadness ~]$ ./t_aio
14 aio_read: Function not implemented
17 Ktrace'ing the process and seeing with my own eyes what was going on, seemed like a good idea.
18 Here comes the fun. I misread the ktrace(1) man page and typed:
20 [beket@sadness ~]$ ktrace -c ./t_aio
23 (My intention was to track the system calls of t_aio.)
26 To setup a vkernel, please consult this [man page](http://leaf.dragonflybsd.org/cgi/web-man?command=vkernel§ion=ANY).
27 It's very straightforward.
29 # Reproduce the problem
30 We boot into our vkernel:
33 # ./boot/kernel -m 64m -r rootimg.01 -I auto:bridge0
37 And then try to reproduce the system freeze:
41 Fatal trap 12: page fault while in kernel mode
42 mp_lock = 00000001; cpuid = 1
43 fault virtual address = 0x0
44 fault code = supervisor read, page not present
45 instruction pointer = 0x1f:0x80aca52
46 stack pointer = 0x10:0x5709d914
47 frame pointer = 0x10:0x5709dbe0
48 processor eflags = interrupt enabled, resume, IOPL = 0
49 current process = 692 (ktrace)
50 current thread = pri 6
52 kernel: type 12 trap, code=4
54 CPU1 stopping CPUs: 0x00000001
56 Stopped at 0x80aca52: movl 0(%eax),%eax
59 This db> prompt is from ddb(4), the interactive kernel debugger.
62 fault virtual address = 0x0
64 field is indicative of a NULL pointer dereference inside the kernel.
66 Let's get a trace of what went wrong:
69 ktrdestroy(57082700,5709dc5c,0,57082700,5709dca0) at 0x80aca52
70 allproc_scan(80aca14,5709dc5c,be,2,0) at 0x80b2e91
71 sys_ktrace(5709dca0,6,0,0,57082700) at 0x80acffe
72 syscall2(5709dd40,6,57082700,0,0) at 0x8214b6d
73 user_trap(5709dd40,570940e8,8214185,0,8215462) at 0x8214d9c
74 go_user(5709dd38,0,0,7b,0) at 0x82151ac
78 We attach to the vkernel process:
80 # ps aux | grep kernel
81 root 25408 0.0 2.3 1053376 17772 p0 IL+ 8:32PM 0:06.51 ./boot/kernel -m 64m -r rootimg.01 -I auto:bridge0
86 Let's get a trace from inside gdb:
89 #0 0x282d4c10 in sigsuspend () from /usr/lib/libc.so.6
90 #1 0x28287eb2 in sigsuspend () from /usr/lib/libthread_xu.so.2
91 #2 0x0821530a in stopsig (nada=24, info=0x40407d2c, ctxp=0x40407a4c) at /usr/src/sys/platform/vkernel/i386/exception.c:112
92 #3 <signal handler called>
93 #4 0x282d4690 in umtx_sleep () from /usr/lib/libc.so.6
94 #5 0x08213bde in cpu_idle () at /usr/src/sys/platform/vkernel/i386/cpu_regs.c:722
95 #6 0x00000000 in ?? ()
98 Why does it differ from the ddb's trace ?
99 Well, when the vkernel is sitting at a db> prompt all vkernel threads representing virtual cpu's except the one handling the db> prompt itself will be suspended in stopsig(). The backtrace only sees one of the N threads.
101 We need to do better this time. Let's break into the kernel _before_ it crashes. sys_ktrace() seems like a good candidate.
106 (gdb) break sys_ktrace
107 Breakpoint 1 at 0x80acf43: file ./machine/thread.h, line 83.
110 Next we type 'c' in the gdb prompt to resume vkernel execution:
115 Now we go to our vkernel and type the offending command:
119 Gdb stops the execution of vkernel and a message pops up in gdb buffer:
121 Breakpoint 1, sys_ktrace (uap=0x573e2ca0) at ./machine/thread.h:83
122 83 __asm ("movl %%fs:globaldata,%0" : "=r" (gd) : "m"(__mycpu__dummy));
125 We navigate through source code with the 'step' and 'next' gdb commands. They are identical, except that 'step' follows function calls. When we meet this call:
127 276 allproc_scan(ktrace_clear_callback, &info);
129 we 'step' inside it. alloproc_scan() iterates through the process list and applies the ktrace_clear_callback() to each one of them.
132 347 if (p->p_tracenode->kn_vp == info->tracenode->kn_vp) {
134 Here p is a pointer to the current process:
136 $1 = (struct proc *) 0x57098c00
138 Let's see if this process is traced:
139 (gdb) print p->p_tracenode
140 $2 = (struct ktrace_node *) 0x0
143 Oops. There is no trace to a vnode for this process.