![[BACK]](/cvsweb/icons/back.gif){kind=icon}
Return to udb.t CVS log ![[TXT]](/cvsweb/icons/text.gif){kind=icon}
![[DIR]](/cvsweb/icons/dir.gif){kind=icon}
Up to [Research Unix] / researchv8dc / cmd / adb|
Return to udb.t CVS log | Up to [Research Unix] / researchv8dc / cmd / adb |
1.1 root 1: .po .5i
2: .bd S B 3
3: .de UX
4: .ie \\n(GA>0 \\$2UNIX\\$1
5: .el \{\
6: .if n \\$2UNIX\\$1*
7: .if t \\$2UNIX\\$1\\f1\(dg\\fP
8: .FS
9: .if n *UNIX
10: .if t \(dgUNIX
11: .ie \\$3=1 is a Footnote of Bell Laboratories.
12: .el is a Trademark of Bell Laboratories.
13: .FE
14: .nr GA 1\}
15: ..
16: .TL
17: Using ADB to Debug the
18: .UX
19: Kernel
20: .AU
21: Samuel J. Leffler
22: .AU
23: William N. Joy
24: .AI
25: Computer Systems Research Group
26: Department of Electrical Engineering and Computer Science
27: University of California, Berkeley
28: Berkeley, California 94720
29: (415) 642-7780
30: .de IR
31: \fI\\$1\fP\\$2
32: ..
33: .de DT
34: .TA 8 16 24 32 40 48 56 64 72 80
35: ..
36: .AB
37: .PP
38: .FS
39: *DEC and VAX are trademarks of
40: Digital Equipment Corporation.
41: .FE
42: This document describes the use of extensions made
43: to the 4.1bsd release of the VAX*
44: .UX
45: debugger adb for the purpose of debugging the
46: .UX
47: kernel.
48: It discusses the changes made to allow
49: standard adb commands to
50: function properly with the kernel and
51: introduces the basics necessary for users
52: to write adb command scripts which
53: may be used to augment the standard adb
54: command set. The examination techniques described here
55: may be applied to running systems, as well as
56: the post-mortem dumps automatically created
57: by the
58: .IR savecore (8)
59: program after a system crash.
60: The reader is expected to have at least a
61: passing familiarity with the debugger command
62: language.
63: .AE
64: .ds LH "Using ADB on the UNIX Kernel
65: .ds RH Introduction
66: .ds CF \*(DY
67: .LP
68: .nr H1 1
69: .ds CH "
70: .bp
71: .nr % 1
72: .ds CH "\(hy \\n(PN \(hy
73: .LG
74: .B
75: .ce
76: 1. INTRODUCTION
77: .sp 2
78: .R
79: .NL
80: .PP
81: Modifications have been made to the
82: standard VAX
83: .UX
84: debugger adb to simplify
85: examination of post-mortem
86: dumps automatically generated following a system crash.
87: These changes may also be used when examining
88: .UX
89: in its normal operation.
90: This document serves as
91: an introduction to the \fBuse\fP of
92: these facilities, and
93: should not be construed as a description
94: of \fIhow to debug the kernel\fP.
95: .NH 2
96: Invocation
97: .PP
98: When examining the
99: .UX
100: kernel a new option, \fB\-k\fP, should be used, e.g.
101: .sp 1
102: .ti +5
103: adb \-k /vmunix /dev/mem
104: .sp 1
105: This flag causes adb to partially simulate
106: the VAX virtual memory hardware when
107: accessing the \fIcore\fP file.
108: In addition the internal state maintained
109: by the debugger is initialized from
110: data structures maintained by the
111: .UX
112: kernel explicitly for debugging\(dd.
113: A post-mortem dump may be examined in a similar
114: fashion,
115: .sp 1
116: .ti +5
117: adb \-k vmunix.? vmcore.?
118: .sp 1
119: where the appropriate version of the saved operating system
120: image and core dump are supplied in place of ``?''.
121: .FS
122: \(dd If the \-k flag is not used when invoking
123: adb the user must explicitly calculate virtual
124: addresses. With the \-k option adb interprets
125: page tables to automatically
126: perform virtual to physical address translation.
127: .FE
128: .NH 2
129: Establishing Context
130: .PP
131: During initialization adb attempts to establish the
132: context of the ``currently active process'' by examining
133: the value of the kernel variable \fImasterpaddr\fP.
134: This variable contains the virtual address of the
135: process context block of the last process which
136: was set executing by the \fISwtch\fP routine.
137: \fImasterpaddr\fP normally provides sufficient information
138: to locate the current stack frame (via the stack
139: pointers found in the context block).
140: By locating the VAX process context block for the
141: process, adb may then perform virtual to
142: physical address translation using that process's
143: in-core page tables.
144: .PP
145: When examining post-mortem dumps establishing the
146: context of the ``currently active process'' is nontrivial.
147: This is due to the different ways in which the
148: VAX may save state after a nonrecoverable error.
149: Crashes may or may not be ``clean'' (i.e.
150: the top of the interrupt stack contains the process's
151: kernel mode stack pointer and program counter);
152: an ``unclean'' crash will occur, for instance,
153: if the interrupt stack overflows.
154: Thus, one must manually try one of two possible techniques
155: to get a stack trace from a post-mortem dump. If the
156: crash was clean the current stack frame is present at the
157: ``top'' of the interrupt stack, \fIintstack\-4\fP,
158: and the command
159: .sp 1
160: .ti +5
161: *(\fBintstack\fP\-4)$c
162: .sp 1
163: will generate a stack trace all the way from the kernel
164: to the top of the user process's stack (e.g. to the
165: \fImain\fP routine in the user process which was running).
166: Otherwise, one must scan through the interrupt stack
167: looking for the stack frame. This is usually indicated
168: by a zero longword entry (the argument count),
169: .sp 1
170: .in +5
171: .nf
172: \fBintstack\fP/\fBX\fP
173: .fi
174: .in -5
175: .sp 1
176: Once the stack frame has been located, the command
177: .sp 1
178: .ti +5
179: \fB*.$c\fP
180: .sp 1
181: will generate a stack trace.
182: An alternate method may be used when a trace of a particular
183: process is required, see section 2.3.
184: .ds RH "Command Scripts
185: .LP
186: .nr H1 2
187: .bp
188: .LG
189: .B
190: .ce
191: 2. ADB COMMAND SCRIPTS
192: .sp 2
193: .R
194: .nr H2 0
195: .NL
196: .NH 2
197: Extending the Formatting Facilities
198: .PP
199: Once the process context has been established, the
200: complete adb command set is available for interpreting
201: data structures. In addition, a number of adb scripts have
202: been created to simplify the structured printing of commonly
203: referenced kernel data structures. The scripts normally
204: reside in
205: the directory \fI/usr/lib/adb\fP, and are invoked
206: with the ``$<'' operator.
207: (A later table lists the ``standard'' scripts.)
208: .PP
209: As an example, consider the following listing which
210: contains a dump of a faulty process's state
211: (our typing is shown emboldened).
212: .sp 1
213: .nf
214: .DT
215: % \fBadb \-k vmunix.17 vmcore.17\fP
216: sbr 8001d064 slr d9c
217: p0br 800efa00 p0lr 34 p1br 7f8efe00 p1lr 1ffff2
218: \fB*(intstack\-4)$c\fP
219: _boot() from 80004025
220: _boot(0,4) from 80004025
221: _panic(80021185) from 800057e2
222: _soreceive(8017478c,0) from 80007c90
223: _read() from 800098d7
224: _syscall() from 8000b6e2
225: _Xsyscall(3,7fffe834,258) from 80000f64
226: ?() from c1c
227: ?() from 26a
228: ?(0,7fffef18,7fffef1c) from 1d3
229: ?() from 2f
230: \fB800021185/s\fP
231: _icpreg+99: receive
232: \fBu$<u\fP
233: _u:
234: _u: ksp usp
235: 7fffff9c 7fffe59c
236: r0 r1 r2 r3
237: 155c00 800237d4 80041800 3
238: r4 r5 r6 r7
239: 0 0 11090 80041800
240: r8 r9 r10 r11
241: 80021244 c 7fffe5b4 80000000
242: ap fp pc psl
243: 7fffffe8 7fffffa4 8000b784 d80004
244: p0br p0lr p1br p1lr
245: 800efa00 4000034 7f8efe00 1ffff2
246: szpt cmap2 sswap
247: 2 94000307 0
248: sigc1 sigc2 sigc3
249: 1af03fb fa007f02 40cbc6c
250: _u+78: arg0 arg1 arg2
251: 3 7fffe834 258
252: _u+8c: segflg error uid gid ruid rgid procp
253: 0 0 4 a 4 a 80041800
254:
255: _u+d4: uap rv1 rv2 ubase
256: 7ffff078 0 1 7fffe834
257: count off cdir rdir
258: 258 150 8003cf00 0
259: _u+f4: pathname
260: .netrc
261: dirp dino entry pdir
262: 3 1395 .netrc0
263: 7ffff11c: ofiles
264: 80040818 80040818 80040818 800406b0
265: 800406d4 800406ec 0 0
266: 0 0 0 0
267: 0 0 0 0
268: 0 0 0 0
269:
270: ofileflg
271: 0 0 0 0 0 0 0 0
272: 0 0 0 0 0 0 0 0
273: 0 0 0 0
274: 7ffff180: sigs
275: 0 360c 1 360c
276: 0 0 0 aae
277: 0 0 0 0
278: 0 0 0 0
279: 0 0 0 0
280: 1 0 0 0
281: 0 0 0 0
282: 0 0 0 0
283:
284: code ar0 prbase prsize
285: 0 80000000 0 0
286:
287: .ne 2
288: 7ffff248: proff prscal eosys sep ttyp
289: 0 0 0 0 800288b4
290:
291: 7ffff258: ttymin ttymaj
292: 0 0
293: 7ffff25e: xmag xtsiz xdsiz xbsiz
294: 3c000000 10000000 108c0000 a680000
295:
296: xssiz entloc relflg
297: 0 0 6c720000
298: 7ffff27e: directory
299: ogin
300: start acflg fpflg cmsk tsiz dsiz
301: 11688 0 12 0 160000 60000
302:
303: 7ffff2a2: ssiz
304: 80000
305: \fB80041800$<proc\fP
306: 80041800: link rlink addr
307: 800237d4 0 800efde0
308: 8004180c: upri pri cpu stat time nice slp cursig
309: 073 073 045 03 023 024 0 0
310: 80041814: sig siga0 siga1 flag
311: 0 80002 45 8001
312: 80041824: uid pgrp pid ppid poip szpt tsize
313: 4 bb bc bb 0 2 1e
314: 80041834: dsize ssize rssize maxrss
315: 16 6 14 3fffff
316: 80041844: swrss swaddr wchan textp
317: 0 0 0 80044ee0
318: 80041854: clktim p0br xlink ticks
319: 0 800efa00 80041720 22
320: 80041864: %cpu ndx idhash pptr
321: +5.1369253545999527e\-02 1c 8 80041720
322: \fB80044ee0$<text\fP
323: 80044ee0: daddr
324: 7e2 0 0 0
325: 0 0 0 0
326: 0 0 0 0
327:
328: ptdaddr size caddr iptr
329: 352 1e 80041800 8003cfa0
330:
331: rssize swrss count ccount flag slptim poip
332: 1a 0 02 02 042 0 0
333: .sp 1
334: .fi
335: .PP
336: The cause of the crash was a ``panic''
337: (see the stack trace) due to the 0
338: argument passed the \fIsoreceive\fP routine. The majority
339: of the dump was done to illustrate the use of two command
340: scripts used to format kernel data structures. The ``u''
341: script, invoked by the command ``u$<u'', is a lengthy series
342: of commands which pretty-prints the user vector. Likewise,
343: ``proc'' and ``text'' are scripts used to format the obvious
344: data structures. Let's quickly examine the ``text'' script (the
345: script has been broken into a number of lines for convenience
346: here; in actuality it is a single line of text).
347: .sp 1
348: .nf
349: \&./"daddr"n12Xn\e
350: "ptdaddr"16t"size"16t"caddr"16t"iptr"n4Xn\e
351: "rssize"8t"swrss"8t"count"8t"ccount"8t"flag"8t"slptim"8t"poip"n2x4bx++n
352: .sp 1
353: .fi
354: The first line produces the list of disk block addresses associated
355: with a swapped out text segment. The ``n'' format forces a new-line
356: character, with 12 hexadecimal integers printed immediately after.
357: Likewise, the remaining two lines of the command format the remainder
358: of the text structure. The expression ``16t'' causes adb to tab
359: to the next column which is a multiple of 16.
360: The last two plus operators are present
361: to round ``.'' to the end of the text structure. This allows the
362: user to reinvoke the format on consecutive text structures without
363: having to be concerned about proper alignment of ``.''.
364: .PP
365: The majority of the scripts provided are of this nature.
366: When possible, the formatting scripts print a data structure
367: with a single format to allow subsequent reuse when interrogating
368: arrays of structures. That is, the previous script could have
369: been written
370: .sp 1
371: .nf
372: \&./"daddr"n12Xn
373: +/"ptdaddr"16t"size"16t"caddr"16t"iptr"n4Xn
374: +/"rssize"8t"swrss"8t"count"8t"ccount"8t"flag"8t"slptim"8t"poip"n2x4bx++n
375: .sp 1
376: .fi
377: but then reuse of the format would have invoked only the last
378: line of the format.
379: .NH 2
380: Traversing Data Structures
381: .PP
382: The adb command language can be used to traverse complex data
383: structures. One such data structure, a linked list, occurs
384: quite often in the kernel. By using adb variables and the
385: normal expression operators it is a simple matter to construct
386: a script which chains down the list printing each element
387: along the way.
388: .PP
389: For instance, the queue of processes awaiting timer events,
390: the callout queue, is printed with the following two scripts:
391: .sp 1
392: .nf
393: .ne 4
394: \fBcallout\fP:
395: .in +5
396: .sp 1
397: calltodo/"time"16t"arg"16t"func"12+
398: *+$<callout.next
399: .sp 1
400: .ne 6
401: .ti -5
402: \fBcallout.next\fP:
403: .sp 1
404: \&./Dpp
405: *+>l
406: ,#<l$<
407: <l$<callout.next
408: .sp 1
409: .in -5
410: .fi
411: .PP
412: The first line of the script \fBcallout\fP starts the traversal
413: at the global symbol
414: \fIcalltodo\fP and prints a set of headings.
415: It then skips the empty portion of the structure used
416: as the head of the queue.
417: The second line then invokes the script \fBcallout.next\fP
418: moving ``.'' to
419: the top of the queue (``*+'' performs the indirection
420: through the link entry of the structure at the head of the queue).
421: .PP
422: \fBcallout.next\fP prints values for each column, then performs
423: a conditional test on the link to the next entry. This test
424: is performed as follows,
425: .IP "*+>l" 9
426: Place the value of the ``link'' in the adb variable ``<l''.
427: .IP ",#<l$<"
428: If the value stored in ``<l'' is non-zero, then the current
429: input stream (i.e. the script \fBcallout.next\fP) is terminated.
430: Otherwise, the expression ``#<l'' will be zero, and the ``$<''
431: will be ignored. That is, the combination of the logical negation
432: operator ``#'', adb variable ``<l'', and ``$<'' operator
433: creates a statement of the form,
434: .sp 1
435: .ce
436: if (!link) exit;
437: .sp 1
438: The remaining line of \fBcallout.next\fP simply reapplies the
439: script on the next element in the linked list.
440: .LP
441: A sample \fIcallout\fP dump is shown below.
442: .nf
443: .sp 1
444: .ne 14
445: % \fBadb \-k /vmunix /dev/mem\fP
446: sbr 8001f864 slr d9c
447: p0br 800efa00 p0lr 8e p1br 7f8efe00 p1lr 1ffff2
448: \fB$<callout\fP
449: _calltodo:
450: _calltodo: time arg func
451: 8004ecfc: 26 0 _dzscan
452: 8004ed0c: 8 0 _upwatch
453: 8004ed1c: 0 0 _ip_timeo
454: 8004ed5c: 0 0 _tcp_timeo
455: 8004ed6c: 0 0 _rkwatch
456: 8004ecfc: 52 0 _dzscan
457: 8004ed2c: 68 _Syssize+70 _tmtimer
458: 8004ed3c: 2920 0 _memenable
459: .fi
460: .sp 1
461: .NH 2
462: Supplying Parameters
463: .PP
464: If one is clever, a command script may use the address
465: and count portions of an adb command as parameters. An example of
466: this is the \fBsetproc\fP script used to switch to the
467: context of a process with a known process-id;
468: .sp 1
469: .ti +5
470: \fB99$<setproc\fP
471: .sp 1
472: The body of \fBsetproc\fP is
473: .sp 1
474: .in +5
475: .nf
476: \&.>4
477: *nproc>l
478: *proc>f
479: $<setproc.nxt
480: .in -5
481: .sp 1
482: .fi
483: while \fBsetproc.nxt\fP is
484: .sp 1
485: .nf
486: .in +5
487: (*(<f+28))&0xffff="pid "X
488: ,#((*(<f+28)&0xffff)-<4)$<setproc.done
489: <l-1>l
490: <f+70>f
491: ,#<l$<
492: $<setproc.nxt
493: .in -5
494: .sp 1
495: .fi
496: The process-id, supplied as the parameter, is stored in the
497: variable ``<4'', the number of processes is placed in ``<l'',
498: and the base of the array of process structures in ``<f''.
499: \fBsetproc.nxt\fP then performs a linear search through the
500: array until it matches the process-id requested, or until
501: it runs out of process structures to check. The script
502: \fBsetproc.done\fP simply establishes the context of the
503: process, then exits.
504: .NH 2
505: Standard Scripts
506: .PP
507: The following table summarizes the command scripts currently
508: available in the directory \fI/usr/lib/adb\fP.
509: .TS
510: center, box;
511: c s s
512: l | l | l
513: lb | l | l.
514: Standard Command Scripts
515: _
516: Name Use Description
517: _
518: buf \fIaddress\fP$<\fBbuf\fP format block I/O buffer
519: callout $<\fBcallout\fP print timer queue
520: clist \fIaddress\fP$<\fBclist\fP format character I/O linked list
521: dino \fIaddress\fP$<\fBdino\fP format directory inode
522: dir \fIaddress\fP$<\fBdir\fP format directory entry
523: file \fIaddress\fP$<\fBfile\fP format open file structure
524: filsys \fIaddress\fP$<\fBfilsys\fP format in-core super block structure
525: findproc \fIpid\fP$<\fBfindproc\fP find process by process id
526: ifnet \fIaddress\fP$<\fBifnet\fP format network interface structure
527: ifuba \fIaddress\fP$<\fBifuba\fP format UNIBUS resource structure
528: inode \fIaddress\fP$<\fBinode\fP format in-core inode structure
529: inpcb \fIaddress\fP$<\fBinpcb\fP format internet protocol control block
530: mact \fIaddress\fP$<\fBmact\fP show ``active'' list of mbuf's
531: mbstat $<\fBmbstat\fP show mbuf statistics
532: mbuf \fIaddress\fP$<\fBmbuf\fP show ``next'' list of mbuf's
533: mount \fIaddress\fP$<\fBmount\fP format mount structure
534: pcb \fIaddress\fP$<\fBpcb\fP format process context block
535: proc \fIaddress\fP$<\fBproc\fP format process table entry
536: setproc \fIpid\fP$<\fBsetproc\fP switch process context to \fIpid\fP
537: socket \fIaddress\fP$<\fBsocket\fP format socket structure
538: tcpcb \fIaddress\fP$<\fBtcpcb\fP format TCP control block
539: text \fIaddress\fP$<\fBtext\fP format text structure
540: traceall $<\fBtraceall\fP show stack trace for all processes
541: tty \fIaddress\fP$<\fBtty\fP format tty structure
542: u \fIaddress\fP$<\fBu\fP format user vector, including pcb
543: vtimes \fIaddress\fP$<\fBvtimes\fP format process times
544: .TE
545: .ds RH "Summary
546: .LP
547: .nr H1 2
548: .bp
549: .LG
550: .B
551: .ce
552: 3. SUMMARY
553: .sp 2
554: .R
555: .nr H2 0
556: .NL
557: .PP
558: The extensions made to adb provide basic support for
559: debugging the
560: .UX
561: kernel by eliminating the need for a user to carry
562: out virtual to physical address translation. A collection
563: of scripts have been written to nicely format the major
564: kernel data structures and aid in switching between
565: process contexts. This has been carried out with
566: only minimal changes to the debugger.
567: .PP
568: More work is needed to provide enough information
569: for the debugger to automatically establish context
570: after a system crash. The system currently does not
571: always save enough state to allow the debugger to reliably
572: locate the stack frame just prior to an exception.
573: .PP
574: More work is also required on the user interface
575: to adb. It appears the inscrutable adb command language
576: has limited widespread use of much of the power of
577: adb. One possibility is to provide a more comprehensible
578: ``adb frontend'', just as \fIbc\fP(1) is used to
579: frontend \fIdc\fP(1).
580: .PP
581: Finally, adb could be significantly improved if it
582: were knowledgeable about a programs data structures.
583: This would eliminate the use of numeric offsets into
584: C structures.
This archive runs on limited infrastructure. Preserving old code on modern bandwidth. Automated agents are requested to crawl responsibly.