1 <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN">
4 <meta name="generator" content="HTML Tidy, see www.w3.org">
5 <title>NTP Debugging Techniques</title>
8 <h3>NTP Debugging Techniques</h3>
10 <img align="left" src="pic/pogo.gif" alt="gif"><a href=
11 "http://www.eecis.udel.edu/~mills/pictures.htm">from <i>Pogo</i>,
14 <p>We make house calls and bring our own bugs.<br clear="left">
18 <p>Once the NTP software distribution has been compiled and
19 installed and the configuration file constructed, the next step is
20 to verify correct operation and fix any bugs that may result.
21 Usually, the command line that starts the daemon is included in the
22 system startup file, so it is executed only at system boot time;
23 however, the daemon can be stopped and restarted from root at any
24 time. Usually, no command-line arguments are required, unless
25 special actions described in the <tt><a href="ntpd.htm">
26 ntpd</a></tt> page are required. Once started, the daemon will
27 begin sending and receiving messages, as specified in the
28 configuration file.</p>
30 <h4>Initial Startup</h4>
32 <p>The best way to verify correct operation is using the <tt><a
33 href="ntpq.htm">ntpq</a></tt> and <tt><a href="ntpdc.htm">
34 ntpdc</a></tt> utility programs, either on the server itself or
35 from another machine elsewhere in the network. The <tt>ntpq</tt>
36 program implements the management functions specified in the NTP
37 specification <a href=
38 "http://www.eecis.udel.edu/~mills/database/rfc/rfc1305/rfc1305c.ps">
39 RFC-1305, Appendix A</a>. The <tt>ntpdc</tt> program implements
40 additional functions not provided in the standard. Both programs
41 can be used to inspect the state variables defined in the
42 specification and, in the case of <tt>ntpdc</tt>, additional ones
43 of interest. In addition, the <tt>ntpdc</tt> program can be used to
44 selectively reconfigure and enable or disable some functions while
45 the daemon is running.</p>
47 <p>In extreme cases with elusive bugs, the daemon can operate in
48 two modes, depending on the presence of the <tt>-d</tt>
49 command-line debug switch. If not present, the daemon detaches from
50 the controlling terminal and proceeds autonomously. If one or more
51 <tt>-d</tt> switches are present, the daemon does not detach and
52 generates special output useful for debugging. In general,
53 interpretation of this output requires reference to the sources.
54 However, a single <tt>-d</tt> does produce only mildly cryptic
55 output and can be very useful in finding problems with
56 configuration and network troubles. With a little experience, the
57 volume of output can be reduced by piping the output to <tt>
58 grep</tt> and specifying the keyword of the trace you want to
61 <p>Some problems are immediately apparent when the daemon first
62 starts running. The most common of these are the lack of a UDP port
63 for NTP (123) in the Unix <tt>/etc/services</tt> file (or
64 equivalent in some systems). Note that NTP does not use TCP in any
65 form. Other problems are apparent in the system log file. The log
66 file should show the startup banner, some cryptic initialization
67 data and the computed precision value. The next most common problem
68 is incorrect DNS names. Check that each DNS name used in the
69 configuration file exists and that the address responds to the Unix
70 <tt>ping</tt> command.</p>
72 <p>When first started, the daemon normally polls the servers listed
73 in the configuration file at 64-s intervals. In order to allow a
74 sufficient number of samples for the NTP algorithms to reliably
75 discriminate between correctly operating servers and possible
76 intruders, at least four valid messages from the majority of
77 servers and peers listed in the configuration file is required
78 before the daemon can set the local clock. However, if the
79 difference between the client time and server time is greater than
80 the panic threshold, which defaults to 1000 s, the daemon will send
81 a message to the system log and shut down without setting the
82 clock. It is necessary to set the local clock to within the panic
83 threshold first, either manually by eyeball and wristwatch and the
84 Unix <tt>date</tt> command, or by the <tt>ntpdate</tt> or <tt>ntpd
85 -q</tt> commands. The panic threshold can be changed by the <tt>
86 tinker panic</tt> command discribed on the <a href="miscopt.htm">
87 Miscellaneous Options</a> page. The panic threshold can be disabled
88 entirely by the <tt>-g</tt> command line option described on the <a
89 href="ntpd.htm">ntpd - Network Time Protocol (NTP) daemon</a>
92 <p>If the difference between local time and server time is less
93 than the panic threshold but greater than the step threshold, which
94 defaults to 125 ms, the daemon will perform a step adjustment;
95 otherwise, it will gradually slew the clock to the nominal time.
96 The step threshold can be changed by the <tt>tinker step</tt>
97 command discribed on the <a href="miscopt.htm">Miscellaneous
98 Options</a> page. The step threshold can be disabled entirely by
99 the <tt>-x</tt> command line option described on the <a href=
100 "ntpd.htm">ntpd - Network Time Protocol (NTP) daemon</a> page. In
101 this case the clock will never be stepped; however, users should
102 understand the implications for doing this in a distributed data
103 network where all processing must be tightly synchronized. See the
104 <a href="leap.htm">NTP Timescale and Leap Seconds</a> page for
105 further information. If a step adjustment is made, the clock
106 discipline algorithm will start all over again, requiring another
107 round of at least four messages as before. This is necessary so
108 that all servers and peers operate on the same set of time
111 <p>The clock discipline algorithm is designed to avoid large noise
112 spikes that might occur on a congested network or access line. If
113 an offset sample exceeds the step threshold, it is ignored and a
114 timer started. If a later sample is below the step threshold, the
115 counter is reset. However, if the counter is greater than the
116 stepout interval, which defaults to 900 s, the next sample will
117 step or slew the time as directed. The stepout threshold can be
118 changed by the <tt>tinker stepout</tt> command discribed on the <a
119 href="miscopt.htm">Miscellaneous Options</a> page.</p>
121 <p>If, as discussed later on this page, for some reason the
122 hardware clock oscillator frequency error is very large, the time
123 errors upon first startup of the daemon may increase over time
124 until exceeding the step threshold, which requires another step
125 correction. However, due to provisions that reduce vulnerability to
126 noise spikes, the second correction will not be done until after
127 the stepout threshold. When the frequency error is very large, it
128 may take a number of cycles like this until converging on the
129 nominal frequency correction. After this, the correction is written
130 to the <tt>ntp.drift</tt> file, which is read upon subsequent
131 restarts, so the herky-jerky cycles should not recur.</p>
133 <h4>Verifying Correct Operation</h4>
135 <p>After starting the daemon, run the <tt>ntpq</tt> program using
136 the <tt>-n</tt> switch, which will avoid possible distractions due
137 to name resolution problems. Use the <tt>pe</tt> command to display
138 a billboard showing the status of configured peers and possibly
139 other clients poking the daemon. After operating for a few minutes,
140 the display should be something like:</p>
144 remote refid st t when poll reach delay offset jitter
145 =====================================================================
146 -isipc6.cairn.ne .GPS1. 1 u 18 64 377 65.592 -5.891 0.044
147 +saicpc-isiepc2. pogo.udel.edu 2 u 241 128 370 10.477 -0.117 0.067
148 +uclpc.cairn.net pogo.udel.edu 2 u 37 64 177 212.111 -0.551 0.187
149 *pogo.udel.edu .GPS1. 1 u 95 128 377 0.607 0.123 0.027
152 <p>The host names or addresses shown in the <tt>remote</tt> column
153 correspond to the server and peer entries listed in the
154 configuration file; however, the DNS names might not agree if the
155 names listed are not the canonical DNS names. The <tt>refid</tt>
156 column shows the current source of synchronization, while the <tt>
157 st</tt> column reveals the stratum, <tt>t</tt> the type (<tt>u</tt>
158 = unicast, <tt>m</tt> = multicast, <tt>l</tt> = local, <tt>-</tt> =
159 don't know), and <tt>poll</tt> the poll interval in seconds. The
160 <tt>when</tt> column shows the time since the peer was last heard
161 in seconds, while the <tt>reach</tt> column shows the status of the
162 reachability register (see RFC-1305) in octal. The remaining
163 entries show the latest delay, offset and jitter in milliseconds.
164 Note that in NTP Version 4 what used to be the <tt>dispersion</tt>
165 column has been replaced by the <tt>jitter</tt> column.</p>
167 <p>The tattletale symbol at the left margin displays the
168 synchronization status of each peer. The currently selected peer is
169 marked <tt>*</tt>, while additional peers designated acceptable for
170 synchronization, but not currently selected, are marked <tt>+</tt>.
171 Peers marked <tt>*</tt> and <tt>+</tt> are included in the weighted
172 average computation to set the local clock; the data produced by
173 peers marked with other symbols are discarded. See the <tt>
174 ntpq</tt> page for the meaning of these symbols.</p>
176 <p>Additional details for each peer separately can be determined by
177 the following procedure. First, use the <tt>as</tt> command to
178 display an index of association identifiers, such as</p>
182 ind assID status conf reach auth condition last_event cnt
183 ===========================================================
184 1 50252 f314 yes yes ok outlyer reachable 1
185 2 50253 f414 yes yes ok candidat reachable 1
186 3 50254 f414 yes yes ok candidat reachable 1
187 4 50255 f614 yes yes ok sys.peer reachable 1
190 <p>Each line in this billboard is associated with the corresponding
191 line in the <tt>pe</tt> billboard above. The <tt>assID</tt> shows
192 the unique identifier for each mobilized association, while the
193 <tt>status</tt> column shows the peer status word in hex, as
194 defined in the NTP specification. Next, use the <tt>rv</tt> command
195 and the respective <tt>assID</tt> identifier to display a detailed
196 synopsis for the selected peer, such as</p>
200 status=f414 reach, conf, auth, sel_candidat, 1 event, event_reach,
201 srcadr=saicpc-isiepc2.cairn.net, srcport=123, dstadr=140.173.1.46,
202 dstport=123, keyid=3816249004, stratum=2, precision=-27,
203 rootdelay=10.925, rootdispersion=12.848, refid=pogo.udel.edu,
204 reftime=bd11b225.133e1437 Sat, Jul 8 2000 13:59:01.075, delay=10.550,
205 offset=-1.357, jitter=0.074, dispersion=1.444, reach=377, valid=7,
206 hmode=1, pmode=1, hpoll=6, ppoll=7, leap=00, flash=00 ok,
207 org=bd11b23c.01385836 Sat, Jul 8 2000 13:59:24.004,
208 rec=bd11b23c.02dc8fb8 Sat, Jul 8 2000 13:59:24.011,
209 xmt=bd11b21a.ac34c1a8 Sat, Jul 8 2000 13:58:50.672,
210 filtdelay= 10.45 10.50 10.63 10.40 10.48 10.43 10.49 11.26,
211 filtoffset= -1.18 -1.26 -1.26 -1.35 -1.35 -1.42 -1.54 -1.81,
212 filtdisp= 0.51 1.47 2.46 3.45 4.40 5.34 6.33 7.28,
213 hostname="miro.time.saic.com", publickey=3171359012, pcookie=0x6629adb2,
214 hcookie=0x61f99cdb, initsequence=61, initkey=0x287b649c,
218 <p>A detailed explanation of the fields in this billboard are
219 beyond the scope of this discussion; however, most variables
220 defined in the NTP Version 3 specification RFC-1305 are available
221 along with others defined for NTP Version 4. This particular
222 example was chosen to illustrate probably the most complex
223 configuration involving symmetric modes and public-key
224 cryptography. As the result of debugging experience, the names and
225 values of these variables may change from time to time. An
226 explanation of the current set is on the <tt>ntpq</tt> page.</p>
228 <p>A useful indicator of miscellaneous problems is the <tt>
229 flash</tt> value, which reveals the state of the various sanity
230 tests on incoming packets. There are currently eleven bits, one for
231 each test, numbered from the right, which is for test 1. If the
232 test fails, the corresponding bit is set to one and zero otherwise.
233 If any bit is set following each processing step, the packet is
234 discarded. The meaning of each test is described on the <tt>
237 <p>The three lines identified as <tt>filtdelay</tt>, <tt>
238 filtoffset</tt> and <tt>filtdisp</tt> reveal the roundtrip delay,
239 clock offset and dispersion for each of the last eight measurement
240 rounds, all in milliseconds. Note that the dispersion, which is an
241 estimate of the error, increases as the age of the sample
242 increases. From these data, it is usually possible to determine the
243 incidence of severe packet loss, network congestion, and unstable
244 local clock oscillators. There are no hard and fast rules here,
245 since every case is unique; however, if one or more of the rounds
246 show large values or change radically from one round to another,
247 the network is probably congested or lossy.</p>
249 <p>Once the daemon has set the local clock, it will continuously
250 track the discrepancy between local time and NTP time and adjust
251 the local clock accordingly. There are two components of this
252 adjustment, time and frequency. These adjustments are automatically
253 determined by the clock discipline algorithm, which functions as a
254 hybrid phase/frequency feedback loop. The behavior of this
255 algorithm is carefully controlled to minimize residual errors due
256 to network jitter and frequency variations of the local clock
257 hardware oscillator that normally occur in practice. However, when
258 started for the first time, the algorithm may take some time to
259 converge on the intrinsic frequency error of the host machine.</p>
261 <p>The state of the local clock itself can be determined using the
262 <tt>rv</tt> command (without the argument), such as</p>
266 status=0644 leap_none, sync_ntp, 4 events, event_peer/strat_chg,
267 version="ntpd 4.0.99j4-r Fri Jul 7 23:38:17 GMT 2000 (1)",
268 processor="i386", system="FreeBSD3.4-RELEASE", leap=00, stratum=2,
269 precision=-27, rootdelay=0.552, rootdispersion=12.532, peer=50255,
271 reftime=bd11b220.ac89f40a Sat, Jul 8 2000 13:58:56.673, poll=6,
272 clock=bd11b225.ee201472 Sat, Jul 8 2000 13:59:01.930, state=4,
273 phase=0.179, frequency=44.298, jitter=0.022, stability=0.001,
274 hostname="barnstable.udel.edu", publickey=3171372095, params=3171372095,
278 <p>An explanation about most of these variables is in the RFC-1305
279 specification. The most useful ones include <tt>clock</tt>, which
280 shows when the clock was last adjusted, and <tt>reftime</tt>, which
281 shows when the server clock of <tt>refid</tt> was last adjusted.
282 The <tt>version</tt>, <tt>processor</tt> and <tt>system</tt> values
283 are very helpful when included in bug reports. The mean millisecond
284 time offset (<tt>phase</tt>) and deviation (<tt>jitter</tt>)
285 monitor the clock quality, while the mean PPM frequency offset
286 (<tt>frequency</tt>) and deviation (<tt>stability</tt>) monitor the
287 clock stability and serve as a useful diagnostic tool. It has been
288 the experience of NTP operators over the years that these data
289 represent useful environment and hardware alarms. If the
290 motherboard fan freezes up or some hardware bit sticks, the system
291 clock is usually the first to notice it.</p>
293 <p>Among the new variables added for NTP Version 4 are the <tt>
294 hostname</tt>, <tt>publickey</tt>, <tt>params</tt> and <tt>
295 refresh</tt>, which are used for the Autokey public-key
296 cryptography described on the <a href="authopt.htm">Authentication
297 Options</a> page. The values show the filestamps, in NTP seconds,
298 that the associated values were created. These are useful in
299 diagnosing problems with cryptographic key consistency and ordering
302 <p>When nothing seems to happen in the <tt>pe</tt> billboard after
303 some minutes, there may be a network problem. One common network
304 problem is an access controlled router on the path to the selected
305 peer or an access controlled server using methods described on the
306 <a href="accopt.htm">Access Control Options</a> page. Another
307 common problem is that the server is down or running in
308 unsynchronized mode due to a local problem. Use the <tt>ntpq</tt>
309 program to spy on the server variables in the same way you can spy
312 <p>Normally, the daemon will adjust the local clock in small steps
313 in such a way that system and user programs are unaware of its
314 operation. The adjustment process operates continuously as long as
315 the apparent clock error exceeds the step threshold for a period
316 longer than the stepout threshold, which for most Internet paths is
317 a very rare event. If the event is simply an outlyer due to an
318 occasional network delay spike, the correction is simply discarded;
319 however, if the apparent time error persists for longer than the
320 stepout threshold of about 17 minutes, the local clock is stepped
321 or slewed to the new value as directed. This behavior is designed
322 to resist errors due to severely congested network paths, as well
323 as errors due to confused radio clocks upon the epoch of a leap
326 <h4>Special Problems</h4>
328 <p>The frequency tolerance of computer clock oscillators can vary
329 widely, which can put a strain on the daemon's ability to
330 compensate for the intrinsic frequency error. While the daemon can
331 handle frequency errors up to 500 parts-per-million (PPM), or 43
332 seconds per day, values much above 100 PPM reduce the headroom and
333 increase the time to learn the particular value and record it in
334 the <tt>ntp.drift</tt> file. In extreme cases before the particular
335 oscillator frequency error has been determined, the residual system
336 time offsets can sweep from one extreme to the other of the 128-ms
337 tracking window only for the behavior to repeat at 900-s intervals
338 until the measurements have converged.</p>
340 <p>In order to determine if excessive frequency error is a problem,
341 observe the nominal <tt>filtoffset</tt> values for a number of
342 rounds and divide by the poll interval. If the result is something
343 approaching 500 PPM, there is a good chance that NTP will not work
344 properly until the frequency error is reduced by some means. A
345 common cause is the hardware time-of-year (TOY) clock chip, which
346 must be disabled when NTP disciplines the software clock. For some
347 systems this can be done using the <tt><a href="tickadj.htm">
348 tickadj</a></tt> utility and the <tt>-s</tt> command line argument.
349 For other systems this can be done using a command in the system
352 <p>If the TOY chip is not the cause, the problem may be that the
353 hardware clock frequency may simply be too slow or two fast. In
354 some systems this might require tweaking a trimmer capacitor on the
355 motherboard. For other systems the clock frequency can be adjusted
356 in increments of 100 PPM using the <tt>tickadj</tt> utility and the
357 <tt>-t</tt> command line argument. Note that the <tt>tickadj</tt>
358 alters certain kernel variables and, while the utility attempts to
359 figure out an acceptable way to do this, there are many cases where
360 <tt>tickadj</tt> is incompatible with a running kernel.</p>
362 <p>Provisions are included in <tt>ntpd</tt> for access controls
363 which deflect unwanted traffic from selected hosts or networks. The
364 controls described on the <a href="accopt.htm">Access Control
365 Options</a> include detailed packet filter operations based on
366 source address and address mask. Normally, filtered packets are
367 dropped without notice other than to increment tally counters.
368 However, the server can configure to generate what is called a
369 kiss-of-death (KOD) packet and send to the client. In case of
370 outright access denied, the KOD is the response to the first client
371 packet. In this case the client association is permanently disabled
372 and the access denied bit (test 4) is set in the flash peer
373 variable mentioned above and a message is sent to the system
376 <p>The access control provisions include a limit on the packet rate
377 from a host or network. If an incoming packet exceeds the limit, it
378 is dropped and a KOD sent to the source. If this occurs after the
379 client association has synchronized, the association is not
380 disabled, but a message is sent to the system log. See the <a href=
381 "accopt.htm">Access Control Options</a> page for further
384 <p>In some reported scenarios an access line may show low to
385 moderate network delays during some period of the day and moderate
386 to high delays during other periods. Often the delay on one
387 direction of transmission dominates, which can result in large time
388 offset errors, sometimes in the range up to a few seconds. It is
389 not usually convenient to run <tt>ntpd</tt> throughout the day in
390 such scenarios, since this could result in several time steps,
391 especially if the condition persists for greater than the stepout
394 <p>The recommended approach in such scenarios is first to calibrate
395 the local clock frequency error by running <tt>ntpd</tt> in
396 continuous mode during the quiet interval and let it write the
397 frequency to the <tt>ntp.drift</tt> file. Then, run <tt>ntpd
398 -q</tt> from a cron job each day at some time in the quiet
399 interval. In systems with the nanokernel or microkernel performance
400 enhancements, including Solaris, Tru64, Linux and FreeBSD, the
401 kernel continuously disciplines the frequency so that the residual
402 correction produced by <tt>ntpd</tt> is usually less than a few
405 <h4>Debugging Checklist</h4>
407 If the <tt>ntpq</tt> or <tt>ntpdc</tt> programs do not show that
408 messages are being received by the daemon or that received messages
409 do not result in correct synchronization, verify the following:
412 <li style="list-style: none"></li>
414 <li>Verify the <tt>/etc/services</tt> file host machine is
415 configured to accept UDP packets on the NTP port 123. NTP is
416 specifically designed to use UDP and does not respond to TCP.</li>
418 <li style="list-style: none"></li>
420 <li>Check the system log for <tt>ntpd</tt> messages about
421 configuration errors, name-lookup failures or initialization
424 <li style="list-style: none"></li>
426 <li>Verify using <tt>ping</tt> or other utility that packets
427 actually do make the round trip between the client and server.
428 Verify using <tt>nslookup</tt> or other utility that the DNS server
429 names do exist and resolve to valid Internet addresses.</li>
431 <li>Using the <tt>ntpdc</tt> program, verify that the packets
432 received and packets sent counters are incrementing. If the sent
433 counter does not increment and the configuration file includes
434 configured servers, something may be wrong in the host network or
435 interface configuration. If this counter does increment, but the
436 received counter does not increment, something may be wrong in the
437 network or the server NTP daemon may not be running or the server
438 itself may be down or not responding.</li>
440 <li style="list-style: none"></li>
442 <li>If both the sent and received counters do increment, but the
443 <tt>reach</tt> values in the <tt>pe</tt> billboard with <tt>
444 ntpq</tt> continues to show zero, received packets are probably
445 being discarded for some reason. If this is the case, the cause
446 should be evident from the <tt>flash</tt> variable as discussed
447 above and on the <tt>ntpq</tt> page.</li>
449 <li style="list-style: none"></li>
451 <li>If the <tt>reach</tt> values in the <tt>pe</tt> billboard show
452 the servers are alive and responding, note the tattletale symbols
453 at the left margin, which indicate the status of each server
454 resulting from the various grooming and mitigation algorithms. The
455 interpretation of these symbols is discussed on the <tt>ntpq</tt>
456 page. After a few minutes of operation, one or another of the
457 reachable server candidates should show a * tattletale symbol. If
458 this doesn't happen, the intersection algorithm, which classifies
459 the servers as truechimers or falsetickers, may be unable to find a
460 majority of truechimers among the server population.</li>
462 <li style="list-style: none"></li>
464 <li>If all else fails, see the FAQ and/or the discussion and
465 briefings at <a href="http://www.eecis.udel.edu/~mills/ntp.htm">
466 Network Time Synchronization Project.</a></li>
470 <a href="index.htm"><img align="left" src="pic/home.gif" alt=
473 <address><a href="mailto:mills@udel.edu">David L. Mills
474 <mills@udel.edu></a></address>
projects / dragonfly.git / blob