FS#13684 - [initscripts] hwclock --adjust should be called to compensate for drift
Attached to Project:
Arch Linux
Opened by Dieter Plaetinck (Dieter_be) - Friday, 06 March 2009, 17:05 GMT
Last edited by Tom Gundersen (tomegun) - Wednesday, 30 March 2011, 23:54 GMT
Opened by Dieter Plaetinck (Dieter_be) - Friday, 06 March 2009, 17:05 GMT
Last edited by Tom Gundersen (tomegun) - Wednesday, 30 March 2011, 23:54 GMT
|
Details
man hwclock says:
The Hardware Clock is usually not very accurate. However, much of its inaccuracy is completely predictable - it gains or loses the same amount of time every day. This is called systematic drift. hwclock's "adjust" function lets you make systematic corrections to correct the systematic drift. (...) It is good to do a hwclock --adjust just before the hwclock --hctosys at system startup time, and maybe periodically while the system is running via cron. I could not find anything about this in /etc/rc.* or /etc/cron.d/* * boot: Since we do a hwclock --hctosys in /etc/rc.sysinit, we should probably add a "hwclock --adjust" just before it too. * runtime: it wouldn't hurt to add a crontab in /etc/rc.daily or something. I'm not sure if it is also needed for people with ntp. |
This task depends upon
Closed by Tom Gundersen (tomegun)
Wednesday, 30 March 2011, 23:54 GMT
Reason for closing: Fixed
Additional comments about closing: Kurt's patches have been applied.
Wednesday, 30 March 2011, 23:54 GMT
Reason for closing: Fixed
Additional comments about closing: Kurt's patches have been applied.
I don't like the idea of adding a cronjob, but there aren't any other options
About the cronjob - I don't think this should hurt, we could just put a file to cron.hourly. This command will only adjust the hardware clock for drift, not change the linux system clock!
If the user would use --set to correct the date himself to a specific date, the adjtime gets updated, but this is another story.
Opinions?
I think when you do hwclock --adjust, the adjtime file does not get update. I think that only happens when you do --set.
What does get updated, is the hardware clock itself.
http://code.phraktured.net/cgit.cgi/initscripts/commit/?id=c206cfd1de94e2e79c1d9d42dc8f44f140d4c3f8
I have a concern about the cron job portion of the fix. There is a kernel 11-minute-mode for auto hw clock synchronization that should do this job when something like ntpd or xntpd is running (ex. starting ntpd should start 11-minute-mode).
Take a look farther down in the hwclock man page at the section "Automatic Hardware Clock Synchronization By the Kernel". This provides a more detailed description and additionally states that if 11-minute-mode is enabled, "don't use hwclock --adjust".
I suggest that the cron job can be removed because the HW clock is already synced and doing it separately in a cron job will conflict with the existing mechanism.
We used --noadjust because:
1) root fs was read-only at the moment we do first hwclock call
2) /etc/adjtime file should be in /var/lib per LFS, so when /var is on another partition - it's not accessible at the moment when root fs is remounted rw
3) see Allen's comment
Still, Allen's comment holds true.
Also, since hwclock --adjust is done before hwclock $HWCLOCK_PARAMS which contains --hctosys {--utc,--localtime} - can --adjust lead to incorrect results in this situation:
1) user's hardware RTC is in localtime
2) user changes his/her timezone
3) hardware clock gets changed in rc.shutdown
4) computer reboots
5) how should --adjust know if RTC drifted for alot of time or it's ok?
Shouldn't --adjust be in the same hwclock $HWCLOCK_PARAMS command so that it knows if it's --utc or --localtime?
Note that hwclock --systohc will reset when the hwclock was last corrected or changed. I.e. in the above scenario, hwclock --adjust will only apply drift since step (3). This is the desired behaviour since step (3) should have put a ntp synchronized time into the RTC at that time (either local or UTC).
Completely disabling the cronjob is not what I'd like, as it's a nice feature for cases where the 11-minute mode is not enabled.
All distros I checked seem to get it either from http://ibiblio.org/pub/Linux/system/admin/time/
or from Debian http://packages.debian.org/source/sid/adjtimex
--- /etc/cron.hourly/adjtime 2009-03-10 22:08:42.000000000 +0100
+++ /tmp/adjtime 2009-06-09 00:21:17.840568903 +0200
@@ -1,3 +1,3 @@
#!/bin/sh
# Update our hwclock for system drift
-/sbin/hwclock --adjust
+11minute || /sbin/hwclock --adjust
I can't really test this, I only know that 11 minute mode is disabled on my machine.
The 11 minute mode should be enabled when the system clock is synched to an outside source (ex. ntpd is started). If the system clock is free running, I'm not sure it makes sense to sync two free running clocks. If the date of the system clock is changed manually, it will be propagated to the HW clock in rc.shutdown anyway.
I guess what I'm saying is: Why not check that 11 minute mode is correctly configured in arch linux (*) rather than create a separate cron job to do the same thing?
(*) Note: I installed adjtimex on debian and it seemed to actually set up 11 minute mode. 11 minute mode was running afterwards, but I suspect installing it and checking the status does not necessarily mean that it was running before it was installed.
1. Set the initial time when the PC is started.
2. Wakeup up the PC when it is shutdown (if ACPI or BIOS wakeup is used).
We now do a hwclock --adjust at startup, so the drift is accounted for in point (1).
rc.shutdown does a hwclock --systohc, so for (2), the hw clock is set on shutdown and any drift before that has no impact (it would be nice to account for it after shutdown, but there isn't really a way to do that)
Is there something else I'm missing? I'm not sure regular --adjusts would provide any value.
In my opinion, correct me if i'm wrong:
1) we work in linux/processes mostly on system time.
2) the 'offset' used by --adjust is manual determined (by two settimes with minimal a day in between)
3) There are several solutions to synchronise the system time (ntp on inet/gps/dcf).
In my thoughts:
- The user (think of ntpd) is responsible to have a synchronised system time.
- But ntp may needs a longer period to update the system clock if there is a huge difference (about 15 minutes).
- hwclock --adjust can only adjust 'seconds'... so its impossible to set correct date (e.g. after CMOS battery empty i can understand).
- embedded systems can shutdown without 'halt' or 'reboot' (so no --systohc is issued and hwclock is never been updated).
This makes for me clear you must use:
1) 11 minutes mode of linux (No idea how this works, can be enabled)...
2) do a regular 'hwclock -wu', for example in a hourly cron.
Also I still have a doubt/question:
Additional to the use of solution 1/2 is it required to put a repeating --adjust?
(for example if a product with linux is shutdown for a week or more... which can be true for embedded devices)
I suspect the hwclock --adjust in /etc/rc.sysinit do not play well with DST (summer time) time shifts when you use localtime in dual boot setups.
I dont know what the correct fix for this problem is but i hope a solution for this can be found especially since it can trigger a fs check at boot, and one point i had to adjust time in bios to be able to boot.
There are a lot of users having this problem lately, do a search for hwclock on forums.
I don't see any reason why one should set the hardware clock to localtime anyway, I know of no operating system that doesn't support using UTC here.
Hopefully something will be done to make DST/localtime problem fixed.
Shall I open a new bug? Is this a known issue?
Reading the man page and playing with hwclock --debug (try it!) i found the following behaviour:
1) hwclock --adjust and --systohc both assume the hardware clock was drifting since the last
--systohc or --adjust invocation recorded
2) hwclock --systohc *only* recalculates the drift rate if at least 24 hrs. elapsed since the previous --systohc
Now, when in 11-minute-mode we *need* to do - all in cron.hourly/adjtime:
3) disable hwclock --adjust since not drifting
4) run hwclock --systohc - 1st reason: eventually recalculate the drift rate shortly after entering 11-minute (end of drifting) - If ntpd stepped the system clock on start, we get a meaningfull updated drift rate, if not, it will change sligtly toward the correct value if ever.
5) run hwclock --systohc - 2nd reason: prevent wrong recalculation of the drift rate while not drifting by updating the recorded time of --systohc
Further more we *could* do
6) in ntpd stop: run hwclock --systohc to record the exact time when drifting started again (not realy needed IMHO, done on shutdown anyway and not much drift would have happend since the last cron invocation if stopped manually)
Further more we *need* for the multiboot case (even when only having two installations of Arch Linux on the box)
7) in rc.sysinit/adjtime.cron: a rc.conf parameter to disable (default) hwclock adjustment since the hardware clock can't be assumed drifting since last shutdown but will be stepped by some other OS - This might also help avoiding weird problems with totally wrong drift rates because of localtime/DST.
Some bugs related to using localtime are simply impossible to solve, even from a theoretical point of view (the main problem is that, due to DST, given a time in localtime and a timezone, we cannot always find the corresponding time in UTC and this causes all sorts of fun and unsolvable problems). I think the correct solution is to strongly discourage using localtime, or even stop supporting it.
@Kurt: that's an interesting summary you made.
I have some comments/questions:
5) I understand from this that if there has been 22 hours since the last --systohc, and then we start calling --systohc ever hour, the drift will never be recalculated. I.e., the 24 hours is not since the last recalculate, but since the last --systohc?
4) if I understand (5) correctly, could you explain further what you mean by "eventually recalculate drift rate", I thought that if the drift rate was not recalculated on the first call to --systohc, it will never be?
Regarding the last point (7):
I don't see how disabling adjtime will help the dual-boot case. Assume you lose 10 seconds per day and (from this OS's point of view) the computer has been off for one day. There are two cases: either the computer has been really off and the clock is 10 seconds behind, or it has been booted into a different OS that has been setting the clock correctly so it is in fact exact. If we do not adjust the clock we are either 10 seconds behind or we are exact, if we do adjust the clock we will either be exact or 10 seconds ahead. How is one better than the other?
The only reason I can think of that we would be better off not adjusting in a dual boot setup is that due to the calculated drift being wrong we will set the clock more wrong compared to not adjusting for drift at all. However, I don't think this is the case as, unless I'm mistaken, dual-booting will cause us to underestimate the drift, so at worst we are as bad as not adjusting at all, but never any worse.
What do you think?
Regarding localtime: full ACK.
5) True. --systohc assumes system clock is correct, so the hardware clock needs to be untouched for a reasonable amount of time to get a meaningful drift value because of the one-second granularity. (Just reboot and try 'hwclock --systohc --test --debug'.)
4) It will be if downtime is more than 24 hours.
7) hwclock adjustment needs an up to date adjustment file. If you don't share the same file between all OS, it will do wrong calculations. So to get correct system time before doing --systohc, you need to
- either: disable adjustment on all OS and use ntp
- or: disable adjustment and --systohc and ntp (only read hardware clock) on all OS except one which does the drift calculation then.
About underestimating the drift: Say the first OS has calculated a correct drift rate. Then you decide to install/clone a second OS and run that for two weeks and afterwards you reboot to change back to the first OS. Now the 2nd OS will update the hardware clock on shutdown and then the first OS will adjust the hardware clock for two weeks of drift. So if loosing 10 seconds per day, the clock is now 140 seconds in the future.
I think simply adding an option to rc.conf to treat the hardware clock as read-only (no adjustment, no --systohc) would simplify things very much when it comes to using ntp and/or dual-boot.
5/4: thanks for the clarification.
7: the problem is that the first OS can never know whether or not the second OS has been running, or if the computer has been truly switched off. It will therefore be impossible to know whether or not to adjust for drift. If we always adjust for drift I think the potential error will be less than if we never adjust for drift, but either way we cannot trust the time we get. I think that, as with localtime, we should just admit that we cannot support this usecase, and add to our documentation that in a dualboot setting you need to use ntp to avoid creating a mess.
Would this work:
At boot/shutdown we always use "--noadjfile --utc", and if the user would like to adjust for drift s/he can enable a daemon that does "--adjust"/"--hctosys" at boot and "--systohc" at shutdown. The recommendation would then be to only enable this if you are not using dualboot nor ntp.
Sounds good to me, except for rc.shutdown which should do nothing at all then.
- rc.conf:
USEHWCLOCK = "yes"
DAEMONS=( clock ... )
- rc.sysinit 1st hwclock part:
if [[ $USEHWCLOCK = "yes" ]]; then
# enable rtc access
...
/sbin/hwclock --hctosys --utc --noadjfile
fi
- Drop 2nd hwclock part from rc.sysinit
- rc.d/clock start:
/sbin/hwclock --adjust --utc
/sbin/hwclock --hctosys --utc
add_daemon clock
- rc.d/clock stop:
/sbin/hwclock --systohc --utc
- rc.shutdown: Drop all hwclock stuff (!)
- cron.hourly/adjtime:
. /etc/rc.conf
. /etc/rc.d/functions
if in_array clock "${DAEMONS[@]}"; then
/sbin/hwclock --adjust --utc
fi
I thought/read about more about this problem and I have one more concern:
A discontinuous ("jumping") system time, might cause weird bugs. Reading the hwclock man page I get the impression that running "hwclock --hctosys" when the system is running would be a bad idea (as it will cause the system time to jump). I guess this potential problem would apply both if we set the systemtime for the second time in rc.sysinit (as lots of stuff will be up and running by then), or if we do it in this new "clock" daemon.
To avoid this problem all together, what about just adjusting the hwclock for drift without reading this into the systemtime? I.e.:
"- rc.d/clock start:
/sbin/hwclock --adjust --utc
add_daemon clock"
This means that the systemtime will be off by the drift since last shutdown, but the drifts between each reboot will not accumulate. I.e., if your hwclock looses 1 second between each reboot and you have rebooted 10 times, then your systemtime will be off by 1 second (rather than 10 as would be the case if you did not adjust). What do you think about this?
Would you be able to make a git patch and post it to <arch-projects@archlinux.org> (with a reference to this discussion)?
From current rc.conf:
# HARDWARECLOCK: set to "UTC" or "localtime", any other value will result
# in the hardware clock being left untouched (useful for virtualization)
If I remember right, some troubles where reported with virtual machines with enabling rtc (or using hwclock ?)
--systohc needs correct system time. I don't think stepping the time is such a big problem as long as no daemons other than udev are up.
- rc.conf:
USEHWCLOCK="yes"
DAEMONS=(hwclock syslog-ng network netfs crond)
I also remember something about virtualization being broken. I'd like to find a proper fix for this, rather than work around it like we are now. Anyway, that's a separate problem, so just leave it like it is for now.
About systohc: I see your point. Hmmm... how annoying.
I will also post this to the projects list.