Arch Linux

Doing it in rc.sysinit seems fine to me - what about rc.shutdown?

I don't like the idea of adding a cronjob, but there aren't any other options

I didn't know about it and it seems fine in rc.sysinit. Do we need read-write access to the filesystem for that?

About the cronjob - I don't think this should hurt, we could just put a file to cron.hourly. This command will only adjust the hardware clock for drift, not change the linux system clock!

For --adjust I think you only need read access to the adjtime file.

If the user would use --set to correct the date himself to a specific date, the adjtime gets updated, but this is another story.

So regarding the adjust cron job - what if someone is ALSO using ntpd or ntpdate? Will this screw things up?

don't think so, ntp sets imho also the drift, at least it reports it when running ntpdate

what's the advantage of putting it in crontab if we already put it in rc.sysinit?

Apparently, the drift needs to be updated every so often. If it's done at start, that's fine, but if the machine has been up for 30 days straight, it's going to drift

i'd say go with it, unless putting it in crontab suddenly hags my system when i'm playing quake(input preffered application here).

http://code.phraktured.net/cgit.cgi/initscripts/commit/?id=8845e281feb25fde3269e453d67e56b405ada9c0

Opinions?

I think the comments are incorrect.
I think when you do hwclock --adjust, the adjtime file does not get update. I think that only happens when you do --set.
What does get updated, is the hardware clock itself.

Comments adjusted:
http://code.phraktured.net/cgit.cgi/initscripts/commit/?id=c206cfd1de94e2e79c1d9d42dc8f44f140d4c3f8

Pushed to master

Thanks Thomas for re-opening this.

I have a concern about the cron job portion of the fix. There is a kernel 11-minute-mode for auto hw clock synchronization that should do this job when something like ntpd or xntpd is running (ex. starting ntpd should start 11-minute-mode).

Take a look farther down in the hwclock man page at the section "Automatic Hardware Clock Synchronization By the Kernel". This provides a more detailed description and additionally states that if 11-minute-mode is enabled, "don't use hwclock --adjust".

I suggest that the cron job can be removed because the HW clock is already synced and doing it separately in a cron job will conflict with the existing mechanism.

Wow, I only now discovered this bug, and I do not agree with the request for --adjust.
We used --noadjust because:
1) root fs was read-only at the moment we do first hwclock call
2) /etc/adjtime file should be in /var/lib per LFS, so when /var is on another partition - it's not accessible at the moment when root fs is remounted rw
3) see Allen's comment

In addition I think --adjust was somehow involved in the famous filesystem mount time bug, but I may be wrong here, need to recheck.

Ah, it seems I got a bit confused here, I checked git and --noadjfile is still here, while --adjust is done after everything is mounted, so it looks like I false-alarmed, sorry about that.
Still, Allen's comment holds true.
Also, since hwclock --adjust is done before hwclock $HWCLOCK_PARAMS which contains --hctosys {--utc,--localtime} - can --adjust lead to incorrect results in this situation:
1) user's hardware RTC is in localtime
2) user changes his/her timezone
3) hardware clock gets changed in rc.shutdown
4) computer reboots
5) how should --adjust know if RTC drifted for alot of time or it's ok?
Shouldn't --adjust be in the same hwclock $HWCLOCK_PARAMS command so that it knows if it's --utc or --localtime?

Re: Adjust needing to know utc or localtime. I don't think this is true. --adjust just looks at the adjust file to find the drift rate, figures out how long it has been since the last update and then adds a correction. Ex. If drift is 1s/day and HWCLOCK is now 5 days later than the last time it was corrected or changed, add 5s. This is independent of the system clock, the timezone and whether the RTC is local or UTC.

Note that hwclock --systohc will reset when the hwclock was last corrected or changed. I.e. in the above scenario, hwclock --adjust will only apply drift since step (3). This is the desired behaviour since step (3) should have put a ntp synchronized time into the RTC at that time (either local or UTC).

Thanks for the explanation. So, the only concern left is about 11-minute mode.

I'd like to have a nice check whether this 11-minute mode is enabled or not. If we have that, we can put a "11-minute-mode-enabled || hwclock --adjust" into the cronjob. However, the adjtimex program mentioned in the hwclock manpage doesn't exist on my system (where is this from?) and I didn't find anything in /proc or /sys. If anyone knows how to check for it easily, I'd like to implement it like this.

Completely disabling the cronjob is not what I'd like, as it's a nice feature for cases where the 11-minute mode is not enabled.

I think it's a separate package that does not exist in Arch.
All distros I checked seem to get it either from http://ibiblio.org/pub/Linux/system/admin/time/
or from Debian http://packages.debian.org/source/sid/adjtimex

According to hwclock's manpage and some code stolen from the adjtimex tool, this code should check whether 11 minute mode is turned on. The following patch would then fix the cronjobs on systems where this mode is enabled:

--- /etc/cron.hourly/adjtime 2009-03-10 22:08:42.000000000 +0100
+++ /tmp/adjtime 2009-06-09 00:21:17.840568903 +0200
@@ -1,3 +1,3 @@
#!/bin/sh
# Update our hwclock for system drift
-/sbin/hwclock --adjust
+11minute || /sbin/hwclock --adjust

I can't really test this, I only know that 11 minute mode is disabled on my machine.

Just a comment:

The 11 minute mode should be enabled when the system clock is synched to an outside source (ex. ntpd is started). If the system clock is free running, I'm not sure it makes sense to sync two free running clocks. If the date of the system clock is changed manually, it will be propagated to the HW clock in rc.shutdown anyway.

I guess what I'm saying is: Why not check that 11 minute mode is correctly configured in arch linux (*) rather than create a separate cron job to do the same thing?

(*) Note: I installed adjtimex on debian and it seemed to actually set up 11 minute mode. 11 minute mode was running afterwards, but I suspect installing it and checking the status does not necessarily mean that it was running before it was installed.

Even if ntp is not running, you might have some drift configured in your drift file (because you ran ntp for some time and are now offline, not running ntp anymore). hwclock --adjust will then still keep your hardware clock in sync, does it make sense to use 11 minute mode then? Does it make sense to always enable it?

My understanding is that the HW clock is used for 2 things:

1. Set the initial time when the PC is started.

2. Wakeup up the PC when it is shutdown (if ACPI or BIOS wakeup is used).

We now do a hwclock --adjust at startup, so the drift is accounted for in point (1).

rc.shutdown does a hwclock --systohc, so for (2), the hw clock is set on shutdown and any drift before that has no impact (it would be nice to account for it after shutdown, but there isn't really a way to do that)

Is there something else I'm missing? I'm not sure regular --adjusts would provide any value.

As I said, Linux adjusts the system clock for systematic drift. --adjust takes care that this adjustment is also reflected in the hardware clock. This would not be necessary when --systohc is called properly on shutdown, however, you don't always shut down cleanly.

So what's the status here? I'm a bit lost in the last comments after this report was re-opened.

I'm not sure about the status, but I came across hwclock problems and ntp.
In my opinion, correct me if i'm wrong:

1) we work in linux/processes mostly on system time.
2) the 'offset' used by --adjust is manual determined (by two settimes with minimal a day in between)
3) There are several solutions to synchronise the system time (ntp on inet/gps/dcf).

In my thoughts:
- The user (think of ntpd) is responsible to have a synchronised system time.
- But ntp may needs a longer period to update the system clock if there is a huge difference (about 15 minutes).
- hwclock --adjust can only adjust 'seconds'... so its impossible to set correct date (e.g. after CMOS battery empty i can understand).
- embedded systems can shutdown without 'halt' or 'reboot' (so no --systohc is issued and hwclock is never been updated).

This makes for me clear you must use:
1) 11 minutes mode of linux (No idea how this works, can be enabled)...
2) do a regular 'hwclock -wu', for example in a hourly cron.

Also I still have a doubt/question:

Additional to the use of solution 1/2 is it required to put a repeating --adjust?
(for example if a product with linux is shutdown for a week or more... which can be true for embedded devices)

I recently experienced that my hw clock has been set incorrectly at boot (huge value in /var/lib/hwclock/adjtime). I saw that the hw clock was correct in bios before boot (saved at previous shutdown)
I suspect the hwclock --adjust in /etc/rc.sysinit do not play well with DST (summer time) time shifts when you use localtime in dual boot setups.

I dont know what the correct fix for this problem is but i hope a solution for this can be found especially since it can trigger a fs check at boot, and one point i had to adjust time in bios to be able to boot.
There are a lot of users having this problem lately, do a search for hwclock on forums.

localtime doesn't play well with DST changes whenever Linux isn't running during the change and no other operating system has changed the hardware clock during the downtime.

I don't see any reason why one should set the hardware clock to localtime anyway, I know of no operating system that doesn't support using UTC here.

Arch wiki says that you should use "localtime" if you dual boot with Windows, are you saying that Windows is fine with UTC ?

It is, Google is your friend. BTW, the wiki is not all-knowing.

That (RealTimeIsUniversal) seem to work fine, thanks.
Hopefully something will be done to make DST/localtime problem fixed.

I notice important clock delays after resuming from suspend/hibernate, like if clock san slower during the offline stage.

Shall I open a new bug? Is this a known issue?

This is a probably a bug in pm-utils. I don't know what pm-utils does, but in hibernate-script, the default is to only restore the system clock on resume, but not set the hardware clock on suspend/hibernate. This behaviour can be altered in the hibernate-configuration. I would assume that pm-utils has similar behaviour. Please refer to pm-utils' manual and configuration files for more help.

@Thomas: thank you for your clarifying reply. Sorry if this is too much off-topic, but I have read that recent kernels do not require any fix-up at all (http://osdir.com/ml/debian-bugs-dist/2009-04/msg09260.html), I am using the latest 2.6.32-rc6 and I have no script in /etc/pm/*/ so I should rather consider this a kernel bug?

No, (re)setting the time after resume is the responsibility of userspace, not the kernel.

OK, that was itching me for quite a long time, here is my scratching :)
Reading the man page and playing with hwclock --debug (try it!) i found the following behaviour:
1) hwclock --adjust and --systohc both assume the hardware clock was drifting since the last
--systohc or --adjust invocation recorded
2) hwclock --systohc *only* recalculates the drift rate if at least 24 hrs. elapsed since the previous --systohc
Now, when in 11-minute-mode we *need* to do - all in cron.hourly/adjtime:
3) disable hwclock --adjust since not drifting
4) run hwclock --systohc - 1st reason: eventually recalculate the drift rate shortly after entering 11-minute (end of drifting) - If ntpd stepped the system clock on start, we get a meaningfull updated drift rate, if not, it will change sligtly toward the correct value if ever.
5) run hwclock --systohc - 2nd reason: prevent wrong recalculation of the drift rate while not drifting by updating the recorded time of --systohc
Further more we *could* do
6) in ntpd stop: run hwclock --systohc to record the exact time when drifting started again (not realy needed IMHO, done on shutdown anyway and not much drift would have happend since the last cron invocation if stopped manually)
Further more we *need* for the multiboot case (even when only having two installations of Arch Linux on the box)
7) in rc.sysinit/adjtime.cron: a rc.conf parameter to disable (default) hwclock adjustment since the hardware clock can't be assumed drifting since last shutdown but will be stepped by some other OS - This might also help avoiding weird problems with totally wrong drift rates because of localtime/DST.

I think there should definitely be a way to disable hwclock --adjust as it can cause problems when Arch Linux is installed on a USB drive and you move between different computers often. The hwclock --adjust call would set the clock to the wrong date/time. Additionally after the time is corrected, a reboot would result in a file system check error with the last mount time being in the future requiring a manual fsck.

Jonathan, that is an entirely different problem. The adjfile is maintained by hwclock, whether or not we call --adjust. In your case, all calls to hwclock in initscripts would have to use --noadjfile (iirc).

I'm late to the party, but here are my two cents:

Some bugs related to using localtime are simply impossible to solve, even from a theoretical point of view (the main problem is that, due to DST, given a time in localtime and a timezone, we cannot always find the corresponding time in UTC and this causes all sorts of fun and unsolvable problems). I think the correct solution is to strongly discourage using localtime, or even stop supporting it.

@Kurt: that's an interesting summary you made.

I have some comments/questions:

5) I understand from this that if there has been 22 hours since the last --systohc, and then we start calling --systohc ever hour, the drift will never be recalculated. I.e., the 24 hours is not since the last recalculate, but since the last --systohc?

4) if I understand (5) correctly, could you explain further what you mean by "eventually recalculate drift rate", I thought that if the drift rate was not recalculated on the first call to --systohc, it will never be?

Regarding the last point (7):

I don't see how disabling adjtime will help the dual-boot case. Assume you lose 10 seconds per day and (from this OS's point of view) the computer has been off for one day. There are two cases: either the computer has been really off and the clock is 10 seconds behind, or it has been booted into a different OS that has been setting the clock correctly so it is in fact exact. If we do not adjust the clock we are either 10 seconds behind or we are exact, if we do adjust the clock we will either be exact or 10 seconds ahead. How is one better than the other?

The only reason I can think of that we would be better off not adjusting in a dual boot setup is that due to the calculated drift being wrong we will set the clock more wrong compared to not adjusting for drift at all. However, I don't think this is the case as, unless I'm mistaken, dual-booting will cause us to underestimate the drift, so at worst we are as bad as not adjusting at all, but never any worse.

What do you think?

@Tom:

Regarding localtime: full ACK.

5) True. --systohc assumes system clock is correct, so the hardware clock needs to be untouched for a reasonable amount of time to get a meaningful drift value because of the one-second granularity. (Just reboot and try 'hwclock --systohc --test --debug'.)

4) It will be if downtime is more than 24 hours.

7) hwclock adjustment needs an up to date adjustment file. If you don't share the same file between all OS, it will do wrong calculations. So to get correct system time before doing --systohc, you need to
- either: disable adjustment on all OS and use ntp
- or: disable adjustment and --systohc and ntp (only read hardware clock) on all OS except one which does the drift calculation then.
About underestimating the drift: Say the first OS has calculated a correct drift rate. Then you decide to install/clone a second OS and run that for two weeks and afterwards you reboot to change back to the first OS. Now the 2nd OS will update the hardware clock on shutdown and then the first OS will adjust the hardware clock for two weeks of drift. So if loosing 10 seconds per day, the clock is now 140 seconds in the future.

I think simply adding an option to rc.conf to treat the hardware clock as read-only (no adjustment, no --systohc) would simplify things very much when it comes to using ntp and/or dual-boot.

@Kurt:
5/4: thanks for the clarification.
7: the problem is that the first OS can never know whether or not the second OS has been running, or if the computer has been truly switched off. It will therefore be impossible to know whether or not to adjust for drift. If we always adjust for drift I think the potential error will be less than if we never adjust for drift, but either way we cannot trust the time we get. I think that, as with localtime, we should just admit that we cannot support this usecase, and add to our documentation that in a dualboot setting you need to use ntp to avoid creating a mess.

Would this work:
At boot/shutdown we always use "--noadjfile --utc", and if the user would like to adjust for drift s/he can enable a daemon that does "--adjust"/"--hctosys" at boot and "--systohc" at shutdown. The recommendation would then be to only enable this if you are not using dualboot nor ntp.

@Tom:
Sounds good to me, except for rc.shutdown which should do nothing at all then.

- rc.conf:
USEHWCLOCK = "yes"
DAEMONS=( clock ... )

- rc.sysinit 1st hwclock part:
if [[ $USEHWCLOCK = "yes" ]]; then
# enable rtc access
...
/sbin/hwclock --hctosys --utc --noadjfile
fi

- Drop 2nd hwclock part from rc.sysinit

- rc.d/clock start:
/sbin/hwclock --adjust --utc
/sbin/hwclock --hctosys --utc
add_daemon clock

- rc.d/clock stop:
/sbin/hwclock --systohc --utc

- rc.shutdown: Drop all hwclock stuff (!)

- cron.hourly/adjtime:
. /etc/rc.conf
. /etc/rc.d/functions
if in_array clock "${DAEMONS[@]}"; then
/sbin/hwclock --adjust --utc
fi

@Kurt: Nice work! I suggest renaming the daemon from "clock" to "hwclock" though. Also, would it be possible to avoid the new USEHWCLOCK variable in rc.conf, I think other distro's enable the rtc devices unconditionally, so why can't we?

I thought/read about more about this problem and I have one more concern:

A discontinuous ("jumping") system time, might cause weird bugs. Reading the hwclock man page I get the impression that running "hwclock --hctosys" when the system is running would be a bad idea (as it will cause the system time to jump). I guess this potential problem would apply both if we set the systemtime for the second time in rc.sysinit (as lots of stuff will be up and running by then), or if we do it in this new "clock" daemon.

To avoid this problem all together, what about just adjusting the hwclock for drift without reading this into the systemtime? I.e.:
"- rc.d/clock start:
/sbin/hwclock --adjust --utc
add_daemon clock"

This means that the systemtime will be off by the drift since last shutdown, but the drifts between each reboot will not accumulate. I.e., if your hwclock looses 1 second between each reboot and you have rebooted 10 times, then your systemtime will be off by 1 second (rather than 10 as would be the case if you did not adjust). What do you think about this?

Would you be able to make a git patch and post it to <arch-projects@archlinux.org> (with a reference to this discussion)?

@Tom: 'hwclock' is also OK for me. :)

From current rc.conf:
# HARDWARECLOCK: set to "UTC" or "localtime", any other value will result
# in the hardware clock being left untouched (useful for virtualization)

If I remember right, some troubles where reported with virtual machines with enabling rtc (or using hwclock ?)

--systohc needs correct system time. I don't think stepping the time is such a big problem as long as no daemons other than udev are up.

- rc.conf:
USEHWCLOCK="yes"
DAEMONS=(hwclock syslog-ng network netfs crond)

@Kurt:
I also remember something about virtualization being broken. I'd like to find a proper fix for this, rather than work around it like we are now. Anyway, that's a separate problem, so just leave it like it is for now.

About systohc: I see your point. Hmmm... how annoying.

Made a patch against git. This one keeps $HARDWARECLOCK for now, but changes the out of the box setting to "UTC" and also adds some warning comments to rc.conf. Whether the cron job adjustment is enabled now depends additionally on existence of /var/run/daemons/hwclock. This way it is also disabled when doing a '/etc/rc.d/hwclock stop' manually.

I will also post this to the projects list.