FS#25599 - [initscripts] fsck output encoding broken when DAEMON_LOCALE is set to "yes"

Attached to Project: Arch Linux
Opened by mkkot (mkkot) - Tuesday, 16 August 2011, 08:50 GMT
Last edited by Tom Gundersen (tomegun) - Wednesday, 02 May 2012, 22:14 GMT
Task Type Bug Report
Category System
Status Closed
Assigned To Tom Gundersen (tomegun)
Architecture x86_64
Severity Very Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

Since you introduced DAEMON_LOCALE in /etc/rc.conf, I set it to ="yes". Now, when starting system, I can observe such strange behaviour:

Udev starts, then fsck checks filesystems, says they are clean, boot process continues. Checking filesystems is problably done by /etc/rc.d/functions:
# Check local filesystems
fsck_all() {
fsck -A -T -C"$FSCK_FD" -a -t "no${NETFS//,/,no},noopts=_netdev" $FORCEFSCK

Fsck output is displayed as it supposed to be, translated, with national characters. Then, probably when next function from /etc/rc.d/functions is called, character encoding from fsck gets broken:
# Function for setting console font if required
set_consolefont()
...

You can see the effect here:
http://vimeo.com/27761144 (when it gets converted) or directly:
http://www.linux-porady.multimo.pl/inne/video/IMG_0177.AVI

As you can see "bloków" is changed to blok[some strange character]w


My rc.conf:

LOCALE="pl_PL.utf8"
HARDWARECLOCK="UTC"
TIMEZONE="Europe/Warsaw"
KEYMAP="pl"
CONSOLEFONT="lat2-16"
CONSOLEMAP="8859-2"
USECOLOR="yes"
DAEMON_LOCALE="yes"
This task depends upon

Closed by  Tom Gundersen (tomegun)
Wednesday, 02 May 2012, 22:14 GMT
Reason for closing:  Fixed
Comment by Tom Gundersen (tomegun) - Tuesday, 16 August 2011, 22:08 GMT
If you do the fsck manually after boot, are the correct characters displayed?
Comment by mkkot (mkkot) - Tuesday, 16 August 2011, 22:14 GMT
That's really strange but yes, in text console (I mean without X) I have no problems with native characters. I can write them and read unaffected. Of course it works in X too.
Comment by Tom Gundersen (tomegun) - Tuesday, 16 August 2011, 22:51 GMT
I have no solution yet, just a couple of comments:

You can delete your CONOSELMAP from rc.conf, as that does not take effect in utf8 mode.

This issue can be reproduced by using the attached file, as follows:

# setfont
# cat font-test
# setfont lat2-16

I guess this issue could be solved by moving the font setting to happen before any localised messages are printed. I'll have to look into if that will cause any problems (if so it will be with a /usr partition).
   font-test (0 KiB)
Comment by mkkot (mkkot) - Tuesday, 16 August 2011, 23:10 GMT
It seems you have detected the source of the problem. I'll look into it while being on my affected system, now writting from my laptop. However, I'm almost sure that CONSOLEMAP does make a difference. In /usr/share/kbd/consoletrans you can find files which names suggest that they have to do something with UTF. While trying to solve this myself, I was messing around with rc.conf and CONSOLEMAP had to be set to 8859-2, as it's stated here: https://wiki.archlinux.org/index.php/Fonts#Changing_the_default_font
(I even made some expansions on that page and wanted to check if CONSOLEMAP really has to be 8859-2. I left this untouched, probably had a good reason. I'll check it again tomorrow.)
Comment by Tom Gundersen (tomegun) - Tuesday, 16 August 2011, 23:40 GMT
If I understood correctly, the CONSOLEMAP is used to translate from the charset of your locale to unicode, but should not be needed if your locale is in utf8. We do ignore CONSOLEMAP in rc.sysinit if your locale is utf8 (I did not do this, and it is from before my time, so I don't know the details about why this is done explicitly).

Let me know what you find out (by the way, you can easily play around on any system using setfont and the file I provided. Calling "setfont" resets to the default, and then you can try calling it with a font and using -m you can also provide a consolemap).
Comment by mkkot (mkkot) - Wednesday, 17 August 2011, 20:02 GMT
Okay, I have found something odd for me and maybe this is the point where we should start:
http://www.linux-porady.multimo.pl/inne/IMG_0178.AVI
And that is how it looks directly after log in (this capture is unfortunately more blurry):
http://www.linux-porady.multimo.pl/inne/IMG_0180.AVI

This video shows your "óóó" file (which is UTF-8) displayed on tty with different setfont configurations. This command changes how fonts are displayed on console and generates wrong encoding on captions already written but in all cases it gives proper encoding when 'cat font-test' is called again. I'm wondering if this is due to CONSOLEMAP variable in rc.conf. Going to check this right now. Unfortunately I'm not sure how I should interpret what is happening so I need some guidance here.
Comment by mkkot (mkkot) - Wednesday, 17 August 2011, 20:10 GMT
You have right with this CONSOLEMAP. Commenting it doesn't change fsck boot behaviour and 'cat font-test' results.
Comment by mkkot (mkkot) - Thursday, 25 August 2011, 13:33 GMT
Nope, it doesn't make a difference only with letter ó. Fsck message changed slightly when nofitied about next filesystem check and there was a broken encoding. So it's important to set CONSOLEMAP="8859-2".

// Edit: okay, I set it to 8859-2, then fsck checked my disk. Before that I removed lost+found folder, thinking it's not needed. So I saw some output about fsck modyfying filesystem and the encoding of those messages was wrong.
Comment by Karol Błażewicz (karol) - Saturday, 28 April 2012, 11:00 GMT
Adding 'consolefont" to HOOKS in /etc/mkinitcpio.conf should fix this.
You need to rebuild the kernel image for the change to take effect.

Loading...