FS#9593 - Accents not working on console

Attached to Project: Arch Linux
Opened by Jorge López (panecillo) - Sunday, 17 February 2008, 10:32 GMT
Last edited by Tobias Powalowski (tpowa) - Tuesday, 04 March 2008, 10:29 GMT
Task Type Bug Report
Category System
Status Closed
Assigned To Tobias Powalowski (tpowa)
Roman Kyrylych (Romashka)
Architecture All
Severity Medium
Priority Normal
Reported Version 2007.08-2
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

Accents are not working on console (e.g. Alt+F1) when using encodings like iso-8859-1 or -2. A forum topic was created (http://bbs.archlinux.org/viewtopic.php?id=43779) within this topic.

Using Arch64.


Additional info:
* package version(s)
I don't know which packages could be related, but here are some package versions (please, ask for more versions if necessary):

coreutils 6.10-2
bash 3.2.033-2
filesystem 2007.11-6

* config and/or log files etc.

$ locale
LANG=es_ES@euro
LC_CTYPE="es_ES@euro"
LC_NUMERIC="es_ES@euro"
LC_TIME="es_ES@euro"
LC_COLLATE=C
LC_MONETARY="es_ES@euro"
LC_MESSAGES="es_ES@euro"
LC_PAPER="es_ES@euro"
LC_NAME="es_ES@euro"
LC_ADDRESS="es_ES@euro"
LC_TELEPHONE="es_ES@euro"
LC_MEASUREMENT="es_ES@euro"
LC_IDENTIFICATION="es_ES@euro"
LC_ALL=


Steps to reproduce:
Just boot, login and try to type an accented character.
Using "unicode_stop" make accented characters to be displayed right, but can't type them yet.
This task depends upon

Closed by  Tobias Powalowski (tpowa)
Tuesday, 04 March 2008, 10:29 GMT
Reason for closing:  Fixed
Comment by Mario DV (MarioTex) - Sunday, 17 February 2008, 19:57 GMT
I happen the same about x86, test with "loadkeys defkeymap es", seems as if kernel missing him table keyboard by default.
Comment by Jorge López (panecillo) - Sunday, 17 February 2008, 21:55 GMT
That's it, after doing it I can type accents with no problems. So maybe it's a kernel issue?
Comment by Mario DV (MarioTex) - Sunday, 17 February 2008, 23:23 GMT
I think that if, seems as if kernel missing the table default keyboard, with "loadkeys us es" also works, anyone know because the kernel missing table default keyboard?
Comment by Tobias Powalowski (tpowa) - Monday, 18 February 2008, 06:25 GMT
you are running a none utf locale and unicode is used, pretty strange
Comment by Mario DV (MarioTex) - Monday, 18 February 2008, 13:56 GMT
With LC_ALL=es_ES.utf8, with kernel 2.6.24 or higher, have never worked accents, if we run "dumpkeys ---compose-only" we see that all compose are null, if we run "loadkey -u us es", all compose are ok, for some reason when compiling the kernel does not include defkeymap of the sources, "/usr/src/linux-2.6.24.2/drivers/char/defkeymap.map", someone knows the reason?
Comment by Tobias Powalowski (tpowa) - Wednesday, 20 February 2008, 21:14 GMT Comment by Roman Kyrylych (Romashka) - Wednesday, 20 February 2008, 21:19 GMT Comment by Mario DV (MarioTex) - Wednesday, 20 February 2008, 22:52 GMT
OK, but currently i can not do this on my rc.conf, KEYMAP="defkeymap es" because at bug #8579
Comment by Roman Kyrylych (Romashka) - Wednesday, 20 February 2008, 23:06 GMT
you can try it with a patch attached at #8579.
Anyway, I'll come with a patch for this issue soon, it should be fixed in the next initscripts release.
Comment by Mario DV (MarioTex) - Wednesday, 20 February 2008, 23:10 GMT
ok, thanks
Comment by Roman Kyrylych (Romashka) - Thursday, 28 February 2008, 11:57 GMT
Mario, please try this patch with initscripts-2008.02-1
Comment by Mario DV (MarioTex) - Thursday, 28 February 2008, 17:50 GMT
My terminal es UTF-8 and the patch works well, thanks.
Also seems right for non UTF-8 terminals.
Comment by Roman Kyrylych (Romashka) - Thursday, 28 February 2008, 18:15 GMT
Jorge, can you confirm this too?
Comment by Jorge López (panecillo) - Thursday, 28 February 2008, 23:14 GMT
I'm using initscripts 2007.11-2, because I don't use testing repos. I hope that doesn't make too much difference on the final result.

I have tried running "kbd_mode -a" by hand but I still can't type accented letters. However, I can type the special character ñ (ñ).
Comment by Roman Kyrylych (Romashka) - Friday, 29 February 2008, 07:53 GMT
what keymap are you using?
and do accents display correctly after kbd_mode -a or unicode_stop is needed?
Comment by Jorge López (panecillo) - Friday, 29 February 2008, 08:04 GMT
unicode_stop is needed to make accents being correctly displayed.

I'm using KEYMAP="es.map.gz".
Comment by Roman Kyrylych (Romashka) - Friday, 29 February 2008, 08:17 GMT
ok, I should try it by myself, just tell me how to type those accents :P
Comment by Jorge López (panecillo) - Friday, 29 February 2008, 08:39 GMT
Hehe, typing them using spanish layout keyboard is as simple as pressing accent key and then the vowel to be accented.

BTW, first comment by Mario told to run 'loadkeys defkeymap es' and that worked, althought accents didn't display right.

I think typing special chars and displaying special chars are 2 different issues:

1. 'loadkeys defkeymap es' made possible to type special chars.
2. 'unicode_stop' made special chars being displayed correctly.

Hope it helps.
Comment by Mario DV (MarioTex) - Friday, 29 February 2008, 18:28 GMT

Jorge I think that if you use kernel 2.6.24 you should change the console to UTF-8.
In /etc/locale.gen descoment --> es_ES.UTF-8 and then run "locale-gen".
In /etc/rc.conf --> LOCALE="es_ES.utf8" and KEYMAP="us es euro2"
And a good source unicode, "LatArCyrHeb-16" or "Terminus" terminus is not installed by default
In rc.conf --> CONSOLEFONT="ter-v16f".

Reset and test.

Until they upgrade initscripts, KEYMAP="us es euro2" not working, meanwhile executed manually
"loadkeys us es euro2"

kernel 2.6.24 and console non-utf8 i think not get along very well.

Salu2.
Comment by Roman Kyrylych (Romashka) - Friday, 29 February 2008, 21:14 GMT
non-UTF-8 consoles work fine on 2.6.24 kernel, just our initscripts had to be adapted to the fact that UTF-8 is the default mode now.
The attached patch (for 2008.02-1) fixes the issue completely (tested with uk_UA.UTF-8 and uk_UA.KOI8-U) and will be included in the next initscripts version.
Comment by Mario DV (MarioTex) - Saturday, 01 March 2008, 02:13 GMT
Why not include in rc.sysinit defkeymap, "/bin/loadkeys -q -u defkeymap $KEYMAP"?

And thus would not be necessary to include in rc.conf ----> KEYMAP="defkeymap es etc..."
Comment by Roman Kyrylych (Romashka) - Saturday, 01 March 2008, 08:59 GMT
Thus bugreport is about issues with non-UTF-8 consoles on 2.6.24 (which are now fixed, I believe,
you suggest to load defkeymap for UTF-8 consoles,
I'm confused.
Could you explain what, in your opinion, defkeymap should fix and in what situations (UTF-8 or not, multiple keymaps, some special situations), and what has changed since previous kernel/initscripts/kbd.

"seems as if kernel missing him table keyboard by default" is incorrect, there were no changes in kernel in this area,
and there's no way to specify "default keyboard layout" at all in kernel.

Jorge - please try my latest patch (to test multiple keymaps with initscripts-2007.11 you can use this patch: http://bugs.archlinux.org/task/8579?getfile=1553 )
Comment by Jorge López (panecillo) - Saturday, 01 March 2008, 09:12 GMT
There was no luck with last patch, Roman. I see a message saying that "es.map.gz" is being loaded in legacy mode, but I still can't type accents, and special characters from program outputs are not being displayed correctly, just a blank square instead of them.
Comment by Roman Kyrylych (Romashka) - Saturday, 01 March 2008, 09:29 GMT
hm, you've said "2. 'unicode_stop' made special chars being displayed correctly." before
unicode_stop contains 3 commands:
* kbd_mode -a
* printf '\e%%@'
* stty -iutf8
the first two are executed by patched initscripts, but not stty -iutf8 (which seems to have no effect).
could you add stty -iutf8 after kbd_mode -a in the patched rc.sysinit and see if something has changed?

Important moment: *if* you are trying your console after switching from X (e.g. Ctrl-Alt-F1), and you don't use a framebuffer
- then you will have to do 'setfont <yourfont>' because for some weird reason font is reset to default in this specific case (at least on my system with nvidia)


"I see a message saying that "es.map.gz" is being loaded in legacy mode"
there's no such message in the new patched rc.sysinit:
- status "Loading Keyboard Map: $KEYMAP in legacy mode" /bin/loadkeys -q $KEYMAP
. . .
+ status "Loading Keyboard Map: $KEYMAP" /bin/loadkeys -q $KEYMAP
Please verify that you've applied the patch correctly.
Or just copy-paste this part of rc.sysinit instead of the old one: http://rafb.net/p/dUzpPh72.html
Comment by Mario DV (MarioTex) - Saturday, 01 March 2008, 17:58 GMT
My English is not very good, the attempts to explain with an example

It does not matter if the console is in UTF8 or non-UTF8

Kernel 2.6.23 in /etc/rc.conf --> KEYMAP="es", if we run:

dumpkeys --compose-only > file ; vim file

compose '´' 'A' to 'Á'
compose '´' 'a' to 'á '
etc ....

Kernel 2.6.24 in /etc/rc.conf --> KEYMAP="es", if we run:

dumpkeys ---compose-only > file ; vim file

compose '´' 'A' to 'ÿ'
compose '´' 'a' to 'ÿ'
compose '´' 'e' to 'ÿ'
compose '´' 'i' to 'ÿ'
..... ...
etc .... Always --> ÿ


If /etc/rc.conf --> KEYMAP="us" this problem does not exist.

Because us.map.gz include :

keymaps 0-2,4-6,8-9,12
alt_is_meta
include "qwerty-layout"
include "linux-with-alt-and-altgr"
include "compose.latin1" <-------------- Compose
include "euro1.map"
strings as usual
.....



But es.map.gz not include "compose.latin1" use compose the kernel
but in kernel 2.6.24 no exist valid compose in the kernel.

In Kernels 2.6.23 default compose was correct and /etc/rc.conf --> KEYMAP="es" worked

But kernels 2.6.24 we need KEYMAP="us es" or KEYMAP="defkeymap es" for accents that work well

Jorge he works well ñ, because ñ is not a compose, ñ is defined as key symbol

dumpkeys |fgrep ntilde ----> keycode 39 = ntilde

Solution: Change "es.map.gz" or include defkeymap in rc.conf, KEYMAP="defkeymap es"

Comment by Roman Kyrylych (Romashka) - Saturday, 01 March 2008, 19:05 GMT
Thank you for the explanation.
Could you please report the issue to bugzilla.kernel.org ?

Now what I've discovered:

Kernel's defkeymap is not changed for years:
http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=history;f=drivers/char/defkeymap.map
Note that the last line is:
compose 'i' 'j' to 'ÿ'

Also note that kernel's defkeymap is (and always was) in UTF-8 while kbd's defkeymap is (and always was) in ISO-8859-1,
but I guess this doesn't matter.

The only commit I've found that could be related to it somehow: http://git.kernel.org/?p=linux/kernel/git/stable/linux-2.6.24.y.git;a=commit;h=04c71976500352d02f60616d2b960267d8c5fe24
I think that some bug was introduced that makes the last compose setting to apply for all keys, but only for bundled defkeymap, because defkeymap that is loaded by loadkeys works.
Kernel 2.6.25rc3-git3 doesn't seem to have a fix for this as far as I see.

I will check if always loading defkeymap in rc.sysinit won't hurt other keymaps and then make a change in the next initscripts version.

IMHO the better solution would be to add 'include "compose.latin1"' to all keymaps that require this, but I guess this will be a long-time process.
Comment by Mario DV (MarioTex) - Saturday, 01 March 2008, 20:35 GMT
> Could you please report the issue to bugzilla.kernel.org ?

Uff! My time and my English is not so much for, but I am going to try.

>I will check if always loading defkeymap in rc.sysinit won't hurt other keymaps and then make a change in the next initscripts >version.

ok Thanks.

>IMHO the better solution would be to add 'include "compose.latin1"' to all keymaps that require this, but I guess this will be a >long-time process.

True.
Comment by Jorge López (panecillo) - Sunday, 02 March 2008, 10:28 GMT
Roman, I think I got confused and tried the patch you mentioned ("to test multiple keymaps with initscripts-2007.11 you can use this patch...") believing it was the same patch but applicable to 2007.11.

I'm not using testing, is there any possibility to have a simple patch to try on my installation?
Comment by Roman Kyrylych (Romashka) - Sunday, 02 March 2008, 14:42 GMT
Jorge, here's a patch for initscripts-2007.11-2
to apply it run this as root:
# cd /etc
# patch -Np0 -i /home/jorge/downloads/accents-fix.patch (or whererver you've downloaded it)
Comment by Mario DV (MarioTex) - Sunday, 02 March 2008, 15:54 GMT
Roman, I report the issue to bugzilla.kernel.org, Bug #10143

I think that the problem has begun with this patch:

http://lkml.org/lkml/2007/10/18/475
Comment by Roman Kyrylych (Romashka) - Sunday, 02 March 2008, 16:04 GMT
Thanks!

hm, nice catch, I suggest to post a comment about that patch to the kernel's bugreport
Comment by Mario DV (MarioTex) - Sunday, 02 March 2008, 22:51 GMT

bugzilla.kernel.org, Bug #10143

> Fix (not tested)

> Aaargl, stupid signed chars... int c = 'é' produces a negative value... The
> attached (untested) patch should just work fine.

They are quick , in win we would have to wait for the next servipack :-).
Comment by Roman Kyrylych (Romashka) - Sunday, 02 March 2008, 23:50 GMT
hm, I wonder if I should revert my "always load defkeymap" commit after applying this patch...
need to test it first anyway
Comment by Mario DV (MarioTex) - Monday, 03 March 2008, 00:54 GMT
Patch seems ok, one option is that the maintainer the kernel the arch
can include the patch, and you do not have to include in rc.sysinit defkeymap.
Comment by Jorge López (panecillo) - Monday, 03 March 2008, 17:33 GMT
Roman, last patch worked right for me. I can see and type accented chars.
Comment by Roman Kyrylych (Romashka) - Monday, 03 March 2008, 22:26 GMT
@Jorge: great, thanks for confirmation!
@Mario: the patch will be included in the next kernel, I think about removing defkeymap loading before new initscripts and kbd are released (which should happen not sooner than a kernel release), though not sure yet...

Loading...