FS#36955 - [glibc] create locales in parallel

Attached to Project: Arch Linux
Opened by Dave Reisner (falconindy) - Monday, 16 September 2013, 19:48 GMT
Last edited by Allan McRae (Allan) - Monday, 14 October 2013, 05:26 GMT
Task Type Feature Request
Category Packages: Core
Status Closed
Assigned To Allan McRae (Allan)
Dave Reisner (falconindy)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

We apparently took /usr/bin/locale-gen from Debian, and it's slow and awful. I mentioned to you that I modified the script to generate locales in parallel, and it's far faster. On my machine...

Before
$ time locale-gen
Generating locales...
pt_PT.UTF-8... done
de_DE.UTF-8... done
en_US.UTF-8... done
en_US.ISO-8859-1... done
ru_RU.UTF-8... done
tr_TR.UTF-8... done
zh_CN.UTF-8... done
cy_GB.UTF-8... done
Generation complete.

real 0m6.759s
user 0m6.191s
sys 0m0.439s

---------------------------------

After:
$ time locale-gen
Generating locales...
pt_PT.UTF-8
de_DE.UTF-8
en_US.UTF-8
tr_TR.UTF-8
ru_RU.UTF-8
cy_GB.UTF-8
zh_CN.UTF-8
en_US.ISO-8859-1
Generation complete.

real 0m1.538s
user 0m6.003s
sys 0m0.418s
This task depends upon

Closed by  Allan McRae (Allan)
Monday, 14 October 2013, 05:26 GMT
Reason for closing:  Implemented
Comment by Eric Belanger (Snowman) - Monday, 16 September 2013, 20:20 GMT
You forgot to mention what modifications you did.
Comment by Dave Reisner (falconindy) - Monday, 16 September 2013, 20:30 GMT
Holy crap. See attached.
   parallel-locale-gen.diff (1.8 KiB)
Comment by Allan McRae (Allan) - Tuesday, 17 September 2013, 01:39 GMT
Changes look fine. I'll pull them next time I do something with glibc.

Notes for me to approach upstream for a unified solution...

Arch, Debian, Gentoo(?): Bash script
Fedora: Lua thing
openSuse: Ships all locales
Comment by Fabian Kosmale (Inkane) - Saturday, 12 October 2013, 07:49 GMT
  • Field changed: Percent Complete (100% → 0%)
Not really a request to reopen, but this might be interesting for you to know: With this patch, locale-gen might be too parallel for some use cases, where many locales are enabled. This might be e.g. the case when you build a live ISO, like we do in Chakra, where we build all locales supported by glibc. With the patch, locale-gen used about 19 GB of RAM (all that the build server had) and 8 cores. Systems with lower resources will probably suffer from swapping or trigger the OOM killer. This use case might however arise to seldom to be of concern. To fix it, one would need some kind of job control. However using (GNU) parallel would pull in a new dependency, and writing a program in C would likely overcomplicate what was a simple job.
Comment by Allan McRae (Allan) - Saturday, 12 October 2013, 09:04 GMT
@Dave: See above. Needs fixed or this gets reverted.
Comment by Allan McRae (Allan) - Saturday, 12 October 2013, 09:05 GMT
@Fabian: Out of interest, how much space does generating all locales take up?
Comment by Fabian Kosmale (Inkane) - Saturday, 12 October 2013, 12:32 GMT
@Allan: The generated locale-archive has a size of 95M according to ls -h.
Comment by Allan McRae (Allan) - Saturday, 12 October 2013, 12:38 GMT
OK... not taking the openSuse approach of shipping them all then!
Comment by Dave Reisner (falconindy) - Saturday, 12 October 2013, 14:24 GMT
Easy enough, see attached patch. I chose 10 rather arbitrarily.
   throttle.diff (0.4 KiB)
Comment by Fabian Kosmale (Inkane) - Saturday, 12 October 2013, 18:51 GMT
@Dave: Maybe one could use something along the lines of $(< /proc/cpuinfo | grep processor | wc -l ) instead of the arbitrary 10?
Comment by Dave Reisner (falconindy) - Saturday, 12 October 2013, 19:16 GMT
Sure, see attached.
   throttle.diff (0.6 KiB)
Comment by Allan McRae (Allan) - Monday, 14 October 2013, 05:22 GMT
Added in SVN. Also set-up to only generate locales when they have an update.

Loading...