localedata: Avoid concurrently written locales in gen-locale.sh
Checks
| Context |
Check |
Description |
| redhat-pt-bot/TryBot-apply_patch |
success
|
Patch applied to master at the time it was sent
|
| linaro-tcwg-bot/tcwg_glibc_build--master-arm |
success
|
Build passed
|
| redhat-pt-bot/TryBot-32bit |
success
|
Build for i686
|
| linaro-tcwg-bot/tcwg_glibc_build--master-aarch64 |
success
|
Build passed
|
| linaro-tcwg-bot/tcwg_glibc_check--master-aarch64 |
fail
|
Test failed
|
| linaro-tcwg-bot/tcwg_glibc_check--master-arm |
fail
|
Patch failed to apply
|
Commit Message
There is no cross-directory exclusion of concurrent $(gen-locales)
usage. Parallel localedef calls can clobber locale data as it is
being loaded by tests.
With --no-hard-links, the separate touch invocation is no longer
required.
---
localedata/gen-locale.sh | 22 +++++++++++++++-------
1 file changed, 15 insertions(+), 7 deletions(-)
Comments
On 08/07/26 09:03, Florian Weimer wrote:
> There is no cross-directory exclusion of concurrent $(gen-locales)
> usage. Parallel localedef calls can clobber locale data as it is
> being loaded by tests.
>
> With --no-hard-links, the separate touch invocation is no longer
> required.
LGTM, thanks.
Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
>
> ---
> localedata/gen-locale.sh | 22 +++++++++++++++-------
> 1 file changed, 15 insertions(+), 7 deletions(-)
>
> diff --git a/localedata/gen-locale.sh b/localedata/gen-locale.sh
> index 4762c04284..b0ce27fbae 100644
> --- a/localedata/gen-locale.sh
> +++ b/localedata/gen-locale.sh
> @@ -32,15 +32,23 @@ generate_locale ()
> out=$3
> flags=$4
> ret=0
> +
> + # Use a staging area to avoid writing to locales concurrently.
> + # While this process is running, $$ is sufficiently unique.
> + # Use --no-hard-links to prevent localedef from accessing
> + # other staging areas.
> + stage="${common_objpfx}localedata/gen-locale.$$.tmp"
> +
> ${localedef_before_env} ${run_program_env} I18NPATH=../localedata \
> - ${localedef_after_env} $flags -f $charmap -i $input \
> - ${common_objpfx}localedata/$out || ret=$?
> - if [ $ret -eq 0 ]; then
> - # The makefile checks the timestamp of the LC_CTYPE file,
> - # but localedef won't have touched it if it was able to
> - # hard-link it to an existing file.
> - touch ${common_objpfx}localedata/$out/LC_CTYPE
> + ${localedef_after_env} $flags --no-hard-links -f $charmap -i $input \
> + $stage || ret=$?
> + if [ $ret -eq 0 ] ; then
> + # Ignore errors in case some other process has created the same locale.
> + # (Some versions of mv do not report an error here.)
How common it these versions?
> + mv -Tn $stage ${common_objpfx}localedata/$out 2>/dev/null || true
> + rm -rf $stage
> else
> + rm -rf $stage
> echo "Charmap: \"${charmap}\" Inputfile: \"${input}\"" \
> "Outputdir: \"${out}\" failed"
> exit 1
>
Florian Weimer <fweimer@redhat.com> writes:
> There is no cross-directory exclusion of concurrent $(gen-locales)
> usage. Parallel localedef calls can clobber locale data as it is
> being loaded by tests.
>
> With --no-hard-links, the separate touch invocation is no longer
> required.
>
> ---
> localedata/gen-locale.sh | 22 +++++++++++++++-------
> 1 file changed, 15 insertions(+), 7 deletions(-)
>
> diff --git a/localedata/gen-locale.sh b/localedata/gen-locale.sh
> index 4762c04284..b0ce27fbae 100644
> --- a/localedata/gen-locale.sh
> +++ b/localedata/gen-locale.sh
> @@ -32,15 +32,23 @@ generate_locale ()
> out=$3
> flags=$4
> ret=0
> +
> + # Use a staging area to avoid writing to locales concurrently.
> + # While this process is running, $$ is sufficiently unique.
Yes, this should be OK.
> + # Use --no-hard-links to prevent localedef from accessing
> + # other staging areas.
> + stage="${common_objpfx}localedata/gen-locale.$$.tmp"
> +
> ${localedef_before_env} ${run_program_env} I18NPATH=../localedata \
> - ${localedef_after_env} $flags -f $charmap -i $input \
> - ${common_objpfx}localedata/$out || ret=$?
> - if [ $ret -eq 0 ]; then
> - # The makefile checks the timestamp of the LC_CTYPE file,
> - # but localedef won't have touched it if it was able to
> - # hard-link it to an existing file.
> - touch ${common_objpfx}localedata/$out/LC_CTYPE
> + ${localedef_after_env} $flags --no-hard-links -f $charmap -i $input \
> + $stage || ret=$?
This could be improved to directly test the command but it can be done
later, no need to do it now.
> + if [ $ret -eq 0 ] ; then
> + # Ignore errors in case some other process has created the same
> locale.
OK.
> + # (Some versions of mv do not report an error here.)
> + mv -Tn $stage ${common_objpfx}localedata/$out 2>/dev/null || true
I'm sorry to ask this, but is mv -T sufficiently portable? It's not
POSIX but I don't recall if we require any extensions for coreutils
elsewhere in glibc.
> + rm -rf $stage
> else
> + rm -rf $stage
> echo "Charmap: \"${charmap}\" Inputfile: \"${input}\"" \
> "Outputdir: \"${out}\" failed"
> exit 1
The patch otherwise looks good, but please wait a brief period for
Adhemerval to comment before committing.
sam
On 08/07/26 09:18, Sam James wrote:
> Florian Weimer <fweimer@redhat.com> writes:
>
>> There is no cross-directory exclusion of concurrent $(gen-locales)
>> usage. Parallel localedef calls can clobber locale data as it is
>> being loaded by tests.
>>
>> With --no-hard-links, the separate touch invocation is no longer
>> required.
>>
>> ---
>> localedata/gen-locale.sh | 22 +++++++++++++++-------
>> 1 file changed, 15 insertions(+), 7 deletions(-)
>>
>> diff --git a/localedata/gen-locale.sh b/localedata/gen-locale.sh
>> index 4762c04284..b0ce27fbae 100644
>> --- a/localedata/gen-locale.sh
>> +++ b/localedata/gen-locale.sh
>> @@ -32,15 +32,23 @@ generate_locale ()
>> out=$3
>> flags=$4
>> ret=0
>> +
>> + # Use a staging area to avoid writing to locales concurrently.
>> + # While this process is running, $$ is sufficiently unique.
>
> Yes, this should be OK.
>
>> + # Use --no-hard-links to prevent localedef from accessing
>> + # other staging areas.
>> + stage="${common_objpfx}localedata/gen-locale.$$.tmp"
>> +
>> ${localedef_before_env} ${run_program_env} I18NPATH=../localedata \
>> - ${localedef_after_env} $flags -f $charmap -i $input \
>> - ${common_objpfx}localedata/$out || ret=$?
>> - if [ $ret -eq 0 ]; then
>> - # The makefile checks the timestamp of the LC_CTYPE file,
>> - # but localedef won't have touched it if it was able to
>> - # hard-link it to an existing file.
>> - touch ${common_objpfx}localedata/$out/LC_CTYPE
>> + ${localedef_after_env} $flags --no-hard-links -f $charmap -i $input \
>> + $stage || ret=$?
>
> This could be improved to directly test the command but it can be done
> later, no need to do it now.
Agree, I think we should fix for the release and this approach seems ok.
>
>> + if [ $ret -eq 0 ] ; then
>> + # Ignore errors in case some other process has created the same
>> locale.
>
> OK.
>
>> + # (Some versions of mv do not report an error here.)
>> + mv -Tn $stage ${common_objpfx}localedata/$out 2>/dev/null || true
>
> I'm sorry to ask this, but is mv -T sufficiently portable? It's not
> POSIX but I don't recall if we require any extensions for coreutils
> elsewhere in glibc.
Hum I was not aware of that, maybe we can adjust it after the release if
required.
>
>> + rm -rf $stage
>> else
>> + rm -rf $stage
>> echo "Charmap: \"${charmap}\" Inputfile: \"${input}\"" \
>> "Outputdir: \"${out}\" failed"
>> exit 1
>
> The patch otherwise looks good, but please wait a brief period for
> Adhemerval to comment before committing.
Looks good to me. Just curious Florian, how did you notice this issue?
* Adhemerval Zanella Netto:
> On 08/07/26 09:03, Florian Weimer wrote:
>> There is no cross-directory exclusion of concurrent $(gen-locales)
>> usage. Parallel localedef calls can clobber locale data as it is
>> being loaded by tests.
>>
>> With --no-hard-links, the separate touch invocation is no longer
>> required.
>
> LGTM, thanks.
>
> Reviewed-by: Adhemerval Zanella <adhemerval.zanella@linaro.org>
Thanks.
>> ---
>> localedata/gen-locale.sh | 22 +++++++++++++++-------
>> 1 file changed, 15 insertions(+), 7 deletions(-)
>>
>> diff --git a/localedata/gen-locale.sh b/localedata/gen-locale.sh
>> index 4762c04284..b0ce27fbae 100644
>> --- a/localedata/gen-locale.sh
>> +++ b/localedata/gen-locale.sh
>> @@ -32,15 +32,23 @@ generate_locale ()
>> out=$3
>> flags=$4
>> ret=0
>> +
>> + # Use a staging area to avoid writing to locales concurrently.
>> + # While this process is running, $$ is sufficiently unique.
>> + # Use --no-hard-links to prevent localedef from accessing
>> + # other staging areas.
>> + stage="${common_objpfx}localedata/gen-locale.$$.tmp"
>> +
>> ${localedef_before_env} ${run_program_env} I18NPATH=../localedata \
>> - ${localedef_after_env} $flags -f $charmap -i $input \
>> - ${common_objpfx}localedata/$out || ret=$?
>> - if [ $ret -eq 0 ]; then
>> - # The makefile checks the timestamp of the LC_CTYPE file,
>> - # but localedef won't have touched it if it was able to
>> - # hard-link it to an existing file.
>> - touch ${common_objpfx}localedata/$out/LC_CTYPE
>> + ${localedef_after_env} $flags --no-hard-links -f $charmap -i $input \
>> + $stage || ret=$?
>> + if [ $ret -eq 0 ] ; then
>> + # Ignore errors in case some other process has created the same locale.
>> + # (Some versions of mv do not report an error here.)
>
> How common it these versions?
I do not know. The behavior is clearly against the documentation.
I don't see an error with coreutils-9.7-9.fc43.x86_64. I'll report it.
It'll probably be treated as an error in the future.
Florian
* Sam James:
>> + # (Some versions of mv do not report an error here.)
>> + mv -Tn $stage ${common_objpfx}localedata/$out 2>/dev/null || true
>
> I'm sorry to ask this, but is mv -T sufficiently portable? It's not
> POSIX but I don't recall if we require any extensions for coreutils
> elsewhere in glibc.
I think I found a portable way to get the expected rename failure.
Well, not really portable because naturally, these details are
unspecified by POSIX. But I don't think we'll need the special options.
I'll send a new patch.
Thanks,
Florian
* Adhemerval Zanella Netto:
> Looks good to me. Just curious Florian, how did you notice this issue?
Tests which use locales are occasionally failing on cfarm120.osuosl.org,
a POWER10 machine with 192 hardware threads. Typically, there is
one test failure per run. It's not always the same test.
Thanks,
Florian
Am Mittwoch, 8. Juli 2026, 22:13:26 Japanische Normalzeit schrieb Florian Weimer:
> * Adhemerval Zanella Netto:
>
> > Looks good to me. Just curious Florian, how did you notice this issue?
>
> Tests which use locales are occasionally failing on cfarm120.osuosl.org,
> a POWER10 machine with 192 hardware threads. Typically, there is
> one test failure per run. It's not always the same test.
Hmmm that would explain odd results on our sparc machine (256 threads) too.
(Didn't post anything yet since I'm updating the host system first.)
>
> Thanks,
> Florian
>
>
@@ -32,15 +32,23 @@ generate_locale ()
out=$3
flags=$4
ret=0
+
+ # Use a staging area to avoid writing to locales concurrently.
+ # While this process is running, $$ is sufficiently unique.
+ # Use --no-hard-links to prevent localedef from accessing
+ # other staging areas.
+ stage="${common_objpfx}localedata/gen-locale.$$.tmp"
+
${localedef_before_env} ${run_program_env} I18NPATH=../localedata \
- ${localedef_after_env} $flags -f $charmap -i $input \
- ${common_objpfx}localedata/$out || ret=$?
- if [ $ret -eq 0 ]; then
- # The makefile checks the timestamp of the LC_CTYPE file,
- # but localedef won't have touched it if it was able to
- # hard-link it to an existing file.
- touch ${common_objpfx}localedata/$out/LC_CTYPE
+ ${localedef_after_env} $flags --no-hard-links -f $charmap -i $input \
+ $stage || ret=$?
+ if [ $ret -eq 0 ] ; then
+ # Ignore errors in case some other process has created the same locale.
+ # (Some versions of mv do not report an error here.)
+ mv -Tn $stage ${common_objpfx}localedata/$out 2>/dev/null || true
+ rm -rf $stage
else
+ rm -rf $stage
echo "Charmap: \"${charmap}\" Inputfile: \"${input}\"" \
"Outputdir: \"${out}\" failed"
exit 1