[v3] localedata: Avoid concurrently written locales in gen-locale.sh
Checks
| Context |
Check |
Description |
| redhat-pt-bot/TryBot-apply_patch |
success
|
Patch applied to master at the time it was sent
|
| linaro-tcwg-bot/tcwg_glibc_build--master-arm |
success
|
Build passed
|
| linaro-tcwg-bot/tcwg_glibc_build--master-aarch64 |
success
|
Build passed
|
| redhat-pt-bot/TryBot-32bit |
success
|
Build for i686
|
| linaro-tcwg-bot/tcwg_glibc_check--master-aarch64 |
success
|
Test passed
|
| linaro-tcwg-bot/tcwg_glibc_check--master-arm |
fail
|
Patch failed to apply
|
Commit Message
There is no cross-directory exclusion of concurrent $(gen-locales)
usage. Parallel localedef calls can clobber locale data as it is
being loaded by tests.
With --no-hard-links, the separate touch invocation is now longer
necessary.
---
v2: Avoid mv options. Probably more portable.
v3: Add missing mkdir call. Test is still running.
localedata/gen-locale.sh | 26 ++++++++++++++++++--------
1 file changed, 18 insertions(+), 8 deletions(-)
base-commit: f142bca0e6d9e501b8e704697a827d0b1c6d771e
Comments
* Florian Weimer:
> There is no cross-directory exclusion of concurrent $(gen-locales)
> usage. Parallel localedef calls can clobber locale data as it is
> being loaded by tests.
>
> With --no-hard-links, the separate touch invocation is now longer
> necessary.
>
> ---
> v2: Avoid mv options. Probably more portable.
> v3: Add missing mkdir call. Test is still running.
> localedata/gen-locale.sh | 26 ++++++++++++++++++--------
> 1 file changed, 18 insertions(+), 8 deletions(-)
>
> diff --git a/localedata/gen-locale.sh b/localedata/gen-locale.sh
> index 4762c04284..013d26eafb 100644
> --- a/localedata/gen-locale.sh
> +++ b/localedata/gen-locale.sh
> @@ -32,17 +32,27 @@ generate_locale ()
> out=$3
> flags=$4
> ret=0
> +
> + # Use a staging area to avoid writing to locales concurrently.
> + # While this process is running, $$ is sufficiently unique.
> + # Use --no-hard-links to prevent localedef from accessing
> + # other staging areas.
> + stage="${common_objpfx}localedata/gen-locale.$$.tmp"
> + mkdir "$stage" 2>/dev/null || true
Oh well. With these changes, --no-hard-links is no longer needed
(no peer directories to check). Should I leave it in?
Thanks,
Florian
On 08/07/26 10:14, Florian Weimer wrote:
> * Florian Weimer:
>
>> There is no cross-directory exclusion of concurrent $(gen-locales)
>> usage. Parallel localedef calls can clobber locale data as it is
>> being loaded by tests.
>>
>> With --no-hard-links, the separate touch invocation is now longer
>> necessary.
>>
>> ---
>> v2: Avoid mv options. Probably more portable.
>> v3: Add missing mkdir call. Test is still running.
>> localedata/gen-locale.sh | 26 ++++++++++++++++++--------
>> 1 file changed, 18 insertions(+), 8 deletions(-)
>>
>> diff --git a/localedata/gen-locale.sh b/localedata/gen-locale.sh
>> index 4762c04284..013d26eafb 100644
>> --- a/localedata/gen-locale.sh
>> +++ b/localedata/gen-locale.sh
>> @@ -32,17 +32,27 @@ generate_locale ()
>> out=$3
>> flags=$4
>> ret=0
>> +
>> + # Use a staging area to avoid writing to locales concurrently.
>> + # While this process is running, $$ is sufficiently unique.
>> + # Use --no-hard-links to prevent localedef from accessing
>> + # other staging areas.
>> + stage="${common_objpfx}localedata/gen-locale.$$.tmp"
>> + mkdir "$stage" 2>/dev/null || true
>
> Oh well. With these changes, --no-hard-links is no longer needed
> (no peer directories to check). Should I leave it in?
I would suggest to remove it, so it makes the command intention clear.
@@ -32,17 +32,27 @@ generate_locale ()
out=$3
flags=$4
ret=0
+
+ # Use a staging area to avoid writing to locales concurrently.
+ # While this process is running, $$ is sufficiently unique.
+ # Use --no-hard-links to prevent localedef from accessing
+ # other staging areas.
+ stage="${common_objpfx}localedata/gen-locale.$$.tmp"
+ mkdir "$stage" 2>/dev/null || true
+
${localedef_before_env} ${run_program_env} I18NPATH=../localedata \
- ${localedef_after_env} $flags -f $charmap -i $input \
- ${common_objpfx}localedata/$out || ret=$?
- if [ $ret -eq 0 ]; then
- # The makefile checks the timestamp of the LC_CTYPE file,
- # but localedef won't have touched it if it was able to
- # hard-link it to an existing file.
- touch ${common_objpfx}localedata/$out/LC_CTYPE
+ ${localedef_after_env} $flags --no-hard-links -f $charmap -i $input \
+ $stage/$out || ret=$?
+ if [ $ret -eq 0 ] ; then
+ # Ignore errors in case some other process has created the same locale.
+ # This rename operation should be atomic, and it should fail if the
+ # $out locale already exists (rename fails with ENOTEMPTY).
+ mv $stage/$out ${common_objpfx}localedata/. 2>/dev/null || true
+ rm -rf $stage
else
+ rm -rf $stage
echo "Charmap: \"${charmap}\" Inputfile: \"${input}\"" \
- "Outputdir: \"${out}\" failed"
+ "Outputdir: \"${out}\" failed (exit status $ret)"
exit 1
fi
}