[RFC,v2,3/6] Implement the %OB specifier - alternative month names (bug 10871)

  Some languages (Slavic, Baltic, etc.) require a genitive case of the
month name when formatting a full date (with the day number) while
they require a nominative case when referring to the month standalone.

strftime() now implements a %OB format specifier which generates an
alternative month name.  For the languages which require both nominative
and genitive case of the month names it is expected that it outputs
the alternative month name which should be a nominative (sic!) case
or whatever is appropriate when formatting the month names standalone
(without a day).  For those languages %B will return the basic month
name but from now it is expected to be a genitive case or whatever is
appropriate when formatting full dates (including both day and month).
This means that all applications using %B to retrieve the month
name standalone should use %OB from now.  For those languages which
do not use different (nominative and genitive) cases of the month
name or do not yet have their locales updated %OB will retrieve
the same string as %B so moving to %OB will be harmless as long
as the version of glibc which supports this feature is used.

strptime() now accepts both nominative and genitive month names.

[BZ #10871]
* time/strftime_l.c: %OB format for alternative month names added.
* time/strptime_l.c: alternative month names also recognized.
---
 time/strftime_l.c | 12 ++++++++++--
 time/strptime_l.c | 28 ++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+), 2 deletions(-)

 # define HERE_AM_STR (_nl_C_LC_TIME.values[_NL_ITEM_INDEX (AM_STR)].string)
@@ -403,6 +405,22 @@ __strptime_internal (const char *rp, const char *fmt,
struct tm *tmp,
 	      if (s.decided !=raw)
 		{
 		  trp = rp;
+#ifdef _LIBC
+		  /* First check if the alt month is provided.  */
+		  if (_NL_CURRENT (LC_TIME, ALTMON_1 + cnt)
+		      && * (_NL_CURRENT (LC_TIME, ALTMON_1 + cnt))
+		      && match_string (_NL_CURRENT (LC_TIME, ALTMON_1 + cnt), trp)
+		      && trp > rp_longest)
+		    {
+		      rp_longest = trp;
+		      cnt_longest = cnt;
+		      if (s.decided == not
+			  && strcmp (_NL_CURRENT (LC_TIME, ALTMON_1 + cnt),
+				     alt_month_name[cnt]))
+			decided_longest = loc;
+		    }
+		  trp = rp;
+#endif
 		  if (match_string (_NL_CURRENT (LC_TIME, MON_1 + cnt), trp)
 		      && trp > rp_longest)
 		    {
@@ -429,6 +447,12 @@ __strptime_internal (const char *rp, const char *fmt,
struct tm *tmp,
 	      if (s.decided != loc
 		  && (((trp = rp, match_string (month_name[cnt], trp))
 		       && trp > rp_longest)
+#ifdef _LIBC
+		      || ((trp = rp, alt_month_name[cnt]
+			  && alt_month_name[cnt][0]
+			  && match_string (alt_month_name[cnt], trp))
+			  && trp > rp_longest)
+#endif
 		      || ((trp = rp, match_string (ab_month_name[cnt], trp))
 			  && trp > rp_longest)))
 		{
@@ -1016,6 +1040,10 @@ __strptime_internal (const char *rp, const char *fmt,
struct tm *tmp,
 	case 'O':
 	  switch (*fmt++)
 	    {
+	    case 'B':
+	      /* Undo the increment and continue.  */
+	      fmt--;
+	      break;
 	    case 'd':
 	    case 'e':
 	      /* Match day of month using alternate numeric symbols.  */
  

On 03/25/2016 01:55 AM, Rafal Luzynski wrote:

> strftime() now implements a %OB format specifier which generates an
> alternative month name.  For the languages which require both nominative
> and genitive case of the month names it is expected that it outputs
> the alternative month name which should be a nominative (sic!) case
> or whatever is appropriate when formatting the month names standalone
> (without a day).  For those languages %B will return the basic month
> name but from now it is expected to be a genitive case or whatever is
> appropriate when formatting full dates (including both day and month).
> This means that all applications using %B to retrieve the month
> name standalone should use %OB from now.  For those languages which
> do not use different (nominative and genitive) cases of the month
> name or do not yet have their locales updated %OB will retrieve
> the same string as %B so moving to %OB will be harmless as long
> as the version of glibc which supports this feature is used.

What's the status here on the POSIX side?  It's marked as Accepted in
the tracker, but the changes apparently haven't been folded into
subsequent editions.

Florian
  

On Tue, Mar 29, 2016 at 11:02:08AM +0200, Florian Weimer wrote:
> On 03/25/2016 01:55 AM, Rafal Luzynski wrote:
> 
> > strftime() now implements a %OB format specifier which generates an
> > alternative month name.  For the languages which require both nominative
> > and genitive case of the month names it is expected that it outputs
> > the alternative month name which should be a nominative (sic!) case
> > or whatever is appropriate when formatting the month names standalone
> > (without a day).  For those languages %B will return the basic month
> > name but from now it is expected to be a genitive case or whatever is
> > appropriate when formatting full dates (including both day and month).
> > This means that all applications using %B to retrieve the month
> > name standalone should use %OB from now.  For those languages which
> > do not use different (nominative and genitive) cases of the month
> > name or do not yet have their locales updated %OB will retrieve
> > the same string as %B so moving to %OB will be harmless as long
> > as the version of glibc which supports this feature is used.
> 
> What's the status here on the POSIX side?  It's marked as Accepted in
> the tracker, but the changes apparently haven't been folded into
> subsequent editions.

I would also like to add this to ISO 30112. Do you have text in standardese
that could be added to the text?

best regards
keld
  

On 03/24/2016 08:55 PM, Rafal Luzynski wrote:
> Some languages (Slavic, Baltic, etc.) require a genitive case of the
> month name when formatting a full date (with the day number) while
> they require a nominative case when referring to the month standalone.
> 
> strftime() now implements a %OB format specifier which generates an
> alternative month name.  For the languages which require both nominative
> and genitive case of the month names it is expected that it outputs
> the alternative month name which should be a nominative (sic!) case
> or whatever is appropriate when formatting the month names standalone
> (without a day).  For those languages %B will return the basic month
> name but from now it is expected to be a genitive case or whatever is
> appropriate when formatting full dates (including both day and month).
> This means that all applications using %B to retrieve the month
> name standalone should use %OB from now.  For those languages which
> do not use different (nominative and genitive) cases of the month
> name or do not yet have their locales updated %OB will retrieve
> the same string as %B so moving to %OB will be harmless as long
> as the version of glibc which supports this feature is used.
> 
> strptime() now accepts both nominative and genitive month names.

Please make sure you read through the "Contribution Checklist"
https://sourceware.org/glibc/wiki/Contribution%20checklist

Do you have copyright assignment with the FSF for glibc?

Your change is more than the accepted 11 lines and thus
legally significant and requiring copyright assignment.
  

On Fri, Mar 25, 2016 at 01:55:13AM +0100, Rafal Luzynski wrote:
[...]
> This means that all applications using %B to retrieve the month
> name standalone should use %OB from now.

Such applications as cal(1) would not be able to print month names
properly in a way that would work with different glibc versions.
Looks like this is a change incompatible in both ways.
  

29.03.2016 16:15 Carlos O'Donell <carlos@redhat.com> wrote:
> [...]
> Please make sure you read through the "Contribution Checklist"
> https://sourceware.org/glibc/wiki/Contribution%20checklist
>
> Do you have copyright assignment with the FSF for glibc?
>
> Your change is more than the accepted 11 lines and thus
> legally significant and requiring copyright assignment.

No. I thought it would not be necessary but if it is then
I will sign.

By the way, are there any other technical issues which must
be fixed or should I assume that it looks OK and will be
accepted as soon as I sign the agreement?

Thank you everybody for your replies and comments.

Regards,

Rafal
  

29.03.2016 16:31 "Dmitry V. Levin" <ldv@altlinux.org> wrote:
>
>
> On Fri, Mar 25, 2016 at 01:55:13AM +0100, Rafal Luzynski wrote:
> [...]
> > This means that all applications using %B to retrieve the month
> > name standalone should use %OB from now.
>
> Such applications as cal(1) would not be able to print month names
> properly in a way that would work with different glibc versions.
> Looks like this is a change incompatible in both ways.

Yes, that's exactly what will happen. Such applications must be
updated. They must start using strftime("%OB") and nl_langinfo(ALTMON_...).
They must either detect the glibc version at runtime and choose
the correct format specifier or require the minimum glibc version
at build time. I'm willing to contact the upstream developers and
provide the instructions how to change their applications.

I encourage you to review and apply this fix early because other
projects may need to adopt to this change.

And, last but not least, thank you for mentioning cal(1),
this is another application which probably will need to be updated.

Best regards,

Rafal
  

On Wed, Mar 30, 2016 at 01:31:21AM +0200, Rafal Luzynski wrote:
> 29.03.2016 16:31 "Dmitry V. Levin" <ldv@altlinux.org> wrote:
> > On Fri, Mar 25, 2016 at 01:55:13AM +0100, Rafal Luzynski wrote:
> > [...]
> > > This means that all applications using %B to retrieve the month
> > > name standalone should use %OB from now.
> >
> > Such applications as cal(1) would not be able to print month names
> > properly in a way that would work with different glibc versions.
> > Looks like this is a change incompatible in both ways.
> 
> Yes, that's exactly what will happen. Such applications must be
> updated. They must start using strftime("%OB") and nl_langinfo(ALTMON_...).
> They must either detect the glibc version at runtime and choose
> the correct format specifier or require the minimum glibc version
> at build time. I'm willing to contact the upstream developers and
> provide the instructions how to change their applications.

In glibc, we don't make changes this way.  If an incompatible ABI change
is introduced, the old ABI remains for compatibility with software linked
with it.

With regards to runtime checks, could you give an example of such a check?
  

Hello Dmitry,

Thank you for your feedback. Please find my answers below:

1.06.2016 12:42 "Dmitry V. Levin" <ldv@altlinux.org> wrote:
>
> On Wed, Mar 30, 2016 at 01:31:21AM +0200, Rafal Luzynski wrote:
> > 29.03.2016 16:31 "Dmitry V. Levin" <ldv@altlinux.org> wrote:
> > > On Fri, Mar 25, 2016 at 01:55:13AM +0100, Rafal Luzynski wrote:
> > > [...]
> > > > This means that all applications using %B to retrieve the month
> > > > name standalone should use %OB from now.
> > >
> > > Such applications as cal(1) would not be able to print month names
> > > properly in a way that would work with different glibc versions.
> > > Looks like this is a change incompatible in both ways.
> >
> > Yes, that's exactly what will happen. Such applications must be
> > updated. They must start using strftime("%OB") and nl_langinfo(ALTMON_...).
> > They must either detect the glibc version at runtime and choose
> > the correct format specifier or require the minimum glibc version
> > at build time. I'm willing to contact the upstream developers and
> > provide the instructions how to change their applications.
>
> In glibc, we don't make changes this way. If an incompatible ABI change
> is introduced, the old ABI remains for compatibility with software linked
> with it.

That's why I was thinking [1] about other solutions (this link discusses
all pros and cons but is a little outdated). Shortly:

1. The simplest solution to ensure the full compatibility would be to
leave strftime("%B",...) and nl_langinfo(MON_...) unchanged and provide
the genitive forms via the new symbols strftime("%OB",...) and
nl_langinfo(ALTMON_...). Sounds nice but this means that glibc would
never be compatible with *BSD [2] (you may say you don't care) and
with POSIX [3] (I guess glibc would rather choose to be compatible.)
This would also mean that such programs as cal(1) will not be broken
but all other programs will be broken. I don't mean that the patch will
break them; I mean they are broken already and such fix will not fix
them, all these programs will have to switch to "%OB" and ALTMON_.
Just think how many programs display the month names standalone,
how many display full date (day + month at least) and how many don't
use dates so they don't care. Worse, the meaning of "%OB" and ALTMON_
in *BSD and Linux would be conversed.

2. I also thought about a smart heuristic algorithm detecting if "%B"
is used together with a day number (so a genitive form is needed)
or standalone (so a nominative form is needed). This leads to the
questions like: what does it mean that a day number and a month
name are close to each other? what is the order (day-month vs.
month-day) required by the language? what to do if a software uses
a reversed day-month order against the language rules? That led me
to proposing a day-month-order locale parameter whose meaning is
difficult to explain and the implementation is tricky. Also this
solution would not work for the programs like date(1) which iterate
over whole format string, split it and call strftime() with each
format specifier separately, out of the context.

I'm afraid that providing full backward compatibility is impossible.
If an application calls strftime("%B",...) or nl_langinfo(MON_...)
we have no way to tell if this particular application actually
meant "%OB" and ALTMON_... or it is indeed correct for it to call
"%B" and MON_.

However, please note that in most cases the current call of
strftime("%B",...) and nl_langinfo(MON_...) produces incorrect results
and will automagically start producing correct results if glibc accepts
my recent solution. I believe there are fewer programs where
strftime("%B") and nl_langinfo(MON_...) are correct now and they will
become incorrect, they will need switching to "%OB" and ALTMON_.

> With regards to runtime checks, could you give an example of such a check?

I meant something like:

    #include <gnu/libc-version.h>

    const char *ver_string = gnu_get_libc_version();
    /* Parse ver_string into ver_major and ver_minor */
    if (ver_major > 2 || ver_major == 2 && ver_minor >= 24)
      {
        /* New glibc including my patch */
        strftime("%B",...);    /* output: month name in a genitive form
                                  or nominative for those languages which
                                  don't need a genitive form */
        strftime("%OB",...);   /* output: month name in a nominative form */
        strftime("%Om",...);   /* not useful in any European language */
        strftime("%d %B",...); /* usually the correct date format */
      }
    else
      {
        /* Old glibc */
        strftime("%B",...);    /* output: month name in a nominative form
                                  even if it should be genitive */
        strftime("%OB",...);   /* output: "%OB" string literally */
        strftime("%Om",...);   /* output: correct genitive form but only
                                  in Ukrainian locale (dirty hack) */
        strftime("%d %B",...); /* many programs do it but the result is
                                  incorrect in many languages */
      }

But this is a solution only for closed source software distributed in
a binary form. For those distributed in source form one may expect that
the current glibc version may be detected at compile time (__GLIBC__
and __GLIBC_MINOR__) and a binary package is provided for every specific
distro containing a specific glibc version.

In case of nl_langinfo it's easier:

    /* In case we have the old headers but want to support new glibc */
    #ifndef ALTMON_1
    #define ALTMON_1 ((nl_item) (((int) _NL_TIME_CODESET) + 1)
    #endif
    /* The same for all other months */

    char *june = nl_langinfo(ALTMON_6);
    if (!*june)
        june = nl_langinfo(MON_6);

This will always be valid and always obligatory because nl_langinfo(ALTMON_...)
will return an empty string both on old systems where ALTMON_... is not
valid and on new systems for the locales which do not need to distinguish
between nominative and genitive form (like English) or have not yet
updated their locale data.

I'm aware that none of my solutions is perfect, I'm just trying to
minimize fallout. I'm open to any other solution which leads to correct
results, is backward compatible, portable, and (preferably but not
obligatorily) simple.

Regards,

Rafal Luzynski

Links:

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=10871#c7
[2] https://www.freebsd.org/cgi/man.cgi?query=strftime&sektion=3
[3] http://austingroupbugs.net/view.php?id=258
  

On 06/01/2016 02:51 PM, Rafal Luzynski wrote:
> I'm afraid that providing full backward compatibility is impossible.
> If an application calls strftime("%B",...) or nl_langinfo(MON_...)
> we have no way to tell if this particular application actually
> meant "%OB" and ALTMON_... or it is indeed correct for it to call
> "%B" and MON_.

Of course glibc cannot tell for sure; it is not an oracle. However, if 
glibc uses the old behavior when the application links to the old 
strftime, and the new behavior when the application links to the new 
strftime, then old executables will have the same old behavior even when 
linked to new glibc, which was Dmitry's point.

Even if the new behavior is standardized and is more likely to be what 
the user wants, there will almost surely be cases where the old behavior 
is preferable (if only to make regression tests pass :-), and the 
natural way to tell programmers about this is to say that old programs 
get the old behavior and new programs get the new one.
  

2.06.2016 00:20 Paul Eggert <eggert@cs.ucla.edu> wrote:
>
> [...] However, if
> glibc uses the old behavior when the application links to the old
> strftime, and the new behavior when the application links to the new
> strftime, then old executables will have the same old behavior even when
> linked to new glibc, which was Dmitry's point.

Aaah, that's it! I did not know such a feature exists. So AFAIU this
means that an executable provides to glibc some info what glibc version
it was originally compiled with and if glibc is newer it is still able
to provide the old behavior for old executables. Sounds nice, I will
have to read more about it. (Any hints, links etc. are welcome.)

> Even if the new behavior is standardized and is more likely to be what
> the user wants, there will almost surely be cases where the old behavior
> is preferable (if only to make regression tests pass :-), and the
> natural way to tell programmers about this is to say that old programs
> get the old behavior and new programs get the new one.

What if a programmer, for example an author of cal(1), just rebuilds
the unmodified source while it should be modified? Is there a way to
tell them they should at least verify their source code?

Anyway, I understand that I have to provide the old behavior for
old executables. You're welcome to provide feedback about other issues.

Regards,

Rafal
  

Rafal Luzynski wrote:
> Sounds nice, I will
> have to read more about it. (Any hints, links etc. are welcome.)

Alas, I don't know of any documentation for that, you can look in the source 
code tho. (I'm by no means an expert in this stuff, alas.)

> What if a programmer, for example an author of cal(1), just rebuilds
> the unmodified source while it should be modified? Is there a way to
> tell them they should at least verify their source code?

Not really. I wouldn't worry much about that problem, to be honest.
  

2.06.2016 09:03 Paul Eggert <eggert@cs.ucla.edu> wrote:
>
> Rafal Luzynski wrote:
> > Sounds nice, I will
> > have to read more about it. (Any hints, links etc. are welcome.)
>
> Alas, I don't know of any documentation for that, you can look in the source
> code tho. (I'm by no means an expert in this stuff, alas.)

Source code is fine. Since you don't know of any documentation I will
not waste my time searching for it.

>
> > What if a programmer, for example an author of cal(1), just rebuilds
> > the unmodified source while it should be modified? Is there a way to
> > tell them they should at least verify their source code?
>
> Not really. I wouldn't worry much about that problem, to be honest.

That's fine for me, too. Please note that distros rebuild the packages
from source for each release, that means that changing the source code
of cal(1) will be needed. As I said before, I'll be happy to contact
these projects and help them port to the new format.

Regards,

Rafal
  

On 03/29/2016 11:02 AM, Florian Weimer wrote:
> On 03/25/2016 01:55 AM, Rafal Luzynski wrote:
>
>> strftime() now implements a %OB format specifier which generates an
>> alternative month name.  For the languages which require both nominative
>> and genitive case of the month names it is expected that it outputs
>> the alternative month name which should be a nominative (sic!) case
>> or whatever is appropriate when formatting the month names standalone
>> (without a day).  For those languages %B will return the basic month
>> name but from now it is expected to be a genitive case or whatever is
>> appropriate when formatting full dates (including both day and month).
>> This means that all applications using %B to retrieve the month
>> name standalone should use %OB from now.  For those languages which
>> do not use different (nominative and genitive) cases of the month
>> name or do not yet have their locales updated %OB will retrieve
>> the same string as %B so moving to %OB will be harmless as long
>> as the version of glibc which supports this feature is used.
>
> What's the status here on the POSIX side?  It's marked as Accepted in
> the tracker, but the changes apparently haven't been folded into
> subsequent editions.

This question has never been answered.  I think it's quite relevant to 
whether we can accept the patch or not.

Florian
  

4.11.2016 14:36 Florian Weimer <fweimer@redhat.com> wrote:
>
>
> On 03/29/2016 11:02 AM, Florian Weimer wrote:
> > On 03/25/2016 01:55 AM, Rafal Luzynski wrote:
> >
> >> strftime() now implements a %OB format specifier which generates an
> >> alternative month name. For the languages which require both nominative
> >> and genitive case of the month names it is expected that it outputs
> >> the alternative month name which should be a nominative (sic!) case
> >> or whatever is appropriate when formatting the month names standalone
> >> (without a day). For those languages %B will return the basic month
> >> name but from now it is expected to be a genitive case or whatever is
> >> appropriate when formatting full dates (including both day and month).
> >> This means that all applications using %B to retrieve the month
> >> name standalone should use %OB from now. For those languages which
> >> do not use different (nominative and genitive) cases of the month
> >> name or do not yet have their locales updated %OB will retrieve
> >> the same string as %B so moving to %OB will be harmless as long
> >> as the version of glibc which supports this feature is used.
> >
> > What's the status here on the POSIX side? It's marked as Accepted in
> > the tracker, but the changes apparently haven't been folded into
> > subsequent editions.
>
> This question has never been answered.

At least partial answer is here:
https://sourceware.org/ml/libc-alpha/2016-10/msg00319.html

> I think it's quite relevant to
> whether we can accept the patch or not.
>
> Florian

Please note that it's a bug that strftime() does not format the
month names in nominative/genitive case (more generally:
standalone/full date). Many other toolkits have the same bug
because they just call strftime() internally. And also it's been
accepted that it's a bug in POSIX that it does not specify %OB.

Is there any alternative solution for the bug?

Regards,

Rafal Luzynski