Improve check against integer wraparound in hcreate_r [BZ #18240]
Commit Message
* Paul Eggert:
> Florian Weimer wrote:
>
>> - if (nel >= SIZE_MAX / sizeof (_ENTRY))
>> + /* This limit is sufficient to avoid unsigned wraparound below,
>> + possibly after truncation to unsigned int. (struct hsearch_data
>> + is part of the public API and uses usigned ints.) */
>> + if (nel >= INT_MAX / sizeof (_ENTRY))
>
> This patch doesn't look right. nel should be bounded by UINT_MAX - 2,
> not by INT_MAX / sizeof (anything). (Not by UINT_MAX, since the code
> computes nel + 1 later; and not by UINT_MAX - 1 since that cannot be
> prime.) Furthermore, calloc will check for size overflow on
> multiplication so hcreate_r need not worry about dividing by sizeof
> (anything). Also, "unsigned" is misspelled in the comment.
>
> How about something like the attached (untested) patch instead?
Fair enough. isprime needs to be fixed as well, like this.
Adhemerval, do we still have time to fix this?
@@ -48,7 +48,7 @@ isprime (unsigned int number)
/* no even number will be passed */
unsigned int div = 3;
- while (div * div < number && number % div != 0)
+ while (div * (unsigned long long) div < number && number % div != 0)
div += 2;
return number % div != 0;