Fortran: error recovery for invalid types in array constructors [PR107000]

  Dear all,

we did not recover well from bad expressions in array constructors,
especially when there was a typespec and a unary '+' or '-', and
when the array constructor was used in an arithmetic expression.

The attached patch introduces an ARITH_INVALID_TYPE that is used
when we try to recover from these errors, and tries to handle
all unary and binary arithmetic expressions.

Regtested on x86_64-pc-linux-gnu.  OK for mainline?

Thanks,
Harald
  

Hello

Le 04/10/2022 à 23:19, Harald Anlauf via Fortran a écrit :
> Dear all,
> 
> we did not recover well from bad expressions in array constructors,
> especially when there was a typespec and a unary '+' or '-', and
> when the array constructor was used in an arithmetic expression.
> 
> The attached patch introduces an ARITH_INVALID_TYPE that is used
> when we try to recover from these errors, and tries to handle
> all unary and binary arithmetic expressions.
> 

In the PR, you noted an inconsistency in the error message reported, 
depending on the presence or lack of an operator.
I'm not sure you saw the suggestion to do the following in the last 
message I posted:

diff --git a/gcc/fortran/arith.cc b/gcc/fortran/arith.cc
index e6e35ef3c42..ed93ddb2882 100644
--- a/gcc/fortran/arith.cc
+++ b/gcc/fortran/arith.cc
@@ -1654,6 +1654,8 @@ eval_intrinsic (gfc_intrinsic_op op,
    else
      rc = reduce_binary (eval.f3, op1, op2, &result);

+  if (rc == ARITH_INVALID_TYPE)
+    goto runtime;

    /* Something went wrong.  */
    if (op == INTRINSIC_POWER && rc == ARITH_PROHIBIT)


In the testcase, it improves the situation slightly.
For example, from:
>     9 |   x = (1.0, 2.0) * [complex :: +'1'] ! { dg-error "Invalid type" }
>       |                1
> Error: Invalid type in arithmetic operation at (1)
to:
>     9 |   x = (1.0, 2.0) * [complex :: +'1'] ! { dg-error "Invalid type" }
>       |                              1
> Error: Operand of unary numeric operator ‘+’ at (1) is UNKNOWN


or from:
>    29 |   print *, 2 * [real :: 0, 1+'1']  ! { dg-error "Invalid type" }
>       |            1
> Error: Invalid type in arithmetic operation at (1)
to:
>    29 |   print *, 2 * [real :: 0, 1+'1']  ! { dg-error "Invalid type" }
>       |                          1
> Error: Operands of binary numeric operator ‘+’ at (1) are INTEGER(4)/CHARACTER(1)

Unfortunately, it doesn't fix the bogus incommensurate arrays errors.
  

Le 05/10/2022 à 10:51, Mikael Morin a écrit :
> 
> Unfortunately, it doesn't fix the bogus incommensurate arrays errors.
> 

The following does.


diff --git a/gcc/fortran/arith.cc b/gcc/fortran/arith.cc
index e6e35ef3c42..2c57c796270 100644
--- a/gcc/fortran/arith.cc
+++ b/gcc/fortran/arith.cc
@@ -1443,7 +1443,7 @@ reduce_binary_aa (arith (*eval) (gfc_expr *, 
gfc_expr *, gfc_expr **),
         gfc_replace_expr (c->expr, r);
      }

-  if (c || d)
+  if (rc == ARITH_OK && (c || d))
      rc = ARITH_INCOMMENSURATE;

    if (rc != ARITH_OK)


There is one last thing that I'm dissatisfied with.
The handling of unknown types should be moved to reduce_binary, because 
the dispatching in reduce_binary doesn't handle EXPR_OP, so even if 
either or both operands are scalar, they are handled by the (array vs 
array) reduce_binary_aa function.  That's confusing.
  

Hi Mikael,

> Gesendet: Mittwoch, 05. Oktober 2022 um 11:23 Uhr
> Von: "Mikael Morin" <morin-mikael@orange.fr>
> An: "Harald Anlauf" <anlauf@gmx.de>, "fortran" <fortran@gcc.gnu.org>, "gcc-patches" <gcc-patches@gcc.gnu.org>
> Betreff: Re: [PATCH] Fortran: error recovery for invalid types in array constructors [PR107000]

> The following does.
>
>
> diff --git a/gcc/fortran/arith.cc b/gcc/fortran/arith.cc
> index e6e35ef3c42..2c57c796270 100644
> --- a/gcc/fortran/arith.cc
> +++ b/gcc/fortran/arith.cc
> @@ -1443,7 +1443,7 @@ reduce_binary_aa (arith (*eval) (gfc_expr *,
> gfc_expr *, gfc_expr **),
>          gfc_replace_expr (c->expr, r);
>       }
>
> -  if (c || d)
> +  if (rc == ARITH_OK && (c || d))
>       rc = ARITH_INCOMMENSURATE;
>
>     if (rc != ARITH_OK)

that's great!  It fixes several rather weird cases.  (There is at least
another PR on the incommensurate arrays, but we should not attempt to
fix everything today.)

> There is one last thing that I'm dissatisfied with.
> The handling of unknown types should be moved to reduce_binary, because
> the dispatching in reduce_binary doesn't handle EXPR_OP, so even if
> either or both operands are scalar, they are handled by the (array vs
> array) reduce_binary_aa function.  That's confusing.

Do you have an example?

Anyway, please find attached an updated patch that incorporates your
two changes and regtests fine on x86_64-pc-linux-gnu.

Even if you disagree, I think this is really a significant step
forwards... (error-recovery wise).

OK for mainline?

Thanks,
Harald
  

Le 05/10/2022 à 23:40, Harald Anlauf a écrit :
> 
>> There is one last thing that I'm dissatisfied with.
>> The handling of unknown types should be moved to reduce_binary, because
>> the dispatching in reduce_binary doesn't handle EXPR_OP, so even if
>> either or both operands are scalar, they are handled by the (array vs
>> array) reduce_binary_aa function.  That's confusing.

Thinking about it again, I'm not sure my suggestion is right here.
> 
> Do you have an example?
> 
No.  Actually, I think it works, but a weird way.


For example, for this case:

[real :: 2] * [real :: +(.true.)]

First there is a "root" invocation of reduce binary with arguments [real 
:: 2] and [real :: +(.true.)]
The root invocation of reduce_binary will call reduce_binary_aa. This is 
normal.

Then reduce_binary_aa calls reduce_binary again with arguments 2 and 
+(.true.).  And reduce_binary calls again reduce_binary_aa with those 
arguments.  This is weird, reduce_binary_aa is supposed to have arrays 
for both arguments.


The same goes for the array vs constant case, reduce_binary_ca (or 
reduce_binary_ac) is invoked with two scalars, while if you look at 
reduce_binary, you would expect that we only get to reduce_binary_ca 
with a scalar constant and an array as arguments.


I think the checks in the three reduce_binary_* functions should be 
moved into their respective loops, so that we detect the invalid type 
just before these weird recursive calls instead of just after entering 
into them.

OK with that change.
  

Hi Mikael,

> Gesendet: Donnerstag, 06. Oktober 2022 um 22:14 Uhr
> Von: "Mikael Morin" <morin-mikael@orange.fr>
> An: "Harald Anlauf" <anlauf@gmx.de>
> Cc: "fortran" <fortran@gcc.gnu.org>, "gcc-patches" <gcc-patches@gcc.gnu.org>
> Betreff: Re: [PATCH, v2] Fortran: error recovery for invalid types in array constructors [PR107000]
>
> Le 05/10/2022 à 23:40, Harald Anlauf a écrit :
> > 
> >> There is one last thing that I'm dissatisfied with.
> >> The handling of unknown types should be moved to reduce_binary, because
> >> the dispatching in reduce_binary doesn't handle EXPR_OP, so even if
> >> either or both operands are scalar, they are handled by the (array vs
> >> array) reduce_binary_aa function.  That's confusing.
> 
> Thinking about it again, I'm not sure my suggestion is right here.
> > 
> > Do you have an example?
> > 
> No.  Actually, I think it works, but a weird way.
> 
> 
> For example, for this case:
> 
> [real :: 2] * [real :: +(.true.)]
> 
> First there is a "root" invocation of reduce binary with arguments [real 
> :: 2] and [real :: +(.true.)]
> The root invocation of reduce_binary will call reduce_binary_aa. This is 
> normal.
> 
> Then reduce_binary_aa calls reduce_binary again with arguments 2 and 
> +(.true.).  And reduce_binary calls again reduce_binary_aa with those 
> arguments.  This is weird, reduce_binary_aa is supposed to have arrays 
> for both arguments.

Am I seeing something different from you?  My gdb says
that one argument of reduce_binary is EXPR_CONSTANT,
the other EXPR_OP and BT_UNKNOWN.  Both rank 0.

> The same goes for the array vs constant case, reduce_binary_ca (or 
> reduce_binary_ac) is invoked with two scalars, while if you look at 
> reduce_binary, you would expect that we only get to reduce_binary_ca 
> with a scalar constant and an array as arguments.
> 
> 
> I think the checks in the three reduce_binary_* functions should be 
> moved into their respective loops, so that we detect the invalid type 
> just before these weird recursive calls instead of just after entering 
> into them.

I think I tried that before, and it didn't work.
There was always one weird case that lead to a bad or
invalid constructor for one of the arrays you want to
look at in the respective loop,  and this is why the
testcase tries to cover everything that I hit then and
there... (hopefully).  So I ended up with the check
before the loop.

What do we actually gain with your suggested change?
Moving the check into the loop does not really make
the code more readable to me.  And the recursion is
needed anyway.

Cheers,
Harald

> OK with that change.
>
  

Le 06/10/2022 à 23:36, Harald Anlauf a écrit :
>>
>> For example, for this case:
>>
>> [real :: 2] * [real :: +(.true.)]
>>
>> First there is a "root" invocation of reduce binary with arguments [real
>> :: 2] and [real :: +(.true.)]
>> The root invocation of reduce_binary will call reduce_binary_aa. This is
>> normal.
>>
>> Then reduce_binary_aa calls reduce_binary again with arguments 2 and
>> +(.true.).  And reduce_binary calls again reduce_binary_aa with those
>> arguments.  This is weird, reduce_binary_aa is supposed to have arrays
>> for both arguments.
> 
> Am I seeing something different from you?  My gdb says
> that one argument of reduce_binary is EXPR_CONSTANT,
> the other EXPR_OP and BT_UNKNOWN.  Both rank 0.
> 
No, I get the same, and the program goes to reduce_binary_aa with those 
arguments; this is the problem.

>> The same goes for the array vs constant case, reduce_binary_ca (or
>> reduce_binary_ac) is invoked with two scalars, while if you look at
>> reduce_binary, you would expect that we only get to reduce_binary_ca
>> with a scalar constant and an array as arguments.
>>
>>
>> I think the checks in the three reduce_binary_* functions should be
>> moved into their respective loops, so that we detect the invalid type
>> just before these weird recursive calls instead of just after entering
>> into them.
> 
> I think I tried that before, and it didn't work.
> There was always one weird case that lead to a bad or
> invalid constructor for one of the arrays you want to
> look at in the respective loop,  and this is why the
> testcase tries to cover everything that I hit then and
> there... (hopefully).  So I ended up with the check
> before the loop.
> 
I see, I'll have a look.

> What do we actually gain with your suggested change?
> Moving the check into the loop does not really make
> the code more readable to me.  And the recursion is
> needed anyway.
> 
I think we gain clarity, consistency.

I try to rephrase again.
 From a high level point of view, to evaluate a binary operator you need 
a specific (one for each operator) function to evaluate the scalar vs 
scalar case, and three generic (they are common to all the operators) 
functions to handle respectively:
  - the scalar vs array case,
  - the array vs scalar case,
  - the array vs array case,
by calling in a loop the scalar specific function.
Here we are only dealing with constants, arrays of constants, arrays of 
arrays, etc, all valid cases.

Your patch introduces support for invalid cases, that is invalid values 
that can't be reduced to a constant.  This is fine, and it works.
What is weird is that the scalar vs invalid scalar case is caught in the 
array vs array function.
  

Am 07.10.22 um 10:01 schrieb Mikael Morin:
> Le 06/10/2022 à 23:36, Harald Anlauf a écrit :
>>>
>>> For example, for this case:
>>>
>>> [real :: 2] * [real :: +(.true.)]
>>>
>>> First there is a "root" invocation of reduce binary with arguments [real
>>> :: 2] and [real :: +(.true.)]
>>> The root invocation of reduce_binary will call reduce_binary_aa. This is
>>> normal.
>>>
>>> Then reduce_binary_aa calls reduce_binary again with arguments 2 and
>>> +(.true.).  And reduce_binary calls again reduce_binary_aa with those
>>> arguments.  This is weird, reduce_binary_aa is supposed to have arrays
>>> for both arguments.
>>
>> Am I seeing something different from you?  My gdb says
>> that one argument of reduce_binary is EXPR_CONSTANT,
>> the other EXPR_OP and BT_UNKNOWN.  Both rank 0.
>>
> No, I get the same, and the program goes to reduce_binary_aa with those
> arguments; this is the problem.
>
>>> The same goes for the array vs constant case, reduce_binary_ca (or
>>> reduce_binary_ac) is invoked with two scalars, while if you look at
>>> reduce_binary, you would expect that we only get to reduce_binary_ca
>>> with a scalar constant and an array as arguments.
>>>
>>>
>>> I think the checks in the three reduce_binary_* functions should be
>>> moved into their respective loops, so that we detect the invalid type
>>> just before these weird recursive calls instead of just after entering
>>> into them.
>>
>> I think I tried that before, and it didn't work.
>> There was always one weird case that lead to a bad or
>> invalid constructor for one of the arrays you want to
>> look at in the respective loop,  and this is why the
>> testcase tries to cover everything that I hit then and
>> there... (hopefully).  So I ended up with the check
>> before the loop.
>>
> I see, I'll have a look.
>
>> What do we actually gain with your suggested change?
>> Moving the check into the loop does not really make
>> the code more readable to me.  And the recursion is
>> needed anyway.
>>
> I think we gain clarity, consistency.
>
> I try to rephrase again.
>  From a high level point of view, to evaluate a binary operator you need
> a specific (one for each operator) function to evaluate the scalar vs
> scalar case, and three generic (they are common to all the operators)
> functions to handle respectively:
>   - the scalar vs array case,
>   - the array vs scalar case,
>   - the array vs array case,
> by calling in a loop the scalar specific function.
> Here we are only dealing with constants, arrays of constants, arrays of
> arrays, etc, all valid cases.
>
> Your patch introduces support for invalid cases, that is invalid values
> that can't be reduced to a constant.  This is fine, and it works.
> What is weird is that the scalar vs invalid scalar case is caught in the
> array vs array function.

OK, that is because reduce_binary dispatches the reduce_binary_*.
We could move the check from reduce_binary_aa to the beginning of
reduce_binary, as with the following change on top of the patch:

diff --git a/gcc/fortran/arith.cc b/gcc/fortran/arith.cc
index 2c57c796270..91e70655ad3 100644
--- a/gcc/fortran/arith.cc
+++ b/gcc/fortran/arith.cc
@@ -1426,10 +1426,6 @@ reduce_binary_aa (arith (*eval) (gfc_expr *,
gfc_expr *, gfc_expr **),
    if (!gfc_check_conformance (op1, op2, _("elemental binary operation")))
      return ARITH_INCOMMENSURATE;

-  if ((op1->expr_type == EXPR_OP && op1->ts.type == BT_UNKNOWN)
-      || (op2->expr_type == EXPR_OP && op2->ts.type == BT_UNKNOWN))
-    return ARITH_INVALID_TYPE;
-
    head = gfc_constructor_copy (op1->value.constructor);
    for (c = gfc_constructor_first (head),
         d = gfc_constructor_first (op2->value.constructor);
@@ -1467,6 +1463,10 @@ static arith
  reduce_binary (arith (*eval) (gfc_expr *, gfc_expr *, gfc_expr **),
                gfc_expr *op1, gfc_expr *op2, gfc_expr **result)
  {
+  if ((op1->expr_type == EXPR_OP && op1->ts.type == BT_UNKNOWN)
+      || (op2->expr_type == EXPR_OP && op2->ts.type == BT_UNKNOWN))
+    return ARITH_INVALID_TYPE;
+
    if (op1->expr_type == EXPR_CONSTANT && op2->expr_type == EXPR_CONSTANT)
      return eval (op1, op2, result);

However, we cannot remove the checks from reduce_binary_ac
or reduce_binary_ca, as the lengthy testcase proves...

Do you like the above better?

Cheers,
Harald
  

Le 07/10/2022 à 20:46, Harald Anlauf a écrit :
> 
> OK, that is because reduce_binary dispatches the reduce_binary_*.
> We could move the check from reduce_binary_aa to the beginning of
> reduce_binary, as with the following change on top of the patch:
> 
> diff --git a/gcc/fortran/arith.cc b/gcc/fortran/arith.cc
> index 2c57c796270..91e70655ad3 100644
> --- a/gcc/fortran/arith.cc
> +++ b/gcc/fortran/arith.cc
> @@ -1426,10 +1426,6 @@ reduce_binary_aa (arith (*eval) (gfc_expr *,
> gfc_expr *, gfc_expr **),
>     if (!gfc_check_conformance (op1, op2, _("elemental binary operation")))
>       return ARITH_INCOMMENSURATE;
> 
> -  if ((op1->expr_type == EXPR_OP && op1->ts.type == BT_UNKNOWN)
> -      || (op2->expr_type == EXPR_OP && op2->ts.type == BT_UNKNOWN))
> -    return ARITH_INVALID_TYPE;
> -
>     head = gfc_constructor_copy (op1->value.constructor);
>     for (c = gfc_constructor_first (head),
>          d = gfc_constructor_first (op2->value.constructor);
> @@ -1467,6 +1463,10 @@ static arith
>   reduce_binary (arith (*eval) (gfc_expr *, gfc_expr *, gfc_expr **),
>                 gfc_expr *op1, gfc_expr *op2, gfc_expr **result)
>   {
> +  if ((op1->expr_type == EXPR_OP && op1->ts.type == BT_UNKNOWN)
> +      || (op2->expr_type == EXPR_OP && op2->ts.type == BT_UNKNOWN))
> +    return ARITH_INVALID_TYPE;
> +
>     if (op1->expr_type == EXPR_CONSTANT && op2->expr_type == EXPR_CONSTANT)
>       return eval (op1, op2, result);
> 
> However, we cannot remove the checks from reduce_binary_ac
> or reduce_binary_ca, as the lengthy testcase proves...
> 
> Do you like the above better?
> 
Yes, definitely, but some less important weirdness remains;
the scalar vs array function catches scalar vs invalid scalar cases.
Let me have a look.