[2/3] x86/mce: Define amd_mce_usable_address()

  Currently, all valid MCA_ADDR values are assumed to be usable on AMD
systems. However, this is not correct in most cases. Notifiers expecting
usable addresses may then operate on inappropriate values.

Define a helper function to do AMD-specific checks for a usable memory
address. List out all known cases.

Signed-off-by: Yazen Ghannam <yazen.ghannam@amd.com>
---
 arch/x86/kernel/cpu/mce/amd.c      | 38 ++++++++++++++++++++++++++++++
 arch/x86/kernel/cpu/mce/core.c     |  3 +++
 arch/x86/kernel/cpu/mce/internal.h |  2 ++
 3 files changed, 43 insertions(+)
  

On 2023/6/13 22:11, Yazen Ghannam wrote:
> Currently, all valid MCA_ADDR values are assumed to be usable on AMD
> systems. However, this is not correct in most cases. Notifiers expecting
> usable addresses may then operate on inappropriate values.
> 
> Define a helper function to do AMD-specific checks for a usable memory
> address. List out all known cases.
> 
> Signed-off-by: Yazen Ghannam <yazen.ghannam@amd.com>
> ---
>  arch/x86/kernel/cpu/mce/amd.c      | 38 ++++++++++++++++++++++++++++++
>  arch/x86/kernel/cpu/mce/core.c     |  3 +++
>  arch/x86/kernel/cpu/mce/internal.h |  2 ++
>  3 files changed, 43 insertions(+)
> 
> diff --git a/arch/x86/kernel/cpu/mce/amd.c b/arch/x86/kernel/cpu/mce/amd.c
> index 1ccfb0c9257f..ca79fa10b844 100644
> --- a/arch/x86/kernel/cpu/mce/amd.c
> +++ b/arch/x86/kernel/cpu/mce/amd.c
> @@ -746,6 +746,44 @@ bool amd_mce_is_memory_error(struct mce *m)
>  	return legacy_mce_is_memory_error(m);
>  }
>  
> +/*
> + * AMD systems do not have an explicit indicator that the value in MCA_ADDR is
> + * a system physical address. Therefore individual cases need to be detected.
> + * Future cases and checks will be added as needed.
> + *
> + * 1) General case
> + *	a) Assume address is not usable.
> + * 2) "Poison" errors
> + *	a) Indicated by MCA_STATUS[43]: POISON. Defined for all banks except legacy
> + *	   Northbridge (bank 4).
> + *	b) Refers to poison consumption in the Core. Does not include "no action",
> + *	   "action optional", or "deferred" error severities.
> + *	c) Will include a usuable address so that immediate action can be taken.
> + * 3) Northbridge DRAM ECC errors
> + *	a) Reported in legacy bank 4 with XEC 8.
> + *	b) MCA_STATUS[43] is *not* defined as POISON in legacy bank 4. Therefore,
> + *	   this bit should not be checked.
[nit]

> + *
> + * NOTE: SMCA UMC memory errors fall into case #1.

hi, Yazen

The address for SMCA UMC memory error is not system physical address, it make sense
to be not usable. But how we deal with the SMCA address? The MCE chain like
uc_decode_notifier will do a sanity check with mce_usable_address and it will not
handle SMCA address.

Thanks.

Best Regards,
Shuai

> + */
> +bool amd_mce_usable_address(struct mce *m)
> +{
> +	/* Check special Northbridge case first. */
> +	if (!mce_flags.smca) {
> +		if (legacy_mce_is_memory_error(m))
> +			return true;
> +		else if (m->bank == 4)
> +			return false;
> +	}
> +
> +	/* Check Poison bit for all other bank types. */
> +	if (m->status & MCI_STATUS_POISON)
> +		return true;
> +
> +	/* Assume address is not usable for all others. */
> +	return false;
> +}
> +
>  static void __log_error(unsigned int bank, u64 status, u64 addr, u64 misc)
>  {
>  	struct mce m;
> diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c
> index 89e2aab5d34d..859ce20dd730 100644
> --- a/arch/x86/kernel/cpu/mce/core.c
> +++ b/arch/x86/kernel/cpu/mce/core.c
> @@ -464,6 +464,9 @@ int mce_usable_address(struct mce *m)
>  	if (!(m->status & MCI_STATUS_ADDRV))
>  		return 0;
>  
> +	if (m->cpuvendor == X86_VENDOR_AMD)
> +		return amd_mce_usable_address(m);
> +
>  	/* Checks after this one are Intel/Zhaoxin-specific: */
>  	if (boot_cpu_data.x86_vendor != X86_VENDOR_INTEL &&
>  	    boot_cpu_data.x86_vendor != X86_VENDOR_ZHAOXIN)
> diff --git a/arch/x86/kernel/cpu/mce/internal.h b/arch/x86/kernel/cpu/mce/internal.h
> index d2412ce2d312..0d4c5b83ed93 100644
> --- a/arch/x86/kernel/cpu/mce/internal.h
> +++ b/arch/x86/kernel/cpu/mce/internal.h
> @@ -207,6 +207,7 @@ extern bool filter_mce(struct mce *m);
>  
>  #ifdef CONFIG_X86_MCE_AMD
>  extern bool amd_filter_mce(struct mce *m);
> +bool amd_mce_usable_address(struct mce *m);
>  
>  /*
>   * If MCA_CONFIG[McaLsbInStatusSupported] is set, extract ErrAddr in bits
> @@ -234,6 +235,7 @@ static __always_inline void smca_extract_err_addr(struct mce *m)
>  
>  #else
>  static inline bool amd_filter_mce(struct mce *m) { return false; }
> +static inline bool amd_mce_usable_address(struct mce *m) { return false; }
>  static inline void smca_extract_err_addr(struct mce *m) { }
>  #endif
>
  

On 6/13/2023 10:19 PM, Shuai Xue wrote:
> 
> 
> On 2023/6/13 22:11, Yazen Ghannam wrote:
>> Currently, all valid MCA_ADDR values are assumed to be usable on AMD
>> systems. However, this is not correct in most cases. Notifiers expecting
>> usable addresses may then operate on inappropriate values.
>>
>> Define a helper function to do AMD-specific checks for a usable memory
>> address. List out all known cases.
>>
>> Signed-off-by: Yazen Ghannam <yazen.ghannam@amd.com>
>> ---
>>   arch/x86/kernel/cpu/mce/amd.c      | 38 ++++++++++++++++++++++++++++++
>>   arch/x86/kernel/cpu/mce/core.c     |  3 +++
>>   arch/x86/kernel/cpu/mce/internal.h |  2 ++
>>   3 files changed, 43 insertions(+)
>>
>> diff --git a/arch/x86/kernel/cpu/mce/amd.c b/arch/x86/kernel/cpu/mce/amd.c
>> index 1ccfb0c9257f..ca79fa10b844 100644
>> --- a/arch/x86/kernel/cpu/mce/amd.c
>> +++ b/arch/x86/kernel/cpu/mce/amd.c
>> @@ -746,6 +746,44 @@ bool amd_mce_is_memory_error(struct mce *m)
>>   	return legacy_mce_is_memory_error(m);
>>   }
>>   
>> +/*
>> + * AMD systems do not have an explicit indicator that the value in MCA_ADDR is
>> + * a system physical address. Therefore individual cases need to be detected.
>> + * Future cases and checks will be added as needed.
>> + *
>> + * 1) General case
>> + *	a) Assume address is not usable.
>> + * 2) "Poison" errors
>> + *	a) Indicated by MCA_STATUS[43]: POISON. Defined for all banks except legacy
>> + *	   Northbridge (bank 4).
>> + *	b) Refers to poison consumption in the Core. Does not include "no action",
>> + *	   "action optional", or "deferred" error severities.
>> + *	c) Will include a usuable address so that immediate action can be taken.
>> + * 3) Northbridge DRAM ECC errors
>> + *	a) Reported in legacy bank 4 with XEC 8.
>> + *	b) MCA_STATUS[43] is *not* defined as POISON in legacy bank 4. Therefore,
>> + *	   this bit should not be checked.
> [nit]
> 
>> + *
>> + * NOTE: SMCA UMC memory errors fall into case #1.
> 
> hi, Yazen
> 
> The address for SMCA UMC memory error is not system physical address, it make sense
> to be not usable. But how we deal with the SMCA address? The MCE chain like
> uc_decode_notifier will do a sanity check with mce_usable_address and it will not
> handle SMCA address.
>

Hi Shuai,

That's correct.

There isn't a good solution today. This will be handled in future changes.

Thanks,
Yazen
  

On 2023/6/14 23:09, Yazen Ghannam wrote:
> On 6/13/2023 10:19 PM, Shuai Xue wrote:
>>
>>
>> On 2023/6/13 22:11, Yazen Ghannam wrote:
>>> Currently, all valid MCA_ADDR values are assumed to be usable on AMD
>>> systems. However, this is not correct in most cases. Notifiers expecting
>>> usable addresses may then operate on inappropriate values.
>>>
>>> Define a helper function to do AMD-specific checks for a usable memory
>>> address. List out all known cases.
>>>
>>> Signed-off-by: Yazen Ghannam <yazen.ghannam@amd.com>
>>> ---
>>>   arch/x86/kernel/cpu/mce/amd.c      | 38 ++++++++++++++++++++++++++++++
>>>   arch/x86/kernel/cpu/mce/core.c     |  3 +++
>>>   arch/x86/kernel/cpu/mce/internal.h |  2 ++
>>>   3 files changed, 43 insertions(+)
>>>
>>> diff --git a/arch/x86/kernel/cpu/mce/amd.c b/arch/x86/kernel/cpu/mce/amd.c
>>> index 1ccfb0c9257f..ca79fa10b844 100644
>>> --- a/arch/x86/kernel/cpu/mce/amd.c
>>> +++ b/arch/x86/kernel/cpu/mce/amd.c
>>> @@ -746,6 +746,44 @@ bool amd_mce_is_memory_error(struct mce *m)
>>>       return legacy_mce_is_memory_error(m);
>>>   }
>>>   +/*
>>> + * AMD systems do not have an explicit indicator that the value in MCA_ADDR is
>>> + * a system physical address. Therefore individual cases need to be detected.
>>> + * Future cases and checks will be added as needed.
>>> + *
>>> + * 1) General case
>>> + *    a) Assume address is not usable.
>>> + * 2) "Poison" errors
>>> + *    a) Indicated by MCA_STATUS[43]: POISON. Defined for all banks except legacy
>>> + *       Northbridge (bank 4).
>>> + *    b) Refers to poison consumption in the Core. Does not include "no action",
>>> + *       "action optional", or "deferred" error severities.
>>> + *    c) Will include a usuable address so that immediate action can be taken.
>>> + * 3) Northbridge DRAM ECC errors
>>> + *    a) Reported in legacy bank 4 with XEC 8.
>>> + *    b) MCA_STATUS[43] is *not* defined as POISON in legacy bank 4. Therefore,
>>> + *       this bit should not be checked.
>> [nit]
>>
>>> + *
>>> + * NOTE: SMCA UMC memory errors fall into case #1.
>>
>> hi, Yazen
>>
>> The address for SMCA UMC memory error is not system physical address, it make sense
>> to be not usable. But how we deal with the SMCA address? The MCE chain like
>> uc_decode_notifier will do a sanity check with mce_usable_address and it will not
>> handle SMCA address.
>>
> 
> Hi Shuai,
> 
> That's correct.
> 
> There isn't a good solution today. This will be handled in future changes.

Hi, Yazen,

Do you have plan to address it? If not, I can help. We meet this problem in our products.

Thanks
Shuai
  

On 6/14/2023 10:12 PM, Shuai Xue wrote:
> 
> 
> On 2023/6/14 23:09, Yazen Ghannam wrote:
>> On 6/13/2023 10:19 PM, Shuai Xue wrote:
>>>
>>>
>>> On 2023/6/13 22:11, Yazen Ghannam wrote:
>>>> Currently, all valid MCA_ADDR values are assumed to be usable on AMD
>>>> systems. However, this is not correct in most cases. Notifiers expecting
>>>> usable addresses may then operate on inappropriate values.
>>>>
>>>> Define a helper function to do AMD-specific checks for a usable memory
>>>> address. List out all known cases.
>>>>
>>>> Signed-off-by: Yazen Ghannam <yazen.ghannam@amd.com>
>>>> ---
>>>>    arch/x86/kernel/cpu/mce/amd.c      | 38 ++++++++++++++++++++++++++++++
>>>>    arch/x86/kernel/cpu/mce/core.c     |  3 +++
>>>>    arch/x86/kernel/cpu/mce/internal.h |  2 ++
>>>>    3 files changed, 43 insertions(+)
>>>>
>>>> diff --git a/arch/x86/kernel/cpu/mce/amd.c b/arch/x86/kernel/cpu/mce/amd.c
>>>> index 1ccfb0c9257f..ca79fa10b844 100644
>>>> --- a/arch/x86/kernel/cpu/mce/amd.c
>>>> +++ b/arch/x86/kernel/cpu/mce/amd.c
>>>> @@ -746,6 +746,44 @@ bool amd_mce_is_memory_error(struct mce *m)
>>>>        return legacy_mce_is_memory_error(m);
>>>>    }
>>>>    +/*
>>>> + * AMD systems do not have an explicit indicator that the value in MCA_ADDR is
>>>> + * a system physical address. Therefore individual cases need to be detected.
>>>> + * Future cases and checks will be added as needed.
>>>> + *
>>>> + * 1) General case
>>>> + *    a) Assume address is not usable.
>>>> + * 2) "Poison" errors
>>>> + *    a) Indicated by MCA_STATUS[43]: POISON. Defined for all banks except legacy
>>>> + *       Northbridge (bank 4).
>>>> + *    b) Refers to poison consumption in the Core. Does not include "no action",
>>>> + *       "action optional", or "deferred" error severities.
>>>> + *    c) Will include a usuable address so that immediate action can be taken.
>>>> + * 3) Northbridge DRAM ECC errors
>>>> + *    a) Reported in legacy bank 4 with XEC 8.
>>>> + *    b) MCA_STATUS[43] is *not* defined as POISON in legacy bank 4. Therefore,
>>>> + *       this bit should not be checked.
>>> [nit]
>>>
>>>> + *
>>>> + * NOTE: SMCA UMC memory errors fall into case #1.
>>>
>>> hi, Yazen
>>>
>>> The address for SMCA UMC memory error is not system physical address, it make sense
>>> to be not usable. But how we deal with the SMCA address? The MCE chain like
>>> uc_decode_notifier will do a sanity check with mce_usable_address and it will not
>>> handle SMCA address.
>>>
>>
>> Hi Shuai,
>>
>> That's correct.
>>
>> There isn't a good solution today. This will be handled in future changes.
> 
> Hi, Yazen,
> 
> Do you have plan to address it? If not, I can help. We meet this problem in our products.
> 

Yes, definitely. The first step is to update the address translation 
code; this is progress. Afterwards, we can find a way to leverage this 
in the MCE notifier flows.

Just curious, how big is the benefit of this preemptive page offline in 
your use cases? That is, compared to page offline as part of poison data 
consumption.

Thanks,
Yazen
  

On 2023/6/15 23:15, Yazen Ghannam wrote:
> On 6/14/2023 10:12 PM, Shuai Xue wrote:
>>
>>
>> On 2023/6/14 23:09, Yazen Ghannam wrote:
>>> On 6/13/2023 10:19 PM, Shuai Xue wrote:
>>>>
>>>>
>>>> On 2023/6/13 22:11, Yazen Ghannam wrote:
>>>>> Currently, all valid MCA_ADDR values are assumed to be usable on AMD
>>>>> systems. However, this is not correct in most cases. Notifiers expecting
>>>>> usable addresses may then operate on inappropriate values.
>>>>>
>>>>> Define a helper function to do AMD-specific checks for a usable memory
>>>>> address. List out all known cases.
>>>>>
>>>>> Signed-off-by: Yazen Ghannam <yazen.ghannam@amd.com>
>>>>> ---
>>>>>    arch/x86/kernel/cpu/mce/amd.c      | 38 ++++++++++++++++++++++++++++++
>>>>>    arch/x86/kernel/cpu/mce/core.c     |  3 +++
>>>>>    arch/x86/kernel/cpu/mce/internal.h |  2 ++
>>>>>    3 files changed, 43 insertions(+)
>>>>>
>>>>> diff --git a/arch/x86/kernel/cpu/mce/amd.c b/arch/x86/kernel/cpu/mce/amd.c
>>>>> index 1ccfb0c9257f..ca79fa10b844 100644
>>>>> --- a/arch/x86/kernel/cpu/mce/amd.c
>>>>> +++ b/arch/x86/kernel/cpu/mce/amd.c
>>>>> @@ -746,6 +746,44 @@ bool amd_mce_is_memory_error(struct mce *m)
>>>>>        return legacy_mce_is_memory_error(m);
>>>>>    }
>>>>>    +/*
>>>>> + * AMD systems do not have an explicit indicator that the value in MCA_ADDR is
>>>>> + * a system physical address. Therefore individual cases need to be detected.
>>>>> + * Future cases and checks will be added as needed.
>>>>> + *
>>>>> + * 1) General case
>>>>> + *    a) Assume address is not usable.
>>>>> + * 2) "Poison" errors
>>>>> + *    a) Indicated by MCA_STATUS[43]: POISON. Defined for all banks except legacy
>>>>> + *       Northbridge (bank 4).
>>>>> + *    b) Refers to poison consumption in the Core. Does not include "no action",
>>>>> + *       "action optional", or "deferred" error severities.
>>>>> + *    c) Will include a usuable address so that immediate action can be taken.
>>>>> + * 3) Northbridge DRAM ECC errors
>>>>> + *    a) Reported in legacy bank 4 with XEC 8.
>>>>> + *    b) MCA_STATUS[43] is *not* defined as POISON in legacy bank 4. Therefore,
>>>>> + *       this bit should not be checked.
>>>> [nit]
>>>>
>>>>> + *
>>>>> + * NOTE: SMCA UMC memory errors fall into case #1.
>>>>
>>>> hi, Yazen
>>>>
>>>> The address for SMCA UMC memory error is not system physical address, it make sense
>>>> to be not usable. But how we deal with the SMCA address? The MCE chain like
>>>> uc_decode_notifier will do a sanity check with mce_usable_address and it will not
>>>> handle SMCA address.
>>>>
>>>
>>> Hi Shuai,
>>>
>>> That's correct.
>>>
>>> There isn't a good solution today. This will be handled in future changes.
>>
>> Hi, Yazen,
>>
>> Do you have plan to address it? If not, I can help. We meet this problem in our products.
>>
> 
> Yes, definitely. The first step is to update the address translation code; this is progress. Afterwards, we can find a way to leverage this in the MCE notifier flows.

Look forward to it.

> 
> Just curious, how big is the benefit of this preemptive page offline in your use cases? That is, compared to page offline as part of poison data consumption.

There are three aspects of benefits if SMCA address detected by scrubber is offlined
in advance:

- Free page: it should be isolated and not allocated by buddy so that the poison data
  will never be consumed.
- In-use page: the heath VMs could be migrated into other heath node if many UCE occurs.
- Mitigate the possibility of nested MCE which is a fatal error.

Thank you.

Best Regards,
Shuai.
  

On 6/16/23 03:59, Shuai Xue wrote:
> On 2023/6/15 23:15, Yazen Ghannam wrote:
>> On 6/14/2023 10:12 PM, Shuai Xue wrote:
>>>
>>> On 2023/6/14 23:09, Yazen Ghannam wrote:
>>>> On 6/13/2023 10:19 PM, Shuai Xue wrote:
>>>>>
>>>>> On 2023/6/13 22:11, Yazen Ghannam wrote:
>>>>>> Currently, all valid MCA_ADDR values are assumed to be usable on AMD
>>>>>> systems. However, this is not correct in most cases. Notifiers expecting
>>>>>> usable addresses may then operate on inappropriate values.
>>>>>>
>>>>>> Define a helper function to do AMD-specific checks for a usable memory
>>>>>> address. List out all known cases.
>>>>>>
>>>>>> Signed-off-by: Yazen Ghannam <yazen.ghannam@amd.com>
>>>>>> ---
>>>>>>     arch/x86/kernel/cpu/mce/amd.c      | 38 ++++++++++++++++++++++++++++++
>>>>>>     arch/x86/kernel/cpu/mce/core.c     |  3 +++
>>>>>>     arch/x86/kernel/cpu/mce/internal.h |  2 ++
>>>>>>     3 files changed, 43 insertions(+)
>>>>>>
>>>>>> diff --git a/arch/x86/kernel/cpu/mce/amd.c b/arch/x86/kernel/cpu/mce/amd.c
>>>>>> index 1ccfb0c9257f..ca79fa10b844 100644
>>>>>> --- a/arch/x86/kernel/cpu/mce/amd.c
>>>>>> +++ b/arch/x86/kernel/cpu/mce/amd.c
>>>>>> @@ -746,6 +746,44 @@ bool amd_mce_is_memory_error(struct mce *m)
>>>>>>         return legacy_mce_is_memory_error(m);
>>>>>>     }
>>>>>>     +/*
>>>>>> + * AMD systems do not have an explicit indicator that the value in MCA_ADDR is
>>>>>> + * a system physical address. Therefore individual cases need to be detected.
>>>>>> + * Future cases and checks will be added as needed.
>>>>>> + *
>>>>>> + * 1) General case
>>>>>> + *    a) Assume address is not usable.
>>>>>> + * 2) "Poison" errors
>>>>>> + *    a) Indicated by MCA_STATUS[43]: POISON. Defined for all banks except legacy
>>>>>> + *       Northbridge (bank 4).
>>>>>> + *    b) Refers to poison consumption in the Core. Does not include "no action",
>>>>>> + *       "action optional", or "deferred" error severities.
>>>>>> + *    c) Will include a usuable address so that immediate action can be taken.
>>>>>> + * 3) Northbridge DRAM ECC errors
>>>>>> + *    a) Reported in legacy bank 4 with XEC 8.
>>>>>> + *    b) MCA_STATUS[43] is *not* defined as POISON in legacy bank 4. Therefore,
>>>>>> + *       this bit should not be checked.
>>>>> [nit]
>>>>>
>>>>>> + *
>>>>>> + * NOTE: SMCA UMC memory errors fall into case #1.
>>>>> hi, Yazen
>>>>>
>>>>> The address for SMCA UMC memory error is not system physical address, it make sense
>>>>> to be not usable. But how we deal with the SMCA address? The MCE chain like
>>>>> uc_decode_notifier will do a sanity check with mce_usable_address and it will not
>>>>> handle SMCA address.
>>>>>
>>>> Hi Shuai,
>>>>
>>>> That's correct.
>>>>
>>>> There isn't a good solution today. This will be handled in future changes.
>>> Hi, Yazen,
>>>
>>> Do you have plan to address it? If not, I can help. We meet this problem in our products.
>>>
>> Yes, definitely. The first step is to update the address translation code; this is progress. Afterwards, we can find a way to leverage this in the MCE notifier flows.
> Look forward to it.
>
>> Just curious, how big is the benefit of this preemptive page offline in your use cases? That is, compared to page offline as part of poison data consumption.
> There are three aspects of benefits if SMCA address detected by scrubber is offlined
> in advance:
>
> - Free page: it should be isolated and not allocated by buddy so that the poison data
>    will never be consumed.
> - In-use page: the heath VMs could be migrated into other heath node if many UCE occurs.

I would also like to add that an application able to take action and 
re-create the impacted memory page (like a database for example), could 
do so before the data is requested by a user.

> - Mitigate the possibility of nested MCE which is a fatal error.
>
> Thank you.
>
> Best Regards,
> Shuai.

HTH,
William