[v12,00/13] x86/resctrl: Support for AMD QoS new features

  New AMD processors can now support following QoS features.

1. Slow Memory Bandwidth Allocation (SMBA)
   With this feature, the QOS enforcement policies can be applied
   to the external slow memory connected to the host. QOS enforcement
   is accomplished by assigning a Class Of Service (COS) to a processor
   and specifying allocations or limits for that COS for each resource
   to be allocated.

   Currently, CXL.memory is the only supported "slow" memory device. With
   the support of SMBA feature the hardware enables bandwidth allocation
   on the slow memory devices.

2. Bandwidth Monitoring Event Configuration (BMEC)
   The bandwidth monitoring events mbm_total_event and mbm_local_event 
   are set to count all the total and local reads/writes respectively.
   With the introduction of slow memory, the two counters are not enough
   to count all the different types are memory events. With the feature
   BMEC, the users have the option to configure mbm_total_event and
   mbm_local_event to count the specific type of events.

   Following are the bitmaps of events supported.
   Bits    Description
     6       Dirty Victims from the QOS domain to all types of memory
     5       Reads to slow memory in the non-local NUMA domain
     4       Reads to slow memory in the local NUMA domain
     3       Non-temporal writes to non-local NUMA domain
     2       Non-temporal writes to local NUMA domain
     1       Reads to memory in the non-local NUMA domain
     0       Reads to memory in the local NUMA domain

This series adds support for these features. Also added a minor cleanup(PATCH 1).

Feature description is available in the specification, "AMD64 Technology Platform Quality of Service Extensions, Revision: 1.03 Publication # 56375
Revision: 1.03 Issue Date: February 2022".

Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
Link: https://www.amd.com/en/support/tech-docs/amd64-technology-platform-quality-service-extensions
---
v12:
 Summary of changes:
 Refreshed on top of tip/master.
 Made the new cpu features SMBA and BMEC invisible from /proc/cpuinfo based on our discussion.
 https://lore.kernel.org/lkml/Y7xjxUj+KnOEJssZ@zn.tnic/
 Updated the Documentation/x86/resctrl.rst with minor changes.

v11:
 https://lore.kernel.org/lkml/20230109164405.569714-2-babu.moger@amd.com/
 Summary of changes:
 Removed cpus_read_lock() from the patch 11 and 12 based on our discussion.
 https://lore.kernel.org/lkml/3dc31a6d-5485-746d-3c49-df7dcd1827e3@intel.com/
 Picked up all the Reviewed-by by Reinette.
 Minor text changes on patch 13.
 
v10:
 https://lore.kernel.org/lkml/20221222233127.910538-1-babu.moger@amd.com/
 Summary of changes:
 1. Moved the patch 12 to 1 ("x86/resctrl: Replace smp_call_function_many() with on_each_cpu_mask()")
 2. No need to write MSR_IA32_EVT_CFG_BASE on all the CPUs. Replaced on_each_cpu_mask with smp_call_function_any.
 3. Updated the subject line of the patches to replace sysfs with resctrl.
 4. Added Reviewed-by for the patches which are reviewed by Reinette.
 5. Added few more comments suggested by Reinette.
  
v9:
 https://lore.kernel.org/lkml/166990882621.17806.16780480657453071426.stgit@bmoger-ubuntu/
 Summary of changes:
 1. Rebased on top of lastest tip/master as of 11/30.
 2. Most of the changes are result of the comments from Fenghua, Reinette and Peter Newman.
 3. Fixed the cpuid dependancy.
 4. Added the __init attribute to rdt_get_mon_l3_config and mbm_config_rftype_init.
 5. Added new function resctrl_arch_reset_rmid_all to clear all rmid statues.
 6. Changed mon_event_config_index_get based on Reinette's comments.
 7. Changed mbm_config_rftype_init to take care of few extra error handling.
 8. Few other minor changes and text changes.

v8:
 https://lore.kernel.org/lkml/166759188265.3281208.11769277079826754455.stgit@bmoger-ubuntu/
 Changes:
 1. Removed init attribute for rdt_cpu_has to make it available for all the files.
 2. Updated the change log for mon_features to correct the names of config files.
 3. Changed configuration file name from mbm_total_config to mbm_total_bytes_config.
    This is more consistant with other changes.
 4. Added lock protection while reading/writing the config file.
 5. Other few minor text changes. I have been missing few comments in last couple of
    revisions. Hope I have addressed all of them this time.

v7:
 https://lore.kernel.org/lkml/166604543832.5345.9696970469830919982.stgit@bmoger-ubuntu/
 Changes:
 Not much of a change. Missed one comment from Reinette from v5. Corrected it now.
 Few format corrections from Sanjaya.

v6:
 https://lore.kernel.org/lkml/166543345606.23830.3120625408601531368.stgit@bmoger-ubuntu/
 Summary of changes:
 1. Rebased on top of lastest tip tree. Fixed few minor conflicts.
 2. Fixed format issue with scattered.c.
 3. Removed config_name from the structure mon_evt. It is not required.
 4. The read/write format for mbm_total_config and mbm_local_config will be same
    as schemata format "id0=val0;id1=val1;...". This is comment from Fenghua.
 5. Added more comments MSR_IA32_EVT_CFG_BASE writng.
 5. Few text changes in resctrl.rst 
 
v5:
  https://lore.kernel.org/lkml/166431016617.373387.1968875281081252467.stgit@bmoger-ubuntu/
  Summary of changes.
  1. Split the series into two. The first two patches are bug fixes. So, sent them separate.
  2. The config files mbm_total_config and mbm_local_config are now under
     /sys/fs/resctrl/info/L3_MON/. Removed these config files from mon groups.
  3. Ran "checkpatch --strict --codespell" on all the patches. Looks good with few known exceptions.
  4. Few minor text changes in resctrl.rst file. 

v4:
  https://lore.kernel.org/lkml/166257348081.1043018.11227924488792315932.stgit@bmoger-ubuntu/
  Got numerios of comments from Reinette Chatre. Addressed most of them. 
  Summary of changes.
  1. Removed mon_configurable under /sys/fs/resctrl/info/L3_MON/.  
  2. Updated mon_features texts if the BMEC is supported.
  3. Added more explanation about the slow memory support.
  4. Replaced smp_call_function_many with on_each_cpu_mask call.
  5. Removed arch_has_empty_bitmaps
  6. Few other text changes.
  7. Removed Reviewed-by if the patch is modified.
  8. Rebased the patches to latest tip.

v3:
  https://lore.kernel.org/lkml/166117559756.6695.16047463526634290701.stgit@bmoger-ubuntu/
  a. Rebased the patches to latest tip. Resolved some conflicts.
     https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git
  b. Taken care of feedback from Bagas Sanjaya.
  c. Added Reviewed by from Mingo.
  Note: I am still looking for comments from Reinette or Fenghua.

v2:
  https://lore.kernel.org/lkml/165938717220.724959.10931629283087443782.stgit@bmoger-ubuntu/
  a. Rebased the patches to latest stable tree (v5.18.15). Resolved some conflicts.
  b. Added the patch to fix CBM issue on AMD. This was originally discussed
     https://lore.kernel.org/lkml/20220517001234.3137157-1-eranian@google.com/

v1:
  https://lore.kernel.org/lkml/165757543252.416408.13547339307237713464.stgit@bmoger-ubuntu/

*** BLURB HERE ***

Babu Moger (13):
  x86/resctrl: Replace smp_call_function_many() with on_each_cpu_mask()
  x86/cpufeatures: Add Slow Memory Bandwidth Allocation feature flag
  x86/resctrl: Add a new resource type RDT_RESOURCE_SMBA
  x86/cpufeatures: Add Bandwidth Monitoring Event Configuration feature
    flag
  x86/resctrl: Include new features in command line options
  x86/resctrl: Detect and configure Slow Memory Bandwidth Allocation
  x86/resctrl: Add __init attribute to rdt_get_mon_l3_config()
  x86/resctrl: Support monitor configuration
  x86/resctrl: Add interface to read mbm_total_bytes_config
  x86/resctrl: Add interface to read mbm_local_bytes_config
  x86/resctrl: Add interface to write mbm_total_bytes_config
  x86/resctrl: Add interface to write mbm_local_bytes_config
  Documentation/x86: Update resctrl.rst for new features

 .../admin-guide/kernel-parameters.txt         |   2 +-
 Documentation/x86/resctrl.rst                 | 147 ++++++++-
 arch/x86/include/asm/cpufeatures.h            |   2 +
 arch/x86/include/asm/msr-index.h              |   2 +
 arch/x86/kernel/cpu/cpuid-deps.c              |   2 +
 arch/x86/kernel/cpu/resctrl/core.c            |  54 ++-
 arch/x86/kernel/cpu/resctrl/ctrlmondata.c     |  13 +-
 arch/x86/kernel/cpu/resctrl/internal.h        |  28 ++
 arch/x86/kernel/cpu/resctrl/monitor.c         |  30 +-
 arch/x86/kernel/cpu/resctrl/rdtgroup.c        | 307 ++++++++++++++++--
 arch/x86/kernel/cpu/scattered.c               |   2 +
 include/linux/resctrl.h                       |  11 +
 12 files changed, 559 insertions(+), 41 deletions(-)
  

Hi Babu,

On 1/13/2023 7:20 AM, Babu Moger wrote:
> New AMD processors can now support following QoS features.
> 
> 1. Slow Memory Bandwidth Allocation (SMBA)
>    With this feature, the QOS enforcement policies can be applied
>    to the external slow memory connected to the host. QOS enforcement
>    is accomplished by assigning a Class Of Service (COS) to a processor
>    and specifying allocations or limits for that COS for each resource
>    to be allocated.
> 
>    Currently, CXL.memory is the only supported "slow" memory device. With
>    the support of SMBA feature the hardware enables bandwidth allocation
>    on the slow memory devices.
> 
> 2. Bandwidth Monitoring Event Configuration (BMEC)
>    The bandwidth monitoring events mbm_total_event and mbm_local_event 
>    are set to count all the total and local reads/writes respectively.
>    With the introduction of slow memory, the two counters are not enough
>    to count all the different types are memory events. With the feature
>    BMEC, the users have the option to configure mbm_total_event and
>    mbm_local_event to count the specific type of events.
> 
>    Following are the bitmaps of events supported.
>    Bits    Description
>      6       Dirty Victims from the QOS domain to all types of memory
>      5       Reads to slow memory in the non-local NUMA domain
>      4       Reads to slow memory in the local NUMA domain
>      3       Non-temporal writes to non-local NUMA domain
>      2       Non-temporal writes to local NUMA domain
>      1       Reads to memory in the non-local NUMA domain
>      0       Reads to memory in the local NUMA domain
> 
> This series adds support for these features. Also added a minor cleanup(PATCH 1).
> 
> Feature description is available in the specification, "AMD64 Technology Platform Quality of Service Extensions, Revision: 1.03 Publication # 56375
> Revision: 1.03 Issue Date: February 2022".
> 
> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
> Link: https://www.amd.com/en/support/tech-docs/amd64-technology-platform-quality-service-extensions

This series looks good to me. I added my "Reviewed-by" tag to patch 13/13. All
the other patches in this series already have my "Reviewed-by" tag.

Thank you very much. 

Reinette
  

On 1/17/23 16:16, Reinette Chatre wrote:
> Hi Babu,
>
> On 1/13/2023 7:20 AM, Babu Moger wrote:
>> New AMD processors can now support following QoS features.
>>
>> 1. Slow Memory Bandwidth Allocation (SMBA)
>>    With this feature, the QOS enforcement policies can be applied
>>    to the external slow memory connected to the host. QOS enforcement
>>    is accomplished by assigning a Class Of Service (COS) to a processor
>>    and specifying allocations or limits for that COS for each resource
>>    to be allocated.
>>
>>    Currently, CXL.memory is the only supported "slow" memory device. With
>>    the support of SMBA feature the hardware enables bandwidth allocation
>>    on the slow memory devices.
>>
>> 2. Bandwidth Monitoring Event Configuration (BMEC)
>>    The bandwidth monitoring events mbm_total_event and mbm_local_event 
>>    are set to count all the total and local reads/writes respectively.
>>    With the introduction of slow memory, the two counters are not enough
>>    to count all the different types are memory events. With the feature
>>    BMEC, the users have the option to configure mbm_total_event and
>>    mbm_local_event to count the specific type of events.
>>
>>    Following are the bitmaps of events supported.
>>    Bits    Description
>>      6       Dirty Victims from the QOS domain to all types of memory
>>      5       Reads to slow memory in the non-local NUMA domain
>>      4       Reads to slow memory in the local NUMA domain
>>      3       Non-temporal writes to non-local NUMA domain
>>      2       Non-temporal writes to local NUMA domain
>>      1       Reads to memory in the non-local NUMA domain
>>      0       Reads to memory in the local NUMA domain
>>
>> This series adds support for these features. Also added a minor cleanup(PATCH 1).
>>
>> Feature description is available in the specification, "AMD64 Technology Platform Quality of Service Extensions, Revision: 1.03 Publication # 56375
>> Revision: 1.03 Issue Date: February 2022".
>>
>> Link: https://bugzilla.kernel.org/show_bug.cgi?id=206537
>> Link: https://www.amd.com/en/support/tech-docs/amd64-technology-platform-quality-service-extensions
> This series looks good to me. I added my "Reviewed-by" tag to patch 13/13. All
> the other patches in this series already have my "Reviewed-by" tag.

Thank you very much.
Babu Moger