mbox series

[RFC,0/2] Node migration between memory tiers

Message ID 20231130214858.1887-1-sthanneeru.opensrc@micron.com
Headers
Series Node migration between memory tiers |

Message

Srinivasulu Opensrc Nov. 30, 2023, 9:48 p.m. UTC 
  From: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>

The memory tiers feature allows nodes with similar memory types
or performance characteristics to be grouped together in a
memory tier. However, there is currently no provision for
moving a node from one tier to another on demand.

This patch series aims to support node migration between tiers
on demand by sysadmin/root user using the provided sysfs for
node migration. Each tier has a start abstract distance(adistance)
and range.

To migrate a node to a tier, the corresponding node’s sysfs
adistance_offset is written with a value corresponding to
the tier’s adistance.

Example: Move node2 to memory tier5 from its default tier(i.e 4)

1. Check default values:
$cat /sys/devices/virtual/memory_tiering/memory_tier4/nodelist
0-2

$cat /sys/devices/system/node/node0/adistance_offset
0
$cat /sys/devices/system/node/node1/adistance_offset
0
$cat /sys/devices/system/node/node2/adistance_offset
0

2. Move node2 to  tier5:

To move node2 from emory_tier4 (adistance=512) to
emory_tier5 (abstract=640), set the `adistance_offset` of
node 2 to 128 (i.e., 512 + 128 = 640).

Tier4 adistance start can be derved from tier-id
(i.e for tier4, 4 << 7 = 512).

$echo 128 > /sys/devices/system/node/node2/adistance_offset
$cat /sys/devices/system/node/node2/adistance_offset
128

3. Verify node2's tier id:

$cat /sys/devices/virtual/memory_tiering/memory_tier5/nodelist
2
$cat /sys/devices/virtual/memory_tiering/memory_tier4/nodelist
0-1

Srinivasulu Thanneeru (2):
  base/node: Add sysfs for adistance_offset
  memory tier: Support node migration between tiers

 drivers/base/node.c          | 51 +++++++++++++++++++++++
 include/linux/memory-tiers.h | 11 +++++
 include/linux/node.h         |  6 +++
 mm/memory-tiers.c            | 79 ++++++++++++++++++++----------------
 4 files changed, 113 insertions(+), 34 deletions(-)

Comments

Huang, Ying Dec. 4, 2023, 8:52 a.m. UTC | #1 
<sthanneeru.opensrc@micron.com> writes:

> From: Srinivasulu Thanneeru <sthanneeru.opensrc@micron.com>
>
> The memory tiers feature allows nodes with similar memory types
> or performance characteristics to be grouped together in a
> memory tier. However, there is currently no provision for
> moving a node from one tier to another on demand.
>
> This patch series aims to support node migration between tiers
> on demand by sysadmin/root user using the provided sysfs for
> node migration. Each tier has a start abstract distance(adistance)
> and range.

We have discussed migrating nodes (in fact nodes of a memory type)
between tiers by sysadmin/root before.  The basic idea comes from
Johannes.  It is summarized in page 11 of [1],

[1] https://lpc.events/event/16/contributions/1209/attachments/1042/1995/Live%20In%20a%20World%20With%20Multiple%20Memory%20Types.pdf

The abstract distance of a memory type (e.g., GPU HBM) can be adjusted
via a sysfs knob (<memory_type>/abstract_distance_offset).

I still think that the memory type is better to be used to change
the abstract distance of nodes.  Do you agree?

--
Best Regards,
Huang, Ying

> To migrate a node to a tier, the corresponding node’s sysfs
> adistance_offset is written with a value corresponding to
> the tier’s adistance.
>
> Example: Move node2 to memory tier5 from its default tier(i.e 4)
>
> 1. Check default values:
> $cat /sys/devices/virtual/memory_tiering/memory_tier4/nodelist
> 0-2
>
> $cat /sys/devices/system/node/node0/adistance_offset
> 0
> $cat /sys/devices/system/node/node1/adistance_offset
> 0
> $cat /sys/devices/system/node/node2/adistance_offset
> 0
>
> 2. Move node2 to  tier5:
>
> To move node2 from emory_tier4 (adistance=512) to
> emory_tier5 (abstract=640), set the `adistance_offset` of
> node 2 to 128 (i.e., 512 + 128 = 640).
>
> Tier4 adistance start can be derved from tier-id
> (i.e for tier4, 4 << 7 = 512).
>
> $echo 128 > /sys/devices/system/node/node2/adistance_offset
> $cat /sys/devices/system/node/node2/adistance_offset
> 128
>
> 3. Verify node2's tier id:
>
> $cat /sys/devices/virtual/memory_tiering/memory_tier5/nodelist
> 2
> $cat /sys/devices/virtual/memory_tiering/memory_tier4/nodelist
> 0-1
>
> Srinivasulu Thanneeru (2):
>   base/node: Add sysfs for adistance_offset
>   memory tier: Support node migration between tiers
>
>  drivers/base/node.c          | 51 +++++++++++++++++++++++
>  include/linux/memory-tiers.h | 11 +++++
>  include/linux/node.h         |  6 +++
>  mm/memory-tiers.c            | 79 ++++++++++++++++++++----------------
>  4 files changed, 113 insertions(+), 34 deletions(-)