[PATCHv2,02/13] kernel/cpu: Add support for declaring CPU offlining not supported

  ACPI MADT doesn't allow to offline CPU after it got woke up.

Currently offlining hotplug prevented based on the confidential
computing attribute which is set for Intel TDX. But TDX is not
the only possible user of the wake up method.

Introduce cpu_hotplug_not_supported() that can be called to indicate
that CPU offlining should be disabled.

This function is going to replace CC_ATTR_HOTPLUG_DISABLED for ACPI
MADT.

Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
---
 include/linux/cpu.h |  2 ++
 kernel/cpu.c        | 13 ++++++++++++-
 2 files changed, 14 insertions(+), 1 deletion(-)
  

On Fri, 2023-10-20 at 18:12 +0300, Kirill A. Shutemov wrote:
> ACPI MADT doesn't allow to offline CPU after it got woke up.
> 
> Currently offlining hotplug prevented based on the confidential
> computing attribute which is set for Intel TDX. But TDX is not
> the only possible user of the wake up method.
> 
> Introduce cpu_hotplug_not_supported() that can be called to indicate
> that CPU offlining should be disabled.
> 
> This function is going to replace CC_ATTR_HOTPLUG_DISABLED for ACPI
> MADT.
> 
> Signed-off-by: Kirill A. Shutemov <kirill.shutemov@linux.intel.com>
> ---
>  include/linux/cpu.h |  2 ++
>  kernel/cpu.c        | 13 ++++++++++++-
>  2 files changed, 14 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/cpu.h b/include/linux/cpu.h
> index f19f56501809..97832ced939d 100644
> --- a/include/linux/cpu.h
> +++ b/include/linux/cpu.h
> @@ -132,6 +132,7 @@ extern void cpus_read_lock(void);
>  extern void cpus_read_unlock(void);
>  extern int  cpus_read_trylock(void);
>  extern void lockdep_assert_cpus_held(void);
> +extern void cpu_hotplug_disable_offlining(void);
>  extern void cpu_hotplug_disable(void);
>  extern void cpu_hotplug_enable(void);
>  void clear_tasks_mm_cpumask(int cpu);
> @@ -147,6 +148,7 @@ static inline void cpus_read_lock(void) { }
>  static inline void cpus_read_unlock(void) { }
>  static inline int  cpus_read_trylock(void) { return true; }
>  static inline void lockdep_assert_cpus_held(void) { }
> +static inline void cpu_hotplug_disable_offlining(void) { }
>  static inline void cpu_hotplug_disable(void) { }
>  static inline void cpu_hotplug_enable(void) { }
>  static inline int remove_cpu(unsigned int cpu) { return -EPERM; }
> diff --git a/kernel/cpu.c b/kernel/cpu.c
> index 6de7c6bb74ee..faebee0050a2 100644
> --- a/kernel/cpu.c
> +++ b/kernel/cpu.c
> @@ -484,6 +484,8 @@ static int cpu_hotplug_disabled;
>  
>  DEFINE_STATIC_PERCPU_RWSEM(cpu_hotplug_lock);
>  
> +static bool cpu_hotplug_offline_disabled;
> +
>  void cpus_read_lock(void)
>  {
>  	percpu_down_read(&cpu_hotplug_lock);
> @@ -543,6 +545,14 @@ static void lockdep_release_cpus_lock(void)
>  	rwsem_release(&cpu_hotplug_lock.dep_map, _THIS_IP_);
>  }
>  
> +/* Declare CPU offlining not supported */
> +void cpu_hotplug_disable_offlining(void)
> +{
> +	cpu_maps_update_begin();
> +	cpu_hotplug_offline_disabled = true;
> +	cpu_maps_update_done();
> +}
> +
>  /*
>   * Wait for currently running CPU hotplug operations to complete (if any) and
>   * disable future CPU hotplug (from sysfs). The 'cpu_add_remove_lock' protects
> @@ -1507,7 +1517,8 @@ static int cpu_down_maps_locked(unsigned int cpu, enum cpuhp_state target)
>  	 * If the platform does not support hotplug, report it explicitly to
>  	 * differentiate it from a transient offlining failure.
>  	 */
> -	if (cc_platform_has(CC_ATTR_HOTPLUG_DISABLED))
> +	if (cc_platform_has(CC_ATTR_HOTPLUG_DISABLED) ||
> +	    cpu_hotplug_offline_disabled)
>  		return -EOPNOTSUPP;
>  	if (cpu_hotplug_disabled)
>  		return -EBUSY;


IMHO it's a little bit odd to have two mechanisms in place, even in this middle
state patch.  Is it better to completely replace CC_ATTR_HOTPLUG_DISABLED with
the new cpu_hotplug_offline_disabled in this patch? You can explicitly call
cpu_hotplug_disable_offlining() in tdx_early_init() so no functional change is
done.

Or I am wondering why cannot just merge this and the next patch together, with a
proper justification?

Btw, IMHO the changelog (this and next patch's) seems didn't explain the true
reason to replace CC_ATTR_HOTPLUG_DISABLED.

	Currently hotplug prevented based on the confidential computing
	attribute which is set for Intel TDX. But TDX is not the only possible
	user of the wake up method.

"TDX is not the only possible user of the wake up method" doesn't mean we need
to replace CC_ATTR_HOTPLUG_DISABLED.  E.g., other CoCo VM type can also select
CC_ATTR_HOTPLUG_DISABLED if it uses MADT wake up method.

To me the true reason is the new MADT wake up version actually brings the
support of offlining cpu, thus it's more suitable to decide whether the CoCo VM
needs to disable CPU offline based on the MADT wake up version, but not the CC_*
attributes that is determined by CoCo VM type.
  

On Mon, Oct 23, 2023 at 09:30:59AM +0000, Huang, Kai wrote:
> IMHO it's a little bit odd to have two mechanisms in place, even in this middle
> state patch.  Is it better to completely replace CC_ATTR_HOTPLUG_DISABLED with
> the new cpu_hotplug_offline_disabled in this patch? You can explicitly call
> cpu_hotplug_disable_offlining() in tdx_early_init() so no functional change is
> done.

I can. But I don't see how it makes a difference.

> Or I am wondering why cannot just merge this and the next patch together, with a
> proper justification?

Because the very next thing reviewers would ask is to split them :P

> Btw, IMHO the changelog (this and next patch's) seems didn't explain the true
> reason to replace CC_ATTR_HOTPLUG_DISABLED.
> 
> 	Currently hotplug prevented based on the confidential computing
> 	attribute which is set for Intel TDX. But TDX is not the only possible
> 	user of the wake up method.
> 
> "TDX is not the only possible user of the wake up method" doesn't mean we need
> to replace CC_ATTR_HOTPLUG_DISABLED.  E.g., other CoCo VM type can also select
> CC_ATTR_HOTPLUG_DISABLED if it uses MADT wake up method.
> 
> To me the true reason is the new MADT wake up version actually brings the
> support of offlining cpu, thus it's more suitable to decide whether the CoCo VM
> needs to disable CPU offline based on the MADT wake up version, but not the CC_*
> attributes that is determined by CoCo VM type.

No. MADT is orthogonal to CoCo. It can be implemented outside of CoCo
environment and CoCo platform can implement other wake up methods. It is
not up to TDX/SEV/whatever to decide if offlining is supported. It is
property of the wakeup method implemented on the platform.
  

On Mon, 2023-10-23 at 18:31 +0300, kirill.shutemov@linux.intel.com wrote:
> On Mon, Oct 23, 2023 at 09:30:59AM +0000, Huang, Kai wrote:
> > IMHO it's a little bit odd to have two mechanisms in place, even in this middle
> > state patch.  Is it better to completely replace CC_ATTR_HOTPLUG_DISABLED with
> > the new cpu_hotplug_offline_disabled in this patch? You can explicitly call
> > cpu_hotplug_disable_offlining() in tdx_early_init() so no functional change is
> > done.
> 
> I can. But I don't see how it makes a difference.

Personally I think this is better because it is odd to have two mechanisms in
place even temporarily especially when we can avoid it.  But no hard opinion. 
Up to you.

> 
> > Or I am wondering why cannot just merge this and the next patch together, with a
> > proper justification?
> 
> Because the very next thing reviewers would ask is to split them :P
> 
> > Btw, IMHO the changelog (this and next patch's) seems didn't explain the true
> > reason to replace CC_ATTR_HOTPLUG_DISABLED.
> > 
> > 	Currently hotplug prevented based on the confidential computing
> > 	attribute which is set for Intel TDX. But TDX is not the only possible
> > 	user of the wake up method.
> > 
> > "TDX is not the only possible user of the wake up method" doesn't mean we need
> > to replace CC_ATTR_HOTPLUG_DISABLED.  E.g., other CoCo VM type can also select
> > CC_ATTR_HOTPLUG_DISABLED if it uses MADT wake up method.
> > 
> > To me the true reason is the new MADT wake up version actually brings the
> > support of offlining cpu, thus it's more suitable to decide whether the CoCo VM
> > needs to disable CPU offline based on the MADT wake up version, but not the CC_*
> > attributes that is determined by CoCo VM type.
> 
> No. MADT is orthogonal to CoCo. It can be implemented outside of CoCo
> environment and CoCo platform can implement other wake up methods. It is
> not up to TDX/SEV/whatever to decide if offlining is supported. It is
> property of the wakeup method implemented on the platform.
> 

Yeah sure.  Can we put this to changelog to make it clearer? :-)
  

On Mon, Oct 23 2023 at 22:07, Kai Huang wrote:
> On Mon, 2023-10-23 at 18:31 +0300, kirill.shutemov@linux.intel.com wrote:
>> On Mon, Oct 23, 2023 at 09:30:59AM +0000, Huang, Kai wrote:
>> > IMHO it's a little bit odd to have two mechanisms in place, even in this middle
>> > state patch.  Is it better to completely replace CC_ATTR_HOTPLUG_DISABLED with
>> > the new cpu_hotplug_offline_disabled in this patch? You can explicitly call
>> > cpu_hotplug_disable_offlining() in tdx_early_init() so no functional change is
>> > done.
>> 
>> I can. But I don't see how it makes a difference.
>
> Personally I think this is better because it is odd to have two mechanisms in
> place even temporarily especially when we can avoid it.  But no hard opinion. 
> Up to you.

It's not at all better because having this clear split makes it simpler
to review and ponder the individual changes.

Thanks,

        tglx
  

On Fri, Oct 20 2023 at 18:12, Kirill A. Shutemov wrote:

Subject: cpu/hotplug: ...

> ACPI MADT doesn't allow to offline CPU after it got woke up.

ACPI MADT is a table ...

This is about the MADT mailbox wakeup method, right?

> Currently offlining hotplug prevented based on the confidential

is prevented

> computing attribute which is set for Intel TDX. But TDX is not
> the only possible user of the wake up method.
>
> Introduce cpu_hotplug_not_supported() that can be called to indicate
> that CPU offlining should be disabled.

Otherwise the changes look good!
  

On Sat, Oct 28, 2023 at 04:12:10PM +0200, Thomas Gleixner wrote:
> On Fri, Oct 20 2023 at 18:12, Kirill A. Shutemov wrote:
> 
> Subject: cpu/hotplug: ...
> 
> > ACPI MADT doesn't allow to offline CPU after it got woke up.
> 
> ACPI MADT is a table ...
> 
> This is about the MADT mailbox wakeup method, right?
> 
> > Currently offlining hotplug prevented based on the confidential
> 
> is prevented
> 
> > computing attribute which is set for Intel TDX. But TDX is not
> > the only possible user of the wake up method.
> >
> > Introduce cpu_hotplug_not_supported() that can be called to indicate
> > that CPU offlining should be disabled.
> 
> Otherwise the changes look good!

Thanks!

Thomas, could you take a look if the last patch in the series is sane?