From patchwork Wed Nov 2 16:13:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 14358 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp8652wru; Wed, 2 Nov 2022 09:23:00 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5NfNQbIIIGoisLmKPV+ClVq75R2BoiHW+H093iv9nFW/U2keLa2N9r42RrRIHKDP7YCAxw X-Received: by 2002:a05:6a00:2350:b0:541:b5bf:2774 with SMTP id j16-20020a056a00235000b00541b5bf2774mr25255005pfj.28.1667406179891; Wed, 02 Nov 2022 09:22:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667406179; cv=none; d=google.com; s=arc-20160816; b=hSvTr44YbouEfZYt9sKrDm/V6AKOZ/as+9P+oOLznqJ3phRAjX0PqDOLR/7Rz4/dEJ l8hNvpFulloeI2bQdHq/nqX4iRSZF+kA/33TvbEYH90fLEjLI6HY97rJ0udZeOWv63j3 +BnFJvD+KClvuGJIYpkqLouhdrhrJoacdN4t6vTXDsOecNvPwoGi8UcoA+O1phN1hnqw fNGbYIvajyXWUTU+awyRPwO+M2iP3F1pb2eUhp/wS+z3QaicOPC++EfwGoxRLIGA7eR7 JAqCTXwp8/aBVpcqnx34voeZZfCP+XtNfRD56s9oxFJ1kBAAGzzMQ+9qLuHjGeziDg0V hOoA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TKetkh4WHRfyiXWR3Ck7PLJHjhMFIGu7qwf9qalJlhM=; b=VOYpLxt84TEH72w5F8KvNPumLyzAr9aGL7vPS9K31YwZyAYpy8tOuuljWq4Rg/jjUn cqSzEqk9NqhYWbOD/OEQtvIahRW9Ybvkxk+Ft4BDzCq3U1sFyDnbay6SHlcSDb7KoHa9 v7nOjH4vML0atFqVSKB4MQrn7MDoHj2xw3tYzkOimSgnEujVYr3DQK3/WHItooWAZ/pt Zmanpl2VQzXGvbvYRajif6enVhwuNS0+e8X8+g3yZomFD7joxVETc8ANN/DzCsc1cKPX +sL1S8x3P7BX9by9g/mvJUP7SIdh28nWWewS6RU6tHdeCGoLZ4zS2DuwFyXB+l++1Q+q 17CQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="HXMN/eED"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d21-20020a056a00199500b0050df0383302si19470011pfl.255.2022.11.02.09.22.43; Wed, 02 Nov 2022 09:22:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="HXMN/eED"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=bytedance.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231697AbiKBQVR (ORCPT + 99 others); Wed, 2 Nov 2022 12:21:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231514AbiKBQUi (ORCPT ); Wed, 2 Nov 2022 12:20:38 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 649A3317D9 for ; Wed, 2 Nov 2022 09:14:58 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id j15so25278167wrq.3 for ; Wed, 02 Nov 2022 09:14:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TKetkh4WHRfyiXWR3Ck7PLJHjhMFIGu7qwf9qalJlhM=; b=HXMN/eEDtXRg0lyznWzotr3Id2ejKM0OY0ZT7J1wWoBooooL9lw9TUkFXLirKbziOx 5ZpT0h0dn7/SEniBN03v1j0ElI5smjuDbOper+qxDTEIT7z3yQ0xvSxgu7JtD29haSWd 6tdgy38Uu6VXqF6Xb55fF0gb6L8X2QfogulXBPzPoMe3u9il7bCA9+XT/bpIVR4myrLy mCI91Rsl8oQq43jfRCxk+92/S4hxm0PbFpFBr0AYOtMEmWTwrJJ+za/3GNgEbLbVp0Md MEeR+3D3ayYWEoCqG+l+YCvPce/e4gZfKY88Ar6tBBM8P+4Zj4owL4VqQC+PYKeR/oQp M6IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TKetkh4WHRfyiXWR3Ck7PLJHjhMFIGu7qwf9qalJlhM=; b=h83xUo8Liyh9mEbT3s/I0dk4d1o638eYjNd6FG66A9NDpyAP7+PQHSEIqz44aCzzWP q8yMWQoUzt8+kBNJ356UnqaXKywLQiOqs74p+bjuDWXp0bBbC/Zx8NpNwhs432AVg0MA dwH7HLPzYY9cRcRbYaVsOP1GHYoEYcPtZI4o8QNqNnZvcZDDK5PJWx/UPb8t3pFzgNUi pnj0ONUqPuI42soxI0CbJ+4yOhXU1SvLLVGNTL2Qjw8JDLZOmpPwAKtriHt0TwVcHDTc JYa+XOZfAGasInzPMUUUiJ6HJdr/cevw/3wCQWFlH4sLUr2Scd4cyf13LfLXOFoMyUtW TQQQ== X-Gm-Message-State: ACrzQf0DFjFfLSauKWTm+NxtRh7dp9mRZ9wEfoq7NIw1Du4OcLuVqBRn on3ly8Kzya8sUV9uoEx4D6mYEOo4bMgUUTB6 X-Received: by 2002:a05:6000:170d:b0:236:6aa1:8a56 with SMTP id n13-20020a056000170d00b002366aa18a56mr15676520wrc.302.1667405663370; Wed, 02 Nov 2022 09:14:23 -0700 (PDT) Received: from usaari01.cust.communityfibre.co.uk ([2a02:6b6a:b4d7:0:6a08:9b26:ab04:7065]) by smtp.gmail.com with ESMTPSA id m17-20020a5d56d1000000b0022cc6b8df5esm13230923wrw.7.2022.11.02.09.14.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 02 Nov 2022 09:14:22 -0700 (PDT) From: Usama Arif To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, linux-doc@vger.kernel.org, virtualization@lists.linux-foundation.org, linux@armlinux.org.uk, yezengruan@huawei.com, catalin.marinas@arm.com, will@kernel.org, maz@kernel.org, steven.price@arm.com, mark.rutland@arm.com Cc: fam.zheng@bytedance.com, liangma@liangbit.com, punit.agrawal@bytedance.com, Usama Arif Subject: [RFC 5/6] KVM: arm64: Support the VCPU preemption check Date: Wed, 2 Nov 2022 16:13:39 +0000 Message-Id: <20221102161340.2982090-6-usama.arif@bytedance.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20221102161340.2982090-1-usama.arif@bytedance.com> References: <20221102161340.2982090-1-usama.arif@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748402102916312769?= X-GMAIL-MSGID: =?utf-8?q?1748402102916312769?= Support the vcpu_is_preempted() functionality under KVM/arm64. This will enhance lock performance on overcommitted hosts (more runnable VCPUs than physical CPUs in the system) as doing busy waits for preempted VCPUs will hurt system performance far worse than early yielding. Signed-off-by: Zengruan Ye Signed-off-by: Usama Arif --- arch/arm64/include/asm/paravirt.h | 2 + arch/arm64/include/asm/spinlock.h | 16 +++- arch/arm64/kernel/paravirt.c | 126 ++++++++++++++++++++++++++++++ arch/arm64/kernel/setup.c | 3 + include/linux/cpuhotplug.h | 1 + 5 files changed, 147 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/paravirt.h b/arch/arm64/include/asm/paravirt.h index 9aa193e0e8f2..4ccb4356c56b 100644 --- a/arch/arm64/include/asm/paravirt.h +++ b/arch/arm64/include/asm/paravirt.h @@ -19,10 +19,12 @@ static inline u64 paravirt_steal_clock(int cpu) } int __init pv_time_init(void); +int __init pv_lock_init(void); #else #define pv_time_init() do {} while (0) +#define pv_lock_init() do {} while (0) #endif // CONFIG_PARAVIRT diff --git a/arch/arm64/include/asm/spinlock.h b/arch/arm64/include/asm/spinlock.h index 0525c0b089ed..7023efa4de96 100644 --- a/arch/arm64/include/asm/spinlock.h +++ b/arch/arm64/include/asm/spinlock.h @@ -10,7 +10,20 @@ /* See include/linux/spinlock.h */ #define smp_mb__after_spinlock() smp_mb() +#define vcpu_is_preempted vcpu_is_preempted + +#ifdef CONFIG_PARAVIRT +#include + +bool dummy_vcpu_is_preempted(int cpu); +DECLARE_STATIC_CALL(pv_vcpu_is_preempted, dummy_vcpu_is_preempted); +static inline bool vcpu_is_preempted(int cpu) +{ + return static_call(pv_vcpu_is_preempted)(cpu); +} + +#else /* * Changing this will break osq_lock() thanks to the call inside * smp_cond_load_relaxed(). @@ -18,10 +31,11 @@ * See: * https://lore.kernel.org/lkml/20200110100612.GC2827@hirez.programming.kicks-ass.net */ -#define vcpu_is_preempted vcpu_is_preempted static inline bool vcpu_is_preempted(int cpu) { return false; } +#endif /* CONFIG_PARAVIRT */ + #endif /* __ASM_SPINLOCK_H */ diff --git a/arch/arm64/kernel/paravirt.c b/arch/arm64/kernel/paravirt.c index 57c7c211f8c7..45bcca87bed7 100644 --- a/arch/arm64/kernel/paravirt.c +++ b/arch/arm64/kernel/paravirt.c @@ -22,6 +22,7 @@ #include #include +#include #include struct static_key paravirt_steal_enabled; @@ -38,7 +39,12 @@ struct pv_time_stolen_time_region { struct pvclock_vcpu_stolen_time __rcu *kaddr; }; +struct pv_lock_state_region { + struct pvlock_vcpu_state __rcu *kaddr; +}; + static DEFINE_PER_CPU(struct pv_time_stolen_time_region, stolen_time_region); +static DEFINE_PER_CPU(struct pv_lock_state_region, lock_state_region); static bool steal_acc = true; static int __init parse_no_stealacc(char *arg) @@ -178,3 +184,123 @@ int __init pv_time_init(void) return 0; } + +static bool native_vcpu_is_preempted(int cpu) +{ + return false; +} + +DEFINE_STATIC_CALL(pv_vcpu_is_preempted, native_vcpu_is_preempted); + +static bool para_vcpu_is_preempted(int cpu) +{ + struct pv_lock_state_region *reg; + __le64 preempted_le; + + reg = per_cpu_ptr(&lock_state_region, cpu); + if (!reg->kaddr) { + pr_warn_once("PV lock enabled but not configured for cpu %d\n", + cpu); + return false; + } + + preempted_le = le64_to_cpu(READ_ONCE(reg->kaddr->preempted)); + + return !!(preempted_le); +} + +static int pvlock_vcpu_state_dying_cpu(unsigned int cpu) +{ + struct pv_lock_state_region *reg; + + reg = this_cpu_ptr(&lock_state_region); + if (!reg->kaddr) + return 0; + + memunmap(reg->kaddr); + memset(reg, 0, sizeof(*reg)); + + return 0; +} + +static int init_pvlock_vcpu_state(unsigned int cpu) +{ + struct pv_lock_state_region *reg; + struct arm_smccc_res res; + + reg = this_cpu_ptr(&lock_state_region); + + arm_smccc_1_1_invoke(ARM_SMCCC_HV_PV_LOCK_PREEMPTED, &res); + + if (res.a0 == SMCCC_RET_NOT_SUPPORTED) { + pr_warn("Failed to init PV lock data structure\n"); + return -EINVAL; + } + + reg->kaddr = memremap(res.a0, + sizeof(struct pvlock_vcpu_state), + MEMREMAP_WB); + + if (!reg->kaddr) { + pr_warn("Failed to map PV lock data structure\n"); + return -ENOMEM; + } + + return 0; +} + +static int kvm_arm_init_pvlock(void) +{ + int ret; + + ret = cpuhp_setup_state(CPUHP_AP_ARM_KVM_PVLOCK_STARTING, + "hypervisor/arm/pvlock:starting", + init_pvlock_vcpu_state, + pvlock_vcpu_state_dying_cpu); + if (ret < 0) { + pr_warn("PV-lock init failed\n"); + return ret; + } + + return 0; +} + +static bool has_kvm_pvlock(void) +{ + struct arm_smccc_res res; + + /* To detect the presence of PV lock support we require SMCCC 1.1+ */ + if (arm_smccc_1_1_get_conduit() == SMCCC_CONDUIT_NONE) + return false; + + arm_smccc_1_1_invoke(ARM_SMCCC_ARCH_FEATURES_FUNC_ID, + ARM_SMCCC_HV_PV_LOCK_FEATURES, &res); + + if (res.a0 != SMCCC_RET_SUCCESS) + return false; + + arm_smccc_1_1_invoke(ARM_SMCCC_HV_PV_LOCK_FEATURES, + ARM_SMCCC_HV_PV_LOCK_PREEMPTED, &res); + + return (res.a0 == SMCCC_RET_SUCCESS); +} + +int __init pv_lock_init(void) +{ + int ret; + + if (is_hyp_mode_available()) + return 0; + + if (!has_kvm_pvlock()) + return 0; + + ret = kvm_arm_init_pvlock(); + if (ret) + return ret; + + static_call_update(pv_vcpu_is_preempted, para_vcpu_is_preempted); + pr_info("using PV-lock preempted\n"); + + return 0; +} \ No newline at end of file diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c index fea3223704b6..05ca07ac5800 100644 --- a/arch/arm64/kernel/setup.c +++ b/arch/arm64/kernel/setup.c @@ -42,6 +42,7 @@ #include #include #include +#include #include #include #include @@ -360,6 +361,8 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) smp_init_cpus(); smp_build_mpidr_hash(); + pv_lock_init(); + /* Init percpu seeds for random tags after cpus are set up. */ kasan_init_sw_tags(); diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index f61447913db9..c0ee11855c73 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -192,6 +192,7 @@ enum cpuhp_state { /* Must be the last timer callback */ CPUHP_AP_DUMMY_TIMER_STARTING, CPUHP_AP_ARM_XEN_STARTING, + CPUHP_AP_ARM_KVM_PVLOCK_STARTING, CPUHP_AP_ARM_CORESIGHT_STARTING, CPUHP_AP_ARM_CORESIGHT_CTI_STARTING, CPUHP_AP_ARM64_ISNDEP_STARTING,