From patchwork Wed Nov 30 23:09:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sean Christopherson X-Patchwork-Id: 28080 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp1205404wrr; Wed, 30 Nov 2022 15:30:25 -0800 (PST) X-Google-Smtp-Source: AA0mqf5q6V5310S2gw1WlZFpKQJ52caITG3YAtUvXjWImNs+RC98bU4lzgpmFLCKbSbP6uQu8C7b X-Received: by 2002:a17:90b:3d90:b0:212:de1c:a007 with SMTP id pq16-20020a17090b3d9000b00212de1ca007mr75343062pjb.30.1669851024657; Wed, 30 Nov 2022 15:30:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669851024; cv=none; d=google.com; s=arc-20160816; b=W3N1K3vqTulUY0XKjJChIiQXfwm0W+fJQvTWPNVaxoy+3rDi+PqKRLzWUNwX1+YzSP wq4Bw2jGAlRVKkInQG8peCzj6w92rJpQ0VehALt+lXZsYJk/B0GRYySZbGy6ZZvfoI+v R1oxuIgrG8uvSCMxfF9Tn0PNmCDgfDrq7p35YTqr7UXL0S8BG2qZD2fpDvgTrShjWzJd PdBlrE4hdu9KMnAuU4mY5jY6q/DwLPdM1BZdb3L77bOW27Jmp8Sgb4aKDT90tyOZDtFb GNpHdjLYkydP2MCoupaj/nhZDJDKVYRDJV3ZV/Y6Cp8DTJRXK/fJQqRNcRHFWbik4OpC 5PXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:reply-to:dkim-signature; bh=f1ges7v12pOnWRvBGxVUQ1iVkaen+bCiwnszqRspz5I=; b=YQhVX53B5/B1Hdje1G0e4rC2LZ48iHA6tQbZ9nhtAslzLEeQ2tK0mjRQWzFt9lVIGT GqqM1iRvPZcalmoStnHpdZvvxX25iprrA8nRLMPn32BY7GKw6AMfF0U8c/0xaAyRQExp SAfR2LAwaz5UrdXAsEtE0BTZDHoWkxg/jSQPKGRoQX3mfShbjgbFO+0/EyXB8qtI27AJ aAh9fEByXgfDXavEHVFI2nBAvOc0gNSklPrDYUjbkzgu1Pue7zRufjwCtuvVJWCXqtte KMaBtlTNcq6hgpByijdfHUIL0rr8Z54q8WMLmLAJVMSS6z2jp7w40ve7Gn/ISY6uRVVB ZejA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=gLeoYszh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f10-20020a636a0a000000b004611cfaca6asi2497163pgc.381.2022.11.30.15.30.11; Wed, 30 Nov 2022 15:30:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=gLeoYszh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230196AbiK3XRx (ORCPT + 99 others); Wed, 30 Nov 2022 18:17:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230229AbiK3XQi (ORCPT ); Wed, 30 Nov 2022 18:16:38 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6119B77435 for ; Wed, 30 Nov 2022 15:11:49 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-3c2837e751fso114282517b3.15 for ; Wed, 30 Nov 2022 15:11:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=f1ges7v12pOnWRvBGxVUQ1iVkaen+bCiwnszqRspz5I=; b=gLeoYszhyNvT3b/sb5uYSwOA01VKPMWAXDjqkZYRdtKGfM6Ek/arh5xQWkT5U5yzoB 5CD5ESLZv4WU1n/4uB06OdwXMxCseYp9IaQQLNVjv3WwPdnRofjslDSFaOM0xGYsKK+C tKX3chEelyUD8sc6Y74rDxDyt3aY3CP6z+j1KZWlrcIRv9diHP2iozhFk/Mchi+TTVgv fEeeQaF+pMBa6yom6DRZvH7p+HjJ0Tg0Rdf0l2eI9+N1rfc39tpiDCp65RGs1ugAXHHv MGoQYmmnBoVyTjRNQhOEvAGKhAuRN2t7CBn2+Bl7DY/FSOnm9dbPq0iYKWwQsZDGX6rY /NUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=f1ges7v12pOnWRvBGxVUQ1iVkaen+bCiwnszqRspz5I=; b=CxnbwEPzCaT6hr1Rd5ZU+RwzIMizV/iAJypN9wGLoXomF5PSn/PJsnDPKnD5BWcIK7 mGTL1iU4HtNpbUJuQn07sCXDrCbvjIPa0xuo+1dhOKRuSQC6TOvvKbXQ8wj6PkCZRbQP fsoSKvbRe6FveN3hOeClHV1K+O1gstr/2Ztk+7SpicNtvXsMZTvunREIOBDnd1bkWfD0 4hO4UALQiczLjfrrZ8vsk4bsPyXJbBZKUlRVawQJELAfKvgGZqzISvXKgcPw58r2Wx1S pTNfKeUFzU68FWXJbJZmTMJ/fhhmiQtcAwO9DSTpY1OKjR/NPJpmZSeMLZCRoxtpDOSV oyLw== X-Gm-Message-State: ANoB5plaDWOsz1mYzW8K09AoIHTPLFWD6b/mCvX9jYk3Is48Tpkox9IO WpSkOffwlNNn+9GYYC6lj/d+ohtcV94= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:a0cc:0:b0:6f7:4be9:89f8 with SMTP id i12-20020a25a0cc000000b006f74be989f8mr14935825ybm.345.1669849861304; Wed, 30 Nov 2022 15:11:01 -0800 (PST) Reply-To: Sean Christopherson Date: Wed, 30 Nov 2022 23:09:32 +0000 In-Reply-To: <20221130230934.1014142-1-seanjc@google.com> Mime-Version: 1.0 References: <20221130230934.1014142-1-seanjc@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog Message-ID: <20221130230934.1014142-49-seanjc@google.com> Subject: [PATCH v2 48/50] KVM: Register syscore (suspend/resume) ops early in kvm_init() From: Sean Christopherson To: Paolo Bonzini , Marc Zyngier , Huacai Chen , Aleksandar Markovic , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Claudio Imbrenda , Matthew Rosato , Eric Farman , Sean Christopherson , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: James Morse , Alexandru Elisei , Suzuki K Poulose , Oliver Upton , Atish Patra , David Hildenbrand , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, Yuan Yao , Cornelia Huck , Isaku Yamahata , " =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= " , Fabiano Rosas , Michael Ellerman , Kai Huang , Chao Gao , Thomas Gleixner X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750965708241695403?= X-GMAIL-MSGID: =?utf-8?q?1750965708241695403?= Register the suspend/resume notifier hooks at the same time KVM registers its reboot notifier so that all the code in kvm_init() that deals with enabling/disabling hardware is bundled together. Opportunstically move KVM's implementations to reside near the reboot notifier code for the same reason. Bunching the code together will allow architectures to opt out of KVM's generic hardware enable/disable logic with minimal #ifdeffery. Signed-off-by: Sean Christopherson --- virt/kvm/kvm_main.c | 68 ++++++++++++++++++++++----------------------- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 674a9dab5411..c12db3839114 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -5154,6 +5154,38 @@ static struct notifier_block kvm_reboot_notifier = { .priority = 0, }; +static int kvm_suspend(void) +{ + /* + * Secondary CPUs and CPU hotplug are disabled across the suspend/resume + * callbacks, i.e. no need to acquire kvm_lock to ensure the usage count + * is stable. Assert that kvm_lock is not held to ensure the system + * isn't suspended while KVM is enabling hardware. Hardware enabling + * can be preempted, but the task cannot be frozen until it has dropped + * all locks (userspace tasks are frozen via a fake signal). + */ + lockdep_assert_not_held(&kvm_lock); + lockdep_assert_irqs_disabled(); + + if (kvm_usage_count) + hardware_disable_nolock(NULL); + return 0; +} + +static void kvm_resume(void) +{ + lockdep_assert_not_held(&kvm_lock); + lockdep_assert_irqs_disabled(); + + if (kvm_usage_count) + WARN_ON_ONCE(__hardware_enable_nolock()); +} + +static struct syscore_ops kvm_syscore_ops = { + .suspend = kvm_suspend, + .resume = kvm_resume, +}; + static void kvm_io_bus_destroy(struct kvm_io_bus *bus) { int i; @@ -5732,38 +5764,6 @@ static void kvm_init_debug(void) } } -static int kvm_suspend(void) -{ - /* - * Secondary CPUs and CPU hotplug are disabled across the suspend/resume - * callbacks, i.e. no need to acquire kvm_lock to ensure the usage count - * is stable. Assert that kvm_lock is not held to ensure the system - * isn't suspended while KVM is enabling hardware. Hardware enabling - * can be preempted, but the task cannot be frozen until it has dropped - * all locks (userspace tasks are frozen via a fake signal). - */ - lockdep_assert_not_held(&kvm_lock); - lockdep_assert_irqs_disabled(); - - if (kvm_usage_count) - hardware_disable_nolock(NULL); - return 0; -} - -static void kvm_resume(void) -{ - lockdep_assert_not_held(&kvm_lock); - lockdep_assert_irqs_disabled(); - - if (kvm_usage_count) - WARN_ON_ONCE(__hardware_enable_nolock()); -} - -static struct syscore_ops kvm_syscore_ops = { - .suspend = kvm_suspend, - .resume = kvm_resume, -}; - static inline struct kvm_vcpu *preempt_notifier_to_vcpu(struct preempt_notifier *pn) { @@ -5879,6 +5879,7 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module) return r; register_reboot_notifier(&kvm_reboot_notifier); + register_syscore_ops(&kvm_syscore_ops); /* A kmem cache lets us meet the alignment requirements of fx_save. */ if (!vcpu_align) @@ -5913,8 +5914,6 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module) kvm_chardev_ops.owner = module; - register_syscore_ops(&kvm_syscore_ops); - kvm_preempt_ops.sched_in = kvm_sched_in; kvm_preempt_ops.sched_out = kvm_sched_out; @@ -5948,6 +5947,7 @@ int kvm_init(unsigned vcpu_size, unsigned vcpu_align, struct module *module) free_cpumask_var(per_cpu(cpu_kick_mask, cpu)); kmem_cache_destroy(kvm_vcpu_cache); out_free_3: + unregister_syscore_ops(&kvm_syscore_ops); unregister_reboot_notifier(&kvm_reboot_notifier); cpuhp_remove_state_nocalls(CPUHP_AP_KVM_ONLINE); return r;