From patchwork Mon Sep 25 17:34:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Mingwei Zhang X-Patchwork-Id: 144550 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:cae8:0:b0:403:3b70:6f57 with SMTP id r8csp1446819vqu; Mon, 25 Sep 2023 12:53:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEQXmGAfPb9JCVh6W2NhzH2ggYQ+GLZNAzRvzWQYPN11qX7aontloHl25x940w/gyr13cmY X-Received: by 2002:a05:6a20:918a:b0:149:9b2f:a79d with SMTP id v10-20020a056a20918a00b001499b2fa79dmr6740988pzd.6.1695671635301; Mon, 25 Sep 2023 12:53:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1695671635; cv=none; d=google.com; s=arc-20160816; b=tIqk81ffYpq4ZnAhpRSgpMmYCMKqkp1i19Njw4a9d4B1360VuFFrCp+LWTYPaNWeQm dzQBhkO4Z1gCjDLHevYfLjeanQzHnZwKVAD4LE9fqX6TC4RT2PRn4UlBwCoktMjQ3X9b kVICKKHZehTISXDBwlEMHmXiNnWODE2UnrWB/wyVO1D0dah0UNB+bbSAM3QNW+AtNtd5 xgsoAuXFnk62/k/l9iLCHOANfxFEmLwGTzCqGGu+HLC2Lxe7DiqEXbHn048szjdfk9oA uGTQuwYvFAI4e27HWNBESwN3Srikd2yGsXosoGDR0RRwZ+rP3JCOmFxdhW4OPs8oJHlc HzIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:cc:to:from:subject :message-id:mime-version:date:reply-to:dkim-signature; bh=+fRgk1GXWHwlF4dpNW6mrZozuiwZl1grW/0QNwyM28U=; fh=K+J2qYD/91bJS7zqTA6goMUf/yDX26DDWcmH+Ph084I=; b=TbAEalDdTRW/nCIhAftA5ooHj2l8iFdPkXxLDMYSJUw5W4ZD8fdVLV5/s5EtPmAc7l 7lIaYWFc7wq5uq2fKd3ea713AoB8OUzhf2fWuvqVQdUyqCYdZfwnMxwUwhA72p5hUOwO ieJKo0ck2eC7QJn1BPKhJxlT9d+hRDKqDo8T2o2NN9CjPdqk+/bcNoN61w6+W5+ib02C Abozi+Mh90eMUQlVNhMkuha8BCs1h56c2nHL40Y1v7fxzQ+6CHbU1Q4GE8MVucNwGSoA yyxSWGLSyOO1YmuEieF/HQI7PDoFZJ/J+W41Rfios1+h4oMBlrN3UNHfc34qHMS3YmnU fG1Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=qwvKM4ua; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from howler.vger.email (howler.vger.email. [2620:137:e000::3:4]) by mx.google.com with ESMTPS id u9-20020a056a00158900b00690bdd81cadsi11555575pfk.155.2023.09.25.12.53.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Sep 2023 12:53:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) client-ip=2620:137:e000::3:4; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20230601 header.b=qwvKM4ua; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:4 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by howler.vger.email (Postfix) with ESMTP id 8D7758334884; Mon, 25 Sep 2023 10:35:05 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at howler.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229595AbjIYRfE (ORCPT + 29 others); Mon, 25 Sep 2023 13:35:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60696 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229735AbjIYRfC (ORCPT ); Mon, 25 Sep 2023 13:35:02 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 510D2115 for ; Mon, 25 Sep 2023 10:34:55 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id d2e1a72fcca58-692bbab7a4fso5805110b3a.1 for ; Mon, 25 Sep 2023 10:34:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1695663295; x=1696268095; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id :mime-version:date:reply-to:from:to:cc:subject:date:message-id :reply-to; bh=+fRgk1GXWHwlF4dpNW6mrZozuiwZl1grW/0QNwyM28U=; b=qwvKM4uaM6fxKpqtc/e7PjgDQjZe2GOpjFcneywf1Q6rADXfv+DNMGmreKS9Gl4gMN cDwofVmXErcIi1pt6bRGOgGFPgURFVGlq2jgT4Z3nP4vJuTt+h3sIU8KbjuzsP54UpKM C1yVm9VHxkRPyDdJcKPpfccW+x4IbEUYHA0Xak6oCyB/P/rpUHxGCcUGDZMDMGpH+nQL eUsp/liTkgLriTGqDtOJWs6SkjVsoL/4FIVaR4+cxx3PX7MvLDYwRJFWZGOG8FDspC+9 yyBV+oz9exydhqMkvzMZU7mvSJNlMO8brZ87ellPfhtN4mOyPufYMFMlDhrJ2RtNJDj6 x4tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1695663295; x=1696268095; h=content-transfer-encoding:cc:to:from:subject:message-id :mime-version:date:reply-to:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=+fRgk1GXWHwlF4dpNW6mrZozuiwZl1grW/0QNwyM28U=; b=ciKZZvkMI8jb8WL5O61xrJi8nB6TvXUh/9SM8ZwMJRvCXXxFsYTO4PiIS3d347vwG8 kemtNjBgn2lzUpLH6uH0CDBajN/cTdO0j+9azud3W0DXv114NfSNKuNn/uBCUiY34duu tM9e0DN+UZz9rxvHkx15JPUahgw1OnddU+4gCE1dRKtX+yWBrZha2treTenMNhwo/FAW Od7RrfXoWYqaDJ26h4ICsgxtP+G9dSAC4WQqkf9nX9ywXppqjWxyEjqsf2nOwrdQdn4i NgMNl/tt/Ye6m+gkL2Ohh/L3F/rRPm+rIGnklxeIFp29NW+5Ub4vjVfr40EM6+aeI7Z/ vl3Q== X-Gm-Message-State: AOJu0YxTng7xjXn43vTnuuFVzcZPZBGA3nYf9DJWzoTAiM/q18lQJIG4 pDAqtsuvJMtuAQMUYJOgMQIU82qBj09B X-Received: from mizhang-super.c.googlers.com ([34.105.13.176]) (user=mizhang job=sendgmr) by 2002:a05:6a00:3a27:b0:690:29c0:ef51 with SMTP id fj39-20020a056a003a2700b0069029c0ef51mr6884pfb.1.1695663294680; Mon, 25 Sep 2023 10:34:54 -0700 (PDT) Reply-To: Mingwei Zhang Date: Mon, 25 Sep 2023 17:34:45 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.42.0.515.g380fc7ccd1-goog Message-ID: <20230925173448.3518223-1-mizhang@google.com> Subject: [PATCH 0/2] Fix the duplicate PMI injections in vPMU From: Mingwei Zhang To: Sean Christopherson , Paolo Bonzini Cc: "H. Peter Anvin" , kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Jim Mattson , Mingwei Zhang , Dapeng Mi , Like Xu , Roman Kagan , Kan Liang , Dapeng1 Mi X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (howler.vger.email [0.0.0.0]); Mon, 25 Sep 2023 10:35:05 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778040580427206646 X-GMAIL-MSGID: 1778040580427206646 When we do stress test on KVM vPMU using Intel vtune, we find the following warning kernel message in the guest VM: [ 1437.487320] Uhhuh. NMI received for unknown reason 20 on CPU 3. [ 1437.487330] Dazed and confused, but trying to continue The Problem =========== The above issue indicates that there are more NMIs injected than guest could recognize. After a month of investigation, we discovered that the bug happened due to minor glitches in two separate parts of the KVM: 1) KVM vPMU mistakenly fires a PMI due to emulated counter overflow even though the overflow has already been fired by the PMI handler on the host [1]. 2) KVM APIC allows multiple injections of PMI at one VM entry which violates Intel SDM. Both glitches contributes to extra injection of PMIs and thus confuses PMI handler in guest VM and causes the above warning messages. The Fixes ========= The patches disallow the multi-PMI injection fundamentally at APIC level. In addition, they also simplify the PMI injection process by removing irq_work and only use KVM_REQ_PMI. The Testing =========== With the series applied, we do not see the above warning messages when stress testing VM with Intel vtune. In addition, we add some kernel printing, all emulated counter overflow happens when hardware counter value is 0 and emulated counter value is 1 (prev_counter is -1). We never observed unexpected prev_counter values we saw in [2]. Note that this series does break the upstream kvm-unit-tests/pmu with the following error: FAIL: Intel: emulated instruction: instruction counter overflow FAIL: Intel: full-width writes: emulated instruction: instruction counter overflow This is a test bug and apply the following diff should fix the issue: We will post the above change soon. [1] commit 9cd803d496e7 ("KVM: x86: Update vPMCs when retiring instructions") [2] https://lore.kernel.org/all/CAL715WL9T8Ucnj_1AygwMgDjOJrttNZHRP9o-KUNfpx1aYZnog@mail.gmail.com/ Versioning ========== The series is in v1. We made some changes: - drop Dapeng's reviewed-by, since code changes. - applies fix up in kvm_apic_local_deliver(). [seanjc] - remove pmc->prev_counter. [seanjc] Previous version (v0) shown as follows: - [APIC patches v0]: https://lore.kernel.org/all/20230901185646.2823254-1-jmattson@google.com/ - [vPMU patch v0]: https://lore.kernel.org/all/ZQ4A4KaSyygKHDUI@google.com/ Jim Mattson (2): KVM: x86: Synthesize at most one PMI per VM-exit KVM: x86: Mask LVTPC when handling a PMI arch/x86/include/asm/kvm_host.h | 1 - arch/x86/kvm/lapic.c | 8 ++++++-- arch/x86/kvm/pmu.c | 27 +-------------------------- arch/x86/kvm/x86.c | 3 +++ 4 files changed, 10 insertions(+), 29 deletions(-) base-commit: 6de2ccc169683bf81feba163834dae7cdebdd826 Signed-off-by: Mingwei Zhang diff --git a/x86/pmu.c b/x86/pmu.c index 0def2869..667e6233 100644 --- a/x86/pmu.c +++ b/x86/pmu.c @@ -68,6 +68,7 @@ volatile uint64_t irq_received; static void cnt_overflow(isr_regs_t *regs) { »......irq_received++; +»......apic_write(APIC_LVTPC, apic_read(APIC_LVTPC) & ~APIC_LVT_MASKED); »......apic_write(APIC_EOI, 0); }