From patchwork Wed Mar 22 20:43:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dmytro Maluka X-Patchwork-Id: 7059 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:604a:0:0:0:0:0 with SMTP id j10csp2570451wrt; Wed, 22 Mar 2023 14:06:20 -0700 (PDT) X-Google-Smtp-Source: AK7set+EcWhm2kPduaDZF2Fxaxz5gEcdrDBamgwc3GEEUCUGC0CB37p4zjIZUzsNeeUXm6jW6FPe X-Received: by 2002:aa7:96cc:0:b0:5a8:abd2:2beb with SMTP id h12-20020aa796cc000000b005a8abd22bebmr4332576pfq.30.1679519180477; Wed, 22 Mar 2023 14:06:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1679519180; cv=none; d=google.com; s=arc-20160816; b=gU5Zj7D3v/XkB98GI2WLS6OaJnAqaYqOu/4ioKHsirFbQYRPrblWGJtfL4z/kazvLm oyU3F6poix6F5cWUQqUpD216TYqSqIIxDAAZiRnbReWfHP6887ZSQxjNWFZZRGqp7U2W 7gpfslwN3+eFQVKY7F80dABRimoA/Bgh1PtkzOxKZaz7lCDSpWoCQGMP/uMXeeFNW/11 n+ZUii3JnZalGuaPh1ICFl0cYRvby6qe5eUuRc7vKdtYB8bwee1ePxtE/JeEEC5bumJe ZQCtiSvfODvmnEaGpaGiE//h0EEqoNRSluOedTm4NzLWXOhuyVfG8E+0tG9cURc2k/3F k6mA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=HRJLzQx1UmsMap1tfRUZIxUpggNdDJZoMHruY0d4CdQ=; b=gCQwJLXxyO6h6pKNQgUcQFBkPOYMwpenLQ09n8DO37ssACf74MIm2nPzwY53c8qmQo PjMNIMcztzIBGW2fIhtUf3dbdIVJoRGM3CbKord7UchOM/6Lf87+rBGMthxE8VKjo2Qo YcC2GrUc+KvUMgTCptxZII91TZ6cNqjkHVZ/yAv6bMlPKfWyMwv9f2LDF9/qkG0f1lR2 eo449jcAgUob49NZk0/mRoFG+bY7rhlHKbYjJvWPF6kiR4z9/0VjNrGdpR+ik4EvaSwh H9HiJZnUGvbHvlE6fPodIc1T7wEkdimjM8VhPbB6329Bx5/ApQb/9VQp/xLh6a/aQEVZ mCUQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@semihalf.com header.s=google header.b=IqONvWg8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=semihalf.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m20-20020a056a00165400b006263931f58fsi14033350pfc.348.2023.03.22.14.06.05; Wed, 22 Mar 2023 14:06:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@semihalf.com header.s=google header.b=IqONvWg8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=semihalf.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231743AbjCVUsU (ORCPT + 99 others); Wed, 22 Mar 2023 16:48:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230496AbjCVUr6 (ORCPT ); Wed, 22 Mar 2023 16:47:58 -0400 Received: from mail-lj1-x229.google.com (mail-lj1-x229.google.com [IPv6:2a00:1450:4864:20::229]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF8B321966 for ; Wed, 22 Mar 2023 13:44:38 -0700 (PDT) Received: by mail-lj1-x229.google.com with SMTP id z42so20284919ljq.13 for ; Wed, 22 Mar 2023 13:44:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf.com; s=google; t=1679517874; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=HRJLzQx1UmsMap1tfRUZIxUpggNdDJZoMHruY0d4CdQ=; b=IqONvWg8n6amjpEoKFJGMl580Jn17yQZsKw5Ijc6YSReCyctQN9b7yzMeGUpiaBxdZ jH06dcHuNJF27mAMQAhHoYkQC0WvtDZDdpo5zTwLRG1qf3ZPIdFg3yryCaiggMBVPyFr xe3CD6EoWyKcg/2D7yLp3OVz+CH4D8OaxYe5fzyyQSl9D3IoloPh7j29yGiEZEQOaGeP rqx4FooeM3YUbdM1FuzHGmnhAvQtFca+2ya9XIJUmJ84HHh747u/TWgxVE7Dj1j6sgWC qViW+Bnbu91lDTGXUxjvHnqthsZ5LmfpIR1d/dBzY+JQPvVhhtIUFUqtK0fle3/pa/OI +dQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679517874; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=HRJLzQx1UmsMap1tfRUZIxUpggNdDJZoMHruY0d4CdQ=; b=un8fumbCOdiRF7vSaxMLQ4slSUFHifwFesxKkFKTPwAcyjPjzT6gTaaYozBexUL+b+ I0d1mt5tzBZXa2eXMA/EffJHMCHVYcZe0goQVlGqHiCVixThLrPXAtW1BiQZ3qhAI4w+ lfeNjJovRQIOuVNRx7yp5CGD5oH9HCm5QmvjZqwuaj6einNmQ4PmEtrupATMVI+h7RVm kd9EAAZjigTJ6f0ijKk/eRneufkvdYRpG12oqyEVDSSvJz03MhfAFeCO2uLZvXWCQ15K w7pGpnHtRYAPmL6VyYFuIVgmrizftNwvRLFYmKb0dO5jMR2WVZs13VCJAvJlMNxMuqDf Q1lw== X-Gm-Message-State: AO0yUKU9RNwxhBKFTqEtGmAhg17B4dF0Fz62BPFiF1jlAdYBOgVAFlfP LUzokc2PRgMah40ejVbMnzoYow== X-Received: by 2002:a05:651c:1047:b0:29b:d220:bbb7 with SMTP id x7-20020a05651c104700b0029bd220bbb7mr2466039ljm.47.1679517874566; Wed, 22 Mar 2023 13:44:34 -0700 (PDT) Received: from dmaluka.office.semihalf.net ([83.142.187.84]) by smtp.gmail.com with ESMTPSA id d16-20020a2eb050000000b0029aa0b6b41asm2585686ljl.115.2023.03.22.13.44.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Mar 2023 13:44:33 -0700 (PDT) From: Dmytro Maluka To: Sean Christopherson , Paolo Bonzini , kvm@vger.kernel.org Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , linux-kernel@vger.kernel.org, Marc Zyngier , Eric Auger , Alex Williamson , Rong L Liu , Zhenyu Wang , Tomasz Nowicki , Grzegorz Jaszczyk , upstream@semihalf.com, Dmitry Torokhov , "Dong, Eddie" , Dmytro Maluka Subject: [PATCH v4 0/2] KVM: x86/ioapic: Fix oneshot interrupts forwarding Date: Wed, 22 Mar 2023 21:43:42 +0100 Message-Id: <20230322204344.50138-1-dmy@semihalf.com> X-Mailer: git-send-email 2.40.0.348.gf938b09366-goog MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761103504273763667?= X-GMAIL-MSGID: =?utf-8?q?1761103504273763667?= From: Dmytro Maluka KVM irqfd based emulation of level-triggered interrupts doesn't work quite correctly in some cases, particularly in the case of interrupts that are handled in a Linux guest as oneshot interrupts (IRQF_ONESHOT). Such an interrupt is acked to the device in its threaded irq handler, i.e. later than it is acked to the interrupt controller (EOI at the end of hardirq), not earlier. Linux keeps such interrupt masked until its threaded handler finishes, to prevent the EOI from re-asserting an unacknowledged interrupt. However, with KVM + vfio (or whatever is listening on the resamplefd) we always notify resamplefd at the EOI, so vfio prematurely unmasks the host physical IRQ, thus a new physical interrupt is fired in the host. This extra interrupt in the host is not a problem per se. The problem is that it is unconditionally queued for injection into the guest, so the guest sees an extra bogus interrupt. [*] There are observed at least 2 user-visible issues caused by those extra erroneous interrupts for a oneshot irq in the guest: 1. System suspend aborted due to a pending wakeup interrupt from ChromeOS EC (drivers/platform/chrome/cros_ec.c). 2. Annoying "invalid report id data" errors from ELAN0000 touchpad (drivers/input/mouse/elan_i2c_core.c), flooding the guest dmesg every time the touchpad is touched. The core issue here is that by the time when the guest unmasks the IRQ, the physical IRQ line is no longer asserted (since the guest has acked the interrupt to the device in the meantime), yet we unconditionally inject the interrupt queued into the guest by the previous resampling. So to fix the issue, we need a way to detect that the IRQ is no longer pending, and cancel the queued interrupt in this case. With IOAPIC we are not able to probe the physical IRQ line state directly (at least not if the underlying physical interrupt controller is an IOAPIC too), so in this patch series we use irqfd resampler for that. Namely, instead of injecting the queued interrupt, we just notify the resampler that this interrupt is done. If the IRQ line is actually already deasserted, we are done. If it is still asserted, a new interrupt will be shortly triggered through irqfd and injected into the guest. In the case if there is no irqfd resampler registered for this IRQ, we cannot fix the issue, so we keep the existing behavior: immediately unconditionally inject the queued interrupt. This patch series fixes the issue for x86 IOAPIC only. In the long run, we can fix it for other irqchips and other architectures too, possibly taking advantage of reading the physical state of the IRQ line, which is possible with some other irqchips (e.g. with arm64 GIC, maybe even with the legacy x86 PIC). [*] In this description we assume that the interrupt is a physical host interrupt forwarded to the guest e.g. by vfio. Potentially the same issue may occur also with a purely virtual interrupt from an emulated device, e.g. if the guest handles this interrupt, again, as a oneshot interrupt. v4: - Cosmetic coding style changes as suggested by Sean. v3: - Completely reworked: instead of postponing resamplefd notify until unmask (to avoid extra interrupts in the host), resample the pending status at unmask to avoid erroneous propagation of those extra interrupts to the guest. Thanks to Marc Zyngier for helping to identify the core issue, which resulted in a simpler and probably more sensible implementation (even though Marc's concern about presenting inaccurate pending status to the guest is a non-issue in the case of IOAPIC, since IOAPIC doesn't present this information anyway). v2: - Fixed compilation failure on non-x86: mask_notifier_list moved from x86 "struct kvm_arch" to generic "struct kvm". - kvm_fire_mask_notifiers() also moved from x86 to generic code, even though it is not called on other architectures for now. - Instead of kvm_irq_is_masked() implemented kvm_register_and_fire_irq_mask_notifier() to fix potential race when reading the initial IRQ mask state. - Renamed for clarity: - irqfd_resampler_mask() -> irqfd_resampler_mask_notify() - kvm_irq_has_notifier() -> kvm_irq_has_ack_notifier() - resampler->notifier -> resampler->ack_notifier - Reorganized code in irqfd_resampler_ack() and irqfd_resampler_mask_notify() to make it easier to follow. - Don't follow unwanted "return type on separate line" style for irqfd_resampler_mask_notify(). Dmytro Maluka (2): KVM: irqfd: Make resampler_list an RCU list KVM: x86/ioapic: Resample the pending state of an IRQ when unmasking arch/x86/kvm/ioapic.c | 36 +++++++++++++++++++++++++--- include/linux/kvm_host.h | 11 +++++++++ include/linux/kvm_irqfd.h | 2 +- virt/kvm/eventfd.c | 49 ++++++++++++++++++++++++++++++++------- 4 files changed, 86 insertions(+), 12 deletions(-)