From patchwork Tue Feb 13 09:32:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oliver Upton X-Patchwork-Id: 200294 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp425009dyb; Tue, 13 Feb 2024 01:38:13 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCWkB1sWKPdwIFoAsLFAULi2d/XhzxyV+PSL9Edsg07qzGAG4TIanbunIYD3YLh74VSUTD3Vb6vVP6N+cyTCzIclHLK+6w== X-Google-Smtp-Source: AGHT+IFbHeQEIbW0f3NcAlgaNle1DuRuY9/IDHuF0ToRhqUXsDig05CaWfVUvOjlH3j/GHjvairj X-Received: by 2002:a05:6e02:1d03:b0:363:8877:81af with SMTP id i3-20020a056e021d0300b00363887781afmr12162785ila.16.1707817093078; Tue, 13 Feb 2024 01:38:13 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707817093; cv=pass; d=google.com; s=arc-20160816; b=pjJIDARX3ztIogntKjvmCawYjWJmBbyppNdcjfCW4D6lmDvGhnqC2SjuiZ7/yRrHBr AHwFciZkRnCW+fftKjhcpqopDpuwtlXGv4zixr5wfmIAB/IF+WweuoA5XTd9wl9WogMb DFLOrT98DFWXEi6L/VahMowe1J7lkAtCyJMcw4eQpsgFfZHBLmgmEb1z4TEpPWq7mh+K fjjzGazV+ULqe4DmCRLNl5mwOLsrqagksNOG2U9/YK+X9W5rJhRsU9iwDEdCArjGPrZa ow8W6P943nYj4q3HvTujpb84a74UPBcl6rb0fC+Af2aSYjyGZFEr4/r/ZKrI2yWfm5LH t0YQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=Y+lMHsCzsaKIbkTveh0NrGUV6sBBBNjPrb67/o/BF7Y=; fh=XB9sD1znGn7GjNS3YNO58k6Qa9Vu3SBKnfh+5exzlOo=; b=Hg2Q6yCGQfIQOCUP2E+H7l19n2TGkDufs76XYaKdm3evyAVxWPvQ5xKXc9ynoSwqIl Go8iKlRcW+ynzIDEcKmWO8BXwj9gvBkL55kR9NPlSd/LvcDvHCOVTNSGjyuPo8tzzblc H5yjKt1FQ98mjTUCFRSOuIdaVleR1AzyQoIiWJA3AUIttxs8knATeO1HyI3XqeGqvOQv 0r0u7QLDc121RSt2IgSeI8lPoQK0bq12A5gCt3bwYGv+EWSr6/DtLLXEqy/h+Ov/eL7A SVNVouInoc+8mHUnYuNTv1xKWQn06BfyhmZ9egFK2ON2uQS6frugfervLe2Ibr2Ccy2U JZIQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=e7t7TNz2; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-63217-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63217-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev X-Forwarded-Encrypted: i=2; AJvYcCV+b9Z+09QyNVGqGyK9ExZS39DpfJWmB6psW2d0S4ERtggLvqx5BAy9qYQ4Z1oGnsvJQSzg6I41539JM9Eh5zBxd+G6gA== Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id e16-20020a656890000000b005cdb499ac69si1726470pgt.53.2024.02.13.01.38.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Feb 2024 01:38:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-63217-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=e7t7TNz2; arc=pass (i=1 spf=pass spfdomain=linux.dev dkim=pass dkdomain=linux.dev dmarc=pass fromdomain=linux.dev); spf=pass (google.com: domain of linux-kernel+bounces-63217-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-63217-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id CD4DB280C29 for ; Tue, 13 Feb 2024 09:38:12 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 41C7B51C44; Tue, 13 Feb 2024 09:33:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="e7t7TNz2" Received: from out-189.mta1.migadu.com (out-189.mta1.migadu.com [95.215.58.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 54325224DE for ; Tue, 13 Feb 2024 09:33:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=95.215.58.189 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707816818; cv=none; b=PSddTWeVmfv74LU5N6OqO13mRpm18bLytD78/dLCoFAgPownBecjMpmwHGW/7O4SccJ4jm92Yas5LTUsGvkNr2MQKDhTPBNvurHXqhCmbldKmgzf6MhAB2BpVM5He0RXb+99Z4fWXXr22ET3W4hwfxYZFwVZJq8+Cq18evfMMrw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707816818; c=relaxed/simple; bh=u654twAWO/lOhw5G+s6F5ghriBPc9GHm8+B9VReBiXc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oQYULhLAIb8297QsAJF20LpprEK8jUm9ULpSj+62BArw7Ze9kFfZbD1iA0Kpj4I+KMJxGb0ORm4Z1LtvYzit8yqliIjo2V17LV7krx8Em6KSitBTGoZB1MmMDlz+LMg0iFNPnubVXBlOvENZf4BiBGdCAnq9x3Jfaewo/0A9Ou4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=e7t7TNz2; arc=none smtp.client-ip=95.215.58.189 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1707816814; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y+lMHsCzsaKIbkTveh0NrGUV6sBBBNjPrb67/o/BF7Y=; b=e7t7TNz2VtcfBlVd5Mjzyd6gJKVWSV0ZzpRHdbUml7x9/IMMNdWc32TLtbmJenkRNzWM9J gUzOaUy8lrCvUK+H25jbrbFVoeSONJHlCVz7Jfs5LveG9YomxxWmOfqm3lI6uD4l6WwE3s /z4mrZYrT42YMUsJUS1OXlW36Q421Ic= From: Oliver Upton To: kvmarm@lists.linux.dev Cc: kvm@vger.kernel.org, Marc Zyngier , James Morse , Suzuki K Poulose , Zenghui Yu , linux-kernel@vger.kernel.org, Oliver Upton Subject: [PATCH v2 12/23] KVM: arm64: vgic-its: Lazily allocate LPI translation cache Date: Tue, 13 Feb 2024 09:32:49 +0000 Message-ID: <20240213093250.3960069-13-oliver.upton@linux.dev> In-Reply-To: <20240213093250.3960069-1-oliver.upton@linux.dev> References: <20240213093250.3960069-1-oliver.upton@linux.dev> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790776015741081311 X-GMAIL-MSGID: 1790776015741081311 Reusing translation cache entries within a read-side critical section is fundamentally incompatible with an rculist. As such, we need to allocate a new entry to replace an eviction and free the removed entry afterwards. Take this as an opportunity to remove the eager allocation of translation cache entries altogether in favor of a lazy allocation model on cache miss. Signed-off-by: Oliver Upton --- arch/arm64/kvm/vgic/vgic-init.c | 3 -- arch/arm64/kvm/vgic/vgic-its.c | 96 +++++++++++++++------------------ include/kvm/arm_vgic.h | 1 + 3 files changed, 45 insertions(+), 55 deletions(-) diff --git a/arch/arm64/kvm/vgic/vgic-init.c b/arch/arm64/kvm/vgic/vgic-init.c index e25672d6e846..660d5ce3b610 100644 --- a/arch/arm64/kvm/vgic/vgic-init.c +++ b/arch/arm64/kvm/vgic/vgic-init.c @@ -305,9 +305,6 @@ int vgic_init(struct kvm *kvm) } } - if (vgic_has_its(kvm)) - vgic_lpi_translation_cache_init(kvm); - /* * If we have GICv4.1 enabled, unconditionnaly request enable the * v4 support so that we get HW-accelerated vSGIs. Otherwise, only diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index 50a9addebeed..a7ba20b57264 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -611,12 +611,20 @@ static struct vgic_irq *vgic_its_check_cache(struct kvm *kvm, phys_addr_t db, return irq; } +/* Default is 16 cached LPIs per vcpu */ +#define LPI_DEFAULT_PCPU_CACHE_SIZE 16 + +static unsigned int vgic_its_max_cache_size(struct kvm *kvm) +{ + return atomic_read(&kvm->online_vcpus) * LPI_DEFAULT_PCPU_CACHE_SIZE; +} + static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, u32 devid, u32 eventid, struct vgic_irq *irq) { + struct vgic_translation_cache_entry *new, *victim = NULL; struct vgic_dist *dist = &kvm->arch.vgic; - struct vgic_translation_cache_entry *cte; unsigned long flags; phys_addr_t db; @@ -624,10 +632,11 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, if (irq->hw) return; - raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); + new = kzalloc(sizeof(*new), GFP_KERNEL_ACCOUNT); + if (!new) + return; - if (unlikely(list_empty(&dist->lpi_translation_cache))) - goto out; + raw_spin_lock_irqsave(&dist->lpi_list_lock, flags); /* * We could have raced with another CPU caching the same @@ -635,22 +644,17 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, * already */ db = its->vgic_its_base + GITS_TRANSLATER; - if (__vgic_its_check_cache(dist, db, devid, eventid)) + if (__vgic_its_check_cache(dist, db, devid, eventid)) { + kfree(new); goto out; + } - /* Always reuse the last entry (LRU policy) */ - cte = list_last_entry(&dist->lpi_translation_cache, - typeof(*cte), entry); - - /* - * Caching the translation implies having an extra reference - * to the interrupt, so drop the potential reference on what - * was in the cache, and increment it on the new interrupt. - */ - if (cte->irq) { - KVM_VM_TRACE_EVENT(kvm, vgic_its_trans_cache_victim, cte->db, - cte->devid, cte->eventid, cte->irq->intid); - vgic_put_irq(kvm, cte->irq); + if (dist->lpi_cache_count >= vgic_its_max_cache_size(kvm)) { + /* Always reuse the last entry (LRU policy) */ + victim = list_last_entry(&dist->lpi_translation_cache, + typeof(*cte), entry); + list_del(&victim->entry); + dist->lpi_cache_count--; } /* @@ -660,16 +664,33 @@ static void vgic_its_cache_translation(struct kvm *kvm, struct vgic_its *its, lockdep_assert_held(&its->its_lock); vgic_get_irq_kref(irq); - cte->db = db; - cte->devid = devid; - cte->eventid = eventid; - cte->irq = irq; + new->db = db; + new->devid = devid; + new->eventid = eventid; + new->irq = irq; /* Move the new translation to the head of the list */ - list_move(&cte->entry, &dist->lpi_translation_cache); + list_add(&new->entry, &dist->lpi_translation_cache); + dist->lpi_cache_count++; out: raw_spin_unlock_irqrestore(&dist->lpi_list_lock, flags); + + if (!victim) + return; + + /* + * Caching the translation implies having an extra reference + * to the interrupt, so drop the potential reference on what + * was in the cache, and increment it on the new interrupt. + */ + if (victim->irq) { + KVM_VM_TRACE_EVENT(kvm, vgic_its_trans_cache_victim, victim->db, + victim->devid, victim->eventid, victim->irq->intid); + vgic_put_irq(kvm, victim->irq); + } + + kfree(victim); } void vgic_its_invalidate_cache(struct kvm *kvm) @@ -1917,33 +1938,6 @@ static int vgic_register_its_iodev(struct kvm *kvm, struct vgic_its *its, return ret; } -/* Default is 16 cached LPIs per vcpu */ -#define LPI_DEFAULT_PCPU_CACHE_SIZE 16 - -void vgic_lpi_translation_cache_init(struct kvm *kvm) -{ - struct vgic_dist *dist = &kvm->arch.vgic; - unsigned int sz; - int i; - - if (!list_empty(&dist->lpi_translation_cache)) - return; - - sz = atomic_read(&kvm->online_vcpus) * LPI_DEFAULT_PCPU_CACHE_SIZE; - - for (i = 0; i < sz; i++) { - struct vgic_translation_cache_entry *cte; - - /* An allocation failure is not fatal */ - cte = kzalloc(sizeof(*cte), GFP_KERNEL_ACCOUNT); - if (WARN_ON(!cte)) - break; - - INIT_LIST_HEAD(&cte->entry); - list_add(&cte->entry, &dist->lpi_translation_cache); - } -} - void vgic_lpi_translation_cache_destroy(struct kvm *kvm) { struct vgic_dist *dist = &kvm->arch.vgic; @@ -1990,8 +1984,6 @@ static int vgic_its_create(struct kvm_device *dev, u32 type) kfree(its); return ret; } - - vgic_lpi_translation_cache_init(dev->kvm); } mutex_init(&its->its_lock); diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 47035946648e..431d05c01a53 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -281,6 +281,7 @@ struct vgic_dist { /* LPI translation cache */ struct list_head lpi_translation_cache; + unsigned int lpi_cache_count; /* used by vgic-debug */ struct vgic_state_iter *iter;