From patchwork Fri Jun 2 17:01:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Colton Lewis X-Patchwork-Id: 102656 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp1174754vqr; Fri, 2 Jun 2023 10:07:51 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6ENBNToKsFx7q0BsOUSmALep16qTBnOumZYnPdreSiCytki2VkgBzR3S6KeWK63VhPioTy X-Received: by 2002:a05:6a20:440e:b0:103:ef39:a832 with SMTP id ce14-20020a056a20440e00b00103ef39a832mr10654414pzb.23.1685725670747; Fri, 02 Jun 2023 10:07:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685725670; cv=none; d=google.com; s=arc-20160816; b=JfuEyE3Fank73QgdOC5AJsHORZ9N/omigoW8ZAdRRLKiz+g0Fv74m/oWHR5tY+tWTM qcogM4NjAFpOf7re2ZxztFweB/3iCA3bK8L8mJejYBw8fH/+/pBzBQ2lfgwNyGBECkdT gzFwiu8ZVCE4Op34BVaPaRAJBW63vv65DDvLUEorx6bKBBeXD6Sr/lOuLK7KuDtMU9Ml BhSMyu/q3pJiIgDZwWqCmW7MxOit1RDHJRmvmGLWLdwWmjbWQEU9JBEoT504XA+MWVxV jFDLLW5cPCT+sxc+nbv2WFnTeQvECy87ns6FnHU2IaSBw4Md6AxtUjYOXpi9H8HuyMA4 7uXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=DZrtPESLe1mA5daPGrUc+60GXQlGnhOLsAUJu36d7rQ=; b=naJNaIip4+AN+v2IKLmE75V1AglFJKm0WepnMfSROJ3qkbXY7TUqf7udIXTnE9Kuo4 IupJRqVCYAHTn/5p0FvBhxfkCiOP2AtiG5Wo/hnOsKMfjr0h7z2OPsUlMPCCvXzHk0Tc sXuLLQgkJOjFFddpKf2Ypdf109au/oGdsd0CTOaLEnaLonaIyTGAQuJr1nIbDeqU6NYr FzSwpa68cdGMoXcHR1d4aHuB9Y+jg9n+ChLOS34vsxNiQTWCGmfRpLTYT4z4hV8w68L0 lfH/+XZ5mZZslJ3NlcwcdQH7oywCt2LVpYYOrMFudmyYOmQoZSKJnu57vNQgauUOD4H+ w9Sw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=Qq3RLBS2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b27-20020a63931b000000b0052868a865d4si1246447pge.553.2023.06.02.10.07.36; Fri, 02 Jun 2023 10:07:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=Qq3RLBS2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236811AbjFBRC6 (ORCPT + 99 others); Fri, 2 Jun 2023 13:02:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236913AbjFBRCx (ORCPT ); Fri, 2 Jun 2023 13:02:53 -0400 Received: from mail-io1-xd4a.google.com (mail-io1-xd4a.google.com [IPv6:2607:f8b0:4864:20::d4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 492BB1AD for ; Fri, 2 Jun 2023 10:02:52 -0700 (PDT) Received: by mail-io1-xd4a.google.com with SMTP id ca18e2360f4ac-76c6c1b16d2so117570439f.1 for ; Fri, 02 Jun 2023 10:02:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685725371; x=1688317371; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DZrtPESLe1mA5daPGrUc+60GXQlGnhOLsAUJu36d7rQ=; b=Qq3RLBS2mk0SnALShNCtm+TQlg+XH/AHekNH7/7QlGAv3Kwq/6fPi6+CvfRavpIs4G RjyTeXe+0RGaZAufFJ8EvFNFa8pOjeTTjjb8sD9rItu1zRExurnNtBOtX6w1QJedD+7I afR/Bn/wNwB2NO7fNerz8ES6XgY4MyN711P03otg/ynXmiG0KHjiA/NCjGChETeqa8Sq PPIW2euhDqJ/s3oFviuWT4EcIxz18d4QuSIOimUwO4KwNQ9xdjmAWN3FZO+WhB+n8/fm Z+b1LzGrVLf99x0xH1gni5y4fL0Jf+jIx8h2sro9EeSCJDdt5AbgU5s/l97G1yjsUE5q 7IyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685725371; x=1688317371; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DZrtPESLe1mA5daPGrUc+60GXQlGnhOLsAUJu36d7rQ=; b=b8l+q3MjEoZExbEeHwSh4MJyHmjBDjz0GWjX10WqsCV4LIRcUo6uRu6G9qyCyn7wGv X80Mnj+hKLHfrv9UwUYyjoUrXMufInvgEkoJ6aAeMR7IRkah1WBzKP87xBLKfhOtOmT1 rRgPwf03p65KKAYWUn+WRn13Ip22xY6q3dlzA3LOJmQEOblI6kZ+c6+ayn4JMYVItxcK gmeZMCYQteBweac+ylqdYQEbCk0MPLcNLOULeOxP7qmcBz9NxOc7stR4SA2bVhjoHYum +1Qd/rZn0sBQwbf2EYpDU2QIHQMfrmGdny5JdVv7eTsuGnR0oKBRPREph6yB1Pdbw4nf zGFg== X-Gm-Message-State: AC+VfDxuJ7oJ/kVGHxi7tsIVHVrwVU1YSgc0AkeHusVL4zmFDz97m1oP L6MGj4J3RzOE0I9QYOShgocO6C6/oAdQZhqSRg== X-Received: from coltonlewis-kvm.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:14ce]) (user=coltonlewis job=sendgmr) by 2002:a6b:5904:0:b0:766:6741:8849 with SMTP id n4-20020a6b5904000000b0076667418849mr1203200iob.3.1685725371731; Fri, 02 Jun 2023 10:02:51 -0700 (PDT) Date: Fri, 2 Jun 2023 17:01:45 +0000 In-Reply-To: <20230602170147.1541355-1-coltonlewis@google.com> Mime-Version: 1.0 References: <20230602170147.1541355-1-coltonlewis@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Message-ID: <20230602170147.1541355-2-coltonlewis@google.com> Subject: [PATCH 1/3] arm64: Add a capability for FEAT_BBM level 2 From: Colton Lewis To: kvm@vger.kernel.org Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, Ricardo Koller X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767611480641548779?= X-GMAIL-MSGID: =?utf-8?q?1767611480641548779?= From: Ricardo Koller Add a new capability to detect "Stage-2 Translation table break-before-make" (FEAT_BBM) level 2. Signed-off-by: Ricardo Koller --- arch/arm64/kernel/cpufeature.c | 11 +++++++++++ arch/arm64/tools/cpucaps | 1 + 2 files changed, 12 insertions(+) -- 2.41.0.rc0.172.g3f132b7071-goog diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index c331c49a7d19c..c538060f7f66b 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2455,6 +2455,17 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .min_field_value = 1, .matches = has_cpuid_feature, }, + { + .desc = "Stage-2 Translation table break-before-make level 2", + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .capability = ARM64_HAS_STAGE2_BBM2, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_EL1_BBM_SHIFT, + .field_width = 4, + .min_field_value = 2, + .matches = has_cpuid_feature, + }, { .desc = "TLB range maintenance instructions", .capability = ARM64_HAS_TLB_RANGE, diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 40ba95472594d..010aca1892642 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -41,6 +41,7 @@ HAS_PAN HAS_RAS_EXTN HAS_RNG HAS_SB +HAS_STAGE2_BBM2 HAS_STAGE2_FWB HAS_TIDCP1 HAS_TLB_RANGE From patchwork Fri Jun 2 17:01:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Colton Lewis X-Patchwork-Id: 102657 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp1175055vqr; Fri, 2 Jun 2023 10:08:17 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6T87BVt5rLcEHVe019sT/sJaLVuzpFMzGaQqIOX0dA5IuSrCBHmkeE2ljxOvZQVw3F8g+r X-Received: by 2002:a17:90a:19c4:b0:256:9b75:4bf0 with SMTP id 4-20020a17090a19c400b002569b754bf0mr545834pjj.17.1685725696769; Fri, 02 Jun 2023 10:08:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685725696; cv=none; d=google.com; s=arc-20160816; b=qcvpDU75WHigNllWYMb+qzpoBY4+fzbbE2unv3AceeFHQsYPMPGCCgWBECbBfWmdqW rmN/Kb7KkrW1V6E4ILRmEvP340vzjdTV0RIbJUyZLsMtpRU4HAj9fVxELeT0q2EBa7+d obXCi920BW/piomfN1ue3dQolM+mNrRc3ZDpJEB04B/KrJcqJT2mlm1NPCkhKsNkHvWz W+we/hu+HGwngsF1TxyGBl0sqNpdV7Kg7sZz6Ft8i1uQgwKAXjy2bC6sYRwhOK9aiSdh 1Zitcnvk3bCbWEcYa3RlsKIqsu9qVReLTpnUNIzKU4ztZedX0tv+XOgck3lXRSOOe5O2 A7BA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=jc0HXGzYvsWKFFqMwew1siGd4KbapSh3HWhZ39XsN18=; b=xsRjMwJUNQlLwhb7l32hOtr9ataVZCWP1st18XBDm3Vr9myGO6K5HpRt9KkER+EjYz /yFMGAIMa9kv2es7wualqP29zlZ188GyoG0JZujt2DgUwLRV9txLCYyD/W/5WFdRIuXZ 4ZhRQqyP9VcrMzRFwdqV8Ye9wXExyLHu8J8R1JZsnVvT/J4Z5/qRcFvb2yTu6Ngo3A4M xoA78Nb2/8Jz5Ul0TG0fizZeq7AKAzRA/3+Msdahp63Anuq22xFZZUBI7YOWGFCehTib bFHg0RMftVwmm3O21uDAJuWqBM9S3upHeyFc1JatiT471Gw6p3eaCpTN3NCJznL0Vxvj UXxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=j+tKfi2k; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y30-20020a63b51e000000b0051fb9c77447si1209928pge.477.2023.06.02.10.08.03; Fri, 02 Jun 2023 10:08:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=j+tKfi2k; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236945AbjFBRDB (ORCPT + 99 others); Fri, 2 Jun 2023 13:03:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236918AbjFBRCy (ORCPT ); Fri, 2 Jun 2023 13:02:54 -0400 Received: from mail-il1-x149.google.com (mail-il1-x149.google.com [IPv6:2607:f8b0:4864:20::149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2BDD01A4 for ; Fri, 2 Jun 2023 10:02:53 -0700 (PDT) Received: by mail-il1-x149.google.com with SMTP id e9e14a558f8ab-33b59d59193so20406405ab.0 for ; Fri, 02 Jun 2023 10:02:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685725372; x=1688317372; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=jc0HXGzYvsWKFFqMwew1siGd4KbapSh3HWhZ39XsN18=; b=j+tKfi2kSo+PoSM8bzNfiRyUHJOyeGZ8cXfsmhGYboQIkXNZcgKm7MC/cWX/C6ZkqG 60f7UABnT6ErAicHdSv8GNsaMsxzvOwv4YmniUVjWIoOAFBY8RnuSFSvJlf10WPLD0Pl 04johD9tWvxiLCZi1WR0w+oOd0+AB/CP+2jYhm7evUJn7pTqemwzVAxdP+R68pydUGOA HIqJ0F9ltOSTrTcu2f3O/9bIynM8h5wdOVo4MLk5r9Ic6t6CxEi8rZbTDEZoGZX1Jbzt PmrdFFznsQ54/dOus3KfJ9Qig75zKJPzP+rVna7qVoudLzvULSQv2dyT4F4wCWi6idZ2 o0vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685725372; x=1688317372; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=jc0HXGzYvsWKFFqMwew1siGd4KbapSh3HWhZ39XsN18=; b=kIhogpxT4Igs7VFT8cgtWASPTzdWghgqaEKGD+O+ig2SdspjQLLKN8oYxKYKgYV2fw dalAgGargmbmom1VzAsLEUMAPP72jZY7lS3E9btzmzD2+3g870PZL5p9/LW34qyndyKq U/9j8rWOjyQfVPzb7XiecKMoEPHGAFFdLqGSCsCHqCVyFEVabTZA/y1xM+PxOZ6Gi+jh MqyhDvl9LRS/rPBsbsbebWCNisyhqmwN+Ws5RLYuHliLg/zSJGxkOgslsQ2rp4/SOiud HmnzuJ9EYTH7qlkkY3btYDL0Z7viQ5OIMeT1PMFvF+VHdkiJ7AZhzBJDISpPY3QZLnrL w9Og== X-Gm-Message-State: AC+VfDz3Q/KyZQtLTRMt/azIV6brjus5AifjLTu7gseFjhLTetOnfxV9 +079IpkWns/wpwuV3DiyJ9p0xPDR/SBQeXgQ+w== X-Received: from coltonlewis-kvm.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:14ce]) (user=coltonlewis job=sendgmr) by 2002:a92:db52:0:b0:335:908b:8f9 with SMTP id w18-20020a92db52000000b00335908b08f9mr4386617ilq.1.1685725372593; Fri, 02 Jun 2023 10:02:52 -0700 (PDT) Date: Fri, 2 Jun 2023 17:01:46 +0000 In-Reply-To: <20230602170147.1541355-1-coltonlewis@google.com> Mime-Version: 1.0 References: <20230602170147.1541355-1-coltonlewis@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Message-ID: <20230602170147.1541355-3-coltonlewis@google.com> Subject: [PATCH 2/3] KVM: arm64: Clear possible conflict aborts From: Colton Lewis To: kvm@vger.kernel.org Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, Colton Lewis X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767611508305445422?= X-GMAIL-MSGID: =?utf-8?q?1767611508305445422?= Clear possible conflict aborts by TLB invalidation targeted to the address that caused the abort. Making use of FEAT_BBM Level 2 creates the possibility of a conflict abort when translating addresses, where multiple entries exist in the TLB for a single input address. Signed-off-by: Colton Lewis --- arch/arm64/include/asm/esr.h | 1 + arch/arm64/kvm/mmu.c | 6 ++++++ 2 files changed, 7 insertions(+) -- 2.41.0.rc0.172.g3f132b7071-goog diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 8487aec9b6587..41336cfa19ff3 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -123,6 +123,7 @@ #define ESR_ELx_FSC_SECC_TTW1 (0x1d) #define ESR_ELx_FSC_SECC_TTW2 (0x1e) #define ESR_ELx_FSC_SECC_TTW3 (0x1f) +#define ESR_ELx_FSC_CONFLICT (0x30) /* ISS field definitions for Data Aborts */ #define ESR_ELx_ISV_SHIFT (24) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 7a68398517c95..96b950f20c8d0 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1591,6 +1591,12 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) return 1; } + if (fault_status == ESR_ELx_FSC_CONFLICT) { + /* We could be at any level. 0 covers all levels. */ + __kvm_tlb_flush_vmid_ipa(vcpu->arch.hw_mmu, fault_ipa, 0); + return 1; + } + trace_kvm_guest_fault(*vcpu_pc(vcpu), kvm_vcpu_get_esr(vcpu), kvm_vcpu_get_hfar(vcpu), fault_ipa); From patchwork Fri Jun 2 17:01:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Colton Lewis X-Patchwork-Id: 102658 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp1175865vqr; Fri, 2 Jun 2023 10:09:33 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6x1dGN4ut+CaPCaNmmHZZR5uWanL5QPP6du0asdzqsAKSTNGpA7MpyDvRBgLX+i2y9S7GU X-Received: by 2002:a05:6a20:c704:b0:105:b75e:9df6 with SMTP id hi4-20020a056a20c70400b00105b75e9df6mr7950812pzb.26.1685725773568; Fri, 02 Jun 2023 10:09:33 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685725773; cv=none; d=google.com; s=arc-20160816; b=FsS6DwBV1J/b1vgOZ8EbmKkPXJkPgWTSl8wr/srRy+OTsmGXdgxBgR5UGZrpu4nZrn hABNsDYFP5Ca/jj55iJiRu3eIY35Qg7UKWDiC3z73m5E4h1QM4/aSeEXBO9FjoK4Iq3k ZgKhONF0/233LGnujWQcBGkP6EDDQZ+9xrqxhwGFi08xMJ8zFaLCXAZ8aEtodTcVvM7v XauCcSF/SR2/Q0sYdpMzJGA8r3vdc5Ig4vQ4t2K0bAjGN2tI7kBgBOuISIQfGk2XynHt 1IHuYmbhVKqWnF640Oqg1UJvW/n2EyRfAhshQsXIvdq4WTF2BnEhPjI4TJFqym7q1iz1 PttA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:dkim-signature; bh=46U2EWEjxn55kWeKsj3lDgLNoam846hFcVZO+vAwdLA=; b=kTOlAVAjL8WxEyss5TrkcamOdAWJzMaA9w0ckOJTh5oBELDZC/M29OeuIGwsmu910G V2d7HPYOI7GlvFhjo1JGMtuWGEuu6kjJDlY30gRVnTDlI9/7uNZQ94IzmXbK2wQZds9F H4E3ThVpuoVC73Igv1tVv0HynTl46r6cECXcCrtkJVvUFu38o+fV9d5jGJIxByIrLdbA E+TMkUmVQtafFtzcBn3Heu8J3HCQVw92n1hQYVr73qI9VyPlZd7wSyof8t8G98BwyCop u6gSa3HyFjtkgO9bvCS7u98mPZMzqGsxxRCcUAtD3vRGQfXxfG65p1LPC+Yqy8fBXJHY mvcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=WAqGXaWP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y30-20020a63b51e000000b0051fb9c77447si1209928pge.477.2023.06.02.10.09.18; Fri, 02 Jun 2023 10:09:33 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=WAqGXaWP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236890AbjFBRDE (ORCPT + 99 others); Fri, 2 Jun 2023 13:03:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236927AbjFBRCz (ORCPT ); Fri, 2 Jun 2023 13:02:55 -0400 Received: from mail-io1-xd49.google.com (mail-io1-xd49.google.com [IPv6:2607:f8b0:4864:20::d49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5EA441A2 for ; Fri, 2 Jun 2023 10:02:54 -0700 (PDT) Received: by mail-io1-xd49.google.com with SMTP id ca18e2360f4ac-777097126abso83073139f.0 for ; Fri, 02 Jun 2023 10:02:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1685725374; x=1688317374; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=46U2EWEjxn55kWeKsj3lDgLNoam846hFcVZO+vAwdLA=; b=WAqGXaWPraSA2daImKoen5B6tdZ+WYHgTjxhUUeeZAsbrniK5QsoJdy6bnYalQom0q MTUHUxan0x428dbLSXOSDczGl+sVACZNDReIhyFVk8qGv8O19J818JqDUV+L39pYvd+O +D0peOAv5LwnSHuhbBKS4on+Spl3k1vhxR2tQ3f+hr+YwUTyNz098zCnNqlglZ99wmXF 03RZCAfLMqzIZPxHZwDc+9cJzrwA0lt7idYGl5Mr8XHmSaECCNan2CtJJnr3I5Ts2P2r magwgDPHCmtfLDNKqCKxBMgHtvBcTUMA++iZxC/lrSlB4KiCVknekWOPC0oT6ZjtI3V8 dGqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1685725374; x=1688317374; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=46U2EWEjxn55kWeKsj3lDgLNoam846hFcVZO+vAwdLA=; b=ipymIVwzk2Z0Icl7YLJOuv9zEYAZ9TD/TBRpEiR1JI6iP4/2xzsjqGAKBhEc8lKe56 yeodxpI7yDtDInCFgCpEi9QZwkz60VGpe0QQjQcy7i8QJ2sa+OeT+i3PGL/cMPTlmF+D PsIgARPUDNZmqKKJmVv2UKpLrDYmSJqE2Fu6yWZEooT7AVcLt1dpeSNrfszoUtYRSQ6e /04kmIh/Qn/IV57KRNs6ClamaPmoGFlMh87MM873Yma2UvOZnQlcZhx1J2tAaDpDHlu7 euJUw84DSYufoRkilBPq+ixpumdxpcupwV3kOgey7BFaE/EmBgQLd1SENcD1a0R4Mnl9 Xt0w== X-Gm-Message-State: AC+VfDyBFGYT3/C4nl2QuocKDsoEL9WhobABzPaB0tl5Hj9ZuVv8phB3 GU9pfgq2bGpemyZzRboYVgIdBOeHLKCuf9v56w== X-Received: from coltonlewis-kvm.c.googlers.com ([fda3:e722:ac3:cc00:2b:ff92:c0a8:14ce]) (user=coltonlewis job=sendgmr) by 2002:a05:6602:1503:b0:777:2367:a7d3 with SMTP id g3-20020a056602150300b007772367a7d3mr1813249iow.0.1685725373854; Fri, 02 Jun 2023 10:02:53 -0700 (PDT) Date: Fri, 2 Jun 2023 17:01:47 +0000 In-Reply-To: <20230602170147.1541355-1-coltonlewis@google.com> Mime-Version: 1.0 References: <20230602170147.1541355-1-coltonlewis@google.com> X-Mailer: git-send-email 2.41.0.rc0.172.g3f132b7071-goog Message-ID: <20230602170147.1541355-4-coltonlewis@google.com> Subject: [PATCH 3/3] KVM: arm64: Skip break phase when we have FEAT_BBM level 2 From: Colton Lewis To: kvm@vger.kernel.org Cc: Catalin Marinas , Will Deacon , Marc Zyngier , Oliver Upton , James Morse , Suzuki K Poulose , Zenghui Yu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev, Colton Lewis X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1767611588488486917?= X-GMAIL-MSGID: =?utf-8?q?1767611588488486917?= Skip the break phase of break-before-make when the CPU has FEAT_BBM level 2. This allows skipping some expensive invalidation and serialization and should result in significant performance improvements when changing block size. The ARM manual section D5.10.1 specifically states under heading "Support levels for changing block size" that FEAT_BBM Level 2 support means changing block size does not break coherency, ordering guarantees, or uniprocessor semantics. Because a compare-and-exchange operation was used in the break phase to serialize access to the PTE, an analogous compare-and-exchange is introduced in the make phase to ensure serialization remains even if the break phase is skipped and proper handling is introduced to account for this function now having a way to fail. Considering the possibility that the new pte has different permissions than the old pte, the minimum necessary tlb invalidations are used. Signed-off-by: Colton Lewis --- arch/arm64/kvm/hyp/pgtable.c | 58 +++++++++++++++++++++++++++++++----- 1 file changed, 51 insertions(+), 7 deletions(-) -- 2.41.0.rc0.172.g3f132b7071-goog diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 8acab89080af9..6778e3df697f7 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -643,6 +643,11 @@ static bool stage2_has_fwb(struct kvm_pgtable *pgt) return !(pgt->flags & KVM_PGTABLE_S2_NOFWB); } +static bool stage2_has_bbm_level2(void) +{ + return cpus_have_const_cap(ARM64_HAS_STAGE2_BBM2); +} + #define KVM_S2_MEMATTR(pgt, attr) PAGE_S2_MEMATTR(attr, stage2_has_fwb(pgt)) static int stage2_set_prot_attr(struct kvm_pgtable *pgt, enum kvm_pgtable_prot prot, @@ -730,7 +735,7 @@ static bool stage2_try_set_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_ * @ctx: context of the visited pte. * @mmu: stage-2 mmu * - * Returns: true if the pte was successfully broken. + * Returns: true if the pte was successfully broken or there is no need. * * If the removed pte was valid, performs the necessary serialization and TLB * invalidation for the old value. For counted ptes, drops the reference count @@ -750,6 +755,10 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, return false; } + /* There is no need to break the pte. */ + if (stage2_has_bbm_level2()) + return true; + if (!stage2_try_set_pte(ctx, KVM_INVALID_PTE_LOCKED)) return false; @@ -771,16 +780,45 @@ static bool stage2_try_break_pte(const struct kvm_pgtable_visit_ctx *ctx, return true; } -static void stage2_make_pte(const struct kvm_pgtable_visit_ctx *ctx, kvm_pte_t new) +static bool stage2_pte_perms_equal(kvm_pte_t p1, kvm_pte_t p2) +{ + u64 perms1 = p1 & KVM_PGTABLE_PROT_RWX; + u64 perms2 = p2 & KVM_PGTABLE_PROT_RWX; + + return perms1 == perms2; +} + +/** + * stage2_try_make_pte() - Attempts to install a new pte. + * + * @ctx: context of the visited pte. + * @new: new pte to install + * + * Returns: true if the pte was successfully installed + * + * If the old pte had different permissions, perform appropriate TLB + * invalidation for the old value. For counted ptes, drops the + * reference count on the containing table page. + */ +static bool stage2_try_make_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu, kvm_pte_t new) { struct kvm_pgtable_mm_ops *mm_ops = ctx->mm_ops; - WARN_ON(!stage2_pte_is_locked(*ctx->ptep)); + if (!stage2_has_bbm_level2()) + WARN_ON(!stage2_pte_is_locked(*ctx->ptep)); + + if (!stage2_try_set_pte(ctx, new)) + return false; + + if (kvm_pte_table(ctx->old, ctx->level)) + kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + else if (kvm_pte_valid(ctx->old) && !stage2_pte_perms_equal(ctx->old, new)) + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa_nsh, mmu, ctx->addr, ctx->level); if (stage2_pte_is_counted(new)) mm_ops->get_page(ctx->ptep); - smp_store_release(ctx->ptep, new); + return true; } static void stage2_put_pte(const struct kvm_pgtable_visit_ctx *ctx, struct kvm_s2_mmu *mmu, @@ -879,7 +917,8 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx, stage2_pte_executable(new)) mm_ops->icache_inval_pou(kvm_pte_follow(new, mm_ops), granule); - stage2_make_pte(ctx, new); + if (!stage2_try_make_pte(ctx, data->mmu, new)) + return -EAGAIN; return 0; } @@ -934,7 +973,9 @@ static int stage2_map_walk_leaf(const struct kvm_pgtable_visit_ctx *ctx, * will be mapped lazily. */ new = kvm_init_table_pte(childp, mm_ops); - stage2_make_pte(ctx, new); + + if (!stage2_try_make_pte(ctx, data->mmu, new)) + return -EAGAIN; return 0; } @@ -1385,7 +1426,10 @@ static int stage2_split_walker(const struct kvm_pgtable_visit_ctx *ctx, * writes the PTE using smp_store_release(). */ new = kvm_init_table_pte(childp, mm_ops); - stage2_make_pte(ctx, new); + + if (!stage2_try_make_pte(ctx, mmu, new)) + return -EAGAIN; + dsb(ishst); return 0; }