From patchwork Tue Jan 2 22:00:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184517 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4705735dyb; Tue, 2 Jan 2024 14:05:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IGRHaovPl5UouiwrzNwoREDVtStJkrvCDf7+AfDk+5POOCyyVia4px1FOf73lBQbl9kBnva X-Received: by 2002:a17:903:32d1:b0:1d4:be64:263f with SMTP id i17-20020a17090332d100b001d4be64263fmr1567632plr.120.1704233119362; Tue, 02 Jan 2024 14:05:19 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233119; cv=none; d=google.com; s=arc-20160816; b=rCFTUpH8LZF8vNxqg/OEuqHvPE+Umwmoh2/Pz6g9zlqk6m5lxCVvX3/v3hKxXB+3S/ z8Iyb+RvTObWgAhzpwXgpRUkzEyPGikHDowAtKaIgQ3bg0gQczF+zGhIQ335IbBSrtjY FXoB812z+b/2Z5v961vBIBbP3WQEZ0Zn0Ki3+XMiyeXn80+EI6RAQFOoPENfiQlRk7Y+ j+f0HW4zShJLh//8SJGe166tyeCkQn/WyPPhei17wwUrgwUKWw3KdrpJNRyqClEuwuIh 0rXwA8KoNzAGvTX7T+7FmLboT+bgml7Toz9HIAZujE2/BUYdXACA1YiyYQhXMPMFaLhl XYoQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=Sx1CrYCYqcy0I3lbTucfKtzktUZzOavzb19InQrqEBM=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=ex1UkBIWxb4br3PjZ6JbbGXiXCiW/mibX3X2m0AgG/tOkGchrDkdZqGF/3XlfNlQmU jWiYQ/IvwxpU/JGJIB0H0p1Ite1ZXRxMf7E/7feCYO1mDuSVPFNyzh/yt7XasGqqjie4 60virrR3FWKwmvESddiWWrN3wcD45zYOA51Pp1cQeQfqaucKYtgtCTqJy7trjeEsa79X zNUy3tXzRgV8FsXgZ7SrDNXikY4ubwQHS1/1bRmq1v9vCQvMZ7DRsnYGmQPf2RuTZsEl dGs1dAWVWFv+TlBsC8Lb7YQsi2t141J3kO4z3c1onIqzwM+Nf6n22JpluSy0v2OL8Ry7 SWwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=k0R4KwlT; spf=pass (google.com: domain of linux-kernel+bounces-14911-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14911-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id h3-20020a170902b94300b001d45e24b91asi14045364pls.572.2024.01.02.14.05.19 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:05:19 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14911-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=k0R4KwlT; spf=pass (google.com: domain of linux-kernel+bounces-14911-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14911-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 2369C281612 for ; Tue, 2 Jan 2024 22:05:19 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 9989418E13; Tue, 2 Jan 2024 22:01:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="k0R4KwlT" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F0E718AF4 for ; Tue, 2 Jan 2024 22:01:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-28ce6eeb4easo310245a91.0 for ; Tue, 02 Jan 2024 14:01:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232898; x=1704837698; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Sx1CrYCYqcy0I3lbTucfKtzktUZzOavzb19InQrqEBM=; b=k0R4KwlTr46S55jHvhzg/o8eollZhzK6pzTHwCz1T8zqKvHQU3BsmWUf4lurfepP/m 4IYn3Ua468pGBIZwAcdXb7+rO3nCvvl5Mo+1W07ES0fRyOjfjm+mdUibpgGnEQwBP3Lk KG8T359hvMvcuU+36tIUT3hc5ov0asYY1zDGJCd1lionGf4/sWZukLGPJMu/z+58IDKZ cyBWCM+M5oJuwwiyQUyt2b0NTyRL1VMYFPrsy49tlGm73PNeqHK+SKg3qKrTUjYsyPVN 4tDrj+rh/CkyVAaHtD8EtNz7AbCQ0olPZ4JKDFmAroinso3n64OkV1H/D/bP3ugOARAb iihA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232898; x=1704837698; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Sx1CrYCYqcy0I3lbTucfKtzktUZzOavzb19InQrqEBM=; b=IsuWJSh15s3LHoOn1AQhByGZtMasWaQ73rHj40PKhqnHkHHXv2TRMuOIIpj32faYg6 e1BzBw5LjF9P1M9vA2jzJQ6nKlcQg/glC2L0uLTCFqkpBoed0EW88FpI3qcmNLOiuTOl KXXH9lugPKe6Iu5xkDZmzMuD+k3soM0BwkujappzW7oAgw6hmZrJneO6H9u8y7SFHhMt xfnipmCfPk0rNKGbR7prBUG0Wmee6ydvihD6vPHjI8kbNri5BLIvJSVm8vd+nIXnYeAe Qcx6AYYUi+Z/GUhrja5lyVaFnSkbb0HTpmIkopwhU5zU4t/o0pounIjNKaU3V92ZXxUw AnVQ== X-Gm-Message-State: AOJu0Yz5p8zt1366m442V/I2bCojlU/qfD+jghYPfzClYjZ6g2l2NYoM i0z9h63+TMWNYKXd/J3RdA++XL8Ozaxr+g== X-Received: by 2002:a17:90a:728e:b0:28c:a9d0:33ff with SMTP id e14-20020a17090a728e00b0028ca9d033ffmr2151486pjg.62.1704232897945; Tue, 02 Jan 2024 14:01:37 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:37 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 01/12] riscv: Flush the instruction cache during SMP bringup Date: Tue, 2 Jan 2024 14:00:38 -0800 Message-ID: <20240102220134.3229156-2-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787017947275396792 X-GMAIL-MSGID: 1787017947275396792 Instruction cache flush IPIs are sent only to CPUs in cpu_online_mask, so they will not target a CPU until it calls set_cpu_online() earlier in smp_callin(). As a result, if instruction memory is modified between the CPU coming out of reset and that point, then its instruction cache may contain stale data. Therefore, the instruction cache must be flushed after the set_cpu_online() synchronization point. Fixes: 08f051eda33b ("RISC-V: Flush I$ when making a dirty page executable") Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - New patch for v4 arch/riscv/kernel/smpboot.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c index d162bf339beb..48af5bd3ec30 100644 --- a/arch/riscv/kernel/smpboot.c +++ b/arch/riscv/kernel/smpboot.c @@ -26,7 +26,7 @@ #include #include -#include +#include #include #include #include @@ -257,9 +257,10 @@ asmlinkage __visible void smp_callin(void) riscv_user_isa_enable(); /* - * Remote TLB flushes are ignored while the CPU is offline, so emit - * a local TLB flush right now just in case. + * Remote cache and TLB flushes are ignored while the CPU is offline, + * so flush them both right now just in case. */ + local_flush_icache_all(); local_flush_tlb_all(); complete(&cpu_running); /* From patchwork Tue Jan 2 22:00:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184518 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4705915dyb; Tue, 2 Jan 2024 14:05:44 -0800 (PST) X-Google-Smtp-Source: AGHT+IGNhBP1sUylKO4+tS7QVHAq7FFzlhebEdBGUHskZveiVoor0UmPKBGmJzq0257hQ5T6yzv6 X-Received: by 2002:a05:6512:925:b0:50e:885d:4d1d with SMTP id f5-20020a056512092500b0050e885d4d1dmr4122869lft.37.1704233143834; Tue, 02 Jan 2024 14:05:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233143; cv=none; d=google.com; s=arc-20160816; b=an/jYlf4RZf3pJn4dTPuitsQ1s+czmH+ZS5/pabL0BCFX45mS5+ukHigJtmBIZubWs x/V6NKOHwoY/gsi8CA8XGx34TNkWWY6zD1R0MAP0E8wv1s08qflzj5EDpCZkQoSbFWas BQ63y5gTHVFqrnAibnNjKObMjIssM1WcF8kA4ySVUWygMOljldqtqJ1i4wJtHyht3dXw cWAf/EyS7F7e9OJ1yI6CEXRDaXefDYynmj0YSIr1o+0U1+QNnEiZDG2RPqcpKMFo4Qms mXFwGufSONxLv5BkWXR7ITJzecn7hrwDzsMJhsqIghqC3M87pTzyuWEm+S1rDGEkppNT R5Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=pynuwdwrgTDQTs9EZPEkGjuJgTxII1R0VKB5FEKmh8A=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=b0u8mGlCq2CsvcGco9yxn6QjyidlXoO1a6uQAycV8KK/cQdrSoCVfmQ7AYwpy2NHge L05wVWEkzB3N0M8L7CyUov+HGXesr+qEBQnMX9PMZbQ9H4zE/u8DwcOtvF4qtzZqH98T Ku1jRMrgkpOopoG4+poMFzbTQYNB2U33iv4q3JvUJkUFjR8XDdz7NQvdKgdSKGQT3dvy a9hhJeGgKumZdp+mg6TsQ/yQDnSlaGV/4eOtrAY9JersQcGs9XC47F9397HWFJHgf3/0 GGPCHlvvfq5IrjlGzzt4kSyB9bB7C4ROh3vtOuZnI9WYum62DKTTOS1DTwIBaMCPejrl 009Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=FfjNMhrQ; spf=pass (google.com: domain of linux-kernel+bounces-14912-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14912-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id wh12-20020a170906fd0c00b00a2829a8fd60si1190894ejb.1053.2024.01.02.14.05.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:05:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14912-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=FfjNMhrQ; spf=pass (google.com: domain of linux-kernel+bounces-14912-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14912-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 450151F2381F for ; Tue, 2 Jan 2024 22:05:43 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 28DF818EA7; Tue, 2 Jan 2024 22:01:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="FfjNMhrQ" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEE5518B0D for ; Tue, 2 Jan 2024 22:01:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-1d3ed1ca402so75965005ad.2 for ; Tue, 02 Jan 2024 14:01:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232899; x=1704837699; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pynuwdwrgTDQTs9EZPEkGjuJgTxII1R0VKB5FEKmh8A=; b=FfjNMhrQ2J03UobrzaKUakTrVdNaNEJx8tGD521XPYAekzCF2+wKX4MPKhMGSK/gN8 A5/UK5wGRYTrb/y2VqWNWIu4O3FctlSLMghABai1+vBkqQpMhWeUEqi8Cx4hlPiMawDK Ue+oDPUDovmJDplqE74hieMY66whud/qCvV79FjxMmYmgUFpzbO0/Is1G9/Kh0qBC1B4 D2x2AIWsdIH/0+odvOSP1jCiv1pzomJNDCMaPfWN6GUpbsCTPCmOqMfuxrV1+YgrW6Of +wD1ZAlTyTwaTgY/vibR/mTHIEfhQ2DnV39lrz3a9ieOzH4FkQseQ1GRck7cWoCnTXDH ZMKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232899; x=1704837699; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pynuwdwrgTDQTs9EZPEkGjuJgTxII1R0VKB5FEKmh8A=; b=Y+xBmar7GaC1K6P573Uitx2y5jRbOMgVHRzBR4Jn4CaX8n3gU6FJXDXXfs8Ixf8wc6 lBK5E/oh4aAaf5osgzwfWnRdJAf+V0TTRN+LGvp3EVAKG9Mb/jY573AZanhSstidt/eG GW9pQy059geDkfn6qrhwTeRPs7tMZqzbmBAisG62MocvraD+dlJBybLBT9bbZzaXBPck rN95CUcUWzvcB4fN+6zWvY55BBgY+l+Tx8kuYwyNQzjAMJ6ba6fRKf+fHxxBsEc+waq2 4iT8CCHXOTNxpnH2v1+l9c80FFX4iNnjzT1p6/qMtItsC7ncSA15oRK4Bqour2pQEonl feSg== X-Gm-Message-State: AOJu0YwsTFIvZqyVxYhNLDz+U58ssYma+zISkNNmWiD8gj9H/Y5I1nSP +/Cg0IB206daqf/+UaO/OLrejuLX4Saa0bttroWrHUsSLdY= X-Received: by 2002:a17:90b:3ec2:b0:28c:1b98:e6e7 with SMTP id rm2-20020a17090b3ec200b0028c1b98e6e7mr8644807pjb.47.1704232899208; Tue, 02 Jan 2024 14:01:39 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:38 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 02/12] riscv: Use IPIs for remote cache/TLB flushes by default Date: Tue, 2 Jan 2024 14:00:39 -0800 Message-ID: <20240102220134.3229156-3-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787017973402248185 X-GMAIL-MSGID: 1787017973402248185 An IPI backend is always required in an SMP configuration, but an SBI implementation is not. For example, SBI will be unavailable when the kernel runs in M mode. Generally, IPIs are assumed to be faster than SBI calls due to the SBI context switch overhead. However, when SBI is used as the IPI backend, then the context switch cost must be paid anyway, and performing the cache/TLB flush directly in the SBI implementation is more efficient than inserting an interrupt to the kernel. This is the only scenario where riscv_ipi_set_virq_range()'s use_for_rfence parameter is false. Thus, it makes sense for remote fences to use IPIs by default, and make the SBI remote fence extension the special case. sbi_ipi_init() already checks riscv_ipi_have_virq_range(), so it only calls riscv_ipi_set_virq_range() when no other IPI device is available. So we can move the static key and drop the use_for_rfence parameter. Furthermore, the static branch only makes sense when CONFIG_RISCV_SBI is enabled. Optherwise, IPIs must be used. Add a fallback definition of riscv_use_sbi_for_rfence() which handles this case and removes the need to check CONFIG_RISCV_SBI elsewhere, such as in cacheflush.c. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - New patch for v4 arch/riscv/include/asm/sbi.h | 4 ++++ arch/riscv/include/asm/smp.h | 15 ++------------- arch/riscv/kernel/sbi-ipi.c | 11 ++++++++++- arch/riscv/kernel/smp.c | 11 +---------- arch/riscv/mm/cacheflush.c | 5 ++--- arch/riscv/mm/tlbflush.c | 31 ++++++++++++++----------------- drivers/clocksource/timer-clint.c | 2 +- 7 files changed, 34 insertions(+), 45 deletions(-) diff --git a/arch/riscv/include/asm/sbi.h b/arch/riscv/include/asm/sbi.h index 0892f4421bc4..aeee0127df76 100644 --- a/arch/riscv/include/asm/sbi.h +++ b/arch/riscv/include/asm/sbi.h @@ -339,8 +339,12 @@ unsigned long riscv_cached_marchid(unsigned int cpu_id); unsigned long riscv_cached_mimpid(unsigned int cpu_id); #if IS_ENABLED(CONFIG_SMP) && IS_ENABLED(CONFIG_RISCV_SBI) +DECLARE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +#define riscv_use_sbi_for_rfence() \ + static_branch_unlikely(&riscv_sbi_for_rfence) void sbi_ipi_init(void); #else +static inline bool riscv_use_sbi_for_rfence(void) { return false; } static inline void sbi_ipi_init(void) { } #endif diff --git a/arch/riscv/include/asm/smp.h b/arch/riscv/include/asm/smp.h index 0d555847cde6..7ac80e9f2288 100644 --- a/arch/riscv/include/asm/smp.h +++ b/arch/riscv/include/asm/smp.h @@ -49,12 +49,7 @@ void riscv_ipi_disable(void); bool riscv_ipi_have_virq_range(void); /* Set the IPI interrupt numbers for arch (called by irqchip drivers) */ -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence); - -/* Check if we can use IPIs for remote FENCEs */ -DECLARE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -#define riscv_use_ipi_for_rfence() \ - static_branch_unlikely(&riscv_ipi_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr); /* Check other CPUs stop or not */ bool smp_crash_stop_failed(void); @@ -104,16 +99,10 @@ static inline bool riscv_ipi_have_virq_range(void) return false; } -static inline void riscv_ipi_set_virq_range(int virq, int nr, - bool use_for_rfence) +static inline void riscv_ipi_set_virq_range(int virq, int nr) { } -static inline bool riscv_use_ipi_for_rfence(void) -{ - return false; -} - #endif /* CONFIG_SMP */ #if defined(CONFIG_HOTPLUG_CPU) && (CONFIG_SMP) diff --git a/arch/riscv/kernel/sbi-ipi.c b/arch/riscv/kernel/sbi-ipi.c index a4559695ce62..1026e22955cc 100644 --- a/arch/riscv/kernel/sbi-ipi.c +++ b/arch/riscv/kernel/sbi-ipi.c @@ -13,6 +13,9 @@ #include #include +DEFINE_STATIC_KEY_FALSE(riscv_sbi_for_rfence); +EXPORT_SYMBOL_GPL(riscv_sbi_for_rfence); + static int sbi_ipi_virq; static void sbi_ipi_handle(struct irq_desc *desc) @@ -72,6 +75,12 @@ void __init sbi_ipi_init(void) "irqchip/sbi-ipi:starting", sbi_ipi_starting_cpu, NULL); - riscv_ipi_set_virq_range(virq, BITS_PER_BYTE, false); + riscv_ipi_set_virq_range(virq, BITS_PER_BYTE); pr_info("providing IPIs using SBI IPI extension\n"); + + /* + * Use the SBI remote fence extension to avoid + * the extra context switch needed to handle IPIs. + */ + static_branch_enable(&riscv_sbi_for_rfence); } diff --git a/arch/riscv/kernel/smp.c b/arch/riscv/kernel/smp.c index 40420afbb1a0..1d06df04eb71 100644 --- a/arch/riscv/kernel/smp.c +++ b/arch/riscv/kernel/smp.c @@ -171,10 +171,7 @@ bool riscv_ipi_have_virq_range(void) return (ipi_virq_base) ? true : false; } -DEFINE_STATIC_KEY_FALSE(riscv_ipi_for_rfence); -EXPORT_SYMBOL_GPL(riscv_ipi_for_rfence); - -void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) +void riscv_ipi_set_virq_range(int virq, int nr) { int i, err; @@ -197,12 +194,6 @@ void riscv_ipi_set_virq_range(int virq, int nr, bool use_for_rfence) /* Enabled IPIs for boot CPU immediately */ riscv_ipi_enable(); - - /* Update RFENCE static key */ - if (use_for_rfence) - static_branch_enable(&riscv_ipi_for_rfence); - else - static_branch_disable(&riscv_ipi_for_rfence); } static const char * const ipi_names[] = { diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 55a34f2020a8..47c485bc7df0 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,7 @@ void flush_icache_all(void) { local_flush_icache_all(); - if (IS_ENABLED(CONFIG_RISCV_SBI) && !riscv_use_ipi_for_rfence()) + if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); @@ -69,8 +69,7 @@ void flush_icache_mm(struct mm_struct *mm, bool local) * with flush_icache_deferred(). */ smp_mb(); - } else if (IS_ENABLED(CONFIG_RISCV_SBI) && - !riscv_use_ipi_for_rfence()) { + } else if (riscv_use_sbi_for_rfence()) { sbi_remote_fence_i(&others); } else { on_each_cpu_mask(&others, ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index e6659d7368b3..09b03bf71e6a 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -73,10 +73,10 @@ static void __ipi_flush_tlb_all(void *info) void flush_tlb_all(void) { - if (riscv_use_ipi_for_rfence()) - on_each_cpu(__ipi_flush_tlb_all, NULL, 1); - else + if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); + else + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); } struct flush_tlb_range_data { @@ -96,7 +96,6 @@ static void __ipi_flush_tlb_range_asid(void *info) static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { - struct flush_tlb_range_data ftd; const struct cpumask *cmask; unsigned long asid = FLUSH_TLB_NO_ASID; bool broadcast; @@ -119,20 +118,18 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, broadcast = true; } - if (broadcast) { - if (riscv_use_ipi_for_rfence()) { - ftd.asid = asid; - ftd.start = start; - ftd.size = size; - ftd.stride = stride; - on_each_cpu_mask(cmask, - __ipi_flush_tlb_range_asid, - &ftd, 1); - } else - sbi_remote_sfence_vma_asid(cmask, - start, size, asid); - } else { + if (!broadcast) { local_flush_tlb_range_asid(start, size, stride, asid); + } else if (riscv_use_sbi_for_rfence()) { + sbi_remote_sfence_vma_asid(cmask, start, size, asid); + } else { + struct flush_tlb_range_data ftd; + + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } if (mm) diff --git a/drivers/clocksource/timer-clint.c b/drivers/clocksource/timer-clint.c index 9a55e733ae99..7ccc16dd6a76 100644 --- a/drivers/clocksource/timer-clint.c +++ b/drivers/clocksource/timer-clint.c @@ -251,7 +251,7 @@ static int __init clint_timer_init_dt(struct device_node *np) } irq_set_chained_handler(clint_ipi_irq, clint_ipi_interrupt); - riscv_ipi_set_virq_range(rc, BITS_PER_BYTE, true); + riscv_ipi_set_virq_range(rc, BITS_PER_BYTE); clint_clear_ipi(); #endif From patchwork Tue Jan 2 22:00:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184526 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4709431dyb; Tue, 2 Jan 2024 14:14:11 -0800 (PST) X-Google-Smtp-Source: AGHT+IEj7Q+Si543kyy/zhqZXgOBQYDrygHehIeBEDBwIiChFHg2+hW50Qr53Bxl40KWnf3zJI2U X-Received: by 2002:a05:6830:1459:b0:6dc:386:a236 with SMTP id w25-20020a056830145900b006dc0386a236mr7411289otp.47.1704233651759; Tue, 02 Jan 2024 14:14:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233651; cv=none; d=google.com; s=arc-20160816; b=rttjeLJ3pZkE9sySH1fmFkdWK7bbpyPFkRE00sg1Yut08FP3ewVfGgpwz52lbD+baC 6JdMmTEUqi+Ja6/N6cb6Gch+Ob37er0CfJW88oZwe8BuxZgTOLCqessS7wcgW1XCtkSV Q2BS7o6KWfFe8KHQGEr5ajWplurfSjWOYbuLP8v458d9pe77oqH1NsrSmVa6cCeh3mcc MzMx5QHyy3BEKxOi7uSMu68bqP8xu9FRforUM3o3SNepH6Y8Xxc9smiuVSL8JXw1Um4p uzW3iQsIQdnJMT6y1PktYLPSHG2/lS5ta+pAAQNTvwb0qHkTv8eAp3m5OzgT6ujZiwFc VUAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=XA8TOHyyeKTc95PFXJRpYatfRR5oPHMuobG+iziT3HE=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=VCpGQkfpfFfOCnwW14tx76UOwdyqXEHZE0GU/8yGNT3aRU2VICUwk+NnO/JbHiNmyT 8pKFW6aCtWrTLWn+aQ616C5r8by5DrlmYiLs3OELcO8XSjsgnE1BwqCDUZkEmbgm53Pp ssACCyh/D9VW4+bQArJdW/fxpyoaWVksgTbArBJsIVqFjAPYsVjuC4l8Tx55PpxfUCa4 pNnp4rHqt/aphoFZortpyDUe4SZfrs+YgnzYbklPMtEy10LMJKmp8DWKgEucn0Wb6Efc w22w9M9ksNx24g2ow3Pkt9xDjC/9voyCIDHWHFLXwmecEnSr3WRWv72PTHElWujBM5Ny 7+kw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=AGqNsbmM; spf=pass (google.com: domain of linux-kernel+bounces-14913-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14913-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id n10-20020a056a0007ca00b006d9abd892fbsi15901576pfu.175.2024.01.02.14.14.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:14:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14913-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=AGqNsbmM; spf=pass (google.com: domain of linux-kernel+bounces-14913-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14913-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id D1325B230BA for ; Tue, 2 Jan 2024 22:05:57 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 0673918EC9; Tue, 2 Jan 2024 22:01:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="AGqNsbmM" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B04C18C13 for ; Tue, 2 Jan 2024 22:01:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-28c7c9b19f1so2514434a91.1 for ; Tue, 02 Jan 2024 14:01:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232900; x=1704837700; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XA8TOHyyeKTc95PFXJRpYatfRR5oPHMuobG+iziT3HE=; b=AGqNsbmMnp9vjyebsFChQjy7nWz6SWLDLP0QAiuNu27kO49irRUptsDDwu9E5V9Ket xvodU/gGJFv9vP5cV6Q8RA8C4/ZF7/VZCDCtJQBuiyuuiJ4ia418Kc4zam429n8/ofuS Uo4UFXB4eRCJsditxh58ECPJ53zpK0LjtIA1qftThpO0uq2LhqaLAgJjWjzzkUlH9NmS 3hbJTa8CZsVqH4yce8yvbbnq0iYoVoCBRNG3qqBh9fSl22ovicWKevmCdZu8bx5sSLCU OmkkQy20Jb0DsOTi5CKI0zsCgdq/J983tSNp8xKiwyEAaignZWNbTGCJ2zZ1fv9T5cu3 jclQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232900; x=1704837700; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XA8TOHyyeKTc95PFXJRpYatfRR5oPHMuobG+iziT3HE=; b=V4td4ENHyyYtcM+wxiSEEbRhkVd9kzd4wISlCEKoeyA8vlg5iLFOFrreiGA8lLZ2LE 3O4S/e+3Em/tuoUbqJ5SHZKPIk7iOeV55uiRa0zwdoD6TbSl2xNY5MBf/Ny4tjDHJ/43 qpAiTu5mdeL89kXl556Zg38xKoTVpQWOa7E7kX6YkCvC6yezt2zvubyeSbuEQfkn7wA6 udWT1JyiiF63FG99pVHwz4tfAdm828O3c7JV7QXgG/sxvFj6J6GHFxx6CCXbJTsYshFq jy5G75HSowy9vKM+omj61B0LtBoOuWOBMSwhJO7jfb7WAMxtWXxWFVRZXEf8rmsHDVH0 8N2A== X-Gm-Message-State: AOJu0YyDVgS/6Foj946cMGA17geeGxh7FA1lraLhsxHPWaPJukZcZ0JT b78N1xOwbYvLuELqWHf99NR/48hHHDkNRQ== X-Received: by 2002:a17:90a:974c:b0:28c:accc:b1db with SMTP id i12-20020a17090a974c00b0028cacccb1dbmr2272863pjw.44.1704232900428; Tue, 02 Jan 2024 14:01:40 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:40 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 03/12] riscv: mm: Broadcast kernel TLB flushes only when needed Date: Tue, 2 Jan 2024 14:00:40 -0800 Message-ID: <20240102220134.3229156-4-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787018506009881029 X-GMAIL-MSGID: 1787018506009881029 __flush_tlb_range() avoids broadcasting TLB flushes when an mm context is only active on the local CPU. Apply this same optimization to TLB flushes of kernel memory when only one CPU is online. This check can be constant-folded when SMP is disabled. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - New patch for v4 arch/riscv/mm/tlbflush.c | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 09b03bf71e6a..2f18fe6fc4f3 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -98,27 +98,23 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, { const struct cpumask *cmask; unsigned long asid = FLUSH_TLB_NO_ASID; - bool broadcast; + unsigned int cpu; if (mm) { - unsigned int cpuid; - cmask = mm_cpumask(mm); if (cpumask_empty(cmask)) return; - cpuid = get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; - if (static_branch_unlikely(&use_asid_allocator)) asid = atomic_long_read(&mm->context.id) & asid_mask; } else { cmask = cpu_online_mask; - broadcast = true; } - if (!broadcast) { + cpu = get_cpu(); + + /* Check if the TLB flush needs to be sent to other CPUs. */ + if (cpumask_any_but(cmask, cpu) >= nr_cpu_ids) { local_flush_tlb_range_asid(start, size, stride, asid); } else if (riscv_use_sbi_for_rfence()) { sbi_remote_sfence_vma_asid(cmask, start, size, asid); @@ -132,8 +128,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, on_each_cpu_mask(cmask, __ipi_flush_tlb_range_asid, &ftd, 1); } - if (mm) - put_cpu(); + put_cpu(); } void flush_tlb_mm(struct mm_struct *mm) From patchwork Tue Jan 2 22:00:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184519 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4706116dyb; Tue, 2 Jan 2024 14:06:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IGuwYhy6uy33hcJrs7nYZuOG4IGoRjkuMM80fF8W8hNDRajo7VdDulLRVX2AoR26yrl7X4F X-Received: by 2002:a50:c282:0:b0:556:17d4:b18b with SMTP id o2-20020a50c282000000b0055617d4b18bmr1629004edf.96.1704233172934; Tue, 02 Jan 2024 14:06:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233172; cv=none; d=google.com; s=arc-20160816; b=ChnmyuS+KZuDp7f6E8SWkrv+be0Q/gcYvTtiZvEGJlw4WI54yjrCH/6EFZb6s5tlzs v10M8SHaqjXBdSlYJOt8PmEM9Ssv+dOBC2Zx1Bp9/TPLWStEcQ6IqxWV9DB9hzlDfei+ YKJkSBWcNTz6JHqmxhjGOr0x8jowvH6Ue5DNQAkkh1/hSZ8YyaiIY6og3PPmhQBZ6+Xv KJpMu7UoZEkpXem5EDGhfIbN6fjbQJ98VG65E/xg7QuaLDE8253/xYvD6qN10gPcrv+A XEkPvNws/+YA+e2M2JagEo3YSQswqee0NCq5wwfQV39T+e2FUpmubxmqr1ruZM02d3ZM y4kQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=YDpWqQ4mc+jX/Roh6ocibeJ1LmLlK7pqWIKc4/L4AE8=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=qmYqARhSXa5nDjF8rbLwpdL4Pr6EjuHy4YP7Sl1I8uq+EZsgegsJpZNKRlISse5/jg lIFdnDCufAOoCu+fDNvODDnIm2w48pNDzMo3y8zHOlkSGtHe7LqdKJbGISOwWmeGgRB/ Jd3juaBBj41S0cu00qCAUV1nDYfD5rpS8SHm/0J/BcArO3U2TrdwrLDTjuU/LfIadTqo HfelBSzgJvJeXtw6lM+Gru0MXAX+wTUUg0DjCKVln+MauWLZ4zdG5NN7o2pk5MhJf+iA de62ygzQrHbTyShQIFT83BRF9rbscBkW6y2ZHiVHgMNnIc1IILHHMQvGsIo4SA2Y6ddT 0Drw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=DgRqkJoN; spf=pass (google.com: domain of linux-kernel+bounces-14914-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14914-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id v20-20020a50a454000000b0054cd1a2460asi11815005edb.672.2024.01.02.14.06.12 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:06:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14914-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=DgRqkJoN; spf=pass (google.com: domain of linux-kernel+bounces-14914-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14914-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 8A0CF1F2249B for ; Tue, 2 Jan 2024 22:06:12 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id DEC1919447; Tue, 2 Jan 2024 22:01:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="DgRqkJoN" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15FB418C35 for ; Tue, 2 Jan 2024 22:01:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-5c229dabbb6so2569289a12.0 for ; Tue, 02 Jan 2024 14:01:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232901; x=1704837701; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YDpWqQ4mc+jX/Roh6ocibeJ1LmLlK7pqWIKc4/L4AE8=; b=DgRqkJoN7jmq4C8Mete+CRT6XrLFu/qo41dxk46PW9v6kz+g8QgJvnk6qO7xoHR+JN vm3DhCNlDOJ+DzdkuZPjd3T1BzPLejafjTfOLF9j0KNqj5ghgmvqV1dIWAECjHALDlXz LoMnfwTR1mPwilWyEifFn4whkLsUqDNqlxffLPsZvoNUXWF0ZBs/8p2SUuuj6DPLkbid W26tySgW6sKFChUQYQsKU8fT2ZkvAJFy4sgT6sK/1jihvVJEIOuLwvfcp4/JDOke5ggL ws9zS7AzG4voaTu2urPb1f2YSLkfXuO1DNTURKWFMuR5yrKzBI4NmEtEdoEZq8f0cxZp 0VjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232901; x=1704837701; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YDpWqQ4mc+jX/Roh6ocibeJ1LmLlK7pqWIKc4/L4AE8=; b=WKqC4CvQFhUZbVjhiOCu5QGJDxhePG1vwLIRK771A9cfW8OiP8/JuwDnbWKuFCx00j UPJVyoaOTyMsZKyXEwC6Pm8evkOSTUAazrFypPTP0qP8N3Aw1IYSOGjX75juwUTqkyaR B/DAIcDDs8SjMVx43FY+uFprarfiOFN2Go1ni0KXQQ7sFvlZZOfykeleVj4QFxlP+7AQ IolhiN891P55EcsCgeccsgSxr/rvD7FyTEv9pQdy5SPqzeuo2azW2/sL6OwZRctgSE/r Sn9c8oXuNn4TlbR4215G0MyFsCZMFDzQWnnncK1E8tZFkUQNR2iFegVe1JNTtYRQP+7j HwCA== X-Gm-Message-State: AOJu0YzkEUSkws0sJRqQAp4MihPl0FEzcQ41BcVUjEZNk0gClqldWGLt Rzabx5d94U8tpAvOu+ice3MOP9xdkfsENCZS4vkccVSGlE0= X-Received: by 2002:a17:90a:6ba6:b0:28c:640b:ce0b with SMTP id w35-20020a17090a6ba600b0028c640bce0bmr4536054pjj.33.1704232901387; Tue, 02 Jan 2024 14:01:41 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:41 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 04/12] riscv: Only send remote fences when some other CPU is online Date: Tue, 2 Jan 2024 14:00:41 -0800 Message-ID: <20240102220134.3229156-5-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787018003519615186 X-GMAIL-MSGID: 1787018003519615186 If no other CPU is online, a local cache or TLB flush is sufficient. These checks can be constant-folded when SMP is disabled. Signed-off-by: Samuel Holland --- Changes in v4: - New patch for v4 arch/riscv/mm/cacheflush.c | 4 +++- arch/riscv/mm/tlbflush.c | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/arch/riscv/mm/cacheflush.c b/arch/riscv/mm/cacheflush.c index 47c485bc7df0..f7933ae88a55 100644 --- a/arch/riscv/mm/cacheflush.c +++ b/arch/riscv/mm/cacheflush.c @@ -21,7 +21,9 @@ void flush_icache_all(void) { local_flush_icache_all(); - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + return; + else if (riscv_use_sbi_for_rfence()) sbi_remote_fence_i(NULL); else on_each_cpu(ipi_remote_fence_i, NULL, 1); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 2f18fe6fc4f3..37b3c93e3c30 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -73,7 +73,9 @@ static void __ipi_flush_tlb_all(void *info) void flush_tlb_all(void) { - if (riscv_use_sbi_for_rfence()) + if (num_online_cpus() < 2) + local_flush_tlb_all(); + else if (riscv_use_sbi_for_rfence()) sbi_remote_sfence_vma_asid(NULL, 0, FLUSH_TLB_MAX_SIZE, FLUSH_TLB_NO_ASID); else on_each_cpu(__ipi_flush_tlb_all, NULL, 1); From patchwork Tue Jan 2 22:00:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184527 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4709686dyb; Tue, 2 Jan 2024 14:14:49 -0800 (PST) X-Google-Smtp-Source: AGHT+IFLC5qmU5Up8UeU2Jxf6oa43VsONbIqEtDyrDqq7C8JABE3hFaQbs5+2dsS/QMjukJoLR63 X-Received: by 2002:a05:6358:a086:b0:174:f664:5771 with SMTP id u6-20020a056358a08600b00174f6645771mr14257512rwn.48.1704233689082; Tue, 02 Jan 2024 14:14:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233689; cv=none; d=google.com; s=arc-20160816; b=fp3BVsVIZG6RLJMJSsA/j13KvhRANSRC8L7oyd4RB4WrWsgvHASYt2po04sa9dHvey YKm7TVmXX7QNK0w3lEUEP9JYI92tCC8q2m9jKOeJtp+jFHezKqteHvW6TksxDzDWpL+e 4IjvOyKtIJ1AqObQOJDDeR3Kp7dcMZTHO1OOmyB9XKPqwxlhvdbhB/6AAP2VIaZb3KI8 usMvtbQD3BhByAdho4dP3GJoktfOkZX7bbOFiMviPX8q2+NpL0OuJjBJeHkRrc/9Ija5 k9+yhdoFF+WE61v+VeYjn+R3UeBnfufxNwKjIvucueD7JpC1lYZqTmxv6BCJXvm23Bt+ SyHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=LtzHkWRF5gALofCSRDnlYgSzHUyaPaF++esfIcBOkBw=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=W2D4l3FZweM/sJ6CkmMq3JkggJnhHI7o2fh+xiO1o0bBCxVMi5uSei6k9j+bFGtUuz xjCpC8LFIt/14uY+1XLbft6+hUF3LA7Xy26NafWVJf8KgbZhqafZNo7e3F8nmhvpUokV 4LsYnVuEP52RMIlWiGjtYqdUrYEjzwLK8Rt3iiIrDT+sit1Fbat3I+8TJ3dtA24i9BtB rWS7yyQ9emcA9lcfSungD4cQDCNjOYkQ51Va0tcpbi5rG/oKKdvBvT/f+I6b3c92TJ4B muXDyXPCcP0GkJ8JhbpxZ6zN5gpAdiXxhzwe1H3vHYMGFXoEPn8ETbUJicGMtOT86aRO FQAw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=DAIC4PEH; spf=pass (google.com: domain of linux-kernel+bounces-14915-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14915-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id a7-20020a056a000c8700b006d9b26765d9si15962514pfv.348.2024.01.02.14.14.48 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:14:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14915-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=DAIC4PEH; spf=pass (google.com: domain of linux-kernel+bounces-14915-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14915-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 13142B228A0 for ; Tue, 2 Jan 2024 22:06:28 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D2B981946F; Tue, 2 Jan 2024 22:01:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="DAIC4PEH" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1AEE18E1C for ; Tue, 2 Jan 2024 22:01:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pj1-f42.google.com with SMTP id 98e67ed59e1d1-28c0d8dd88bso4065989a91.2 for ; Tue, 02 Jan 2024 14:01:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232902; x=1704837702; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LtzHkWRF5gALofCSRDnlYgSzHUyaPaF++esfIcBOkBw=; b=DAIC4PEHXK6zN41yGxbL8WtnEyXB7nYGu49/aORPHuv2xTX+4si3aH4of6n+waXs+n QRpUA/BGpBPkwGoGWxZ5S9NRdjoXPFJNTiGuzXbsM+k4srZHnZZ4pj0iLwfUdSsFcDb6 PGPtfbLTo8KNJ0NVUQiJBf6s7tYRjqyobBZ8SCazTKU4VkIMM1wSSov7ZMAnJVXxqG1q OleZkGQ+YCQ0EkF1344I0kMlE1SjqEsrWAToUpADyIXqPMxJf4vXa6E0By2FNGttP7Zn yUcBaJgG8E4Pe8Hec4TCWqq+sPfy8ngGJ2wXfbAS6M+NeLPZbSJRuw1H728uJlnpPdHI +eyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232902; x=1704837702; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LtzHkWRF5gALofCSRDnlYgSzHUyaPaF++esfIcBOkBw=; b=I3nbBbO8vE2U/gaedSgg4TVm15v3A5EOz7z75h8f/gxHjBJno+gl5qp7YVXmYl8tU7 HktOFiCu1c80wp6VekZC1cKEdLlyaxq/AkVnDqljQfg0gKAFMKL8x5MEatWOxIPaGgHu w3alK/e75gRQgPe3tw9+Eggc4mrgaWTx2tVxKz18hV1ZkibDRCMW8FQY7Cn74a7z8MxS yp5OXLw0k6IaN/OE+Cnrju1T3zufnuivT5q5yRZrk50/2BKSh10+MLlvqzcj8ONT2aYN pjjE7bp0PaivvNV/VtpcxKLuzRxikUT9jkm0FeGuGglIKN2ezaZWFuX8yrIjk/kXRILP CpcQ== X-Gm-Message-State: AOJu0YxWQW8edkcra5Kaikg6tJZZrybPCprPgyD/msjNaMpJdjOO6ydm XytC8chEGN/AcRb1Dfy0Ueqgv04u/Ch4TA== X-Received: by 2002:a17:90a:740a:b0:286:7f0d:6254 with SMTP id a10-20020a17090a740a00b002867f0d6254mr6006468pjg.63.1704232902374; Tue, 02 Jan 2024 14:01:42 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:42 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 05/12] riscv: mm: Combine the SMP and UP TLB flush code Date: Tue, 2 Jan 2024 14:00:42 -0800 Message-ID: <20240102220134.3229156-6-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787018544612465714 X-GMAIL-MSGID: 1787018544612465714 In SMP configurations, all TLB flushing narrower than flush_tlb_all() goes through __flush_tlb_range(). Do the same in UP configurations. This allows UP configurations to take advantage of recent improvements to the code in tlbflush.c, such as support for huge pages and flushing multiple-page ranges. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - Merge the two copies of __flush_tlb_range() and rely on the compiler to optimize out the broadcast path (both clang and gcc do this) - Merge the two copies of flush_tlb_all() and rely on constant folding Changes in v2: - Move the SMP/UP merge earlier in the series to avoid build issues - Make a copy of __flush_tlb_range() instead of adding ifdefs inside - local_flush_tlb_all() is the only function used on !MMU (smpboot.c) arch/riscv/include/asm/tlbflush.h | 29 +++-------------------------- arch/riscv/mm/Makefile | 5 +---- 2 files changed, 4 insertions(+), 30 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 8f3418c5f172..7712ffe2f6c4 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -27,12 +27,7 @@ static inline void local_flush_tlb_page(unsigned long addr) { ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); } -#else /* CONFIG_MMU */ -#define local_flush_tlb_all() do { } while (0) -#define local_flush_tlb_page(addr) do { } while (0) -#endif /* CONFIG_MMU */ -#if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, @@ -46,26 +41,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif -#else /* CONFIG_SMP && CONFIG_MMU */ - -#define flush_tlb_all() local_flush_tlb_all() -#define flush_tlb_page(vma, addr) local_flush_tlb_page(addr) - -static inline void flush_tlb_range(struct vm_area_struct *vma, - unsigned long start, unsigned long end) -{ - local_flush_tlb_all(); -} - -/* Flush a range of kernel pages */ -static inline void flush_tlb_kernel_range(unsigned long start, - unsigned long end) -{ - local_flush_tlb_all(); -} - -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ +#else /* CONFIG_MMU */ +#define local_flush_tlb_all() do { } while (0) +#endif /* CONFIG_MMU */ #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/Makefile b/arch/riscv/mm/Makefile index 3a4dfc8babcf..96e65c571ce8 100644 --- a/arch/riscv/mm/Makefile +++ b/arch/riscv/mm/Makefile @@ -13,15 +13,12 @@ endif KCOV_INSTRUMENT_init.o := n obj-y += init.o -obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o +obj-$(CONFIG_MMU) += extable.o fault.o pageattr.o tlbflush.o obj-y += cacheflush.o obj-y += context.o obj-y += pgtable.o obj-y += pmem.o -ifeq ($(CONFIG_MMU),y) -obj-$(CONFIG_SMP) += tlbflush.o -endif obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_KASAN) += kasan_init.o From patchwork Tue Jan 2 22:00:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184520 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4706355dyb; Tue, 2 Jan 2024 14:06:43 -0800 (PST) X-Google-Smtp-Source: AGHT+IFf4ocr1lcOgBHfgFYmUF2TsB6G5dL4kAUrrxUp0/nE+ssvWYun/P61gw7ldZhx1V1SCc33 X-Received: by 2002:a05:6512:2106:b0:50e:9767:1488 with SMTP id q6-20020a056512210600b0050e97671488mr1796486lfr.46.1704233203548; Tue, 02 Jan 2024 14:06:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233203; cv=none; d=google.com; s=arc-20160816; b=ZBvEMcqxcJZFhLu2ZD3YxaBnPENOSU360yFJekKWUWuH9eYNBKAOpIMR4eXLc6kNPS 9w2dfl1DMkBMHsu7rrZuhZ3480H5jVCxcuvv8I9Z1xELVWIR5QVmAx7j9P+1sQbdBFa/ reibLGAT+0GUC/PW42mioZcn7LRoezSnwCpxOJReBnDNaWfYJC6H2Seqf/xr5vxaFlF2 ql6Uu7i+jIHdE5SMr9Lm6HDXd6h22aoictqSoDWK0qN35gujVfRhRWezwX/n7KaLBeqh /c/EBp7nhWrI5nuvOVsu2zHAV4evCCOgCbcPFqIDJFfNRiNDCSt8nXyN3KoGeylAQ++z gCnw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=6c+wYTacIgINHvokNTIzxqzbFxSPL5G6VDrdebR45nU=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=kEnxOCj19HFofDwCq+5hxwgPs3Z2o6Xl+aCOMIHMt9kGoR48SX45aAm8NTtmfJbSLb QtdEsWBiYjUfjQYXA1XFI/DG0ankfaeVLFrtCVDEvsLAJVd6mcbE3/EW/aIJ02+ksvgE cnkp+eTlEdd6aj7L6RRpzoAGEXy5cdRXmTfebt+BP5RTNPP65ei9m8a9C/hvVkljbbcD rnKl8p0AG7SrDVGft6G+MBLxoui12WQBj+VTd2YbrrAKy0fvyBFVVMjlzWCZ7kMMkZkT gZ28R0aP08eOiw6KXeyvlTWGUrn9yakXJYSPwfMmVWKLhJMtBjQeW1cX3tCf4a8b6JVz zNdA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=HP1wacbR; spf=pass (google.com: domain of linux-kernel+bounces-14916-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14916-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id v20-20020a50a454000000b0054cd1a2460asi11815005edb.672.2024.01.02.14.06.43 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:06:43 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14916-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=HP1wacbR; spf=pass (google.com: domain of linux-kernel+bounces-14916-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14916-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 00AE01F21AFF for ; Tue, 2 Jan 2024 22:06:43 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 05230199B6; Tue, 2 Jan 2024 22:01:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="HP1wacbR" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pg1-f178.google.com (mail-pg1-f178.google.com [209.85.215.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07A7418EA1 for ; Tue, 2 Jan 2024 22:01:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pg1-f178.google.com with SMTP id 41be03b00d2f7-53fa455cd94so2562418a12.2 for ; Tue, 02 Jan 2024 14:01:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232903; x=1704837703; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6c+wYTacIgINHvokNTIzxqzbFxSPL5G6VDrdebR45nU=; b=HP1wacbRDgJCzBv7AGUbFOLF9b1VWCqm1Dfy4j8RUI11g4HsIFZaPeN2dsxBhW64Oh HMp2yVVjhGcS3SVOsqJn6mI9NV+3gzBTAlo9QY9MowPlpIM0mq4/ft/JxoFh3xZ5Mrf7 EiR0pvh1HashwVTzQI5rlrk8FQVzG56V9eVwCjuMN2hsAS4Eqab5Rq+0S67UoBSD3tKr krCs+qnAjt16JeI5rs0ovGOpNiE5LlV/wNvluwlCzV78Ara8ksKlyKMDKmS/MuC/E7KS 6Y6vpwcTtP08CLaQ0EYhOk4N1nhM8ZGxMa1NLVETUNdPSEvD0WUXaaalr0VOCo88aPgU 5NZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232903; x=1704837703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6c+wYTacIgINHvokNTIzxqzbFxSPL5G6VDrdebR45nU=; b=bElJ4hXujPMdCM+Feux5OoLSe3nZ7Ed12G60a4UM/o3+Dgh8uKVCPFoRRkWDtr3toh xul84wcIDyVZxhTRbv50gxomQFc2T0NV655IY2dRGfT0oY2pHJ1roErFcoFOw/Ne/7+L 30sctMeUCSqmPtj47GtWUSvqxJCcqoK0bGb5feOldS19cE78HmkqBAAHydX6Kxxjy4LC lPPKDow7FedXQTvwIObgiCAcpupLy/150T4GfV62WdU3s+XYHd8lrcbbcUhmghGALJRz JWi/JQ+q6cPgd5xR/9lt0ksA293Utr46Ve3kMop52cN6ryenzgNaqXP8WcuOexH5PKCE UZnA== X-Gm-Message-State: AOJu0YxMRd9rULtoF14/eQKo2By0JS3q9kmyNnKDW0vfaCkUFi3CzXj7 r3oIJiAECOY7d/9xfRJH+jKvGByx+eH1qQ== X-Received: by 2002:a17:902:d584:b0:1d4:5f9:d3d7 with SMTP id k4-20020a170902d58400b001d405f9d3d7mr8348439plh.127.1704232903384; Tue, 02 Jan 2024 14:01:43 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:43 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 06/12] riscv: Apply SiFive CIP-1200 workaround to single-ASID sfence.vma Date: Tue, 2 Jan 2024 14:00:43 -0800 Message-ID: <20240102220134.3229156-7-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787018035322620713 X-GMAIL-MSGID: 1787018035322620713 commit 3f1e782998cd ("riscv: add ASID-based tlbflushing methods") added calls to the sfence.vma instruction with rs2 != x0. These single-ASID instruction variants are also affected by SiFive errata CIP-1200. Until now, the errata workaround was not needed for the single-ASID sfence.vma variants, because they were only used when the ASID allocator was enabled, and the affected SiFive platforms do not support multiple ASIDs. However, we are going to start using those sfence.vma variants regardless of ASID support, so now we need alternatives covering them. Signed-off-by: Samuel Holland --- (no changes since v2) Changes in v2: - Rebase on Alexandre's "riscv: tlb flush improvements" series v5 arch/riscv/include/asm/errata_list.h | 12 +++++++++++- arch/riscv/include/asm/tlbflush.h | 19 ++++++++++++++++++- arch/riscv/mm/tlbflush.c | 23 ----------------------- 3 files changed, 29 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/errata_list.h b/arch/riscv/include/asm/errata_list.h index 83ed25e43553..6781460ae564 100644 --- a/arch/riscv/include/asm/errata_list.h +++ b/arch/riscv/include/asm/errata_list.h @@ -44,11 +44,21 @@ ALTERNATIVE(__stringify(RISCV_PTR do_page_fault), \ CONFIG_ERRATA_SIFIVE_CIP_453) #else /* !__ASSEMBLY__ */ -#define ALT_FLUSH_TLB_PAGE(x) \ +#define ALT_SFENCE_VMA_ASID(asid) \ +asm(ALTERNATIVE("sfence.vma x0, %0", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (asid) : "memory") + +#define ALT_SFENCE_VMA_ADDR(addr) \ asm(ALTERNATIVE("sfence.vma %0", "sfence.vma", SIFIVE_VENDOR_ID, \ ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ : : "r" (addr) : "memory") +#define ALT_SFENCE_VMA_ADDR_ASID(addr, asid) \ +asm(ALTERNATIVE("sfence.vma %0, %1", "sfence.vma", SIFIVE_VENDOR_ID, \ + ERRATA_SIFIVE_CIP_1200, CONFIG_ERRATA_SIFIVE_CIP_1200) \ + : : "r" (addr), "r" (asid) : "memory") + /* * _val is marked as "will be overwritten", so need to set it to 0 * in the default case. diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 7712ffe2f6c4..002c4c2620f3 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -22,10 +22,27 @@ static inline void local_flush_tlb_all(void) __asm__ __volatile__ ("sfence.vma" : : : "memory"); } +static inline void local_flush_tlb_all_asid(unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ASID(asid); + else + local_flush_tlb_all(); +} + /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { - ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); + ALT_SFENCE_VMA_ADDR(addr); +} + +static inline void local_flush_tlb_page_asid(unsigned long addr, + unsigned long asid) +{ + if (asid != FLUSH_TLB_NO_ASID) + ALT_SFENCE_VMA_ADDR_ASID(addr, asid); + else + local_flush_tlb_page(addr); } void flush_tlb_all(void); diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 37b3c93e3c30..292d7cf3c4f6 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -7,29 +7,6 @@ #include #include -static inline void local_flush_tlb_all_asid(unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma x0, %0" - : - : "r" (asid) - : "memory"); - else - local_flush_tlb_all(); -} - -static inline void local_flush_tlb_page_asid(unsigned long addr, - unsigned long asid) -{ - if (asid != FLUSH_TLB_NO_ASID) - __asm__ __volatile__ ("sfence.vma %0, %1" - : - : "r" (addr), "r" (asid) - : "memory"); - else - local_flush_tlb_page(addr); -} - /* * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. From patchwork Tue Jan 2 22:00:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184521 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4706407dyb; Tue, 2 Jan 2024 14:06:51 -0800 (PST) X-Google-Smtp-Source: AGHT+IH58lETt3eVVuBFhoK/mgo+NnnIJm7vfnv85C7IgLwgmegUQmeMuJ4xC33qmPiQYPzAlYTy X-Received: by 2002:a05:620a:178d:b0:77f:6810:dcff with SMTP id ay13-20020a05620a178d00b0077f6810dcffmr218699qkb.28.1704233211379; Tue, 02 Jan 2024 14:06:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233211; cv=none; d=google.com; s=arc-20160816; b=LCVSvlL/OBFGaQv2Iv0Dkotkp4upOA64sXezR/9GN/hJOTDJ1Vih95GSPYPy0zP0Tb slaVKCDngehpgHIvRAj+7QtO7Wt5Nb8bqfzVcviVlVzeVZI6DnQnVu1SifVejiFpQNN+ K04Pfbzp/T1cx0ZDm+bGCqfQAbbMCyIeYBalLuowmYdcW1HLxyFYTKv3IGKrLNgjdxyf 1BvPzG2nJX3pnoQA9nxCDJGQGef+HXOKfHyUNWxmlAbS2N1FarFNXOEO2ohpmfhWUaDX vMYus5Y3KCVLZEhn63VO2Otg8vn36fdn+ERrSe994ArCSfvuaoCLZVHOTiMPmWLxYx7Y ICmg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=1tIOva8KFaIwh3i+98q+ysPtJyT8vpZl2kFy86xHGzc=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=WTuYFYbtW8WLEYmvwro1otb6E3/fiIIkTVdaCPgm+CF4BfOR5n60Xig6prPVeOmQzw 1NN+RZpVGcp5K+4J/vh/dq+3IiC4nNgaBgEDAsjlXgcVsWDKWmhg5bi33w/A3TntBqvz Fr5WIUPOeJaIQqJa0M1R0PipKdR0So4PHwCtVAmH0FuGY8eETPWj733CO80UOJFpyRc/ E/AFl9lGXHCjsMmApBd67DKvTYgcXbS9QsHa059zhMQNMwrdnqGIS36JRYQKVgAWmGaL nDXjlzgfFpUaqMbIkDKAMpkq9VL43eUdCsDQREvFwnK4kEEJHh/jldqNvw20VzlP0UK/ uQUA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=TP2bfssn; spf=pass (google.com: domain of linux-kernel+bounces-14917-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14917-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id q9-20020a05620a038900b0077d6c62779fsi25961621qkm.206.2024.01.02.14.06.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:06:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14917-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=TP2bfssn; spf=pass (google.com: domain of linux-kernel+bounces-14917-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14917-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 2E4261C22C40 for ; Tue, 2 Jan 2024 22:06:51 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id C753A18B04; Tue, 2 Jan 2024 22:01:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="TP2bfssn" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E460D18EC7 for ; Tue, 2 Jan 2024 22:01:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-28c0df4b42eso5243380a91.1 for ; Tue, 02 Jan 2024 14:01:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232904; x=1704837704; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1tIOva8KFaIwh3i+98q+ysPtJyT8vpZl2kFy86xHGzc=; b=TP2bfssnGxqJPMtzP4cLt7lxFe+FsXfD55VmnK823h/myUpbpnj3vf8GqR5ZQ0yfqI 6dnsPojh9MFPy8oWLimIWNgwCwX8DLcAZWmnHp5zdTI+jBQJHMFTBB7kgxMunmrYGCrr Rtu9tCidaSaWo7IRhQ5giGKYdxsnxoFFYRwOGG/2CmjILC910XSkSXvUDNE8t8tALJfN l7vUcJyOsFKIls+6W1a9xmQkpLt2lRR+zzPjypvne2gz24wELloosE3mIuyHI6Al+DJd VQqWeY0Bc4PDtMOHsnZ8TR01/wQ5Oy50fDyWDo6aGWbbt9IAha9EjAgawjrWtTKglPsN Ik9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232904; x=1704837704; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1tIOva8KFaIwh3i+98q+ysPtJyT8vpZl2kFy86xHGzc=; b=vqq2oxBRzE+LVNo7nej4fgOORmZ8IKBiuyGkRMYOJy4zlilKaOOI4lr7I5OlOvQH5U FheWZSIw8X8n0gRsx3Dy72+9F5gllC3OcsFlWw6RjC/HVv4vVEti+dEcPWnGOlItNlhl BxJ/dB0lOYqy18FmCqaU3WJJ7+FDfMf96oz2V72IVh1tJ59u2m60qhl5sYx3sTczroJF av5MzzrV3bFop9WReXqLI2brp6R/htpapKqaxJBRoXkwGWHJR2fjANy4xPiDMAtSskTP uk1FiWJa7lX9JGC0H3lmrulAh1mXFRRwegKd3uWG7c1ZeZgZQMqvV2Ad2Rmx3jYb6tuR r44w== X-Gm-Message-State: AOJu0Yy1hJb7pv2KXMqF95R8vRUFT4AprjeQihesJIQCo9gRq8cAgfB1 YS5PzqhN8Bgx6Hzo9ehY9nVMFI8xYNmXyQ== X-Received: by 2002:a17:90a:d90b:b0:28c:300d:d858 with SMTP id c11-20020a17090ad90b00b0028c300dd858mr127429pjv.20.1704232904383; Tue, 02 Jan 2024 14:01:44 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:44 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 07/12] riscv: Avoid TLB flush loops when affected by SiFive CIP-1200 Date: Tue, 2 Jan 2024 14:00:44 -0800 Message-ID: <20240102220134.3229156-8-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787018043846003139 X-GMAIL-MSGID: 1787018043846003139 Since implementations affected by SiFive errata CIP-1200 always use the global variant of the sfence.vma instruction, they only need to execute the instruction once. The range-based loop only hurts performance. Signed-off-by: Samuel Holland --- Changes in v4: - Only set tlb_flush_all_threshold when CONFIG_MMU=y. Changes in v3: - New patch for v3 arch/riscv/errata/sifive/errata.c | 5 +++++ arch/riscv/include/asm/tlbflush.h | 2 ++ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/riscv/errata/sifive/errata.c b/arch/riscv/errata/sifive/errata.c index 3d9a32d791f7..716cfedad3a2 100644 --- a/arch/riscv/errata/sifive/errata.c +++ b/arch/riscv/errata/sifive/errata.c @@ -42,6 +42,11 @@ static bool errata_cip_1200_check_func(unsigned long arch_id, unsigned long imp return false; if ((impid & 0xffffff) > 0x200630 || impid == 0x1200626) return false; + +#ifdef CONFIG_MMU + tlb_flush_all_threshold = 0; +#endif + return true; } diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index 002c4c2620f3..d9913590f82e 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -58,6 +58,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end); void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); #endif + +extern unsigned long tlb_flush_all_threshold; #else /* CONFIG_MMU */ #define local_flush_tlb_all() do { } while (0) #endif /* CONFIG_MMU */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 292d7cf3c4f6..76b24d4ed4ab 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -11,7 +11,7 @@ * Flush entire TLB if number of entries to be flushed is greater * than the threshold below. */ -static unsigned long tlb_flush_all_threshold __read_mostly = 64; +unsigned long tlb_flush_all_threshold __read_mostly = 64; static void local_flush_tlb_range_threshold_asid(unsigned long start, unsigned long size, From patchwork Tue Jan 2 22:00:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184530 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4710694dyb; Tue, 2 Jan 2024 14:17:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IFRb+7h5kCWSckOUGOFsjPYxJrBK5WxJz0fK98sNHvmbMU9EoJgmfm1U5LaK6ySDqzxE05w X-Received: by 2002:a17:90a:7108:b0:28b:9b19:315c with SMTP id h8-20020a17090a710800b0028b9b19315cmr8736691pjk.61.1704233842520; Tue, 02 Jan 2024 14:17:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233842; cv=none; d=google.com; s=arc-20160816; b=s4NPluLvywg1V3A4Mf+d+J72mRyx+cq5aAveSUMkG1XytcbEGuh525v6jphuDHQ5jM ZK+k0cwtvybTZx2SAztmg3eYRkFtTw6VuCEsXh9co/MtYc/QneHENiKFohizssq2UpBL D7oMaVcZQ1D87YHvLDYoQ9AZ/3SkVBrTgFay1YXwXSRNX11ueIvPxU9iTqCS/PP3LBgn 42ZNBs9fCqzmOeeYcwtVgUB2D56pk4PW08dfKFFcGBq/OnbA6+U05JbJHakxHH6RXsWc Cb5fe7Y9THpnYKUbwNV5QWLPNVyGoZ2/c5GCYxxnDPFL8x1xpbWm5cbDp7xUsHh8Sn9g Gfdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=d7sRColwM16ZDpkHsKhbx/H1UjX1V2ZP/gk6hKZVk/k=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=v0VYKV1cOYzZDJtjzQg2pq1mMnKpOxeQPHN2z3hxggqPBCkyeGDBnmkn6ODGsIPGbk xT3G/y1K58ThFFkRfNJuPKCeiAbTUETqR+vqIsc9yKb4pvWWnlFmywed4qdKVf/baB4g ztH3ynsnxNWMxmwHcHvuhBOxLnp+oZQFGeXQuKqOXx+ytpa4bXGYjW5o/ver2gT13gPT oLk2fh+whR7n0IndWNHNPZRfnCcpF4wYJDvE/twXIJJ9+Og/8rDrhmhMg+rLSNOoNV1S to/CblGMy/ObBcbHX0aFYoRqDcQweZ9z5WfD3y7acGwQaqzb5tOnrdZF+NOCot3CKdh9 Mi1Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b="bfZ/XZFv"; spf=pass (google.com: domain of linux-kernel+bounces-14918-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14918-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id bt7-20020a17090af00700b00278f6d616aasi140500pjb.71.2024.01.02.14.17.22 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:17:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14918-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b="bfZ/XZFv"; spf=pass (google.com: domain of linux-kernel+bounces-14918-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14918-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 12714285BBF for ; Tue, 2 Jan 2024 22:07:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D3E3219BDE; Tue, 2 Jan 2024 22:01:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="bfZ/XZFv" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9E8619471 for ; Tue, 2 Jan 2024 22:01:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-5ce2170b716so4730641a12.1 for ; Tue, 02 Jan 2024 14:01:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232905; x=1704837705; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d7sRColwM16ZDpkHsKhbx/H1UjX1V2ZP/gk6hKZVk/k=; b=bfZ/XZFvnuIlbRaPhiMl3TU27P/xztx2cS1Kj3kl7/uEFDQZZurqnQ7s5bw2wKOnts EnnRARcDRCwsp1t0lPh/w3kOVUQ4wm3Hx6w2VbuBDZLSO1tJqkqgm65OVRk3jsXb4J5K 3bp3foZ24OKfppgCAsSRyyj0zx1SMWtrD/sdH7EgHQdwKNgI+AKhKQDeL0lBLjBnALX8 05mhfIeLGkoP519SliEsfudgx3id0R2pYdMK8dumUCCU9u33jNZGrvKewxWtDJ6vx4Mt 9KI+Cy0mY9j6YvYVbRnNgty3tyzrf75iSVY6xQwb+4cbzVDAlWs9nh5890ZpAKZSIvSB Fx9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232905; x=1704837705; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d7sRColwM16ZDpkHsKhbx/H1UjX1V2ZP/gk6hKZVk/k=; b=GYHt4aHlJpW+vD6HieHAqP160JSB2b8xLci43pfadxe4Exd/2lxcVaIaoFei2kxqAO earJhVHLT8SVGYWzpytPjOPlAdk50o0QLI/EE91sV93SGHhUZKlUQ+3CVZ0WtPgPjnPI kTqw8t2SRxMLoJ4SFe3ichrk1qipcvOsOgUksQelpRH8xYTEEu3s2ebv0zvnr6c3zY9I mHJFpNuH6l7qQi1XHMeUTXNJ6qXi2PcdzbYuc2m51ViA/b3kYS8aOrAe8b15cvl4FSr2 ZXOrz1LbVZ5H3dp0f4swlEwCvJ7cmNYMDWPmoBjtksDNvJtdql80zISiUrK60IUmFSWT t4OA== X-Gm-Message-State: AOJu0YwHMJpX88ccFdCAwTO9ta2UntQQsaIisRTbZFwMcwc8EOw4NlTp LcLMyK+VQetkz2ZH5ShSo4flF+QvkfrT7ft940nhVjaDtoU= X-Received: by 2002:a05:6a20:a107:b0:197:2fc0:7e25 with SMTP id q7-20020a056a20a10700b001972fc07e25mr2866566pzk.3.1704232905452; Tue, 02 Jan 2024 14:01:45 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:45 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 08/12] riscv: mm: Introduce cntx2asid/cntx2version helper macros Date: Tue, 2 Jan 2024 14:00:45 -0800 Message-ID: <20240102220134.3229156-9-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787018705793086785 X-GMAIL-MSGID: 1787018705793086785 When using the ASID allocator, the MM context ID contains two values: the ASID in the lower bits, and the allocator version number in the remaining bits. Use macros to make this separation more obvious. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/include/asm/mmu.h | 3 +++ arch/riscv/mm/context.c | 12 ++++++------ arch/riscv/mm/tlbflush.c | 2 +- 3 files changed, 10 insertions(+), 7 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index 355504b37f8e..a550fbf770be 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,6 +26,9 @@ typedef struct { #endif } mm_context_t; +#define cntx2asid(cntx) ((cntx) & asid_mask) +#define cntx2version(cntx) ((cntx) & ~asid_mask) + void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); #endif /* __ASSEMBLY__ */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 217fd4de6134..43d005f63253 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -81,7 +81,7 @@ static void __flush_context(void) if (cntx == 0) cntx = per_cpu(reserved_context, i); - __set_bit(cntx & asid_mask, context_asid_map); + __set_bit(cntx2asid(cntx), context_asid_map); per_cpu(reserved_context, i) = cntx; } @@ -102,7 +102,7 @@ static unsigned long __new_context(struct mm_struct *mm) lockdep_assert_held(&context_lock); if (cntx != 0) { - unsigned long newcntx = ver | (cntx & asid_mask); + unsigned long newcntx = ver | cntx2asid(cntx); /* * If our current CONTEXT was active during a rollover, we @@ -115,7 +115,7 @@ static unsigned long __new_context(struct mm_struct *mm) * We had a valid CONTEXT in a previous life, so try to * re-use it if possible. */ - if (!__test_and_set_bit(cntx & asid_mask, context_asid_map)) + if (!__test_and_set_bit(cntx2asid(cntx), context_asid_map)) return newcntx; } @@ -168,7 +168,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) */ old_active_cntx = atomic_long_read(&per_cpu(active_context, cpu)); if (old_active_cntx && - ((cntx & ~asid_mask) == atomic_long_read(¤t_version)) && + (cntx2version(cntx) == atomic_long_read(¤t_version)) && atomic_long_cmpxchg_relaxed(&per_cpu(active_context, cpu), old_active_cntx, cntx)) goto switch_mm_fast; @@ -177,7 +177,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) /* Check that our ASID belongs to the current_version. */ cntx = atomic_long_read(&mm->context.id); - if ((cntx & ~asid_mask) != atomic_long_read(¤t_version)) { + if (cntx2version(cntx) != atomic_long_read(¤t_version)) { cntx = __new_context(mm); atomic_long_set(&mm->context.id, cntx); } @@ -191,7 +191,7 @@ static void set_mm_asid(struct mm_struct *mm, unsigned int cpu) switch_mm_fast: csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | - ((cntx & asid_mask) << SATP_ASID_SHIFT) | + (cntx2asid(cntx) << SATP_ASID_SHIFT) | satp_mode); if (need_flush_tlb) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 76b24d4ed4ab..5ec621545c69 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -85,7 +85,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, return; if (static_branch_unlikely(&use_asid_allocator)) - asid = atomic_long_read(&mm->context.id) & asid_mask; + asid = cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask = cpu_online_mask; } From patchwork Tue Jan 2 22:00:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184523 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4706701dyb; Tue, 2 Jan 2024 14:07:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IG5GoqwHx5C4Kt/msbAUwju1gJvWsvn8c68+FTrlW7HedNRUbQoKB6lM2LD3tZkbChWOU3r X-Received: by 2002:a17:906:74cc:b0:a28:61f7:bbdb with SMTP id z12-20020a17090674cc00b00a2861f7bbdbmr524931ejl.71.1704233248489; Tue, 02 Jan 2024 14:07:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233248; cv=none; d=google.com; s=arc-20160816; b=rPs4Y4xWtIHNxbkCmzqdXw9pn9hTk+k/TWdcFlIlgxhZq+Svr7VzWjYTYcVCHso7Nh GkcEBGEK3LKX8B0/4aP9eyCouAR3AwGqohOVzCRhv0diY4qzXUV0uw4R2nmKfQW1X86u XIB04NoInoZln++LQwRnr7z5tUDafaF1pzPpGs5noZCkfYkrWKW6cqhKoixhSruaQ44m 6BuIVkjdg0L6RiasVkAQv46O/iaDdtmUQw20UOpf+2Yu4znFvmy9CDA5PFwlGJ4hIo2W uGEu/UxLJttJ5CO3CYeKCeh5IKLtWiP5eHXdwKQ1nMu/9N0zLzEE+CF759jYhGE/VzwC Xe3w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=g2cmoIK/tf9yPohv/YvO5QQvGunkgXtONjIGDQZEX58=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=yiPfXZKBlql8bVN48WB7PeGNvZGT3IzR2NkYF9s9K7PtSr02ufKnu/TOaI/W8aS6Le 8WJJElCTZVDYm3BxssztTE5GuB4u+DRS1w9uDoUWTobERWWI4K55DlfxZ7pzAK+2HOwq YUejx/3SoUggKNaqEP+XqrqN06p0VhB0cuu3G5UmM84/QpmFiIODIP3BIpEpFIZwtsdV uf8HiTVrXxVTxIA47dEuzr0huOCafDZRHYU9YJoJx4EXlnoUNmrQ9QAdrISbdYXxpqDt +pauBs9oPP2Fyaql5aREkeewrNhkHWIqmIzsKJaNANU03pfXmY9avr7voHqVV54Zl8T1 PMig== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=NKqqWxPa; spf=pass (google.com: domain of linux-kernel+bounces-14919-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14919-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id ly12-20020a170906af4c00b00a2855f23902si692708ejb.508.2024.01.02.14.07.28 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:07:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14919-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=NKqqWxPa; spf=pass (google.com: domain of linux-kernel+bounces-14919-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14919-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id E88A31F21AFF for ; Tue, 2 Jan 2024 22:07:27 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1E35B1A287; Tue, 2 Jan 2024 22:01:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="NKqqWxPa" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1AB461947A for ; Tue, 2 Jan 2024 22:01:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-28bc7155755so3952296a91.2 for ; Tue, 02 Jan 2024 14:01:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232906; x=1704837706; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g2cmoIK/tf9yPohv/YvO5QQvGunkgXtONjIGDQZEX58=; b=NKqqWxPaj9rcV1IWAEDEU5QgBO1QwiNOgePEN9zSk7nzNbpWa44amkz2NbAO6NLdHJ MLMvcUQ4n3FWjNLww45EzULW9lgASED1bsOKehFf9/sLOsXYlIsRRPLlMWQPAzz3pndL N3I9aLOTxJ+0sXdLCvpQlk6IuLCik+2236voMxhb8/Yz1qGbOIdAK34fcs8SFsBFW5VT xtncr81K2RPu27HOCTYsAS1XUumIxuNDtA78pj/zp/T45YydmbEu2OfyEOxCJ9S1a79i BdTjkNPY9HL72mdjFvJq4nMloRpUIfk37237cZfmt+vARzPch6FZxki2j+JN9RFiJmL8 u8+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232906; x=1704837706; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=g2cmoIK/tf9yPohv/YvO5QQvGunkgXtONjIGDQZEX58=; b=g8Px5+hBA5tqkHUcGnjR9fbo46CrvGxSqphTEixYCDDbrzjj1hsJa1bgQ9y8syRDEA oEABHh3/UEyX/6+MwMmKuNEfTUxXY8ipUSMnjAL1O9lne2uke6DAP9LOx0B6ITyoGr6R 3v8WWMfjCEpCkiVtjKJaARVaS8BDZwSxaOmelm1urKlqrDYnNxgH8H0/0sLaPIgnMoaJ s4T/+agPjWY+r+ne2+whoeHY6znjjNMwQzmo+BLT9YD0ad8FbpvWsTPNKI1+G1LZlvlc kzppI4BuycbVXGDfyXe1IC9CrmeW80VaipE2QThMweFL0hTSOcUgUGMEBEIiBKL9zq1C 4wAw== X-Gm-Message-State: AOJu0YzeX87//xDNABoytTYrZxNjDKgMmTXNCFCH+rxPpa3QuEjqfGJW ueRBvoI4y/AzGxHov0W/lfjSDaeBUU/EOw== X-Received: by 2002:a17:90a:134f:b0:28c:b1bd:3c42 with SMTP id y15-20020a17090a134f00b0028cb1bd3c42mr2130687pjf.55.1704232906433; Tue, 02 Jan 2024 14:01:46 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:46 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 09/12] riscv: mm: Use a fixed layout for the MM context ID Date: Tue, 2 Jan 2024 14:00:46 -0800 Message-ID: <20240102220134.3229156-10-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787018082946029830 X-GMAIL-MSGID: 1787018082946029830 Currently, the size of the ASID field in the MM context ID dynamically depends on the number of hardware-supported ASID bits. This requires reading a global variable to extract either field from the context ID. Instead, allocate the maximum possible number of bits to the ASID field, so the layout of the context ID is known at compile-time. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/include/asm/mmu.h | 4 ++-- arch/riscv/include/asm/tlbflush.h | 2 -- arch/riscv/mm/context.c | 6 ++---- 3 files changed, 4 insertions(+), 8 deletions(-) diff --git a/arch/riscv/include/asm/mmu.h b/arch/riscv/include/asm/mmu.h index a550fbf770be..dc0273f7905f 100644 --- a/arch/riscv/include/asm/mmu.h +++ b/arch/riscv/include/asm/mmu.h @@ -26,8 +26,8 @@ typedef struct { #endif } mm_context_t; -#define cntx2asid(cntx) ((cntx) & asid_mask) -#define cntx2version(cntx) ((cntx) & ~asid_mask) +#define cntx2asid(cntx) ((cntx) & SATP_ASID_MASK) +#define cntx2version(cntx) ((cntx) & ~SATP_ASID_MASK) void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz, pgprot_t prot); diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index d9913590f82e..5bfd37cfd8c3 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -15,8 +15,6 @@ #define FLUSH_TLB_NO_ASID ((unsigned long)-1) #ifdef CONFIG_MMU -extern unsigned long asid_mask; - static inline void local_flush_tlb_all(void) { __asm__ __volatile__ ("sfence.vma" : : : "memory"); diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43d005f63253..b5170ac1b742 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -22,7 +22,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long asid_bits; static unsigned long num_asids; -unsigned long asid_mask; static atomic_long_t current_version; @@ -128,7 +127,7 @@ static unsigned long __new_context(struct mm_struct *mm) goto set_asid; /* We're out of ASIDs, so increment current_version */ - ver = atomic_long_add_return_relaxed(num_asids, ¤t_version); + ver = atomic_long_add_return_relaxed(BIT(SATP_ASID_BITS), ¤t_version); /* Flush everything */ __flush_context(); @@ -247,7 +246,6 @@ static int __init asids_init(void) /* Pre-compute ASID details */ if (asid_bits) { num_asids = 1 << asid_bits; - asid_mask = num_asids - 1; } /* @@ -255,7 +253,7 @@ static int __init asids_init(void) * at-least twice more than CPUs */ if (num_asids > (2 * num_possible_cpus())) { - atomic_long_set(¤t_version, num_asids); + atomic_long_set(¤t_version, BIT(SATP_ASID_BITS)); context_asid_map = bitmap_zalloc(num_asids, GFP_KERNEL); if (!context_asid_map) From patchwork Tue Jan 2 22:00:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184524 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4706809dyb; Tue, 2 Jan 2024 14:07:39 -0800 (PST) X-Google-Smtp-Source: AGHT+IEZXQbB9KuC8BkK5ZHehGRLsHczvalzjEYlUvqTZvozaimvnnTevb7wsLwoGKSNhIcOq25o X-Received: by 2002:a17:906:10d1:b0:a27:ff1d:2d72 with SMTP id v17-20020a17090610d100b00a27ff1d2d72mr1794600ejv.90.1704233259727; Tue, 02 Jan 2024 14:07:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233259; cv=none; d=google.com; s=arc-20160816; b=woplVphCTpV1Ut3M8nEv33IE8IprnpoEbUAI2xoK+UtVYFB6/HPS2VK1TaLTHJujV5 vkAqFIp/B/T+KhkUekNCcVOzqaBQVWVLBQ2gPxRcepkv/kKaHBEboFOaKDHeKiC000R8 eoGlYgxnQY2Hl253AHStDB+geZ/vsKRkvAADDiivDI+jl5Bl7qxGWY9hMwjscyF2GibU MhzPwnYT6nzUvn3/nsTMuSxPMYDS03vgZRd4esmKQ7TnNEZ357Yugw9kUUaZvnxPElGJ IcKwTOlRI0mCqv1Osn8wk4Yq4Kca5RKTjslNj+2mlGPqOcZ5M6hbb5PsHkYHa84FiqVy XoFg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=FlojwujQvNxoRTzQVfrIJOgCsWHsQN1qDJgE313X4QGokGdzwCoIfTsvchIpV4w4zo yDelGB81639X2Xsid+eO+RKRCJvMzQBLBEzkjdT/QXZayGaZY0yJW6Jm3cfOY99yEIhS vGEWmeF2GhYZ+BBmt7sV+xzt21/8Gq2OrhWJaWfqOoTqGX7/V0ANfJY1ACrYQWRUDg2g YOM7OYGYrn10jLPT7Lsu8lbHgDrrUmHcN8mthpuEcPBNpWggnl2htsNSBe8tRuhzaSqL KpzIBrTaqUoiv+bqwV40PLmBJ8ZO2KwgqTDgoaE/NrDX79y/K2hzIl+TCfqMxPQtTrOc nVUg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=Fq3hbOFI; spf=pass (google.com: domain of linux-kernel+bounces-14920-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14920-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id wj6-20020a170907050600b00a2697f77293si10098174ejb.232.2024.01.02.14.07.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:07:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14920-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=Fq3hbOFI; spf=pass (google.com: domain of linux-kernel+bounces-14920-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14920-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 593901F234AF for ; Tue, 2 Jan 2024 22:07:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B77891A594; Tue, 2 Jan 2024 22:01:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="Fq3hbOFI" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7B6F199B3 for ; Tue, 2 Jan 2024 22:01:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-28bc7155755so3952305a91.2 for ; Tue, 02 Jan 2024 14:01:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232907; x=1704837707; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=Fq3hbOFITwhrCF1xo3JKafxvDiY2hTS9MG3k7e4U6FsfOrftpP2sgWuYYotBePj36L r7sCj50iDICiKIuU1qwmr09yvMpY20VBxLYXG6yxBqvnvwQPjeC0V7wW6LpAEulpXfiE v3Rz8ObmvE59Bu3mCceu1lmwgUwSHxmcM3KheDBO2Bmz2pZ00PTqqqC/NBcNH2AGHXrt ND4lYHzancP0+KLfkwpx7ENF8Yp1DWN4O9hCWx2N3pse78IR1KJZszpuBAtog0GqQ1M9 IIHzSvnEP9FlF50k0LH9O7g8vqLSyjk/7xHMrW4TFcp5v9AsH5gaJ73h2GIzwZb6UHoj vzfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232907; x=1704837707; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O8uQuAx5HbeGrqX4Ip/qQY7nv6lrxw4qacCJFbpKbko=; b=aiYRk/sAsIbXlwN2X6acgr4GekvqfPejipnlsTraEbESE0JsuRhKjP0ZfgNPH2pLId JrkrBroK4KvwCSNpSBrQnZqYA87pR8VldWoNc2fLvmcSHwbjwZT1UgOxWYlOhCg55zpc AKFnPTLXuCopmurQVyTePWO49C6GkqsNWQteuJtntoRB3gwVHzoHQOGgmytlwe5LIx9w H2RfUP+3iZuc/eXehSonT98k13J/ezCNWZRavWpv+KLnYDePJO5TKZ6LNI+xtte113Gx XxtlAe7n53zjVhIauTU7UpQUwcuOt+AXAWMTpScjBtNX07uE8Vq4H3cpnQruOiE/CzDe b5oA== X-Gm-Message-State: AOJu0YwC0s7Hbc9S/XXeGyZTNRlbMZC3fMjmmPwJn6M40U5leNSMeRdm zPbu7CAwxVoD1RAW4bNWdS1sVjbfi1EiUkgbFEq0z2pD/Zs= X-Received: by 2002:a17:90b:33cc:b0:28a:f2cc:e008 with SMTP id lk12-20020a17090b33cc00b0028af2cce008mr6619129pjb.54.1704232907387; Tue, 02 Jan 2024 14:01:47 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:47 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 10/12] riscv: mm: Make asid_bits a local variable Date: Tue, 2 Jan 2024 14:00:47 -0800 Message-ID: <20240102220134.3229156-11-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787018094525113801 X-GMAIL-MSGID: 1787018094525113801 This variable is only used inside asids_init(). Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/mm/context.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index b5170ac1b742..43a8bc2d5af4 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -20,7 +20,6 @@ DEFINE_STATIC_KEY_FALSE(use_asid_allocator); -static unsigned long asid_bits; static unsigned long num_asids; static atomic_long_t current_version; @@ -226,7 +225,7 @@ static inline void set_mm(struct mm_struct *prev, static int __init asids_init(void) { - unsigned long old; + unsigned long asid_bits, old; /* Figure-out number of ASID bits in HW */ old = csr_read(CSR_SATP); From patchwork Tue Jan 2 22:00:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184529 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4710314dyb; Tue, 2 Jan 2024 14:16:18 -0800 (PST) X-Google-Smtp-Source: AGHT+IGi+w3lH4RXa/jzIScPBIZxWBvmgvskAUYKBkGcaDoY8xwn7/WaEHUlHA2j8k+OmW4ApBUL X-Received: by 2002:a17:903:4289:b0:1d3:f2ec:a43f with SMTP id ju9-20020a170903428900b001d3f2eca43fmr7681740plb.27.1704233778616; Tue, 02 Jan 2024 14:16:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233778; cv=none; d=google.com; s=arc-20160816; b=X9yK179ay/Q1N+DX1WhZRbwTRlBPzeIoQhwmTLY7afEQkxe05/ThuOm5X+Vs5LcqQl PSNL88J0gqHpyvcrz9ibrKKXkzPQHbt/3mzVycPzOCNA+z8eFsBRvbcMzsfDYObw0R0e emKtW/UqLzxIDQCddhOmpJHD1PgroYzTNALv2SC0IkPGozqK3LfGiORHrgRR3vZnzTKo iLD6/adkrWBoVLWrpHEGr+2DQbOPEnE0e5MmTGenLgtROhLmIheSQyzzJmpTL03GrIQ9 yDHPYul4QhcHUKN6fIo5DPBghtLR/uXh595FeRf6fl41q/YO5PESRgVNr7NZkeQJDaK+ ENtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=H+s6l04qYh/a4w1hea8IkdLxYk4K6Rnl6qCSJeSJcnLX1+Vk0iySCjpcQxCjuuMmOU VJIxSTJ5mw1/eVV6+oFZTvsvTwpZILSZmyvZ9Z4gtLDwNIoW1Rlij0ShEU3udneqkarR KMtw1gcu32pjMQemEX3DHJHxkcl7PD+UEC0XN4xG27owbYDLk06KdMIzRWRqNLlDuk6A 20C8CGQeOtulQaEoXvyuqySWnWOmOSoLuiv6VoGtoYUfX2DxFSASmVWEykBLMQJMT9mt M+UNXCsx9D/sFtDJBzFxEzOxuweXcIx9UcDxoXW4idBFIxqnhBVVgmWPjHzYjPo4qcXT EviA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=NX5xwfDX; spf=pass (google.com: domain of linux-kernel+bounces-14921-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14921-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id b18-20020a170903229200b001d4bd22adafsi3025205plh.318.2024.01.02.14.16.18 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:16:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14921-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=NX5xwfDX; spf=pass (google.com: domain of linux-kernel+bounces-14921-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14921-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 7C68AB22F47 for ; Tue, 2 Jan 2024 22:07:50 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A7D631A5B0; Tue, 2 Jan 2024 22:01:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="NX5xwfDX" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7DCF18B0D for ; Tue, 2 Jan 2024 22:01:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-28ca63fd071so1666557a91.3 for ; Tue, 02 Jan 2024 14:01:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232908; x=1704837708; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=NX5xwfDXegsIZdZeD3oMwQwMHll3o+dryBRf6MwNR6F6IX4VNOGzcoupwUi0hInOhr h6sF53qbIWdDZj6Qgu4O+pdeZ+C9CECw8Y7lwEU4TONlqJ9Q8wa8OzdNURP/c/bf2QQc ro0tzJ7vCnxgAUvDgSgLW9K4RFISSOKH43jgvpkaFvYEEtN4TMa+QSZhoNyyTWW+sgc8 YNS5kZFkfxv2lcIiMzufz2EQ8x+nGWpgKxnoG87ULrnUExvhQiFHPka+bgs7NfgAkuLe bQe19ylYJbvD8Xl448n8aB/05Vt5ui5sGcmfLmpAonKKDwtmm37SxcSLb0u68c9NTOFx rVWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232908; x=1704837708; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MROdnZxFJAn7brM6ye/jIKPJcvVCbRqlgsjUSxtP0PY=; b=UFqMNXlOfX78Q7InhSbI15FSi2sm+mJwX/taC0GkxQbM/6aP6Luk1Qahjrx3lf/gjN iMZtGkeiFeyrb3rHzlzsbkYkXCBnhlAM+L+/i8GMr2/SDhM7WofFLxSilR/+ppoKo5EA wGDCJO76jeWKxdE/eRmi8RLrvb/j8zgYiltkFIHCg/DAEc5ZY5Mkjwm/nEkr5nNFztUR PdzJbSSVkuQP+FRwXP5rl2rLhizl2qxr2j6kJBslYmh2LRqnhT8tvhLliBEbIem75xfE ZvUe37x2Ic5j9F2DtxdqjW3MROZ+ORGrujKeAmmRa3iP6CWMiflPQEpDePC2Iv/vUn0g iNGA== X-Gm-Message-State: AOJu0YxOUlChG7ngwFnuj78SiRsgv46pNcq0Z+H4pVIa8552e+AVby6V tE9EAKHBb7LD00JnNaxcApNF3hvAC3N93g== X-Received: by 2002:a17:90a:5c82:b0:28c:2e1f:8bf2 with SMTP id r2-20020a17090a5c8200b0028c2e1f8bf2mr4930306pji.84.1704232908323; Tue, 02 Jan 2024 14:01:48 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:48 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 11/12] riscv: mm: Preserve global TLB entries when switching contexts Date: Tue, 2 Jan 2024 14:00:48 -0800 Message-ID: <20240102220134.3229156-12-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787018638676306699 X-GMAIL-MSGID: 1787018638676306699 If the CPU does not support multiple ASIDs, all MM contexts use ASID 0. In this case, it is still beneficial to flush the TLB by ASID, as the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. This optimization is recommended by the RISC-V privileged specification: If the implementation does not provide ASIDs, or software chooses to always use ASID 0, then after every satp write, software should execute SFENCE.VMA with rs1=x0. In the common case that no global translations have been modified, rs2 should be set to a register other than x0 but which contains the value zero, so that global translations are not flushed. It is not possible to apply this optimization when using the ASID allocator, because that code must flush the TLB for all ASIDs at once when incrementing the version number. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- (no changes since v1) arch/riscv/mm/context.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 43a8bc2d5af4..3ca9b653df7d 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -200,7 +200,7 @@ static void set_mm_noasid(struct mm_struct *mm) { /* Switch the page table and blindly nuke entire local TLB */ csr_write(CSR_SATP, virt_to_pfn(mm->pgd) | satp_mode); - local_flush_tlb_all(); + local_flush_tlb_all_asid(0); } static inline void set_mm(struct mm_struct *prev, From patchwork Tue Jan 2 22:00:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Samuel Holland X-Patchwork-Id: 184528 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:6f82:b0:100:9c79:88ff with SMTP id tb2csp4710121dyb; Tue, 2 Jan 2024 14:15:42 -0800 (PST) X-Google-Smtp-Source: AGHT+IHEgZpcTNobqz/+Hqz66Pc/FYH254fxUL5mMP5om3CgxI5WuC5k2IEuwUncQhjcVVEKNOen X-Received: by 2002:a05:6e02:1cad:b0:360:ec9:c932 with SMTP id x13-20020a056e021cad00b003600ec9c932mr18557210ill.61.1704233742211; Tue, 02 Jan 2024 14:15:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1704233742; cv=none; d=google.com; s=arc-20160816; b=ZLTlVi6RLYrqNbfiGfEb3Kdr0JblWymEexzlUt9jPgRymwroACIPW4DRc2XsLITspL 6aESUNNWD6Fv0QjoEln74z7o8LvLDWSk05stcNK9yzlhRc/iAutuQhziwGcp8WeeKprv xJQwr32AbT53FtVNOrSt8Xc30CGtbh2oUkszNEnHmyRwQDNW1d6QIo6gWptSg6mtUFBf 9834+FjeSaVT1P6sgtqxJ7YimV94Z4JUkjYVAAdYhInRMjNXOY+7Mx2lI8N+8Ow1ss6C 3Q2oE2o5MKpB3w9OHtAJ7oKj9gtshVaho0H3KYe0hx0DyWcCpyuICJi2sx19KCmUD0kX sR9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=ntXY1BdgwMpYCVbC/qFvmg3dz3t/ZzKIeqKl93B+ekg=; fh=dlotjpk72cYcmDz0+AxIEXTz48ooVBc8lhZ+ndNwuOg=; b=wHFypB/UNREWwqxrJ+EfvpuYSn9CSsSHhp+23sYrHkGSjCVCcHMy/FS5qHShSBWeCq qOlc8v/VDwJJWQOHFMQLO+rB2FqXZczPaCYT5jSFMHja3dDTvsXtXYUvWSuiNgiZ8kfq Z29wCyM6ljK8W1x3Yh/GSHv2VKguX3SL40dt2IMkzX8fRPgFe8k9GbLB251KJLPMfNH4 29ezP3Bpx/tWh7tjdNtUF8nKW379P6zrImbR3JenITHOacun/6AUNrAEVglCJ/kguTDB wdI1MvJyHaSuSC0RPQemETZ810z/oySwaS5aZj4c1ZrwegKyBeU/D8lT0bXPAg4o/P2t ihzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=S+xnwffw; spf=pass (google.com: domain of linux-kernel+bounces-14922-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14922-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id b8-20020a63d808000000b005c660acad7asi20841789pgh.4.2024.01.02.14.15.42 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:15:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-14922-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@sifive.com header.s=google header.b=S+xnwffw; spf=pass (google.com: domain of linux-kernel+bounces-14922-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-14922-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=sifive.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 9C8B62862D8 for ; Tue, 2 Jan 2024 22:07:56 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B9AE41A703; Tue, 2 Jan 2024 22:01:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=sifive.com header.i=@sifive.com header.b="S+xnwffw" X-Original-To: linux-kernel@vger.kernel.org Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0070199DF for ; Tue, 2 Jan 2024 22:01:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=sifive.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=sifive.com Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-28ce6eeb4easo310365a91.0 for ; Tue, 02 Jan 2024 14:01:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sifive.com; s=google; t=1704232909; x=1704837709; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ntXY1BdgwMpYCVbC/qFvmg3dz3t/ZzKIeqKl93B+ekg=; b=S+xnwffwYxZ2FpgRDUEM5aG5lh6apSSIe8QE1YEuvg5HNZl2uvXS5xB3I4h68PXJtp gDoasIZL+E6vJDxsAlaAz2qiuZGnidTBKhLbKUVB9MCqyezHINS1vk9XB1P8QhCeQ/Gk 1F3vkO36JKmC6wu3aDyGwRtHq/T4uIGYQI8W7lwNy99bF6j1+kb6RN8jomX5tdXoDy0z vv5Imvd5qeKtPNXykUn7nqE3fbpEknvZSukERFlYHljwch8UPjc169Yq0S62vKZRE1+2 +sL6iybOEqjROZIlxUaOrgJTurzWQkqGwTWUSYMSkJZ+grzlapuwMTGlafSeAc4TDLZ5 juuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704232909; x=1704837709; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ntXY1BdgwMpYCVbC/qFvmg3dz3t/ZzKIeqKl93B+ekg=; b=wUFzXaOKgPQ6bpVNRdd3sXVSphJIgLT4XAs1Hg6wuk8igZ47iWWsJfoBFkPc2rw7HU X6YbehGy/PwEp08QENOlEm7ESgzXRzWS1JJeNE1CZaOwAhGYpB6+bnR7bXzt46K+FwRp LjFjU8YANW+/tBaQqBl3m9hziCm5Qn6SZooUnlwIeIiSRqXnIpRwHCaYnVbXv3toXfVv mTyRX/e6VWaw2o5Yju0ChSbfl4ZCqIHPoLZgVcO/oI3bta1WQHrxooAabJjpIJNm1bg3 ADqN3DVVKGxgOJnxIx7nssthW7NIKhsH1Z6AhZeaCqssk69jT27puFZ1AOz2WwKD0tLg bENA== X-Gm-Message-State: AOJu0Yxjvt/cT1av97KCBOP0oJOF7Cfc0U7dZzR3wCqymxwaSWrUni1I yuHESkk5hUG5ZAfGWnlSFpzncjCxxtwGQw== X-Received: by 2002:a17:90a:bc46:b0:28b:31f5:9668 with SMTP id t6-20020a17090abc4600b0028b31f59668mr5241043pjv.30.1704232909372; Tue, 02 Jan 2024 14:01:49 -0800 (PST) Received: from sw06.internal.sifive.com ([4.53.31.132]) by smtp.gmail.com with ESMTPSA id r59-20020a17090a43c100b0028ce507cd7dsm101724pjg.55.2024.01.02.14.01.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Jan 2024 14:01:49 -0800 (PST) From: Samuel Holland To: Palmer Dabbelt , linux-riscv@lists.infradead.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Alexandre Ghiti , Samuel Holland Subject: [PATCH v4 12/12] riscv: mm: Always use an ASID to flush mm contexts Date: Tue, 2 Jan 2024 14:00:49 -0800 Message-ID: <20240102220134.3229156-13-samuel.holland@sifive.com> X-Mailer: git-send-email 2.42.0 In-Reply-To: <20240102220134.3229156-1-samuel.holland@sifive.com> References: <20240102220134.3229156-1-samuel.holland@sifive.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1787018600204715972 X-GMAIL-MSGID: 1787018600204715972 Even if multiple ASIDs are not supported, using the single-ASID variant of the sfence.vma instruction preserves TLB entries for global (kernel) pages. So it is always more efficient to use the single-ASID code path. Signed-off-by: Samuel Holland Reviewed-by: Alexandre Ghiti --- Changes in v4: - There is now only one copy of __flush_tlb_range() Changes in v2: - Update both copies of __flush_tlb_range() arch/riscv/include/asm/mmu_context.h | 2 -- arch/riscv/mm/context.c | 3 +-- arch/riscv/mm/tlbflush.c | 3 +-- 3 files changed, 2 insertions(+), 6 deletions(-) diff --git a/arch/riscv/include/asm/mmu_context.h b/arch/riscv/include/asm/mmu_context.h index 7030837adc1a..b0659413a080 100644 --- a/arch/riscv/include/asm/mmu_context.h +++ b/arch/riscv/include/asm/mmu_context.h @@ -33,8 +33,6 @@ static inline int init_new_context(struct task_struct *tsk, return 0; } -DECLARE_STATIC_KEY_FALSE(use_asid_allocator); - #include #endif /* _ASM_RISCV_MMU_CONTEXT_H */ diff --git a/arch/riscv/mm/context.c b/arch/riscv/mm/context.c index 3ca9b653df7d..20057085ab8a 100644 --- a/arch/riscv/mm/context.c +++ b/arch/riscv/mm/context.c @@ -18,8 +18,7 @@ #ifdef CONFIG_MMU -DEFINE_STATIC_KEY_FALSE(use_asid_allocator); - +static DEFINE_STATIC_KEY_FALSE(use_asid_allocator); static unsigned long num_asids; static atomic_long_t current_version; diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 5ec621545c69..39d80f56d292 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -84,8 +84,7 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, if (cpumask_empty(cmask)) return; - if (static_branch_unlikely(&use_asid_allocator)) - asid = cntx2asid(atomic_long_read(&mm->context.id)); + asid = cntx2asid(atomic_long_read(&mm->context.id)); } else { cmask = cpu_online_mask; }