From patchwork Tue Nov 1 14:33:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anup Patel X-Patchwork-Id: 13689 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp3009998wru; Tue, 1 Nov 2022 07:37:25 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6xe/ZS4gHwJaBNymmPMVrMQSKGKkmDBkyfiBLIqs5C/L16lpwNMIPHUtCYZCJ0U+YPj9y4 X-Received: by 2002:a17:902:d4ce:b0:186:c8b9:c301 with SMTP id o14-20020a170902d4ce00b00186c8b9c301mr19906329plg.27.1667313445277; Tue, 01 Nov 2022 07:37:25 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667313445; cv=none; d=google.com; s=arc-20160816; b=M9uPmPIcQ4WMMmROTShHTxpILYDMwDHz09/efhdftyyxiFIEm38fll/B6VhjLvO0zq uib4lq3uqkq7F+RcMvjNIw+Igltni6j+joG1CsnhmtIzQbFj2fcMLz5lRk0zqAd41eKI RthPy4nxKEugVvDoG00wUo33DrB0Qw3DATNaasNTECFRMDgQBHN86h7hnaxvDSlLZvSZ RJUZ786uwaUWNDXNIAFaQKc9Jf1GwoTzpx/Bw5VZujIJWFffOsVhPGMyqRYT6pMZPFRU O66cAEN03vDDe3KKy8Y5kBAmTJnKlje0UX3WIePYXDtTKe5asEbKSsrNtEJQpMZMr4J2 NdTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=BjmiwAV+qkEddhUojBQVUbZIbS/TIGSArMUYW9Y9o7E=; b=YfOTkVhE8MO/lJVY+nBCF9RQd6GmWTbJI0kX1+vZCmZfpeZtmzz0uKYIFjgyVe3HDt lxiOEjbAR/6dSYPEtigza+MPpp7SxucAcudoj56R4nJoRc5Vzi1fm0kVEsexoNtjQSPp OLvv1h/7mxOsAM0QO1eqSPc66E+VH+9jme5NOyh9mmR4iXjdJcY/vfnjs66jphPL1e9a bUnP0Ok3t/lJnKvF/u2g/KnOqigB02Ham2ZfSakBlUFkmdOFwU9Xa1vLpAVE+OY/WV/4 14cfhMPb3Ewx3zyWa8nOVo5onX7zC7/6hjxRx1IfypnRS4cJOO6JrNXqUcFnwVeLErub FlTQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@ventanamicro.com header.s=google header.b=Zy5sdTrM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w184-20020a6382c1000000b0046248881576si842236pgd.157.2022.11.01.07.37.12; Tue, 01 Nov 2022 07:37:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@ventanamicro.com header.s=google header.b=Zy5sdTrM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230005AbiKAOfR (ORCPT + 99 others); Tue, 1 Nov 2022 10:35:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230331AbiKAOe7 (ORCPT ); Tue, 1 Nov 2022 10:34:59 -0400 Received: from mail-pg1-x533.google.com (mail-pg1-x533.google.com [IPv6:2607:f8b0:4864:20::533]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3039B1C119 for ; Tue, 1 Nov 2022 07:34:57 -0700 (PDT) Received: by mail-pg1-x533.google.com with SMTP id g129so13582666pgc.7 for ; Tue, 01 Nov 2022 07:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ventanamicro.com; s=google; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BjmiwAV+qkEddhUojBQVUbZIbS/TIGSArMUYW9Y9o7E=; b=Zy5sdTrMf0M3pjYa/MT2umcZ0Oy093AmxiJWAZCUyMOcrYjMskX3ojkSBUlq18/80d dEI1FtpwIJiACRD/VD5OE735DGA4vZpabreavado648cZKP9wy4GvGvZLtzu4uTKkr0n +1/wsZCJScfAmXqZEtH+EYmoYWjqKEJxyqC/bcduFnDx7FNK8WkgyXZ7Myxeo92gfMEr 9fxKUh40MhaM7KjmRXCcQRi5eh9x9mI2d2JQaxd5O3pOBaqFd4ZpfkVUfRmLb+sCdTqA MRJP6HUdEx8bdfwtqfSOX/LuMLCyMrTiilEZ88mrI+9g5+pOxNKaP8KM6/bTiU/CxK7f puug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BjmiwAV+qkEddhUojBQVUbZIbS/TIGSArMUYW9Y9o7E=; b=EFWJrsOvol7DY3eyi6ADxWGFus/jn/k16Ne8z0ovL4L4jFhcZj/RNJrdY+HW81Cj6o 4RUsrZxlOkIxMV6iG2LDSFE90WCHOE+CyVS3FqT7K3ZLabg9rtY6FpaKD4ZOyeEZHZ4d sbYE8M117//gjoQM8/GV5FHEU63/e+SfdIy3c/6+ur6rx45sG4bkg7ChQlT9INnXyFz2 n01kTNpoFQ8UwdfDO7bNhgWTvlto4cOjrGD2r9aBnmFoLTukJDLRlBFtL1k2sF8PUhjT v2WNvrBoqQgt2SE4ArEBkxyU3vHBHiMlkRmLILSvmLowbo9sZSu4ZNisTs0X+3MY0kDc 2rJQ== X-Gm-Message-State: ACrzQf09amScztV17IQ/UaNcmWnTonrRMOM/XuQLzdxG8mwyG46lmH1E KR9Hd3RG930mTlYdc4XuW1s7hoC8qiPyoQ== X-Received: by 2002:a05:6a00:a94:b0:562:dcbb:47c3 with SMTP id b20-20020a056a000a9400b00562dcbb47c3mr20315835pfl.79.1667313296494; Tue, 01 Nov 2022 07:34:56 -0700 (PDT) Received: from anup-ubuntu64-vm.. ([171.76.80.52]) by smtp.gmail.com with ESMTPSA id o20-20020a170903009400b0018685aaf41dsm6449055pld.18.2022.11.01.07.34.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 07:34:56 -0700 (PDT) From: Anup Patel To: Palmer Dabbelt , Paul Walmsley , Thomas Gleixner , Marc Zyngier , Daniel Lezcano Cc: Atish Patra , Alistair Francis , Anup Patel , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Anup Patel Subject: [PATCH v10 6/7] RISC-V: Use IPIs for remote TLB flush when possible Date: Tue, 1 Nov 2022 20:03:59 +0530 Message-Id: <20221101143400.690000-7-apatel@ventanamicro.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221101143400.690000-1-apatel@ventanamicro.com> References: <20221101143400.690000-1-apatel@ventanamicro.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748304863297339122?= X-GMAIL-MSGID: =?utf-8?q?1748304863297339122?= If we have specialized interrupt controller (such as AIA IMSIC) which allows supervisor mode to directly inject IPIs without any assistance from M-mode or HS-mode then using such specialized interrupt controller, we can do remote TLB flushes directly from supervisor mode instead of using the SBI RFENCE calls. This patch extends remote TLB flush functions to use supervisor mode IPIs whenever direct supervisor mode IPIs.are supported by interrupt controller. Signed-off-by: Anup Patel Reviewed-by: Atish Patra --- arch/riscv/mm/tlbflush.c | 93 +++++++++++++++++++++++++++++++++------- 1 file changed, 78 insertions(+), 15 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 37ed760d007c..27a7db8eb2c4 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -23,14 +23,62 @@ static inline void local_flush_tlb_page_asid(unsigned long addr, : "memory"); } +static inline void local_flush_tlb_range(unsigned long start, + unsigned long size, unsigned long stride) +{ + if (size <= stride) + local_flush_tlb_page(start); + else + local_flush_tlb_all(); +} + +static inline void local_flush_tlb_range_asid(unsigned long start, + unsigned long size, unsigned long stride, unsigned long asid) +{ + if (size <= stride) + local_flush_tlb_page_asid(start, asid); + else + local_flush_tlb_all_asid(asid); +} + +static void __ipi_flush_tlb_all(void *info) +{ + local_flush_tlb_all(); +} + void flush_tlb_all(void) { - sbi_remote_sfence_vma(NULL, 0, -1); + if (riscv_use_ipi_for_rfence()) + on_each_cpu(__ipi_flush_tlb_all, NULL, 1); + else + sbi_remote_sfence_vma(NULL, 0, -1); +} + +struct flush_tlb_range_data { + unsigned long asid; + unsigned long start; + unsigned long size; + unsigned long stride; +}; + +static void __ipi_flush_tlb_range_asid(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range_asid(d->start, d->size, d->stride, d->asid); +} + +static void __ipi_flush_tlb_range(void *info) +{ + struct flush_tlb_range_data *d = info; + + local_flush_tlb_range(d->start, d->size, d->stride); } -static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, - unsigned long size, unsigned long stride) +static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, + unsigned long size, unsigned long stride) { + struct flush_tlb_range_data ftd; struct cpumask *cmask = mm_cpumask(mm); unsigned int cpuid; bool broadcast; @@ -45,19 +93,34 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, unsigned long asid = atomic_long_read(&mm->context.id); if (broadcast) { - sbi_remote_sfence_vma_asid(cmask, start, size, asid); - } else if (size <= stride) { - local_flush_tlb_page_asid(start, asid); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = asid; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range_asid, + &ftd, 1); + } else + sbi_remote_sfence_vma_asid(cmask, + start, size, asid); } else { - local_flush_tlb_all_asid(asid); + local_flush_tlb_range_asid(start, size, stride, asid); } } else { if (broadcast) { - sbi_remote_sfence_vma(cmask, start, size); - } else if (size <= stride) { - local_flush_tlb_page(start); + if (riscv_use_ipi_for_rfence()) { + ftd.asid = 0; + ftd.start = start; + ftd.size = size; + ftd.stride = stride; + on_each_cpu_mask(cmask, + __ipi_flush_tlb_range, + &ftd, 1); + } else + sbi_remote_sfence_vma(cmask, start, size); } else { - local_flush_tlb_all(); + local_flush_tlb_range(start, size, stride); } } @@ -66,23 +129,23 @@ static void __sbi_tlb_flush_range(struct mm_struct *mm, unsigned long start, void flush_tlb_mm(struct mm_struct *mm) { - __sbi_tlb_flush_range(mm, 0, -1, PAGE_SIZE); + __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { - __sbi_tlb_flush_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); } void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PAGE_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __sbi_tlb_flush_range(vma->vm_mm, start, end - start, PMD_SIZE); + __flush_tlb_range(vma->vm_mm, start, end - start, PMD_SIZE); } #endif