From patchwork Tue Jul 11 07:54:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 118312 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp321202vqm; Tue, 11 Jul 2023 01:19:47 -0700 (PDT) X-Google-Smtp-Source: APBJJlGi6H8ee6d3KOe7LWslI0js82Wd2Nd5YJTZRgc62u65eUaPR24ik/68Ui42P5ygyeXE8ewH X-Received: by 2002:a17:907:9722:b0:97e:32e:c1e with SMTP id jg34-20020a170907972200b0097e032e0c1emr19695175ejc.55.1689063586825; Tue, 11 Jul 2023 01:19:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689063586; cv=none; d=google.com; s=arc-20160816; b=f+3NmdK6z4wuEn6cfXtnL3lXBQMfSjA1zmNOh5MWru1SWSHa++gxeUjqLUHCqfDdlv j1rQPgojo2R32wQGB2ZJWTe0diuirF6hiQ00ySx6GN8k5sosREO4OuBuPeWfZv6UjvAh pPjTE8hwVKN4JTa2uMhQwO2S6wV993mcWmMaxbvJn6UgEA+q2U0dEJSDievhuQLNGZis TBGYiXx+KKdbebNG2z0RSgQ2/AjgH2P9uCunk/t89+EB9v2FVdJoNfmtk552JowDRd/F +vJadXh4srfC6knI1M5NUbVSv7z7goXmt1ME+nc08DYRX8nCFUFHryze3Rw/LIynC8jg RWEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/EU/iMG6yg29uBo51A2iBmY9OZkda2o1/b9d0LOiq1U=; fh=9KG1GxoJIHO8DA9/W8zOCR/cizSACkxrp9LVRbza3AA=; b=Ygz9OgcBxGFJvWtQboGO4dFq8AxTyyJG2eR7JXbiwguqRu+A/TNDbibGxTp3DeuguZ 1Dy3U1tdlXX7hEAlHJFbZb/PQ8oML7M/xj98DSRXiIzIFsMqE7Pdk+GOok98gF73i6dz SsAU4wqDEYIMkPaT10T8mzFkjBACNprz9P2mEDIU7d6fnZqzr7EJ9DH0LVr2dckCg4Yv 7P+lXA1IaIMHpSx7jMZ8xs5ITlnK9IcQ3Exc7CPsaJXf15HCZtbZPUgD9rcGiyWpNCZb I4gZj8FmAiv2gF1TjI1Df0E+gaeig9gMCBBcfgAEEThtyWGa/J2ncWDFzQdBn3e8SwcM 35vQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=NCoKYaei; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ga15-20020a170906b84f00b0098e1efaafc5si1558320ejb.195.2023.07.11.01.19.23; Tue, 11 Jul 2023 01:19:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=NCoKYaei; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231210AbjGKHzm (ORCPT + 99 others); Tue, 11 Jul 2023 03:55:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230236AbjGKHzk (ORCPT ); Tue, 11 Jul 2023 03:55:40 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B91E8CF for ; Tue, 11 Jul 2023 00:55:39 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id 5b1f17b1804b1-3fbea14700bso55060165e9.3 for ; Tue, 11 Jul 2023 00:55:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689062138; x=1691654138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/EU/iMG6yg29uBo51A2iBmY9OZkda2o1/b9d0LOiq1U=; b=NCoKYaeiv1nFsZu1kkN9IKONZ0ywf8MrhYyLoYAG7Q6i/1MQkq4oT+ftSJXO49RKyX ACt2ggo37pYi+8YgPL8PJ6dxHo9tA4/bLupo3TXiy9URZ4wDKkU28wB9BygUOld8z2e/ 3zc8DUiMdWOgQtJ0Ndn4UV5Li6hfcg8Kga3ZZhjNakNirjxP7y+vfmb9LrM9PCGhIPob Hdzs0ngha48Jpks5P0qe+0HhpMg4tmNZJB4wqnH4M1bv2Zwy3CCAuPi3okrxJTIKbojW InsOcPcSYd20w6iszei4mg1pDK4LCzlrLwfCY1fjznaJUncNDxte3J8oxZxhilbHxfFV 6g/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689062138; x=1691654138; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/EU/iMG6yg29uBo51A2iBmY9OZkda2o1/b9d0LOiq1U=; b=Nf8HJZFhc+g5mpGppFD5PXujxw7TjduAh6JsROvBSrBE3hZ9M9k8YSYAFGjmyKiX7T 7OrwpYyR9a2ErjZMbGrRBMJrS7F3iqm6l1Aexe14yNoSgXKRr93O1OK6xa44qpmQXuXo YtKW50YHoQiagQRYZaPdXy4iS+v7Twh0n3zoK2V9Nm0LTbCV30RO+O6BzRZwQ87GeGdB vXTZgdfVzXYxBePx+Pf+rgQWq20WNnKuR0gM0VZsLZ+r/JN4y9ZFZXvUPF6N/tB/aWwu 6oV2aYB973X9TGyX3r4VvbzWeWNHfan90wCJDW6THj9w4i4CYEuicMiStH7QeFYL3ec7 zy4Q== X-Gm-Message-State: ABy/qLbbahB5ki+8+buTahFJ/2ybbGs70urmIfwTybiw/8/ZTvHWHWNk sL2j6u1qtENlx+mHEuglp9UZjA== X-Received: by 2002:a05:600c:b5a:b0:3fc:60:7dbf with SMTP id k26-20020a05600c0b5a00b003fc00607dbfmr9944380wmr.41.1689062138206; Tue, 11 Jul 2023 00:55:38 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id f19-20020a7bcc13000000b003fa973e6612sm12317248wmh.44.2023.07.11.00.55.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 00:55:38 -0700 (PDT) From: Alexandre Ghiti To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Mayuresh Chitale , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 1/4] riscv: Improve flush_tlb() Date: Tue, 11 Jul 2023 09:54:31 +0200 Message-Id: <20230711075434.10936-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711075434.10936-1-alexghiti@rivosinc.com> References: <20230711075434.10936-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771111539430129751 X-GMAIL-MSGID: 1771111539430129751 For now, flush_tlb() simply calls flush_tlb_mm() which results in a flush of the whole TLB. So let's use mmu_gather fields to provide a more fine-grained flush of the TLB. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/tlb.h | 6 +++++- arch/riscv/include/asm/tlbflush.h | 3 +++ arch/riscv/mm/tlbflush.c | 7 +++++++ 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 120bcf2ed8a8..3373dcf0f413 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -15,7 +15,11 @@ static void tlb_flush(struct mmu_gather *tlb); static inline void tlb_flush(struct mmu_gather *tlb) { - flush_tlb_mm(tlb->mm); + if (tlb->fullmm || tlb->need_flush_all) + flush_tlb_mm(tlb->mm); + else + flush_tlb_mm_range(tlb->mm, tlb->start, tlb->end, + tlb_get_unmap_size(tlb)); } #endif /* _ASM_RISCV_TLB_H */ diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index a09196f8de68..f5c4fb0ae642 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -32,6 +32,8 @@ static inline void local_flush_tlb_page(unsigned long addr) #if defined(CONFIG_SMP) && defined(CONFIG_MMU) void flush_tlb_all(void); void flush_tlb_mm(struct mm_struct *mm); +void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, + unsigned long end, unsigned int page_size); void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); @@ -52,6 +54,7 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, } #define flush_tlb_mm(mm) flush_tlb_all() +#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() #endif /* !CONFIG_SMP || !CONFIG_MMU */ /* Flush a range of kernel pages */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 77be59aadc73..fa03289853d8 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -132,6 +132,13 @@ void flush_tlb_mm(struct mm_struct *mm) __flush_tlb_range(mm, 0, -1, PAGE_SIZE); } +void flush_tlb_mm_range(struct mm_struct *mm, + unsigned long start, unsigned long end, + unsigned int page_size) +{ + __flush_tlb_range(mm, start, end - start, page_size); +} + void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) { __flush_tlb_range(vma->vm_mm, addr, PAGE_SIZE, PAGE_SIZE); From patchwork Tue Jul 11 07:54:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 118311 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp319793vqm; Tue, 11 Jul 2023 01:16:01 -0700 (PDT) X-Google-Smtp-Source: APBJJlE5Zc7b0wZ/WoJpcUfqTs1KXPMsAqfoztVTT/tI4PnbZr6UhofoOEwpay4ol26Gq9z5lW0H X-Received: by 2002:a05:6512:348b:b0:4f8:7807:a612 with SMTP id v11-20020a056512348b00b004f87807a612mr11678797lfr.61.1689063361582; Tue, 11 Jul 2023 01:16:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689063361; cv=none; d=google.com; s=arc-20160816; b=MjainQ2k6GxaX84zSVWlEmStx1srrYa/alKOhKQEI+G1ZYQ8CuwTAJhD+6vB6l9cSt YODTVyRV+IoZfrjNfdpP6WvWPZKrXDlGqXUXF9nswMNTFV1QtJ6InjzCGsdkOwneMuoV gw8FqI5/djS2gvF6ksO+kJMBqdRa9QBplS/troSNfWXY2T3EWuClaGpLa2RTGzv3qnQW mULQP/HYJfJxxswGLx6hoWUSQQnXejPFegWCgIQI0JIjAgVugdTVQ/VRST5j8SjtSbxX s2YHAL75CdaKB/cld5hgYzVeDLJ1ijjAt1wUq1zqtBErOF0Fb3rb4tNsYdFks5+hSCKy k9oA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FofimiyN+7gyzGEdPyrBP2skR/4a1H3RZbKcHwnJwCU=; fh=9KG1GxoJIHO8DA9/W8zOCR/cizSACkxrp9LVRbza3AA=; b=jvdqyNZVyfpBxncqDHAxiLraRu9CjrE7BhKDsuegNSdTarNPiGFaaZCARLU954IUfc h+nkBLW+irXKxvs1SHMzDQ1k04ZOD66bVHMAF0cJEDi3lVJWnAvSWieDLM7p6Q03Q1rW bZ5NjnmgPxlFgjZJeDwYfRY2pAX5Jz40tgbokFyofDIAqybx3uTPWWE8bARAkHWjwB8X TznLo6XOQLERH1NQQrXOrh8XQoli9C+mXeffO66amHtG02KwvpO50Qy/naLo2GGjuTjs KseVBfpGTAY52MmiCCmXfJWw3WPDYZLFF1TV0yZZOcPjK63fHyGhX0JRu9tZlAb+zlCI aesQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=Qf1MDi4I; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k19-20020aa7c393000000b0051e2cc05ba9si1527982edq.306.2023.07.11.01.15.38; Tue, 11 Jul 2023 01:16:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=Qf1MDi4I; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231503AbjGKH4p (ORCPT + 99 others); Tue, 11 Jul 2023 03:56:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231509AbjGKH4m (ORCPT ); Tue, 11 Jul 2023 03:56:42 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8D661A8 for ; Tue, 11 Jul 2023 00:56:40 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-3fc0aecf15bso28410575e9.1 for ; Tue, 11 Jul 2023 00:56:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689062199; x=1691654199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FofimiyN+7gyzGEdPyrBP2skR/4a1H3RZbKcHwnJwCU=; b=Qf1MDi4IQVSgi+mVmNSDLBvvWhfd8ba6Ayo48Gj0+fuhQZ4V9G61dBcMO//cFS1/fC eMTYYy06UvyotO+oVGCBq6zI9sY1LeYU01nJRoZJ+JLckfsVKJMM97nPklKyDZRE3HOu TVkWiZ1s6BeKrYLS2wuPUnzE9wyBrK5LdHaTtNHXVs67A6vQfZbIP4oSE43hkg/e5u/I R82FjYAOaShi8TnwHOG++346KF4aekyWodyFCqKGj8HPeSDtbX8x88W80pqGP3MBU7wD 2AUgLCJEZ6IWj8TTYNHFpXz8+pCuX+wIU15tPfBynjH9x67d68/tJq305rfx6T581tDJ NCkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689062199; x=1691654199; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FofimiyN+7gyzGEdPyrBP2skR/4a1H3RZbKcHwnJwCU=; b=dln7TNWGQ+VVkuWvNFCKcmHgklokzhprinyx+AZMFWxzxVhHJcwWUwVUoeuoOXJ/z6 zfp80pEn/yD1CaJ9L9c8xJSytV7MFLiu0wB6X9+D4wbhdXFBwk5dCtpNXyLmzxVjA07v 4A87SGJScbr+leo5+8XtTk3UxlWyiU+OeH6Bs23AscuVI+rQvMhzqFXuy0LYr5T5nL20 6Cuo5Z5S3EpEJHKh8O0r0M8fWIgRP/YbnLEPlmNgi7TOYNv8ewfNjRE+xDZCwdXpIFqM 66YabW6aHq4ibfgy5Th3FfpQd0sYpn0847XjXdZN+OaKTL95c3hurOSB2lqVpwZHRe9f 0bpw== X-Gm-Message-State: ABy/qLZkj86a/aASTN8kXrToLcpnpy0ManoOC8AxLJmZhy6Z/ZUxHaAu dOXZ5cOwtay4eZBbVIIMw4olWQ== X-Received: by 2002:adf:f052:0:b0:30a:a15d:eb2f with SMTP id t18-20020adff052000000b0030aa15deb2fmr16274589wro.3.1689062199178; Tue, 11 Jul 2023 00:56:39 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id p8-20020adfe608000000b003141f3843e6sm1489099wrm.90.2023.07.11.00.56.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 00:56:39 -0700 (PDT) From: Alexandre Ghiti To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Mayuresh Chitale , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 2/4] riscv: Improve flush_tlb_range() for hugetlb pages Date: Tue, 11 Jul 2023 09:54:32 +0200 Message-Id: <20230711075434.10936-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711075434.10936-1-alexghiti@rivosinc.com> References: <20230711075434.10936-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771111303661734466 X-GMAIL-MSGID: 1771111303661734466 flush_tlb_range() uses a fixed stride of PAGE_SIZE and in its current form, when a hugetlb mapping needs to be flushed, flush_tlb_range() flushes the whole tlb: so set a stride of the size of the hugetlb mapping in order to only flush the hugetlb mapping. Note that THPs are directly handled by flush_pmd_tlb_range(). Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/tlbflush.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index fa03289853d8..3e4acef1f6bc 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include @@ -147,7 +148,14 @@ void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { - __flush_tlb_range(vma->vm_mm, start, end - start, PAGE_SIZE); + unsigned long stride_shift; + + stride_shift = is_vm_hugetlb_page(vma) ? + huge_page_shift(hstate_vma(vma)) : + PAGE_SHIFT; + + __flush_tlb_range(vma->vm_mm, + start, end - start, 1 << stride_shift); } #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, From patchwork Tue Jul 11 07:54:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 118308 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp318780vqm; Tue, 11 Jul 2023 01:13:37 -0700 (PDT) X-Google-Smtp-Source: APBJJlFTX+GceLixNwLBss+auLXBjEBY0wfHdRf6UDUgNdu2vZ7rc2X6EuwpXvTlmPOmx1K6zt5D X-Received: by 2002:a05:6402:d3:b0:51d:9195:400f with SMTP id i19-20020a05640200d300b0051d9195400fmr14522286edu.17.1689063216612; Tue, 11 Jul 2023 01:13:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689063216; cv=none; d=google.com; s=arc-20160816; b=EoN3Fm1WYethCfysaN6mhPR5of6AdyqoLs5Z6FKGAJUftjmoFl9YpWKxuJL5fpvJdT bN9ZucybI3ljzUBteDuJOBOIn55xVO6sCcibIUDm8PKs4Tmdy6fp3FGtQhIhz0wnq2d/ g3Ru/h0eIzFzOq92/98NqEFw+a+CMml5ZMtNeuLfKIQsBNcV83beHStKUADZCK/hRDk9 8ZtFPBq87imlo6xuZbbMmZrzqMT9ZHXMC+UFY2Yy78iakyJCCorl34j7BqCrUeTbgTVz kGuMCmfGqs3Y9bEjGZsUuU2jpdEAnaWsDIcaPbiHsqMy/0y7LyvYyHbCF9D1v5hGomPO D1Nw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+GXjLhe5TXUta+EXxk7AgfyYmiNiXB9g5OSSIlBPGEA=; fh=9KG1GxoJIHO8DA9/W8zOCR/cizSACkxrp9LVRbza3AA=; b=aNyrIelqk4oo+8GsmjLmO/+e6u4uetj5Cui6egaCAj6GtrZ1v+91hSeO32Xmsc6gFV HN3YIhuRbjyiu2XmNanRW6JCYUtv8QAaHpwwaj7z4Pp9jGzbsmkw8pJ/tlWK0zaIEd0+ DqX4DrdwSCY7do8YAhMWX46djxZ3N/nVjWOZLoZMzT8KkSsqscak/dRrt8vmK4ZdVzSM 7Dj8ZMWkU46qgW0LIYkO1AdrXIvVYo5XZ5YfUu+3xvHUg2CLiBFEVsvBeF68UcoKopsz z0YfRSiBuf+1aCIFGPhhxkpIOstMpPc3YVtxKJ76xZzVXGYw5r7xk41LU7iSIlYi47/n qXhw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=jpjsGIvH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z8-20020aa7d408000000b0051dd1d87973si1533514edq.353.2023.07.11.01.13.12; Tue, 11 Jul 2023 01:13:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=jpjsGIvH; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231516AbjGKH5p (ORCPT + 99 others); Tue, 11 Jul 2023 03:57:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231509AbjGKH5o (ORCPT ); Tue, 11 Jul 2023 03:57:44 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B16B9F7 for ; Tue, 11 Jul 2023 00:57:41 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id ffacd0b85a97d-31434226a2eso6268370f8f.1 for ; Tue, 11 Jul 2023 00:57:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689062260; x=1691654260; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+GXjLhe5TXUta+EXxk7AgfyYmiNiXB9g5OSSIlBPGEA=; b=jpjsGIvHWDXwtmenwjPw+yleh84wYF4Gde1Y36TPhdybnF0jpwSLd/aOo7CFMtSuDK ScyvlYS80l2vcQ9apidNHokPdNcKdGvUFOz16EWh6HyNXWzuR/QaMrKyDRVuWrs51yKx skZgZTUZjD6cFKSOYgb53UN64gLFvysjjqU8dLz9nE2+HJHAHGf48tinF7oqdJtBI4pc Et4wuuECz5m6cPYyXZRudVvVIj9AKfqmjpYZWVzHlfwOhzfYW2aebgP/Fu+HS+8C0bcV XDhilVqCMXb407/A4iXWplI1yIn5lQwUah0v2s5df1KU/cDtaR7kCadJjP9Y4QwOQKtB s4Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689062260; x=1691654260; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+GXjLhe5TXUta+EXxk7AgfyYmiNiXB9g5OSSIlBPGEA=; b=lPCKwYhJuSaufSqmWui2uFJY+DNjvLlWzz8M/YmKl9M1tsp1O/By4Urk7nBNTgL08w NiVNLYlYDHML8yX8b13KgGF/DF7smqIio0BkMz5f8fZIIz95TiSHOxYr2odGMzX+Rs1s NWKn3m/zcdlOTKR/j3byv1dsNbz9bdFIugzqEMp2NTcdaFXPZ8yCuDW0IH5Fo6t25MT0 KnKpFp9+VI/VBJ/1ywN9Z4+NU0dCkuk5kR0cYjOmLJdZ+m3Sejxb/GvuyI2sSik+rqnL 4ac1EIP637rOXXgKc21t+UO5lEiEhOHht4ldYwzJPaHFY60hh7NZJeFRuxV2sKVLP2Nm x6ew== X-Gm-Message-State: ABy/qLaanBianLXEYwKtv7IhS4LwB9MyMDIHawNBCpBhHfIXQzcSXqI1 2Yf/YMvzhMV1+eE9D6xk8BLvig== X-Received: by 2002:a5d:4404:0:b0:314:46cb:880a with SMTP id z4-20020a5d4404000000b0031446cb880amr18276011wrq.28.1689062260118; Tue, 11 Jul 2023 00:57:40 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id t16-20020a5d49d0000000b003143bb5ecd5sm1499978wrs.69.2023.07.11.00.57.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 00:57:39 -0700 (PDT) From: Alexandre Ghiti To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Mayuresh Chitale , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 3/4] riscv: Make __flush_tlb_range() loop over pte instead of flushing the whole tlb Date: Tue, 11 Jul 2023 09:54:33 +0200 Message-Id: <20230711075434.10936-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711075434.10936-1-alexghiti@rivosinc.com> References: <20230711075434.10936-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771111150984713622 X-GMAIL-MSGID: 1771111150984713622 Currently, when the range to flush covers more than one page (a 4K page or a hugepage), __flush_tlb_range() flushes the whole tlb. Flushing the whole tlb comes with a greater cost than flushing a single entry so we should flush single entries up to a certain threshold so that: threshold * cost of flushing a single entry < cost of flushing the whole tlb. This threshold is microarchitecture dependent and can/should be overwritten by vendors. Co-developed-by: Mayuresh Chitale Signed-off-by: Mayuresh Chitale Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/tlbflush.c | 41 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 39 insertions(+), 2 deletions(-) diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 3e4acef1f6bc..de61ecaa218a 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -24,13 +24,48 @@ static inline void local_flush_tlb_page_asid(unsigned long addr, : "memory"); } +/* + * Flush entire TLB if number of entries to be flushed is greater + * than the threshold below. Platforms may override the threshold + * value based on marchid, mvendorid, and mimpid. + */ +unsigned long tlb_flush_all_threshold __read_mostly = 64; + +static void local_flush_tlb_range_threshold_asid(unsigned long start, + unsigned long size, + unsigned long stride, + unsigned long asid) +{ + u16 nr_ptes_in_range = DIV_ROUND_UP(size, stride); + int i; + + if (nr_ptes_in_range > tlb_flush_all_threshold) { + if (asid != -1) + local_flush_tlb_all_asid(asid); + else + local_flush_tlb_all(); + return; + } + + for (i = 0; i < nr_ptes_in_range; ++i) { + if (asid != -1) + local_flush_tlb_page_asid(start, asid); + else + local_flush_tlb_page(start); + start += stride; + } +} + static inline void local_flush_tlb_range(unsigned long start, unsigned long size, unsigned long stride) { if (size <= stride) local_flush_tlb_page(start); - else + else if (size == (unsigned long)-1) local_flush_tlb_all(); + else + local_flush_tlb_range_threshold_asid(start, size, stride, -1); + } static inline void local_flush_tlb_range_asid(unsigned long start, @@ -38,8 +73,10 @@ static inline void local_flush_tlb_range_asid(unsigned long start, { if (size <= stride) local_flush_tlb_page_asid(start, asid); - else + else if (size == (unsigned long)-1) local_flush_tlb_all_asid(asid); + else + local_flush_tlb_range_threshold_asid(start, size, stride, asid); } static void __ipi_flush_tlb_all(void *info) From patchwork Tue Jul 11 07:54:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 118304 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a6b2:0:b0:3e4:2afc:c1 with SMTP id c18csp314862vqm; Tue, 11 Jul 2023 01:04:52 -0700 (PDT) X-Google-Smtp-Source: APBJJlEgNnVAoZ6D5gIzYDb0aKwJrNU9N3sd7+IC4tbfpCVdeR24h8kXbDnPu+5iXbTKvC6At8WX X-Received: by 2002:a17:906:924d:b0:98e:7a7:60b0 with SMTP id c13-20020a170906924d00b0098e07a760b0mr14402532ejx.65.1689062692525; Tue, 11 Jul 2023 01:04:52 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1689062692; cv=none; d=google.com; s=arc-20160816; b=RlXX0liyvB7fiTcQFa5PJ6dp38ZwZcKDlxCQM5agQMuO1ifVvsRQ2xaFb3C2Zbb1/j 3M36ENtH2mFO/cKP2venZmfJ8L36YQXo3pSMYAPeZvegUanEW0S8iiqfk8pgLm9bIECU bluRmuQH+DR2xJl31/wraaDntnSx10tc9vZAVKy3/aMTaeFgvEOOOSxDlKTKfEOjrQCc K3wyaXSw/pAOUL0AbcnQ77Fwv6+lnX+dYEAzHDaJKBzL5dTdHjijhVikZVYVjyG7X3i+ kq5IMUdC100tjZ9e2n7MgD/MqNqMnWbRx0sogUvyEx85idiTBw7xyqgQzep+krtzjJb5 dGJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YDYIXf8sIKLPYejGQ8O6uVDYRyye9psJA+Q5PmU0G6c=; fh=9KG1GxoJIHO8DA9/W8zOCR/cizSACkxrp9LVRbza3AA=; b=fLTkTwP51GMZ8ClW2saGJIbi53/+M9Pylhz6t5nDXnAl2YY3XbxqFi5UBhW4agAPcK DNA1480biCGLcOrZfGg6v5MrdPXpYCWpPv//7P1tO6k/6WVWzE3EzLCE02Vn5tZCrlsN XXXjJbFyQIlk7cYIBypcQoyDqHwrxUdVnwH//AxFmnNqZBGIo9eTcM+bxAH1YAEzrKTf 0Z44IRXzP5YzYXhEGTCpLBJwxIm51qQ+orqKvd9AOpmS+eLrKGmUY4LtPEOio3dErC5V dhlc+ekMd8Tk0+AiQTo9mNY61koUpmYtNeob1NNY6rcLEC4K0L2LNAYzu5ZVstuVlCMg H3Fg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=w92bWwP4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s16-20020a170906bc5000b0098e2edf7479si1513511ejv.858.2023.07.11.01.04.28; Tue, 11 Jul 2023 01:04:52 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20221208.gappssmtp.com header.s=20221208 header.b=w92bWwP4; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231651AbjGKH6t (ORCPT + 99 others); Tue, 11 Jul 2023 03:58:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60104 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231608AbjGKH6r (ORCPT ); Tue, 11 Jul 2023 03:58:47 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CB0610C7 for ; Tue, 11 Jul 2023 00:58:42 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-3fbc5d5742bso59639795e9.2 for ; Tue, 11 Jul 2023 00:58:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20221208.gappssmtp.com; s=20221208; t=1689062321; x=1691654321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YDYIXf8sIKLPYejGQ8O6uVDYRyye9psJA+Q5PmU0G6c=; b=w92bWwP4/B9Yg8nTudHNREZgu5pw/Y+pQ/5HeoGML42uuWQbv8zilzfmgXbkrCeUXJ BigOw37ZeE+XRBcUme8MnQpOGaTDa7wH4BCEBKoFhYCZ+hVEXq0Q3Mv/yS9fBPm22IAZ 5H2e5BzVRS345P78IGjkr7Sg6L606PPnBu/yLKT3L5E04/awzbxsIVShbm8PjQD8fNKG cq982sPP3LSK8cemVb1+wFEr0xgVkryUn7u3N0cBuFOQEhKyoy/uauDpQuDZhIntYloR KCSkuNsDO2zpLsRgyBdBNuZUfDREeDfyFDyDnHp6qvfQoWIgJOr3FntAg4EIsSB3wIGx QgAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689062321; x=1691654321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YDYIXf8sIKLPYejGQ8O6uVDYRyye9psJA+Q5PmU0G6c=; b=ITMfwsNEprd8bKmvbaIXRGvifSHGseOG+J262ir22gdy2i5TIBvK3w6vaA4UEnZd1F PmlDw2V/oap/6uKRRAvxq3laH6phVtDc76o0DDfs5kt+r7zYrj0NKzRB5ZAUkp/Seykg cMxKNzBXe1KJ82y453TTnQVSj4odlsVKCRBaWSCRJY+A/W0eAh6asVogXtyANtcLiWmz SFyf8cvTyCkvFkSiC6/IyqH+Y0WropJsnfE76I8SZk7l0QnAKGTfYIKGs49hixgEBFPx WUTvp1fQ1BVhvFguMr+IZwOceR7hlJ79T4ADWYqFVhjKKFXxsVwTbqhXTehjDolowroT CoTA== X-Gm-Message-State: ABy/qLavrah5wwkVyyUPrPbzyi36SjjTDrsvrmBfex6LaxtgODN2Y9LG 2yb++tZPxdPfaNX4HH5xIsblRg== X-Received: by 2002:a7b:c38b:0:b0:3fc:524:e80a with SMTP id s11-20020a7bc38b000000b003fc0524e80amr10779950wmj.18.1689062321091; Tue, 11 Jul 2023 00:58:41 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id 12-20020a05600c020c00b003fbfa6066acsm1750168wmi.40.2023.07.11.00.58.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 11 Jul 2023 00:58:40 -0700 (PDT) From: Alexandre Ghiti To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Mayuresh Chitale , Vincent Chen , Paul Walmsley , Palmer Dabbelt , Albert Ou , linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org Cc: Alexandre Ghiti Subject: [PATCH 4/4] riscv: Improve flush_tlb_kernel_range() Date: Tue, 11 Jul 2023 09:54:34 +0200 Message-Id: <20230711075434.10936-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711075434.10936-1-alexghiti@rivosinc.com> References: <20230711075434.10936-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771110601983200665 X-GMAIL-MSGID: 1771110601983200665 This function used to simply flush the whole tlb of all harts, be more subtile and try to only flush the range. The problem is that we can only use PAGE_SIZE as stride since we don't know the size of the underlying mapping and then this function will be improved only if the size of the region to flush is < threshold * PAGE_SIZE. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/tlbflush.h | 11 +++++----- arch/riscv/mm/tlbflush.c | 35 +++++++++++++++++++++++-------- 2 files changed, 32 insertions(+), 14 deletions(-) diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index f5c4fb0ae642..7426fdcd8ec5 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -37,6 +37,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, void flush_tlb_page(struct vm_area_struct *vma, unsigned long addr); void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end); +void flush_tlb_kernel_range(unsigned long start, unsigned long end); #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, @@ -53,15 +54,15 @@ static inline void flush_tlb_range(struct vm_area_struct *vma, local_flush_tlb_all(); } -#define flush_tlb_mm(mm) flush_tlb_all() -#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() -#endif /* !CONFIG_SMP || !CONFIG_MMU */ - /* Flush a range of kernel pages */ static inline void flush_tlb_kernel_range(unsigned long start, unsigned long end) { - flush_tlb_all(); + local_flush_tlb_all(); } +#define flush_tlb_mm(mm) flush_tlb_all() +#define flush_tlb_mm_range(mm, start, end, page_size) flush_tlb_all() +#endif /* !CONFIG_SMP || !CONFIG_MMU */ + #endif /* _ASM_RISCV_TLBFLUSH_H */ diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index de61ecaa218a..07cfed83bec8 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -117,18 +117,27 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, unsigned long size, unsigned long stride) { struct flush_tlb_range_data ftd; - struct cpumask *cmask = mm_cpumask(mm); - unsigned int cpuid; + struct cpumask *cmask, full_cmask; bool broadcast; - if (cpumask_empty(cmask)) - return; + if (mm) { + unsigned int cpuid; + + cmask = mm_cpumask(mm); + if (cpumask_empty(cmask)) + return; + + cpuid = get_cpu(); + /* check if the tlbflush needs to be sent to other CPUs */ + broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; + } else { + cpumask_setall(&full_cmask); + cmask = &full_cmask; + broadcast = true; + } - cpuid = get_cpu(); - /* check if the tlbflush needs to be sent to other CPUs */ - broadcast = cpumask_any_but(cmask, cpuid) < nr_cpu_ids; if (static_branch_unlikely(&use_asid_allocator)) { - unsigned long asid = atomic_long_read(&mm->context.id) & asid_mask; + unsigned long asid = mm ? atomic_long_read(&mm->context.id) & asid_mask : 0; if (broadcast) { if (riscv_use_ipi_for_rfence()) { @@ -162,7 +171,8 @@ static void __flush_tlb_range(struct mm_struct *mm, unsigned long start, } } - put_cpu(); + if (mm) + put_cpu(); } void flush_tlb_mm(struct mm_struct *mm) @@ -194,6 +204,13 @@ void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, __flush_tlb_range(vma->vm_mm, start, end - start, 1 << stride_shift); } + +void flush_tlb_kernel_range(unsigned long start, + unsigned long end) +{ + __flush_tlb_range(NULL, start, end, PAGE_SIZE); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end)