From patchwork Thu Dec 7 15:03:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 17694 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp4841614vqy; Thu, 7 Dec 2023 07:04:12 -0800 (PST) X-Google-Smtp-Source: AGHT+IFifFxHSIEH6tbMG79CHrY8tlVbm/oQdJxuLbDYja8EN/Usy+MLDiqa90hh/G80MZ4R13Wp X-Received: by 2002:a17:90b:4a01:b0:286:ccc1:486 with SMTP id kk1-20020a17090b4a0100b00286ccc10486mr2513117pjb.27.1701961452298; Thu, 07 Dec 2023 07:04:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701961452; cv=none; d=google.com; s=arc-20160816; b=QV+PZFrOlZ/Tnqr95yzCurcEig6rxRVwCjDKgYsXvoPTWKlfwYgA+Eaea0JAw21Bil lwP6HpWASBiwe3TFR+6B6NQBeiUmPaSn3B/8LFD/pDQyeGWgIK8neWBQGmBEWm4GLEJ9 efVF5W97dh3ViNSvBoa/VoK1ffkaV3uRlDsbrWNMaqHQOCw4H+cMO5a7G7gME7o42lnK m7/2pfaL0IUhkninoOY4UYcRAHyAkwWIbzVNoDMmtXnUrYwaGa+lSSoHw8mrjfI2ys4O 9TcG/8g43p8G6axE4oV71NUxGTYVJ4JCfjsD4lHo8wdhl69ROy7AUwqJnHY2G8kyO5kv PUCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=nzUGYpwYAIocf74DCpTux4YFllX1H7F6zPZyiGHWp1Y=; fh=ghJZAeZ4zrru54jVI3XtuZXbadJtp4HIJMVSTZEe2l4=; b=fskEorlU2ai6KE5Kd5lwWsgpNyr4ejnC4E9GzWc4Dwv99nGviuVCHoXf79+wdYHKOM pBrVzJkqxihfMF/BmBZdaUQJ2GXE8ZtUzZTn+rLC6gTzKXiea7TCs+B+N+klbT3rep47 HfUmvyJ24QP7IU6iHytleKNt8nm8UAxUBoiMf74ua1vAHQt90x7ILGM/azKu2n8G/CUJ A1OkYn8e8+2nxLP8uM8t/aDzsin1ykLYMr6ED3Q/RDSg5EKaUHcDtIXH+qHBJQhlFCZo xYo6BNsEWYDzJ+CnRmDTPA0UAaTJrETZGCXAO+TFq23vP8vmAgejbUOq5pwMQmQuvPhU hQ8A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=GLL16fO2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id n8-20020a654508000000b005c61d17e6aasi1324375pgq.122.2023.12.07.07.04.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:04:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=GLL16fO2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id E532B80B1222; Thu, 7 Dec 2023 07:03:57 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233041AbjLGPDt (ORCPT + 99 others); Thu, 7 Dec 2023 10:03:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233052AbjLGPDs (ORCPT ); Thu, 7 Dec 2023 10:03:48 -0500 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF5B910C0 for ; Thu, 7 Dec 2023 07:03:52 -0800 (PST) Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-40c09d62b70so12540405e9.1 for ; Thu, 07 Dec 2023 07:03:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961431; x=1702566231; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=nzUGYpwYAIocf74DCpTux4YFllX1H7F6zPZyiGHWp1Y=; b=GLL16fO29Orq5Jz30/saRHWQMHiNkAiqhbqsgY8aiyAaU0vEXnE+0M/K1ydxsZ73XM SlUV0ARtINt/6M3b/YChKOd7VwvQUhi169uNz08xR4Li1JflFw3Je2+teSA4H4I0cCPW K39pBh8Oy2OvAZwLnca5NHaGY7ZEZOtMYQCdGB4RgJKCuGaBOJ56jCXXv/wyGkGf9o4M MuhRBDAbMNRd815kI8aeNcCLSiHgfm9lv/jR8EXh8IpQo2UhqubuJm0/aD+190RkeeLW liGDlm5uVli0iD0w/1z+2x0wuE2UFOd7Q+dXR93vK1W/UEXWRhhQ/I/Ucx6qGvPsx7Sq MZdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961431; x=1702566231; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nzUGYpwYAIocf74DCpTux4YFllX1H7F6zPZyiGHWp1Y=; b=FOVQ6i6zGkWYomUamU2lK92bSLXKJaZEpOtIi/zJBHOCd1VwpSgZ+O0/sp0DwQGemq SNfBqZ16iBNM806J3W48ZVEjW5Z7kGDXwt6hgb9qqRXtmP8UuUHw5O6JSC+JCr24scNq UQBJgrGzsWKfqQNmET7odp53iJiIKj+QuvntJRX9dzk3Jo0KLFsRmDdwXuvkcXCkITS8 sI2lvS5ju3B5Bwc21eOsx1JN4UMvNEuzphNj7hjl6i8JpLmvMfEB/beIKOi82rfLaaOa U+Iq3RbdgTJLgbqZhCTzb/er+7h5vJjRQFmq5em/Pn9v4Wp9bQlSAqZplBmC9nZ/IcU7 R9pQ== X-Gm-Message-State: AOJu0Yy7N/l1/N/9HcjLOSy+rhKWo27VhknJwUywS0sZueY0TYkTul2l 4roXrl6XNKBCEUXd0Y1LhFrt7g== X-Received: by 2002:a05:600c:198b:b0:40c:5ee:2dda with SMTP id t11-20020a05600c198b00b0040c05ee2ddamr1118146wmq.177.1701961430948; Thu, 07 Dec 2023 07:03:50 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id u13-20020a05600c19cd00b0040b42df75fcsm2187533wmq.39.2023.12.07.07.03.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:03:50 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 0/4] Remove preventive sfence.vma Date: Thu, 7 Dec 2023 16:03:44 +0100 Message-Id: <20231207150348.82096-1-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Thu, 07 Dec 2023 07:03:58 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784635932031858529 X-GMAIL-MSGID: 1784635932031858529 In RISC-V, after a new mapping is established, a sfence.vma needs to be emitted for different reasons: - if the uarch caches invalid entries, we need to invalidate it otherwise we would trap on this invalid entry, - if the uarch does not cache invalid entries, a reordered access could fail to see the new mapping and then trap (sfence.vma acts as a fence). We can actually avoid emitting those (mostly) useless and costly sfence.vma by handling the traps instead: - for new kernel mappings: only vmalloc mappings need to be taken care of, other new mapping are rare and already emit the required sfence.vma if needed. That must be achieved very early in the exception path as explained in patch 1, and this also fixes our fragile way of dealing with vmalloc faults. - for new user mappings: that can be handled in the page fault path as done in patch 3. Patch 2 is certainly a TEMP patch which allows to detect at runtime if a uarch caches invalid TLB entries. Patch 4 is a TEMP patch which allows to expose through debugfs the different sfence.vma that are emitted, which can be used for benchmarking. On our uarch that does not cache invalid entries and a 6.5 kernel, the gains are measurable: * Kernel boot: 6% * ltp - mmapstress01: 8% * lmbench - lat_pagefault: 20% * lmbench - lat_mmap: 5% On uarchs that cache invalid entries, the results are more mitigated and need to be explored more thoroughly (if anyone is interested!): that can be explained by the extra page faults, which depending on "how much" the uarch caches invalid entries, could kill the benefits of removing the preventive sfence.vma. Ved Shanbhogue has prepared a new extension to be used by uarchs that do not cache invalid entries, which will certainly be used instead of patch 2. Thanks to Ved and Matt Evans for triggering the discussion that led to this patchset! That's an RFC, so please don't mind the checkpatch warnings and dirty comments. It applies on 6.6. Any feedback, test or relevant benchmark are welcome :) Alexandre Ghiti (4): riscv: Stop emitting preventive sfence.vma for new vmalloc mappings riscv: Add a runtime detection of invalid TLB entries caching riscv: Stop emitting preventive sfence.vma for new userspace mappings TEMP: riscv: Add debugfs interface to retrieve #sfence.vma arch/arm64/include/asm/pgtable.h | 2 +- arch/mips/include/asm/pgtable.h | 6 +- arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 +- arch/riscv/include/asm/cacheflush.h | 19 ++- arch/riscv/include/asm/pgtable.h | 45 ++++--- arch/riscv/include/asm/thread_info.h | 5 + arch/riscv/include/asm/tlbflush.h | 4 + arch/riscv/kernel/asm-offsets.c | 5 + arch/riscv/kernel/entry.S | 94 +++++++++++++ arch/riscv/kernel/sbi.c | 12 ++ arch/riscv/mm/init.c | 126 ++++++++++++++++++ arch/riscv/mm/tlbflush.c | 17 +++ include/linux/pgtable.h | 8 +- mm/memory.c | 12 +- 14 files changed, 331 insertions(+), 32 deletions(-)