From patchwork Thu Dec 7 15:03:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 175193 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp4842498vqy; Thu, 7 Dec 2023 07:05:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IEZsB4bnlWE3MvMgF1XAjE8e8aW0JzTNMlHJp2Oe52f7Oqpiywk+WDBDXa/3RUgJ+dhL2wI X-Received: by 2002:a17:90b:1a88:b0:286:9cdc:c2e1 with SMTP id ng8-20020a17090b1a8800b002869cdcc2e1mr2090572pjb.13.1701961508689; Thu, 07 Dec 2023 07:05:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701961508; cv=none; d=google.com; s=arc-20160816; b=hKFeRazCHCqAU5ytBEe95gpcEsgoLUbldWaTLs2Pjouw+2XIiuH9/DQP4A0r1Ca1UM GvYSIPoA79i+FqQ5CC98XMgKeeSeb/a/tTFVFXcxlTtfGPWRg7lqDgKMLRDjUAjrRBSI MDLl2RAWTQoc22MK6b24VmU1FVk5nmNz3/Efj0PW15EM4g189DOQANL8W4fVxZT2RE83 qJvNBJyB5N5EFt9UxbsBoLcqeakDM+AiDqNIAgglAXIwPxGrDmHDVfyTFt8oEQ0aSSVF LYaCqvlXRa9TcwRJeZTpqjpvojQxEbNrXnTFNz/VVZDodAw9ZtzfXZptrhlKeds4MwrZ 4qMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=pjO7lbfVOd3z5ovuMphYRbL/oKKYVGVFcCWOfr+CdbY=; fh=ghJZAeZ4zrru54jVI3XtuZXbadJtp4HIJMVSTZEe2l4=; b=oW2//qVP64ItvUwAyAV/tn0VWJFzHxQ3FsROUsDjx/heztwre3P7CfJRXi6b6O5BTs AR7yTfZ37ehFfhTjnHzEHCo8jJg7KWtam6/8IDsoHqfaTVERUUmWZr8/6kVhfIY4pDU4 ukWcnmVrbMDq+Rz0jcfei7BnA9/CT93IpJhQAr/MeBC+rVp8BbBC7yxO2CVOM4196G7g M8C5lwC2Omt3yQe/g/OTqNZDZSMzM1QLjvhnfG3d8GoWXCWsPRWCAY3HZHmU8Q41kqNJ Uxrn55ExfoBOk1SRA4rkDW4TcvFZKd5aD1+RTCe9ZGssF6Q8+bwfr0okR+KVFAX7G+nv EokQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b="bECG/SlL"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id n10-20020a17090ac68a00b00286a8518f0bsi1248874pjt.78.2023.12.07.07.05.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:05:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b="bECG/SlL"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 490BB81CB0E3; Thu, 7 Dec 2023 07:04:58 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1442917AbjLGPEt (ORCPT + 99 others); Thu, 7 Dec 2023 10:04:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233041AbjLGPEs (ORCPT ); Thu, 7 Dec 2023 10:04:48 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2CFED5B for ; Thu, 7 Dec 2023 07:04:53 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id ffacd0b85a97d-33349b3f99aso1072669f8f.0 for ; Thu, 07 Dec 2023 07:04:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961492; x=1702566292; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pjO7lbfVOd3z5ovuMphYRbL/oKKYVGVFcCWOfr+CdbY=; b=bECG/SlLscm1TC+uVskRZgBAUEPuiIM5TxzbM6h8sm13Nk6I4iQmhDTxy4uWQNiqIm VlkIfKhbLyIi2jh2y0kb6WJfpqvlRvI2uWoDweJV+Q+R5w6w02PXSabcf3xgWMuqKzWX cOzOmZV1Eo0/4lkH9szfrPid858AFd/IcWZMXuAffFE8+SDl3V9xqRyc0XGR8dQrDPt1 Ynefe8bBM9hIkYErxolJh/e1+m2y8nvvxQe+p7ixtSTC2Yk/qABZn4PAnV7dq7Ihsv4K HcL/Sr/ljTHEhNYey/lySGnBGPuWDko7tfDVp6WGDXAoml4oqFNuriPn0VsSwQQtdRU4 rRUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961492; x=1702566292; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pjO7lbfVOd3z5ovuMphYRbL/oKKYVGVFcCWOfr+CdbY=; b=nosU/xlUPtpeUKyk7w7BVkgXun6uIQOfwgqm11NvsSvPEnw3hIqnNw7AGy64zFtVhG ISslpobHOnjfpXt2lpRHgB6SkfAFR8xNvngRvHuwFkW3+L+RhImBhU5MrdwF1/k1hF+i whCWn9XhaOS6/shOLA7iZ7tbx0qMA519p4rzQ+qHQmnbgWQo8OABtYjBAQp/40yxC2vg 2NWoEH0wMl4HqPcdYwPgZGuiBV9doiB8zArqLTqdpVGpWMMbSthKmIrISU8rksiptugo QLiMDlCRhUMzT0P/+OZ8XqaDcLKRQpC/oiNTmTnhILF+YKs19VHXotNSlY7QJklUAzRW W0GQ== X-Gm-Message-State: AOJu0YyBYoXbwJakvkjpzIIHoYMUtRj5VqHK4t4utons4SDBdmSwwVOG ts0j9gYfhpnRvM87XG7SOSPKjQ== X-Received: by 2002:a05:600c:152:b0:40c:711:f492 with SMTP id w18-20020a05600c015200b0040c0711f492mr1593121wmm.181.1701961492195; Thu, 07 Dec 2023 07:04:52 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id gw18-20020a05600c851200b004053e9276easm2206857wmb.32.2023.12.07.07.04.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:04:51 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 1/4] riscv: Stop emitting preventive sfence.vma for new vmalloc mappings Date: Thu, 7 Dec 2023 16:03:45 +0100 Message-Id: <20231207150348.82096-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231207150348.82096-1-alexghiti@rivosinc.com> References: <20231207150348.82096-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 07 Dec 2023 07:04:58 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784635990760542241 X-GMAIL-MSGID: 1784635990760542241 In 6.5, we removed the vmalloc fault path because that can't work (see [1] [2]). Then in order to make sure that new page table entries were seen by the page table walker, we had to preventively emit a sfence.vma on all harts [3] but this solution is very costly since it relies on IPI. And even there, we could end up in a loop of vmalloc faults if a vmalloc allocation is done in the IPI path (for example if it is traced, see [4]), which could result in a kernel stack overflow. Those preventive sfence.vma needed to be emitted because: - if the uarch caches invalid entries, the new mapping may not be observed by the page table walker and an invalidation may be needed. - if the uarch does not cache invalid entries, a reordered access could "miss" the new mapping and traps: in that case, we would actually only need to retry the access, no sfence.vma is required. So this patch removes those preventive sfence.vma and actually handles the possible (and unlikely) exceptions. And since the kernel stacks mappings lie in the vmalloc area, this handling must be done very early when the trap is taken, at the very beginning of handle_exception: this also rules out the vmalloc allocations in the fault path. Note that for now, we emit a sfence.vma even for uarchs that do not cache invalid entries as we have no means to know that: that will be fixed in the next patch. Link: https://lore.kernel.org/linux-riscv/20230531093817.665799-1-bjorn@kernel.org/ [1] Link: https://lore.kernel.org/linux-riscv/20230801090927.2018653-1-dylan@andestech.com [2] Link: https://lore.kernel.org/linux-riscv/20230725132246.817726-1-alexghiti@rivosinc.com/ [3] Link: https://lore.kernel.org/lkml/20200508144043.13893-1-joro@8bytes.org/ [4] Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/cacheflush.h | 19 +++++- arch/riscv/include/asm/thread_info.h | 5 ++ arch/riscv/kernel/asm-offsets.c | 5 ++ arch/riscv/kernel/entry.S | 94 ++++++++++++++++++++++++++++ arch/riscv/mm/init.c | 2 + 5 files changed, 124 insertions(+), 1 deletion(-) diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/asm/cacheflush.h index 3cb53c4df27c..a916cbc69d47 100644 --- a/arch/riscv/include/asm/cacheflush.h +++ b/arch/riscv/include/asm/cacheflush.h @@ -37,7 +37,24 @@ static inline void flush_dcache_page(struct page *page) flush_icache_mm(vma->vm_mm, 0) #ifdef CONFIG_64BIT -#define flush_cache_vmap(start, end) flush_tlb_kernel_range(start, end) +extern u64 new_vmalloc[]; +extern char _end[]; +#define flush_cache_vmap flush_cache_vmap +static inline void flush_cache_vmap(unsigned long start, unsigned long end) +{ + if ((start < VMALLOC_END && end > VMALLOC_START) || + (start < MODULES_END && end > MODULES_VADDR)) { + int i; + + /* + * We don't care if concurrently a cpu resets this value since + * the only place this can happen is in handle_exception() where + * an sfence.vma is emitted. + */ + for (i = 0; i < NR_CPUS / sizeof(u64) + 1; ++i) + new_vmalloc[i] = -1ULL; + } +} #endif #ifndef CONFIG_SMP diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h index 1833beb00489..8fe12fa6c329 100644 --- a/arch/riscv/include/asm/thread_info.h +++ b/arch/riscv/include/asm/thread_info.h @@ -60,6 +60,11 @@ struct thread_info { long user_sp; /* User stack pointer */ int cpu; unsigned long syscall_work; /* SYSCALL_WORK_ flags */ + /* + * Used in handle_exception() to save a0, a1 and a2 before knowing if we + * can access the kernel stack. + */ + unsigned long a0, a1, a2; }; /* diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index d6a75aac1d27..340c1c84560d 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -34,10 +34,15 @@ void asm_offsets(void) OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]); OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]); OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]); + + OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu); OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags); OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp); + OFFSET(TASK_TI_A0, task_struct, thread_info.a0); + OFFSET(TASK_TI_A1, task_struct, thread_info.a1); + OFFSET(TASK_TI_A2, task_struct, thread_info.a2); OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]); diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 143a2bb3e697..3a3c7b563816 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -14,6 +14,88 @@ #include #include +.macro new_vmalloc_check + REG_S a0, TASK_TI_A0(tp) + REG_S a1, TASK_TI_A1(tp) + REG_S a2, TASK_TI_A2(tp) + + csrr a0, CSR_CAUSE + /* Exclude IRQs */ + blt a0, zero, _new_vmalloc_restore_context + /* Only check new_vmalloc if we are in page/protection fault */ + li a1, EXC_LOAD_PAGE_FAULT + beq a0, a1, _new_vmalloc_kernel_address + li a1, EXC_STORE_PAGE_FAULT + beq a0, a1, _new_vmalloc_kernel_address + li a1, EXC_INST_PAGE_FAULT + bne a0, a1, _new_vmalloc_restore_context + +_new_vmalloc_kernel_address: + /* Is it a kernel address? */ + csrr a0, CSR_TVAL + bge a0, zero, _new_vmalloc_restore_context + + /* Check if a new vmalloc mapping appeared that could explain the trap */ + + /* + * Computes: + * a0 = &new_vmalloc[BIT_WORD(cpu)] + * a1 = BIT_MASK(cpu) + */ + REG_L a2, TASK_TI_CPU(tp) + /* + * Compute the new_vmalloc element position: + * (cpu / 64) * 8 = (cpu >> 6) << 3 + */ + srli a1, a2, 6 + slli a1, a1, 3 + la a0, new_vmalloc + add a0, a0, a1 + /* + * Compute the bit position in the new_vmalloc element: + * bit_pos = cpu % 64 = cpu - (cpu / 64) * 64 = cpu - (cpu >> 6) << 6 + * = cpu - ((cpu >> 6) << 3) << 3 + */ + slli a1, a1, 3 + sub a1, a2, a1 + /* Compute the "get mask": 1 << bit_pos */ + li a2, 1 + sll a1, a2, a1 + + /* Check the value of new_vmalloc for this cpu */ + ld a2, 0(a0) + and a2, a2, a1 + beq a2, zero, _new_vmalloc_restore_context + + ld a2, 0(a0) + not a1, a1 + and a1, a2, a1 + sd a1, 0(a0) + + /* Only emit a sfence.vma if the uarch caches invalid entries */ + la a0, tlb_caching_invalid_entries + lb a0, 0(a0) + beqz a0, _new_vmalloc_no_caching_invalid_entries + sfence.vma +_new_vmalloc_no_caching_invalid_entries: + // debug + la a0, nr_sfence_vma_handle_exception + li a1, 1 + amoadd.w a0, a1, (a0) + // end debug + REG_L a0, TASK_TI_A0(tp) + REG_L a1, TASK_TI_A1(tp) + REG_L a2, TASK_TI_A2(tp) + csrw CSR_SCRATCH, x0 + sret + +_new_vmalloc_restore_context: + REG_L a0, TASK_TI_A0(tp) + REG_L a1, TASK_TI_A1(tp) + REG_L a2, TASK_TI_A2(tp) +.endm + + SYM_CODE_START(handle_exception) /* * If coming from userspace, preserve the user thread pointer and load @@ -25,6 +107,18 @@ SYM_CODE_START(handle_exception) _restore_kernel_tpsp: csrr tp, CSR_SCRATCH + + /* + * The RISC-V kernel does not eagerly emit a sfence.vma after each + * new vmalloc mapping, which may result in exceptions: + * - if the uarch caches invalid entries, the new mapping would not be + * observed by the page table walker and an invalidation is needed. + * - if the uarch does not cache invalid entries, a reordered access + * could "miss" the new mapping and traps: in that case, we only need + * to retry the access, no sfence.vma is required. + */ + new_vmalloc_check + REG_S sp, TASK_TI_KERNEL_SP(tp) #ifdef CONFIG_VMAP_STACK diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 0798bd861dcb..379403de6c6f 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -36,6 +36,8 @@ #include "../kernel/head.h" +u64 new_vmalloc[NR_CPUS / sizeof(u64) + 1]; + struct kernel_mapping kernel_map __ro_after_init; EXPORT_SYMBOL(kernel_map); #ifdef CONFIG_XIP_KERNEL From patchwork Thu Dec 7 15:03:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 175194 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp4843504vqy; Thu, 7 Dec 2023 07:06:12 -0800 (PST) X-Google-Smtp-Source: AGHT+IGe5VXVS820JpYDN/aPPR1yYqy8LwVcOKh4TnW2iXqv/QFMr/oZGFBhgAn7fjoUDebZpD13 X-Received: by 2002:a17:902:e742:b0:1cf:d2c3:2865 with SMTP id p2-20020a170902e74200b001cfd2c32865mr3243199plf.40.1701961572024; Thu, 07 Dec 2023 07:06:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701961572; cv=none; d=google.com; s=arc-20160816; b=qVGEq6f4FAJ9W+qrROiMMm9Ka7Nuds+HpoJBXt+HCf+xUEH8R/2IUp4IQtslRYcRCE 61YflV0Jx9/8AV28lNqpjR3Cvh3Hr4Gh0jCdDoGx4t8iZP9hiPaMITsVaVcrFlcBaSaO Zn7LK2jedHOV7WLzel/a4CRPYVQKSuKb0ngut4corjKPE9560rXO5hrCyfoiykCBWWgK wC107zSQRub0NmFZI/ZG82Szf8XxSemSVEHRTOedf9mkSWi0FeTIDaM5r/UdyaRYA3Nn CyfWMGSnRsCsUsp2UOoxtH1t9ox3o3h+YRI/aj6LsVRTqWEYMFhYG3aRetLwpGsM5ZGQ zDDw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=LVN+VzhbTBys9mVhAQGqZxkaIjV/7TRqzMN+a14FBcU=; fh=ghJZAeZ4zrru54jVI3XtuZXbadJtp4HIJMVSTZEe2l4=; b=LMVmIYwxP84XV4f77+s54Ftt8V+oHfJqfVIblyVoEb+JhV6FdAOP0Yf4NI+lm2kOCl WiLuc8JhpwBTQ6Ootrh0C21YjSKhERMYO5BbELHEzZEMZ1MnxIuZ9+ZqYJUFyU+EvXTG 3WzwCuMPmQE/vzCyEFXJZuquPYKf3uluZMvXeLGv/iHT1oIBxDJVYIZFpLuQsHP39J38 AVnmxvQzGmFjgZyJW7cYtbmUwAUZlL9nAKP26KbAbCpiY+F2SbdGjqkkpTyZSz10nw8v AioHsANgaEFqg+nfDHB9RPFv9+wc8dOEyrodutbjGJNyWihRu20Y+0RTXRRf+kHnJVRJ C1Bg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=EyZdHs8Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id i34-20020a635422000000b005c688ef7549si1341042pgb.98.2023.12.07.07.06.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:06:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=EyZdHs8Z; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 7B2F681CDBA5; Thu, 7 Dec 2023 07:06:07 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1443396AbjLGPF5 (ORCPT + 99 others); Thu, 7 Dec 2023 10:05:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1443171AbjLGPFt (ORCPT ); Thu, 7 Dec 2023 10:05:49 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15DD0170E for ; Thu, 7 Dec 2023 07:05:55 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id ffacd0b85a97d-33349b3f99aso1074041f8f.0 for ; Thu, 07 Dec 2023 07:05:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961553; x=1702566353; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LVN+VzhbTBys9mVhAQGqZxkaIjV/7TRqzMN+a14FBcU=; b=EyZdHs8Zqz8f/yWq6LPZvF4MI2eXyquhSLxJV7WGLWRx107tdftsJjL2Id6BSOC0qE lWBupDT7/zSqVzewHMq7mukHteV25QmszD6/rhsDboRJHAXtDkinT3yEP5LKUzdYEOsU b6Np79lb7XikBilN5KN/9y4FwtXBhQHN6tKSniWWLvN+DmmU2dIRBnCTfsKbZtC6v0lm cVNjI/UFTT1/41688M+KsqPN7WZ3q9BxHJwdDCL48gZx4Iq59fv1QDhhgkgGJ8S+zRk1 o1TqNOV8/JqqYgFaweabOrT5wifF5FmzgwhIbPfxL0T166v1gHYLxk9ng+ZCZpi+YYIB JIEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961553; x=1702566353; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LVN+VzhbTBys9mVhAQGqZxkaIjV/7TRqzMN+a14FBcU=; b=DXchl/4n8pJ9VoLKR6ISmvCkH0p4iYNgRPOW4DgFq/hMI/2RgCADErIZV07XY2ngod ej94YujxLjxV+5jeQSMPyUZdfNRYrOb62mdlVdsB3G3eMEfaGiMAElN5KoiiLCOIJczu C2ZRQ3GeCsbfAbfThMDinSa3kV3phMR7blAk5Ztvo7/L0F1XgDeWaepqY4FMla0d8LIK FuTyRx4F8s0QqGLpHldxpKbAANVL9XzRZalYjNAEkxFLaR5ST97O1BVb6jdyfUQunprJ 0XuBP96tYouogKW/fERGf6OxP8yMM2/G66eSPxqkndmrZrKfv22QjkYQ0s+5EzUe1cqv llhg== X-Gm-Message-State: AOJu0YwsFQJxagCgRC7npvJRQw8pQpoIoz9KdUN2QgKEs06pVf/bnOiM n4vrsInOcBCASENgaRxmSH8D9g== X-Received: by 2002:adf:ea82:0:b0:333:1907:c2a3 with SMTP id s2-20020adfea82000000b003331907c2a3mr1503404wrm.21.1701961553264; Thu, 07 Dec 2023 07:05:53 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id p13-20020a056000018d00b00333415503a7sm1644486wrx.22.2023.12.07.07.05.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:05:53 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 2/4] riscv: Add a runtime detection of invalid TLB entries caching Date: Thu, 7 Dec 2023 16:03:46 +0100 Message-Id: <20231207150348.82096-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231207150348.82096-1-alexghiti@rivosinc.com> References: <20231207150348.82096-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 07 Dec 2023 07:06:07 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784636057331302946 X-GMAIL-MSGID: 1784636057331302946 This mechanism allows to completely bypass the sfence.vma introduced by the previous commit for uarchs that do not cache invalid TLB entries. Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/init.c | 124 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 124 insertions(+) diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c index 379403de6c6f..2e854613740c 100644 --- a/arch/riscv/mm/init.c +++ b/arch/riscv/mm/init.c @@ -56,6 +56,8 @@ bool pgtable_l5_enabled = IS_ENABLED(CONFIG_64BIT) && !IS_ENABLED(CONFIG_XIP_KER EXPORT_SYMBOL(pgtable_l4_enabled); EXPORT_SYMBOL(pgtable_l5_enabled); +bool tlb_caching_invalid_entries; + phys_addr_t phys_ram_base __ro_after_init; EXPORT_SYMBOL(phys_ram_base); @@ -750,6 +752,18 @@ static void __init disable_pgtable_l4(void) satp_mode = SATP_MODE_39; } +static void __init enable_pgtable_l5(void) +{ + pgtable_l5_enabled = true; + satp_mode = SATP_MODE_57; +} + +static void __init enable_pgtable_l4(void) +{ + pgtable_l4_enabled = true; + satp_mode = SATP_MODE_48; +} + static int __init print_no4lvl(char *p) { pr_info("Disabled 4-level and 5-level paging"); @@ -826,6 +840,112 @@ static __init void set_satp_mode(uintptr_t dtb_pa) memset(early_pud, 0, PAGE_SIZE); memset(early_pmd, 0, PAGE_SIZE); } + +/* Determine at runtime if the uarch caches invalid TLB entries */ +static __init void set_tlb_caching_invalid_entries(void) +{ +#define NR_RETRIES_CACHING_INVALID_ENTRIES 50 + uintptr_t set_tlb_caching_invalid_entries_pmd = ((unsigned long)set_tlb_caching_invalid_entries) & PMD_MASK; + // TODO the test_addr as defined below could go into another pud... + uintptr_t test_addr = set_tlb_caching_invalid_entries_pmd + 2 * PMD_SIZE; + pmd_t valid_pmd; + u64 satp; + int i = 0; + + /* To ease the page table creation */ + disable_pgtable_l5(); + disable_pgtable_l4(); + + /* Establish a mapping for set_tlb_caching_invalid_entries() in sv39 */ + create_pgd_mapping(early_pg_dir, + set_tlb_caching_invalid_entries_pmd, + (uintptr_t)early_pmd, + PGDIR_SIZE, PAGE_TABLE); + + /* Handle the case where set_tlb_caching_invalid_entries straddles 2 PMDs */ + create_pmd_mapping(early_pmd, + set_tlb_caching_invalid_entries_pmd, + set_tlb_caching_invalid_entries_pmd, + PMD_SIZE, PAGE_KERNEL_EXEC); + create_pmd_mapping(early_pmd, + set_tlb_caching_invalid_entries_pmd + PMD_SIZE, + set_tlb_caching_invalid_entries_pmd + PMD_SIZE, + PMD_SIZE, PAGE_KERNEL_EXEC); + + /* Establish an invalid mapping */ + create_pmd_mapping(early_pmd, test_addr, 0, PMD_SIZE, __pgprot(0)); + + /* Precompute the valid pmd here because the mapping for pfn_pmd() won't exist */ + valid_pmd = pfn_pmd(PFN_DOWN(set_tlb_caching_invalid_entries_pmd), PAGE_KERNEL); + + local_flush_tlb_all(); + satp = PFN_DOWN((uintptr_t)&early_pg_dir) | SATP_MODE_39; + csr_write(CSR_SATP, satp); + + /* + * Set stvec to after the trapping access, access this invalid mapping + * and legitimately trap + */ + // TODO: Should I save the previous stvec? +#define ASM_STR(x) __ASM_STR(x) + asm volatile( + "la a0, 1f \n" + "csrw " ASM_STR(CSR_TVEC) ", a0 \n" + "ld a0, 0(%0) \n" + ".align 2 \n" + "1: \n" + : + : "r" (test_addr) + : "a0" + ); + + /* Now establish a valid mapping to check if the invalid one is cached */ + early_pmd[pmd_index(test_addr)] = valid_pmd; + + /* + * Access the valid mapping multiple times: indeed, we can't use + * sfence.vma as a barrier to make sure the cpu did not reorder accesses + * so we may trap even if the uarch does not cache invalid entries. By + * trying a few times, we make sure that those uarchs will see the right + * mapping at some point. + */ + + i = NR_RETRIES_CACHING_INVALID_ENTRIES; + +#define ASM_STR(x) __ASM_STR(x) + asm_volatile_goto( + "la a0, 1f \n" + "csrw " ASM_STR(CSR_TVEC) ", a0 \n" + ".align 2 \n" + "1: \n" + "addi %0, %0, -1 \n" + "blt %0, zero, %l[caching_invalid_entries] \n" + "ld a0, 0(%1) \n" + : + : "r" (i), "r" (test_addr) + : "a0" + : caching_invalid_entries + ); + + csr_write(CSR_SATP, 0ULL); + local_flush_tlb_all(); + + /* If we don't trap, the uarch does not cache invalid entries! */ + tlb_caching_invalid_entries = false; + goto clean; + +caching_invalid_entries: + csr_write(CSR_SATP, 0ULL); + local_flush_tlb_all(); + + tlb_caching_invalid_entries = true; +clean: + memset(early_pg_dir, 0, PAGE_SIZE); + memset(early_pmd, 0, PAGE_SIZE); + + enable_pgtable_l4(); + enable_pgtable_l5(); +} #endif /* @@ -1072,6 +1192,7 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa) #endif #if defined(CONFIG_64BIT) && !defined(CONFIG_XIP_KERNEL) + set_tlb_caching_invalid_entries(); set_satp_mode(dtb_pa); #endif @@ -1322,6 +1443,9 @@ static void __init setup_vm_final(void) local_flush_tlb_all(); pt_ops_set_late(); + + pr_info("uarch caches invalid entries: %s", + tlb_caching_invalid_entries ? "yes" : "no"); } #else asmlinkage void __init setup_vm(uintptr_t dtb_pa) From patchwork Thu Dec 7 15:03:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 175198 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp4844405vqy; Thu, 7 Dec 2023 07:07:13 -0800 (PST) X-Google-Smtp-Source: AGHT+IE+14kw3lCJ4km0TBCFJLoCWkeRNg/4olpebVD5vk0efrtcA9pnBibnpWsg10vNAzcy5o0M X-Received: by 2002:a05:6a00:8c03:b0:6ce:6b7c:ba1b with SMTP id ih3-20020a056a008c0300b006ce6b7cba1bmr2855825pfb.60.1701961632998; Thu, 07 Dec 2023 07:07:12 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701961632; cv=none; d=google.com; s=arc-20160816; b=sAMaGac5dy+Ky+LNcwyXfZn+R1fYx+l6GLxK2eHo8qUm1IIYJ4XGwWika1wvmXcjR8 bJD/yZ5ZcOU6fXlopXUWblA3+lcGDLxyktzwLHXsVt0wH1sS+JWw4vNIU4Rd6BkEvpba RWSAYUuU4NqJnS1Wmvkaqt1Tak4Vr4q0XZUPrqqXxSp9IjH0gsg9UgPl5njqBVGfHkXs 2IXxCLzjnaMWb9WiHXP1Z+HUbKaYqc6oYO37ay3TRWBh1V+04a4VnJj6CDRtSrZ11iIN M8F+l87dNK22dNGtGfG4QDcxYWbRT85biZZ3IXBCU6S103uVJUyp112xGSJml2Jr2oTs Ucyg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=EWcoco88tut+CY63Y/9PXHuBFD03h05uf7rbyZFiPT4=; fh=ghJZAeZ4zrru54jVI3XtuZXbadJtp4HIJMVSTZEe2l4=; b=hWjklyYmS3WjPl5j5ahNmOWVbXf8Zm6bUcGrqvZ6RaHPi7A8fzWQiFOAj8wHXKvtZa ASbF3wxrWFjAmMNo2zhJ7FEyRzZDoPUWpkZ1UhXhjf+l+c4+tx3JIgAIjM5Qs1R6RyMm AMOW6f8tdcAfg44tpWoDLpdqfSCQL9N2R/xDzxcHLslROnJKA9TDPgCs+t3riw4BLC9g DAv57nJMVbyTTtWs16XOJrAn3UQRrtWTFgivcWj0I5fMP3P5k2ssCXR+7WqocZnEf9Jf t+TwtFYXXuuJOIIny+bf+O73OAXYlNMJHRUBnsrLC0Ehv8hKDfua5dl3jP0mDltBmkzp KJzw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=YWVt+2yO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from groat.vger.email (groat.vger.email. [23.128.96.35]) by mx.google.com with ESMTPS id b2-20020a056a00114200b006ce823785a2si1392519pfm.134.2023.12.07.07.07.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:07:12 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) client-ip=23.128.96.35; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=YWVt+2yO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.35 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 0C7B6806BE18; Thu, 7 Dec 2023 07:07:01 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1443171AbjLGPGw (ORCPT + 99 others); Thu, 7 Dec 2023 10:06:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232943AbjLGPGu (ORCPT ); Thu, 7 Dec 2023 10:06:50 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 180EA128 for ; Thu, 7 Dec 2023 07:06:56 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id ffacd0b85a97d-33350fcb2c7so752270f8f.2 for ; Thu, 07 Dec 2023 07:06:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961614; x=1702566414; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=EWcoco88tut+CY63Y/9PXHuBFD03h05uf7rbyZFiPT4=; b=YWVt+2yOSzLKdBA6mFxuQnKPoBnm59Qrl4iFRlKaQjOfKdrnHVoPUl/taei2v0CntL T/WWc3PIORGPZHXfTfgGEm/D9uV/LUaZmT2dEytAZEgjPof3K5/pQzZV4WERP8LBV4on Z5cAgODFO65IvGVUDbdOyEDpSsZOBHfkNcZOOWL3liAD81quM2qrtulYeA4JqbnDwRyE YEJmhMHNNcr6B2EOo/MCkBJ6rB43y7WR5L1lCIZOoUg4gr71KD2qW5tuA6ooLrOYykF8 pVVAxjhkJ7c1QV2OZG9NUNeOEqUEZRRqKazaseNHrFunCBZBSAlMQM+lyfesSjy9HSyV Pdyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961614; x=1702566414; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EWcoco88tut+CY63Y/9PXHuBFD03h05uf7rbyZFiPT4=; b=mMwUuJEzGglgScjuUp0z6G4kPzs9cCb0Sdds5K5AEMmVBZ4HNJLUJadoFgDzaTVBC1 TqMvLZzvjvy+ZnvrtYgFQpfGXEC7CeCgTrKHVCne7iK7Utu/af9wxJI+aG/+JIsowWDA vCpDh5ps+VbtjaPDV8WMj6aE7yNppBGp4DPF4fltuzurNye7A/6g8aIYhHQcxZOBerEj CORlk+C87NXUyskaQTqTY4nnqv6+xt4OIowd+CCKBGMkLlOk5l0s2rFGIUr5ZVuEVZEn aV1FgI+CsIQz+SZo4ajLPfm3I3O1/KvoO4pPPl+zONufLRWH+7Ozm9VuDQmkQZYphloD clFA== X-Gm-Message-State: AOJu0YywPPw+SPfy0ZDaQDgGnMtrIRHe6gQq4y9b94EjhX3zV1ZnYgJQ isymt0J338s4GFUCH658S6e4Zw== X-Received: by 2002:a5d:67c5:0:b0:333:3867:c5aa with SMTP id n5-20020a5d67c5000000b003333867c5aamr1932506wrw.20.1701961614467; Thu, 07 Dec 2023 07:06:54 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id a8-20020adffb88000000b003335e67e574sm1649359wrr.78.2023.12.07.07.06.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:06:54 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 3/4] riscv: Stop emitting preventive sfence.vma for new userspace mappings Date: Thu, 7 Dec 2023 16:03:47 +0100 Message-Id: <20231207150348.82096-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231207150348.82096-1-alexghiti@rivosinc.com> References: <20231207150348.82096-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Thu, 07 Dec 2023 07:07:01 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784636121271799367 X-GMAIL-MSGID: 1784636121271799367 The preventive sfence.vma were emitted because new mappings must be made visible to the page table walker, either the uarch caches invalid entries or not. Actually, there is no need to preventively sfence.vma on new mappings for userspace, this should be handled only in the page fault path. This allows to drastically reduce the number of sfence.vma emitted: * Ubuntu boot to login: Before: ~630k sfence.vma After: ~200k sfence.vma * ltp - mmapstress01 Before: ~45k After: ~6.3k * lmbench - lat_pagefault Before: ~665k After: 832 (!) * lmbench - lat_mmap Before: ~546k After: 718 (!) The only issue with the removal of sfence.vma in update_mmu_cache() is that on uarchs that cache invalid entries, those won't be invalidated until the process takes a fault: so that's an additional fault in those cases. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 2 +- arch/mips/include/asm/pgtable.h | 6 +-- arch/powerpc/include/asm/book3s/64/tlbflush.h | 8 ++-- arch/riscv/include/asm/pgtable.h | 43 +++++++++++-------- include/linux/pgtable.h | 8 +++- mm/memory.c | 12 +++++- 6 files changed, 48 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7f7d9b1df4e5..728f25f529a5 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -57,7 +57,7 @@ static inline bool arch_thp_swp_supported(void) * fault on one CPU which has been handled concurrently by another CPU * does not need to perform additional invalidation. */ -#define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0) +#define flush_tlb_fix_spurious_write_fault(vma, address, ptep) do { } while (0) /* * ZERO_PAGE is a global shared page that is always zero: used diff --git a/arch/mips/include/asm/pgtable.h b/arch/mips/include/asm/pgtable.h index 430b208c0130..84439fe6ed29 100644 --- a/arch/mips/include/asm/pgtable.h +++ b/arch/mips/include/asm/pgtable.h @@ -478,9 +478,9 @@ static inline pgprot_t pgprot_writecombine(pgprot_t _prot) return __pgprot(prot); } -static inline void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, - unsigned long address, - pte_t *ptep) +static inline void flush_tlb_fix_spurious_write_fault(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) { } diff --git a/arch/powerpc/include/asm/book3s/64/tlbflush.h b/arch/powerpc/include/asm/book3s/64/tlbflush.h index 1950c1b825b4..7166d56f90db 100644 --- a/arch/powerpc/include/asm/book3s/64/tlbflush.h +++ b/arch/powerpc/include/asm/book3s/64/tlbflush.h @@ -128,10 +128,10 @@ static inline void flush_tlb_page(struct vm_area_struct *vma, #define flush_tlb_page(vma, addr) local_flush_tlb_page(vma, addr) #endif /* CONFIG_SMP */ -#define flush_tlb_fix_spurious_fault flush_tlb_fix_spurious_fault -static inline void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, - unsigned long address, - pte_t *ptep) +#define flush_tlb_fix_spurious_write_fault flush_tlb_fix_spurious_write_fault +static inline void flush_tlb_fix_spurious_write_fault(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) { /* * Book3S 64 does not require spurious fault flushes because the PTE diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index b2ba3f79cfe9..89aa5650f104 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -472,28 +472,20 @@ static inline void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, unsigned long address, pte_t *ptep, unsigned int nr) { - /* - * The kernel assumes that TLBs don't cache invalid entries, but - * in RISC-V, SFENCE.VMA specifies an ordering constraint, not a - * cache flush; it is necessary even after writing invalid entries. - * Relying on flush_tlb_fix_spurious_fault would suffice, but - * the extra traps reduce performance. So, eagerly SFENCE.VMA. - */ - while (nr--) - local_flush_tlb_page(address + nr * PAGE_SIZE); } #define update_mmu_cache(vma, addr, ptep) \ update_mmu_cache_range(NULL, vma, addr, ptep, 1) #define __HAVE_ARCH_UPDATE_MMU_TLB -#define update_mmu_tlb update_mmu_cache +static inline void update_mmu_tlb(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) +{ + flush_tlb_range(vma, address, address + PAGE_SIZE); +} static inline void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - pte_t *ptep = (pte_t *)pmdp; - - update_mmu_cache(vma, address, ptep); } #define __HAVE_ARCH_PTE_SAME @@ -548,13 +540,26 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty) { - if (!pte_same(*ptep, entry)) + if (!pte_same(*ptep, entry)) { __set_pte_at(ptep, entry); - /* - * update_mmu_cache will unconditionally execute, handling both - * the case that the PTE changed and the spurious fault case. - */ - return true; + /* Here only not svadu is impacted */ + flush_tlb_page(vma, address); + return true; + } + + return false; +} + +extern u64 nr_sfence_vma_handle_exception; +extern bool tlb_caching_invalid_entries; + +#define flush_tlb_fix_spurious_read_fault flush_tlb_fix_spurious_read_fault +static inline void flush_tlb_fix_spurious_read_fault(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) +{ + if (tlb_caching_invalid_entries) + flush_tlb_page(vma, address); } #define __HAVE_ARCH_PTEP_GET_AND_CLEAR diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index af7639c3b0a3..7abaf42ef612 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -931,8 +931,12 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) # define pte_accessible(mm, pte) ((void)(pte), 1) #endif -#ifndef flush_tlb_fix_spurious_fault -#define flush_tlb_fix_spurious_fault(vma, address, ptep) flush_tlb_page(vma, address) +#ifndef flush_tlb_fix_spurious_write_fault +#define flush_tlb_fix_spurious_write_fault(vma, address, ptep) flush_tlb_page(vma, address) +#endif + +#ifndef flush_tlb_fix_spurious_read_fault +#define flush_tlb_fix_spurious_read_fault(vma, address, ptep) #endif /* diff --git a/mm/memory.c b/mm/memory.c index 517221f01303..5cb0ccf0c03f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5014,8 +5014,16 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * with threads. */ if (vmf->flags & FAULT_FLAG_WRITE) - flush_tlb_fix_spurious_fault(vmf->vma, vmf->address, - vmf->pte); + flush_tlb_fix_spurious_write_fault(vmf->vma, vmf->address, + vmf->pte); + else + /* + * With the pte_same(ptep_get(vmf->pte), entry) check + * that calls update_mmu_tlb() above, multiple threads + * faulting at the same time won't get there. + */ + flush_tlb_fix_spurious_read_fault(vmf->vma, vmf->address, + vmf->pte); } unlock: pte_unmap_unlock(vmf->pte, vmf->ptl); From patchwork Thu Dec 7 15:03:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 175201 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:bcd1:0:b0:403:3b70:6f57 with SMTP id r17csp4845425vqy; Thu, 7 Dec 2023 07:08:22 -0800 (PST) X-Google-Smtp-Source: AGHT+IGCS3RXVG0bH2+CNRXo70j1hpa/c+3o5Wxr15SY/jtUrDi1Yc2KJIMkbH37e5UCXq9L2dj8 X-Received: by 2002:a05:6a20:841f:b0:148:f952:552b with SMTP id c31-20020a056a20841f00b00148f952552bmr3483830pzd.51.1701961702075; Thu, 07 Dec 2023 07:08:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1701961702; cv=none; d=google.com; s=arc-20160816; b=k1vyz5nZGLQQuSQHqs2zg2J6VNnukqF5ej1hruhQtUM6iuasshajoboOM2mjfYS3Sp bEeQqXu8/wEWQki0uTPwFyhsAzphsAV6I7UGMhik3d2vW2C29aZELtJ0un9iYVshy+3Q qhmjxhlS7Lf/OJmmg3Frda3KnxuYWPJf0ExF+yPVWsDqt2G8gjwGF6fpldxRLGRYUgSQ Nh1RQS0Ra2Fuwzt5RxGQnHCz9DzwqN1fdqN3CjN0n/iS1tC1oNhpuYxs5tGM7l/SQHw6 R5uee4oVCive25s+GbtzOj+cvrYpeZCDrgNqW2RDiE75sBpXK0aEAEtoqgQ3VkaLeAmf fLQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Copp2ndGZ68t6KATBAuS2R34UMtC0q0chiu6mkWyRdw=; fh=ghJZAeZ4zrru54jVI3XtuZXbadJtp4HIJMVSTZEe2l4=; b=LLA7jxnbhvJz+BA28EO6EEhE5tR01+6xD5Ha/yzoK7RBnqvfW/yLR00VvaE34I2YPk xXlO/MOFiy/rgtSPGdZlUWjyXklPpuAM1mDJCYbpyF7cF4lGDvI5rcvhMkm/XgBEmSDL ltr7Amf8bogqvQOYijssaYxXHFBn3wgMyOfr+mJFwGmdwc7amfCrOoNOECXCZRMMDVkP lVdBo2hVM7zQZTHVm2GGrXUg5xzJOxvB2gpT6nDcRLJNfRbCHQ3OCvuSSl22NYnevj5c OZioZS3pgJI9Wp3U/7MtQA9d13YYulhSu6p79ocMgWOCHKZWNjtmgGmWGRs6A13mPIns 0pcQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=u8t5vxmE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id c3-20020a6566c3000000b005c6689f9d81si1297084pgw.215.2023.12.07.07.08.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:08:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=u8t5vxmE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 9EC3580AD514; Thu, 7 Dec 2023 07:08:02 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1443254AbjLGPHx (ORCPT + 99 others); Thu, 7 Dec 2023 10:07:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40494 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1442917AbjLGPHv (ORCPT ); Thu, 7 Dec 2023 10:07:51 -0500 Received: from mail-lf1-x130.google.com (mail-lf1-x130.google.com [IPv6:2a00:1450:4864:20::130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5562D1AD for ; Thu, 7 Dec 2023 07:07:57 -0800 (PST) Received: by mail-lf1-x130.google.com with SMTP id 2adb3069b0e04-50bfd8d5c77so971000e87.1 for ; Thu, 07 Dec 2023 07:07:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1701961675; x=1702566475; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Copp2ndGZ68t6KATBAuS2R34UMtC0q0chiu6mkWyRdw=; b=u8t5vxmEmHrKhlXGx9Y1N22VRqEFQuUadlfMtoo8YlaN1NJBs4b/TI9x6wdcbrDmBr 6o+Dq6kP9F/uNNwM6YxO1CGcDP7qK32Qpk+G2r9A9sMO0JDQeB44UvlKaXvCUBc059bN NOiEocdglH4p1FidzTJeE+aIFXQpuZtYPGwQZxpELUowMXVbTv6b/9p7HFmo2YdouxH2 zbjCTJqHU/b4WJQ0pCo6ldSOPSyPKWVu24Tt4RI+KjISKwg8ljY/WsOXs/mSCrik4XGo V78oY1Rt2tpGheqk4yjuj/sDv3vWdtCN9THm+uvqy1W6KJMfV+h4tBJayW3Yjn5vbXJM NjgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701961675; x=1702566475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Copp2ndGZ68t6KATBAuS2R34UMtC0q0chiu6mkWyRdw=; b=llWcqIbl5PDTalrj6J+yxpt1OFtKxQ97ZbONG0+gnv3JothHduFhVGB+WIjUehH0yt Mhah+6GlFmUxv7R2gRDnB7J5xUGZHs8ui4cphPb4ipjaJCz+WS3PvJGQUCKM8+U8169E wOi1EKIH8jXxkH5g8PhS4LSMc9xdQwJDRMauKRV9v1ycYiZlnsszSfNo3tbBKyl8h4+r TvNhdahiL6ccr6cI/a9NQNlSrns/M1WTjqgZ86XryVhXXP3gsCzXIrlV7wIAMbRAamMt gaPhP5YVpvdg1WWMRJLFL7IseXaj0eAt58f3ddNCjLZ+mw8oEYo5/HFML93eG4SvDPmn nxSA== X-Gm-Message-State: AOJu0YzmFdaFH0OzAy/pu44fcuJrYA6B0W2npLe4iUeCBMdnoW9VWO6F EybjIGS1FY3+TITZe37zYzcUxA== X-Received: by 2002:ac2:5147:0:b0:50b:debe:d35d with SMTP id q7-20020ac25147000000b0050bdebed35dmr702137lfd.133.1701961675502; Thu, 07 Dec 2023 07:07:55 -0800 (PST) Received: from alex-rivos.home (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id i11-20020a5d438b000000b003332ef77db4sm1647628wrq.44.2023.12.07.07.07.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Dec 2023 07:07:55 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH RFC/RFT 4/4] TEMP: riscv: Add debugfs interface to retrieve #sfence.vma Date: Thu, 7 Dec 2023 16:03:48 +0100 Message-Id: <20231207150348.82096-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20231207150348.82096-1-alexghiti@rivosinc.com> References: <20231207150348.82096-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 07 Dec 2023 07:08:02 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1784636193548479699 X-GMAIL-MSGID: 1784636193548479699 This is useful for testing/benchmarking. Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/pgtable.h | 6 ++++-- arch/riscv/include/asm/tlbflush.h | 4 ++++ arch/riscv/kernel/sbi.c | 12 ++++++++++++ arch/riscv/mm/tlbflush.c | 17 +++++++++++++++++ 4 files changed, 37 insertions(+), 2 deletions(-) diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 89aa5650f104..b0855a620cfd 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -550,7 +550,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, return false; } -extern u64 nr_sfence_vma_handle_exception; +extern u64 nr_sfence_vma_spurious_read; extern bool tlb_caching_invalid_entries; #define flush_tlb_fix_spurious_read_fault flush_tlb_fix_spurious_read_fault @@ -558,8 +558,10 @@ static inline void flush_tlb_fix_spurious_read_fault(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - if (tlb_caching_invalid_entries) + if (tlb_caching_invalid_entries) { + __sync_fetch_and_add(&nr_sfence_vma_spurious_read, 1UL); flush_tlb_page(vma, address); + } } #define __HAVE_ARCH_PTEP_GET_AND_CLEAR diff --git a/arch/riscv/include/asm/tlbflush.h b/arch/riscv/include/asm/tlbflush.h index a09196f8de68..f419ec9d2207 100644 --- a/arch/riscv/include/asm/tlbflush.h +++ b/arch/riscv/include/asm/tlbflush.h @@ -14,14 +14,18 @@ #ifdef CONFIG_MMU extern unsigned long asid_mask; +extern u64 nr_sfence_vma, nr_sfence_vma_all, nr_sfence_vma_all_asid; + static inline void local_flush_tlb_all(void) { + __sync_fetch_and_add(&nr_sfence_vma_all, 1UL); __asm__ __volatile__ ("sfence.vma" : : : "memory"); } /* Flush one page from local TLB */ static inline void local_flush_tlb_page(unsigned long addr) { + __sync_fetch_and_add(&nr_sfence_vma, 1UL); ALT_FLUSH_TLB_PAGE(__asm__ __volatile__ ("sfence.vma %0" : : "r" (addr) : "memory")); } #else /* CONFIG_MMU */ diff --git a/arch/riscv/kernel/sbi.c b/arch/riscv/kernel/sbi.c index c672c8ba9a2a..ac1617759583 100644 --- a/arch/riscv/kernel/sbi.c +++ b/arch/riscv/kernel/sbi.c @@ -376,6 +376,8 @@ int sbi_remote_fence_i(const struct cpumask *cpu_mask) } EXPORT_SYMBOL(sbi_remote_fence_i); +extern u64 nr_sfence_vma, nr_sfence_vma_all, nr_sfence_vma_all_asid; + /** * sbi_remote_sfence_vma() - Execute SFENCE.VMA instructions on given remote * harts for the specified virtual address range. @@ -389,6 +391,11 @@ int sbi_remote_sfence_vma(const struct cpumask *cpu_mask, unsigned long start, unsigned long size) { + if (size == (unsigned long)-1) + __sync_fetch_and_add(&nr_sfence_vma_all, 1UL); + else + __sync_fetch_and_add(&nr_sfence_vma, ALIGN(size, PAGE_SIZE) / PAGE_SIZE); + return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA, cpu_mask, start, size, 0, 0); } @@ -410,6 +417,11 @@ int sbi_remote_sfence_vma_asid(const struct cpumask *cpu_mask, unsigned long size, unsigned long asid) { + if (size == (unsigned long)-1) + __sync_fetch_and_add(&nr_sfence_vma_all_asid, 1UL); + else + __sync_fetch_and_add(&nr_sfence_vma, ALIGN(size, PAGE_SIZE) / PAGE_SIZE); + return __sbi_rfence(SBI_EXT_RFENCE_REMOTE_SFENCE_VMA_ASID, cpu_mask, start, size, asid, 0); } diff --git a/arch/riscv/mm/tlbflush.c b/arch/riscv/mm/tlbflush.c index 77be59aadc73..75a3e2dff16a 100644 --- a/arch/riscv/mm/tlbflush.c +++ b/arch/riscv/mm/tlbflush.c @@ -3,11 +3,16 @@ #include #include #include +#include #include #include +u64 nr_sfence_vma, nr_sfence_vma_all, nr_sfence_vma_all_asid, + nr_sfence_vma_handle_exception, nr_sfence_vma_spurious_read; + static inline void local_flush_tlb_all_asid(unsigned long asid) { + __sync_fetch_and_add(&nr_sfence_vma_all_asid, 1); __asm__ __volatile__ ("sfence.vma x0, %0" : : "r" (asid) @@ -17,6 +22,7 @@ static inline void local_flush_tlb_all_asid(unsigned long asid) static inline void local_flush_tlb_page_asid(unsigned long addr, unsigned long asid) { + __sync_fetch_and_add(&nr_sfence_vma, 1); __asm__ __volatile__ ("sfence.vma %0, %1" : : "r" (addr), "r" (asid) @@ -149,3 +155,14 @@ void flush_pmd_tlb_range(struct vm_area_struct *vma, unsigned long start, __flush_tlb_range(vma->vm_mm, start, end - start, PMD_SIZE); } #endif + +static int debugfs_nr_sfence_vma(void) +{ + debugfs_create_u64("nr_sfence_vma", 0444, NULL, &nr_sfence_vma); + debugfs_create_u64("nr_sfence_vma_all", 0444, NULL, &nr_sfence_vma_all); + debugfs_create_u64("nr_sfence_vma_all_asid", 0444, NULL, &nr_sfence_vma_all_asid); + debugfs_create_u64("nr_sfence_vma_handle_exception", 0444, NULL, &nr_sfence_vma_handle_exception); + debugfs_create_u64("nr_sfence_vma_spurious_read", 0444, NULL, &nr_sfence_vma_spurious_read); + return 0; +} +device_initcall(debugfs_nr_sfence_vma);