From patchwork Tue Dec 27 03:08:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36746 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200476wrt; Mon, 26 Dec 2022 19:09:39 -0800 (PST) X-Google-Smtp-Source: AMrXdXsVfZHV1yFJPkm5SAp/34GqUF/12GuomqXfir5Bm6jmArHFq/uPXrN2f6idaX/c273/hPIb X-Received: by 2002:a17:90a:b288:b0:225:c2b4:5742 with SMTP id c8-20020a17090ab28800b00225c2b45742mr14762247pjr.34.1672110579546; Mon, 26 Dec 2022 19:09:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110579; cv=none; d=google.com; s=arc-20160816; b=0bDkDdyCpYiG63I3LhJVygcQwzp3C0SWh63DoQ2UDFCNhd0+ZRsBMmdezs+6IGZ/pE /F0RBc21hRZbCuHYvKb8T4P41DLAeZaUNlmPjTAAXhnE3IRa7SOaBlBZVLqqBq3+V43o 4/PgvdHaYutVRUb7JaEHXisIq/tZiNadPQm5/GVYUexhII6ZTpYWu5ThD1tgMNHXD0wV BXHe2FO3kPR/xTGt9nvbbUJS/Wo2wZvAQu9MP8jXA+sNroSJag5oYLg5IY3nLQBoONqp kZa9h3Kw0LGGFR+kpFMCRQFMBdDrnzIaPtdAnkUL1i1wa6cXg5XSB77BGVxzSUy3+wCg G7YQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3oEAak2Bk2D6xV/o1T1HeLHo/xDtSBqAhlKUcWH+Zxo=; b=eelfuFIDRs+rbfN65bciUJ4yzwsKS1wvXFpl+JoG3+eMVGK9sC2G9Xr5BB/HHNKbIN boQB0jFmZsNDPMq0cM9fcXg+MDweD8KZxmM5cAX5lWQcAnK6aUGzKrPgOdhjRwEtnmEK blF/ktjZvu5zNZtK44LDSeZPtScB7E14Q8h0DAw9mdfsifewGQ+ht5ljKmAIcvtievUZ 2oKDNFVWAxCj4HwZeofM7wqe3O3zOR3LKiJXN9WaY3cT9KL2CbokQYw2doNSpegGfUfA 37m/7bpt2Wibgud444TJCWrOGAiJ1nlh8T3ZAosiPW+tF2qwz2b3PY2hvZztg5dE/hrY sHzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Tc9jXW+9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id nv14-20020a17090b1b4e00b0020af2b9dd80si13736416pjb.69.2022.12.26.19.09.27; Mon, 26 Dec 2022 19:09:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Tc9jXW+9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229934AbiL0DI5 (ORCPT + 99 others); Mon, 26 Dec 2022 22:08:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229730AbiL0DIv (ORCPT ); Mon, 26 Dec 2022 22:08:51 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 55D03EA5 for ; Mon, 26 Dec 2022 19:08:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110530; x=1703646530; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=RtCIdg8lGqV2ZfTQEm450k++h2Jppj7vK0WjZsYLBB4=; b=Tc9jXW+9Us16qBC9de3bu6MpqQ3urXbbPwP8N6GoX5nnTlQjFrAQLTH8 p/Eo5I1hOikIRLOpF2U/JuSt778O69NCwbAIyxjBzweEVCUM+3AeEGSZ/ TiycKnfRd2P6DjIgGm4csODFF8JE8k/S8KQjn0A1c0GjOwqn9Gb12N4EK KopZNgBevj0KXt6+s3sdqxOc6/m446Y1hGZVKn4i6yjWcFffnKA6YyfZc fS6Tgvza4cFfU0km0ebkhbKoFRCdR8vOJLbBdeBQStchRoGb0GpVgJcD5 GMWobYaiLLrQ0mEfF7pS6vlKvrliKtoTj1vGr28GrkWDyqVA8aKc9kGWS g==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="321869746" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="321869746" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:47 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="652917515" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="652917515" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:39 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 84BC110BBAD; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 01/16] x86: Allow atomic MM_CONTEXT flags setting Date: Tue, 27 Dec 2022 06:08:14 +0300 Message-Id: <20221227030829.12508-2-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335022887186561?= X-GMAIL-MSGID: =?utf-8?q?1753335022887186561?= So far there's no need in atomic setting of MM context flags in mm_context_t::flags. The flags set early in exec and never change after that. LAM enabling requires atomic flag setting. Convert the field to unsigned long and MM_CONTEXT_* accesses with __set_bit() and test_bit(). No functional changes. Signed-off-by: Kirill A. Shutemov --- arch/x86/entry/vsyscall/vsyscall_64.c | 2 +- arch/x86/include/asm/mmu.h | 6 +++--- arch/x86/include/asm/mmu_context.h | 2 +- arch/x86/kernel/process_64.c | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/x86/entry/vsyscall/vsyscall_64.c b/arch/x86/entry/vsyscall/vsyscall_64.c index 4af81df133ee..ed320e145cf9 100644 --- a/arch/x86/entry/vsyscall/vsyscall_64.c +++ b/arch/x86/entry/vsyscall/vsyscall_64.c @@ -317,7 +317,7 @@ static struct vm_area_struct gate_vma __ro_after_init = { struct vm_area_struct *get_gate_vma(struct mm_struct *mm) { #ifdef CONFIG_COMPAT - if (!mm || !(mm->context.flags & MM_CONTEXT_HAS_VSYSCALL)) + if (!mm || test_bit(MM_CONTEXT_HAS_VSYSCALL, &mm->context.flags)) return NULL; #endif if (vsyscall_mode == NONE) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 5d7494631ea9..efa3eaee522c 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -9,9 +9,9 @@ #include /* Uprobes on this MM assume 32-bit code */ -#define MM_CONTEXT_UPROBE_IA32 BIT(0) +#define MM_CONTEXT_UPROBE_IA32 0 /* vsyscall page is accessible on this MM */ -#define MM_CONTEXT_HAS_VSYSCALL BIT(1) +#define MM_CONTEXT_HAS_VSYSCALL 1 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. @@ -39,7 +39,7 @@ typedef struct { #endif #ifdef CONFIG_X86_64 - unsigned short flags; + unsigned long flags; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index b8d40ddeab00..53ef591a6166 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -181,7 +181,7 @@ static inline void arch_exit_mmap(struct mm_struct *mm) static inline bool is_64bit_mm(struct mm_struct *mm) { return !IS_ENABLED(CONFIG_IA32_EMULATION) || - !(mm->context.flags & MM_CONTEXT_UPROBE_IA32); + !test_bit(MM_CONTEXT_UPROBE_IA32, &mm->context.flags); } #else static inline bool is_64bit_mm(struct mm_struct *mm) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 4e34b3b68ebd..8b06034e8c70 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -671,7 +671,7 @@ void set_personality_64bit(void) task_pt_regs(current)->orig_ax = __NR_execve; current_thread_info()->status &= ~TS_COMPAT; if (current->mm) - current->mm->context.flags = MM_CONTEXT_HAS_VSYSCALL; + __set_bit(MM_CONTEXT_HAS_VSYSCALL, ¤t->mm->context.flags); /* TBD: overwrites user setup. Should have two bits. But 64bit processes have always behaved this way, @@ -708,7 +708,7 @@ static void __set_personality_ia32(void) * uprobes applied to this MM need to know this and * cannot use user_64bit_mode() at that time. */ - current->mm->context.flags = MM_CONTEXT_UPROBE_IA32; + __set_bit(MM_CONTEXT_UPROBE_IA32, ¤t->mm->context.flags); } current->personality |= force_personality32; From patchwork Tue Dec 27 03:08:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36744 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200412wrt; Mon, 26 Dec 2022 19:09:29 -0800 (PST) X-Google-Smtp-Source: AMrXdXuFSGOxlgkF4X/4AgWXvbY1HSjXTULQbbku7OPlzAWtybMhw1VyADmG12ocrgmXuQlQOeQW X-Received: by 2002:a17:90b:812:b0:219:f993:7f30 with SMTP id bk18-20020a17090b081200b00219f9937f30mr37814248pjb.32.1672110569335; Mon, 26 Dec 2022 19:09:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110569; cv=none; d=google.com; s=arc-20160816; b=WTHs2VpnobQYlEy0574Rp8qFZrM17qPvws2sqBustayqGddYmP5A/rG0aRb/UNC54A XeaquHSas58XMSqq6KRYbAoOsvMCR2RITsB9is7K4zXYL96i04d13+GJklAADcgqv3ua zsU15Bfw3TMKq/EfKNXiJdE4dl6ePGIUmnAHjhE4Ct1U9Cl93bZJihBjOApDqeiIrqPD Z53y1BxeRudngl2pe8Cp+zNpJ1/pxp6rwnsJgJnTPfhyioNTtvk3zJIDCazrWmr7Oyvu pLJE59qvI8XHuYJeBnELgfkBeUUF2iNG4ToawAmu9cNqJD72VRF+9wQUZPjFcMwdSsiW Iw6A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HiQmt79j+yP5ckm1kw0E99dGcnfCgkwWTWtqPyabNRA=; b=WCDzAiTASBubNFyBMn8OUfLlwxJkti9zITkvBJTpCeNzQWUxYVUdSzrcvoIbpmPRW5 YDHU23PdeJBArXGwYONr7tvpL8XFlGChk8H5Twmx1zanudcXINYeFjnkUwUcvoH1UD8g ZCJJbIt4kY65J04fHbTb7XSJviEde43TExQT2YZveB7ITNC7v49KE83o+oq07ALeEpRW i9T3s9Ystt6/adLTfRGCQJBaxuVsrJEcjNVBmufi0/18Ga0NJ6yWApNAqBjqW/w3mbLx NQori4OxGxIUNjqKsHsQPvKrvV9x5dTA55C4N/IsKumshAAD/1X56XNSskJnm+JV8eO9 LeFA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lmrozRjP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lp5-20020a17090b4a8500b00218a9f90590si17884988pjb.2.2022.12.26.19.09.17; Mon, 26 Dec 2022 19:09:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=lmrozRjP; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229724AbiL0DIs (ORCPT + 99 others); Mon, 26 Dec 2022 22:08:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38174 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229574AbiL0DIo (ORCPT ); Mon, 26 Dec 2022 22:08:44 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 042F3EA5 for ; Mon, 26 Dec 2022 19:08:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110523; x=1703646523; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=gEzeHerPaWTs/tIk/qtNJxiWNCV7jnSRQjRqxLLucY8=; b=lmrozRjPyLdejz0lZR9AtOCd7P7L7iyEWEhkk+yomB75eVxMiPno0k63 IBASRkKKhn7uWMPFJk10CrNVzGGt/wC1CDfFcMx49sIRNp12vGo6hh60i lkktvLo//Lu1Tzy5iU/vI4+mmknO/Er+ZEoSYoKZ7+7f3etPZFV0bg1SH dP4igVPzo8EtZa/YXLPCpwtnIJx8p7Ia+9QzN/l6ErSYlIt5783DY/8yo Qah7p8H1mJKyicHF9W732RSxeAEPeAp+Gw6PpZMo+Cuk/UAshro/1L0RM DU16TQvEIEvzMVmO1sMDnyPJJI0yBlYuILRoX3FARtIJw0+k1jv3l8OZ3 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="300994650" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="300994650" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:43 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="741605192" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="741605192" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:39 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 8FB6410BBAF; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 02/16] x86: CPUID and CR3/CR4 flags for Linear Address Masking Date: Tue, 27 Dec 2022 06:08:15 +0300 Message-Id: <20221227030829.12508-3-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335012154746569?= X-GMAIL-MSGID: =?utf-8?q?1753335012154746569?= Enumerate Linear Address Masking and provide defines for CR3 and CR4 flags. Signed-off-by: Kirill A. Shutemov Reviewed-by: Alexander Potapenko Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/include/asm/cpufeatures.h | 1 + arch/x86/include/asm/processor-flags.h | 2 ++ arch/x86/include/uapi/asm/processor-flags.h | 6 ++++++ 3 files changed, 9 insertions(+) diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index 61012476d66e..bc662c80b99d 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -314,6 +314,7 @@ #define X86_FEATURE_CMPCCXADD (12*32+ 7) /* "" CMPccXADD instructions */ #define X86_FEATURE_AMX_FP16 (12*32+21) /* "" AMX fp16 Support */ #define X86_FEATURE_AVX_IFMA (12*32+23) /* "" Support for VPMADD52[H,L]UQ */ +#define X86_FEATURE_LAM (12*32+26) /* Linear Address Masking */ /* AMD-defined CPU features, CPUID level 0x80000008 (EBX), word 13 */ #define X86_FEATURE_CLZERO (13*32+ 0) /* CLZERO instruction */ diff --git a/arch/x86/include/asm/processor-flags.h b/arch/x86/include/asm/processor-flags.h index a7f3d9100adb..d8cccadc83a6 100644 --- a/arch/x86/include/asm/processor-flags.h +++ b/arch/x86/include/asm/processor-flags.h @@ -28,6 +28,8 @@ * On systems with SME, one bit (in a variable position!) is stolen to indicate * that the top-level paging structure is encrypted. * + * On systemms with LAM, bits 61 and 62 are used to indicate LAM mode. + * * All of the remaining bits indicate the physical address of the top-level * paging structure. * diff --git a/arch/x86/include/uapi/asm/processor-flags.h b/arch/x86/include/uapi/asm/processor-flags.h index c47cc7f2feeb..d898432947ff 100644 --- a/arch/x86/include/uapi/asm/processor-flags.h +++ b/arch/x86/include/uapi/asm/processor-flags.h @@ -82,6 +82,10 @@ #define X86_CR3_PCID_BITS 12 #define X86_CR3_PCID_MASK (_AC((1UL << X86_CR3_PCID_BITS) - 1, UL)) +#define X86_CR3_LAM_U57_BIT 61 /* Activate LAM for userspace, 62:57 bits masked */ +#define X86_CR3_LAM_U57 _BITULL(X86_CR3_LAM_U57_BIT) +#define X86_CR3_LAM_U48_BIT 62 /* Activate LAM for userspace, 62:48 bits masked */ +#define X86_CR3_LAM_U48 _BITULL(X86_CR3_LAM_U48_BIT) #define X86_CR3_PCID_NOFLUSH_BIT 63 /* Preserve old PCID */ #define X86_CR3_PCID_NOFLUSH _BITULL(X86_CR3_PCID_NOFLUSH_BIT) @@ -132,6 +136,8 @@ #define X86_CR4_PKE _BITUL(X86_CR4_PKE_BIT) #define X86_CR4_CET_BIT 23 /* enable Control-flow Enforcement Technology */ #define X86_CR4_CET _BITUL(X86_CR4_CET_BIT) +#define X86_CR4_LAM_SUP_BIT 28 /* LAM for supervisor pointers */ +#define X86_CR4_LAM_SUP _BITUL(X86_CR4_LAM_SUP_BIT) /* * x86-64 Task Priority Register, CR8 From patchwork Tue Dec 27 03:08:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36747 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200497wrt; Mon, 26 Dec 2022 19:09:46 -0800 (PST) X-Google-Smtp-Source: AMrXdXuzfrb46p+hbAsJLG3UzUJaInJUL5BnDk3GdNLTXZUGfaqnrwNYy/Tdli7+td280jdEdLDY X-Received: by 2002:a17:90b:2712:b0:219:e1ae:8dc2 with SMTP id px18-20020a17090b271200b00219e1ae8dc2mr22761687pjb.35.1672110585861; Mon, 26 Dec 2022 19:09:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110585; cv=none; d=google.com; s=arc-20160816; b=hiAg+g3IZnzUHb8fbZMw2xUaE06RIWECMEhO9MUT3jB+UibB3T6xOe1SH8NbEXJdnq VSqMLcBwL/LFEkmoHZn/eWEzbxbxCKSeq1Nl3UR9t+pJ3I5tkJATE/nYWtL/lR6TOkk5 MPugL2XxtMy7dONaLmg/XsUVja+QVKN1teEGRco8hdyo14Q5JZtcGwpdUVrQthMO8kEJ yrTDZ1lIfmFVKyIiPtiSSS6iqbUdTCJO/Gniy0iSwBuKUCXc0z4t+vK3kHsj7oY1F8vg mISfdFeLJiGzzT0rZIZECIb/Lb1gto9nFgTHLOsVRKtkieBjL7FKi/dpZAngAOw3p77O OZzw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=TLSpYEgnDH9ROQMR7p9dYcZRjMvuIn2UqTC9mkeJoYE=; b=rMcgcHYQ0CGwlM4a75x72YUnAf5iEbuRDd1Ewvy8aEuFu+XVkkbE8UFJnUm02M3Q++ b50LfqPM7h6ax56v1976waQTNc0EfIU51kTb9oA7Rp/vIPmGmut24xxwao0tZhUkZolK RB4lGrt7MXBzSAFG17LYBcJqXYacO3aMrvsKORs21tU2paY2XJ7iGvVU6G0X4JE6OkCF kMxr1+xJCla5RzDEg4zaJqiSxB4EzeOn8fARbCaiG0jxRBXBRYyiKwn0RMbpBRm1jnNw /aOihtT7RF6MHEEVeR4CTzhhKMfUSABojlYetMYYkmxuW1BNkCMF3hRw2et95/gkKOGt P6Sg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OZt75NyF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l186-20020a6391c3000000b00478e7f82dd3si13018400pge.543.2022.12.26.19.09.33; Mon, 26 Dec 2022 19:09:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=OZt75NyF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229850AbiL0DJF (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229738AbiL0DIv (ORCPT ); Mon, 26 Dec 2022 22:08:51 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D81E25CB for ; Mon, 26 Dec 2022 19:08:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110530; x=1703646530; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=GYyU7jiAE/wXQ+l2FTpkdDQvnKTCwJ1AjTQPGEdwZVw=; b=OZt75NyFipR2CpzPCOVM1CNdLi4AWQW/tog4jXETLZhaFPB47DdtnU5W tPEa4oGwpJJXZQqVohr0PlV+LS2dywE5BBIKlamzoXgOsCXV6W1eyWSfn cU3q637qF9HkuCrLQ7h6Quf1yMD9wErZeXEyuqvyx88AVmOtPHQ/dYITM BrkPRptBsieD7hdraByYYznRLKKcNwtUeWhQ/87sOnXJdyUEjsAmzwDGd TBAfxp7yWm+2tkvPzGlRlVXhlJ5hUlmGyCnCEX4DhCBuPVT3pp5aYN6gZ WSHRWH72UJIU3iVjYHLr68qUD6MCL9LyTOvDSpEmjle0xyuSgL7KCKBRK Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="321869755" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="321869755" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:47 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="652917518" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="652917518" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:39 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 9ABCD10BBB0; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 03/16] x86/mm: Handle LAM on context switch Date: Tue, 27 Dec 2022 06:08:16 +0300 Message-Id: <20221227030829.12508-4-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335029523184186?= X-GMAIL-MSGID: =?utf-8?q?1753335029523184186?= Linear Address Masking mode for userspace pointers encoded in CR3 bits. The mode is selected per-process and stored in mm_context_t. switch_mm_irqs_off() now respects selected LAM mode and constructs CR3 accordingly. The active LAM mode gets recorded in the tlb_state. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 3 ++ arch/x86/include/asm/mmu_context.h | 24 ++++++++++++++ arch/x86/include/asm/tlbflush.h | 34 +++++++++++++++++++ arch/x86/mm/tlb.c | 53 +++++++++++++++++++++--------- 4 files changed, 98 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index efa3eaee522c..9a046aacad8d 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -40,6 +40,9 @@ typedef struct { #ifdef CONFIG_X86_64 unsigned long flags; + + /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ + unsigned long lam_cr3_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 53ef591a6166..464cca41d20a 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -91,6 +91,29 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) } #endif +#ifdef CONFIG_X86_64 +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return READ_ONCE(mm->context.lam_cr3_mask); +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ + mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; +} + +#else + +static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) +{ + return 0; +} + +static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) +{ +} +#endif + #define enter_lazy_tlb enter_lazy_tlb extern void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk); @@ -168,6 +191,7 @@ static inline int arch_dup_mmap(struct mm_struct *oldmm, struct mm_struct *mm) { arch_dup_pkeys(oldmm, mm); paravirt_arch_dup_mmap(oldmm, mm); + dup_lam(oldmm, mm); return ldt_dup_context(oldmm, mm); } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index cda3118f3b27..662598dea937 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -101,6 +101,16 @@ struct tlb_state { */ bool invalidate_other; +#ifdef CONFIG_X86_64 + /* + * Active LAM mode. + * + * X86_CR3_LAM_U57/U48 shifted right by X86_CR3_LAM_U57_BIT or 0 if LAM + * disabled. + */ + u8 lam; +#endif + /* * Mask that contains TLB_NR_DYN_ASIDS+1 bits to indicate * the corresponding user PCID needs a flush next time we @@ -357,6 +367,30 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd) } #define huge_pmd_needs_flush huge_pmd_needs_flush +#ifdef CONFIG_X86_64 +static inline unsigned long tlbstate_lam_cr3_mask(void) +{ + unsigned long lam = this_cpu_read(cpu_tlbstate.lam); + + return lam << X86_CR3_LAM_U57_BIT; +} + +static inline void set_tlbstate_cr3_lam_mask(unsigned long mask) +{ + this_cpu_write(cpu_tlbstate.lam, mask >> X86_CR3_LAM_U57_BIT); +} + +#else + +static inline unsigned long tlbstate_lam_cr3_mask(void) +{ + return 0; +} + +static inline void set_tlbstate_cr3_lam_mask(u64 mask) +{ +} +#endif #endif /* !MODULE */ static inline void __native_tlb_flush_global(unsigned long cr4) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index c1e31e9a85d7..9d1e7a5f141c 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -154,26 +154,30 @@ static inline u16 user_pcid(u16 asid) return ret; } -static inline unsigned long build_cr3(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { + unsigned long cr3 = __sme_pa(pgd) | lam; + if (static_cpu_has(X86_FEATURE_PCID)) { - return __sme_pa(pgd) | kern_pcid(asid); + VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); + cr3 |= kern_pcid(asid); } else { VM_WARN_ON_ONCE(asid != 0); - return __sme_pa(pgd); } + + return cr3; } -static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid) +static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, + unsigned long lam) { - VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); /* * Use boot_cpu_has() instead of this_cpu_has() as this function * might be called during early boot. This should work even after * boot because all CPU's the have same capabilities: */ VM_WARN_ON_ONCE(!boot_cpu_has(X86_FEATURE_PCID)); - return __sme_pa(pgd) | kern_pcid(asid) | CR3_NOFLUSH; + return build_cr3(pgd, asid, lam) | CR3_NOFLUSH; } /* @@ -274,15 +278,16 @@ static inline void invalidate_user_asid(u16 asid) (unsigned long *)this_cpu_ptr(&cpu_tlbstate.user_pcid_flush_mask)); } -static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, bool need_flush) +static void load_new_mm_cr3(pgd_t *pgdir, u16 new_asid, unsigned long lam, + bool need_flush) { unsigned long new_mm_cr3; if (need_flush) { invalidate_user_asid(new_asid); - new_mm_cr3 = build_cr3(pgdir, new_asid); + new_mm_cr3 = build_cr3(pgdir, new_asid, lam); } else { - new_mm_cr3 = build_cr3_noflush(pgdir, new_asid); + new_mm_cr3 = build_cr3_noflush(pgdir, new_asid, lam); } /* @@ -491,6 +496,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, { struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); u64 next_tlb_gen; @@ -520,7 +526,8 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * isn't free. */ #ifdef CONFIG_DEBUG_VM - if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid))) { + if (WARN_ON_ONCE(__read_cr3() != build_cr3(real_prev->pgd, prev_asid, + tlbstate_lam_cr3_mask()))) { /* * If we were to BUG here, we'd be very likely to kill * the system so hard that we don't see the call trace. @@ -552,9 +559,15 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, * instruction. */ if (real_prev == next) { + /* Not actually switching mm's */ VM_WARN_ON(this_cpu_read(cpu_tlbstate.ctxs[prev_asid].ctx_id) != next->context.ctx_id); + /* + * If this races with another thread that enables lam, 'new_lam' + * might not match tlbstate_lam_cr3_mask(). + */ + /* * Even in lazy TLB mode, the CPU should stay set in the * mm_cpumask. The TLB shootdown code can figure out from @@ -622,15 +635,16 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, barrier(); } + set_tlbstate_cr3_lam_mask(new_lam); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); - load_new_mm_cr3(next->pgd, new_asid, true); + load_new_mm_cr3(next->pgd, new_asid, new_lam, true); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, TLB_FLUSH_ALL); } else { /* The new ASID is already up to date. */ - load_new_mm_cr3(next->pgd, new_asid, false); + load_new_mm_cr3(next->pgd, new_asid, new_lam, false); trace_tlb_flush(TLB_FLUSH_ON_TASK_SWITCH, 0); } @@ -691,6 +705,10 @@ void initialize_tlbstate_and_flush(void) /* Assert that CR3 already references the right mm. */ WARN_ON((cr3 & CR3_ADDR_MASK) != __pa(mm->pgd)); + /* LAM expected to be disabled */ + WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)); + WARN_ON(mm_lam_cr3_mask(mm)); + /* * Assert that CR4.PCIDE is set if needed. (CR4.PCIDE initialization * doesn't work like other CR4 bits because it can only be set from @@ -699,8 +717,8 @@ void initialize_tlbstate_and_flush(void) WARN_ON(boot_cpu_has(X86_FEATURE_PCID) && !(cr4_read_shadow() & X86_CR4_PCIDE)); - /* Force ASID 0 and force a TLB flush. */ - write_cr3(build_cr3(mm->pgd, 0)); + /* Disable LAM, force ASID 0 and force a TLB flush. */ + write_cr3(build_cr3(mm->pgd, 0, 0)); /* Reinitialize tlbstate. */ this_cpu_write(cpu_tlbstate.last_user_mm_spec, LAST_USER_MM_INIT); @@ -708,6 +726,7 @@ void initialize_tlbstate_and_flush(void) this_cpu_write(cpu_tlbstate.next_asid, 1); this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen); + set_tlbstate_cr3_lam_mask(0); for (i = 1; i < TLB_NR_DYN_ASIDS; i++) this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0); @@ -1071,8 +1090,10 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) */ unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + unsigned long cr3 = + build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, + this_cpu_read(cpu_tlbstate.loaded_mm_asid), + tlbstate_lam_cr3_mask()); /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); From patchwork Tue Dec 27 03:08:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36745 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200415wrt; Mon, 26 Dec 2022 19:09:31 -0800 (PST) X-Google-Smtp-Source: AMrXdXuNvj2piQjt7E4wOlbC4EHtWHSqQgkAfNY9lg64xWBFmw5NGsgdDgEl7lzjLASjFqtFnASg X-Received: by 2002:a17:902:c40a:b0:189:851d:644c with SMTP id k10-20020a170902c40a00b00189851d644cmr33026380plk.40.1672110571299; Mon, 26 Dec 2022 19:09:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110571; cv=none; d=google.com; s=arc-20160816; b=dQrTqloYroj7IxrHlLqVy/O/+P57lYwpyrsGgvDZy/CKKmQ2GvwKBwwT4KgGQU5MgV M1HdB/xcp4PG5ta0d57svyuaqU0FH/CpXDvLBTOmxOmhH9jhOS5osIxUaJDdXDGTDR9d umkb08a49KwJiL7GpDRZH/8Puc4yR7Im0Kaf3Rbz33D2+OW4ZN9ihYT9dKPmZUHCxoA5 U9vWAcrwjcXZBUEobgVnwtcikabuGbG6J+J6zWWh0Dlzt9LNNR61DZy0ZSPhDbXySmkP bq4n9pYIh49EUHFSrHNhUuzObjJcV47ezJT2pHXeD1qrmp5b27wCSPz11zJ/QW1mpnzf TGkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=C7HJvEkFakmMiC+4clhqvBp0TSytYpgLWXa6owkCvnA=; b=JcfkJYxwAO5S5NPH0I49/wUchR8Z8kD45wMqknAD3XLf4eD9k/yWI08mzoZ49Bx//5 w53wCBihG4j7RCf08KoHaLBwRMsDe0JyE5pf2s0TTZvvVXBRfgtYceSXEysMHbzJv+zT qt9v+SMJxI/GmHS0c3hFPl8ExU+c8wZvwHWm13HwO2gk8cOQb+fHqhxPh/QiKUcWJNH9 DdTdCzdQcJ1YOo1KHvHlVyySJ01HNy7SE0emRc/L0Raz+jJeescCzLwSkQhLSmkgR09/ oyu3qiiLdDi3eafGzcikRN/QdyUKVMtL0BlF3aAlEgb0rdRyVp4pvFeNH7hXm384SQQ2 vZcA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RjA+djRe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t2-20020a634442000000b004928a5abf77si12591971pgk.387.2022.12.26.19.09.18; Mon, 26 Dec 2022 19:09:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=RjA+djRe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229775AbiL0DIw (ORCPT + 99 others); Mon, 26 Dec 2022 22:08:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229578AbiL0DIp (ORCPT ); Mon, 26 Dec 2022 22:08:45 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D18FD1093 for ; Mon, 26 Dec 2022 19:08:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110524; x=1703646524; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=73Lfy6YiNOngUHGIq+SvJHDmwJJPCjL8Zm/vzQL6EVI=; b=RjA+djReJLpqDM9GDVEMyPMWp5ONOXGa4jW5DXuPkmKeTCQ3i8O7Bnv3 7Fj5qnYqohkV5pGwhCB3HjHzLjMAQnR/SehdG8WRVY7Xna9/Jh4ZeIsp+ TYUDRToXhxl/fK72tXVnPAqeVaCkHOERiN6fPxJBVV7J037sqrIed0pyB G3z8kg+j9lw4tkN+y2bHd6tTjlLHHB94oFX00sAGh1+XOIRRpUf2pmShf /Vc+aMkvJ0Y57qeFvJOgRoXNiS9/rjFZRPmQA7K7AUJIL9r3TEAa/nTeT 10+TavEklwR7aQu8cyMuwxFfLpVoexyuRQFxvU0h7tr5ujXHCOkatnaND Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="300994658" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="300994658" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:43 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="741605194" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="741605194" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:39 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id A1E7910BBB1; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 04/16] mm: Introduce untagged_addr_remote() Date: Tue, 27 Dec 2022 06:08:17 +0300 Message-Id: <20221227030829.12508-5-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335014195620325?= X-GMAIL-MSGID: =?utf-8?q?1753335014195620325?= untagged_addr() removes tags/metadata from the address and brings it to the canonical form. The helper is implemented on arm64 and sparc. Both of them do untagging based on global rules. However, Linear Address Masking (LAM) on x86 introduces per-process settings for untagging. As a result, untagged_addr() is now only suitable for untagging addresses for the current proccess. The new helper untagged_addr_remote() has to be used when the address targets remote process. It requires the mmap lock for target mm to be taken. Export dump_mm() as there's now the first user for it: VFIO can be compiled as module and untagged_addr_remote() triggers dump_mm() via mmap_assert_locked(). Signed-off-by: Kirill A. Shutemov --- arch/sparc/include/asm/uaccess_64.h | 2 ++ drivers/vfio/vfio_iommu_type1.c | 2 +- fs/proc/task_mmu.c | 2 +- include/linux/mm.h | 11 ----------- include/linux/uaccess.h | 22 ++++++++++++++++++++++ mm/debug.c | 1 + mm/gup.c | 4 ++-- mm/madvise.c | 5 +++-- mm/migrate.c | 2 +- 9 files changed, 33 insertions(+), 18 deletions(-) diff --git a/arch/sparc/include/asm/uaccess_64.h b/arch/sparc/include/asm/uaccess_64.h index 94266a5c5b04..b825a5dd0210 100644 --- a/arch/sparc/include/asm/uaccess_64.h +++ b/arch/sparc/include/asm/uaccess_64.h @@ -8,8 +8,10 @@ #include #include +#include #include #include +#include #include #include diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 23c24fe98c00..daf34f957b7b 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -573,7 +573,7 @@ static int vaddr_get_pfns(struct mm_struct *mm, unsigned long vaddr, goto done; } - vaddr = untagged_addr(vaddr); + vaddr = untagged_addr_remote(mm, vaddr); retry: vma = vma_lookup(mm, vaddr); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index e35a0398db63..25786a808901 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1693,7 +1693,7 @@ static ssize_t pagemap_read(struct file *file, char __user *buf, /* watch out for wraparound */ start_vaddr = end_vaddr; if (svpfn <= (ULONG_MAX >> PAGE_SHIFT)) - start_vaddr = untagged_addr(svpfn << PAGE_SHIFT); + start_vaddr = untagged_addr_remote(mm, svpfn << PAGE_SHIFT); /* Ensure the address is inside the task */ if (start_vaddr > mm->task_size) diff --git a/include/linux/mm.h b/include/linux/mm.h index f3f196e4d66d..6b28eb9c6ea2 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -96,17 +96,6 @@ extern int mmap_rnd_compat_bits __read_mostly; #include #include -/* - * Architectures that support memory tagging (assigning tags to memory regions, - * embedding these tags into addresses that point to these memory regions, and - * checking that the memory and the pointer tags match on memory accesses) - * redefine this macro to strip tags from pointers. - * It's defined as noop for architectures that don't support memory tagging. - */ -#ifndef untagged_addr -#define untagged_addr(addr) (addr) -#endif - #ifndef __pa_symbol #define __pa_symbol(x) __pa(RELOC_HIDE((unsigned long)(x), 0)) #endif diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h index afb18f198843..bfdadf5f8bbb 100644 --- a/include/linux/uaccess.h +++ b/include/linux/uaccess.h @@ -10,6 +10,28 @@ #include +/* + * Architectures that support memory tagging (assigning tags to memory regions, + * embedding these tags into addresses that point to these memory regions, and + * checking that the memory and the pointer tags match on memory accesses) + * redefine this macro to strip tags from pointers. + * + * Passing down mm_struct allows to define untagging rules on per-process + * basis. + * + * It's defined as noop for architectures that don't support memory tagging. + */ +#ifndef untagged_addr +#define untagged_addr(addr) (addr) +#endif + +#ifndef untagged_addr_remote +#define untagged_addr_remote(mm, addr) ({ \ + mmap_assert_locked(mm); \ + untagged_addr(addr); \ +}) +#endif + /* * Architectures should provide two primitives (raw_copy_{to,from}_user()) * and get rid of their private instances of copy_{to,from}_user() and diff --git a/mm/debug.c b/mm/debug.c index 7f8e5f744e42..3c1b490c7e2b 100644 --- a/mm/debug.c +++ b/mm/debug.c @@ -215,6 +215,7 @@ void dump_mm(const struct mm_struct *mm) mm->def_flags, &mm->def_flags ); } +EXPORT_SYMBOL_GPL(dump_mm); static bool page_init_poisoning __read_mostly = true; diff --git a/mm/gup.c b/mm/gup.c index f45a3a5be53a..e28d787ba8f8 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1091,7 +1091,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!nr_pages) return 0; - start = untagged_addr(start); + start = untagged_addr_remote(mm, start); VM_BUG_ON(!!pages != !!(gup_flags & (FOLL_GET | FOLL_PIN))); @@ -1265,7 +1265,7 @@ int fixup_user_fault(struct mm_struct *mm, struct vm_area_struct *vma; vm_fault_t ret; - address = untagged_addr(address); + address = untagged_addr_remote(mm, address); if (unlocked) fault_flags |= FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE; diff --git a/mm/madvise.c b/mm/madvise.c index a56a6d17e201..90cd4a442fd2 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -1407,8 +1407,6 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh size_t len; struct blk_plug plug; - start = untagged_addr(start); - if (!madvise_behavior_valid(behavior)) return -EINVAL; @@ -1440,6 +1438,9 @@ int do_madvise(struct mm_struct *mm, unsigned long start, size_t len_in, int beh mmap_read_lock(mm); } + start = untagged_addr_remote(mm, start); + end = start + len; + blk_start_plug(&plug); error = madvise_walk_vmas(mm, start, end, behavior, madvise_vma_behavior); diff --git a/mm/migrate.c b/mm/migrate.c index a4d3fc65085f..2cdffeff9ffe 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1839,7 +1839,7 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, goto out_flush; if (get_user(node, nodes + i)) goto out_flush; - addr = (unsigned long)untagged_addr(p); + addr = (unsigned long)untagged_addr_remote(mm, p); err = -ENODEV; if (node < 0 || node >= MAX_NUMNODES) From patchwork Tue Dec 27 03:08:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36749 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200549wrt; Mon, 26 Dec 2022 19:09:55 -0800 (PST) X-Google-Smtp-Source: AMrXdXs5UAnh18belNZGywYktFntmixOFxFiMCVudLvrw66/Hag5wTbDItku0mDq1oa/vntCiPBZ X-Received: by 2002:a05:6a20:be11:b0:9d:efbe:a117 with SMTP id ge17-20020a056a20be1100b0009defbea117mr23037019pzb.39.1672110595435; Mon, 26 Dec 2022 19:09:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110595; cv=none; d=google.com; s=arc-20160816; b=eWic7bmchgxNcMpY6Te+7buh8xDcExwILxV85WOqGhIvigYiVCVGQBveLrV7K1JFr/ u2MtsfAokg8SXnnFc3fLfHeizajrAnft03fa/tXMYTjJTrSw7PI9anE4Y0bIuLHTQWZ7 ojbr1Ybc0Vsp6+uwI0fbC5cujpeYuZrWKA4Wj4se/7LbuIZkiNaT3uZxZQpgOJs4ntso 3kcHGXsAmwZ7HVHB1XZ0L491jUVsUdVzSgd8yC8BcEHiit8Z9Y/nSikZT7jPDWjAlSMc rrVs+Rc6BObRJ5DrUXJkoUF7qaEU71cE7id/rraaYbxu+AsInYZO742SR3t4DAh98JNB 6w+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=qmz2rANUZ4KIs+oy6HIjixRfmrtC4y0ZixDKV4/MIyc=; b=qJGuzCaRJV9MCSdIV2aJG/tQpkWY8DD5avLkXyw1XLS11XfRqC6Fz64ull2tk1GmSW NpDepPrFIH9venlcUlBOJwXkPo9oHqSopyoAsUU6iyNGwYCZn85CHBkJ4aLfPtxYFTJN mAZmf3X0tc8CuADLBiUY6p2Wyv4I9GfAsCkt+QrPjlo0PT4bPcv17/YhJRs4amYCjByp GeMZC1fgf+Ua0kzVhE6mQN8BUXHuOinVlQoOIPcERPEkUVApFADG3uR5jxgXO1vefRRt vZ+uznfk0Ay4avtEmkLzLGF5G4q240EjbXkb9FZGtS9t3tfnDZ/FgMJ2GbGXuUzx56b1 zNIg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=P93IF+hn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l9-20020a654489000000b004785d1fa649si13664583pgq.432.2022.12.26.19.09.43; Mon, 26 Dec 2022 19:09:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=P93IF+hn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230180AbiL0DJN (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38222 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229767AbiL0DIw (ORCPT ); Mon, 26 Dec 2022 22:08:52 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B3201261C for ; Mon, 26 Dec 2022 19:08:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110530; x=1703646530; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=yO+iUKN0r9KVMUOqq6nIAT1LjszIJDhKiqjV1XdDapw=; b=P93IF+hn1r49BrfN7TffaFJdPRSeLGuBGI4X540L42IswYyzhjHoyMW7 gGD7XMp2E2VGhxZg0/+HjcONWtdnT8UMA4F/bKaxU7xlXQVLDX8SBxjNs ROwdQVmqgzMBLLrXwAL9sS+f1SDaMx0HGdpPqoLUI/0ypHYj+Q66JaF+a HDvOgF364lh5Uxr5QTBF/TTz04X7RX4Ex5tSPF6HL217RypFKZOcRMhfW QALRM4vKZ5ErsXgGs4LyGz3Elu5io2LxCk6tFpznggHvccnA8OZJ3oZsj GjGpGJiw59/cMtplushuKPmQnEKOoYsMG0rMHuscOmQvvUBALAxz/Aqyu A==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="300994688" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="300994688" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="741605215" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="741605215" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:45 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id ABB7210BBB2; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 05/16] x86/uaccess: Provide untagged_addr() and remove tags before address check Date: Tue, 27 Dec 2022 06:08:18 +0300 Message-Id: <20221227030829.12508-6-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335039623488144?= X-GMAIL-MSGID: =?utf-8?q?1753335039623488144?= untagged_addr() is a helper used by the core-mm to strip tag bits and get the address to the canonical shape. In only handles userspace addresses. The untagging mask is stored in mmu_context and will be set on enabling LAM for the process. The tags must not be included into check whether it's okay to access the userspace address. Strip tags in access_ok(). get_user() and put_user() don't use access_ok(), but check access against TASK_SIZE directly in assembly. Strip tags, before calling into the assembly helper. Signed-off-by: Kirill A. Shutemov Acked-by: Peter Zijlstra (Intel) Acked-by: Andy Lutomirski Tested-by: Alexander Potapenko --- arch/x86/include/asm/mmu.h | 3 ++ arch/x86/include/asm/mmu_context.h | 11 +++++++ arch/x86/include/asm/uaccess.h | 47 +++++++++++++++++++++++++++--- arch/x86/kernel/process.c | 3 ++ 4 files changed, 60 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 9a046aacad8d..ed72fcd2292d 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -43,6 +43,9 @@ typedef struct { /* Active LAM mode: X86_CR3_LAM_U48 or X86_CR3_LAM_U57 or 0 (disabled) */ unsigned long lam_cr3_mask; + + /* Significant bits of the virtual address. Excludes tag bits. */ + u64 untag_mask; #endif struct mutex lock; diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 464cca41d20a..71581cb4811b 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -100,6 +100,12 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { mm->context.lam_cr3_mask = oldmm->context.lam_cr3_mask; + mm->context.untag_mask = oldmm->context.untag_mask; +} + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ + mm->context.untag_mask = -1UL; } #else @@ -112,6 +118,10 @@ static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) { } + +static inline void mm_reset_untag_mask(struct mm_struct *mm) +{ +} #endif #define enter_lazy_tlb enter_lazy_tlb @@ -138,6 +148,7 @@ static inline int init_new_context(struct task_struct *tsk, mm->context.execute_only_pkey = -1; } #endif + mm_reset_untag_mask(mm); init_new_context_ldt(mm); return 0; } diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index 1cc756eafa44..cbb463e9344f 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -21,6 +22,37 @@ static inline bool pagefault_disabled(void); # define WARN_ON_IN_IRQ() #endif +#ifdef CONFIG_X86_64 +/* + * Mask out tag bits from the address. + * + * Magic with the 'sign' allows to untag userspace pointer without any branches + * while leaving kernel addresses intact. + */ +#define __untagged_addr(mm, addr) ({ \ + u64 __addr = (__force u64)(addr); \ + s64 sign = (s64)__addr >> 63; \ + __addr &= READ_ONCE((mm)->context.untag_mask) | sign; \ + (__force __typeof__(addr))__addr; \ +}) + +#define untagged_addr(addr) __untagged_addr(current->mm, addr) + +#define untagged_addr_remote(mm, addr) ({ \ + mmap_assert_locked(mm); \ + __untagged_addr(mm, addr); \ +}) + +#define untagged_ptr(ptr) ({ \ + u64 __ptrval = (__force u64)(ptr); \ + __ptrval = untagged_addr(__ptrval); \ + (__force __typeof__(ptr))__ptrval; \ +}) +#else +#define untagged_addr(addr) (addr) +#define untagged_ptr(ptr) (ptr) +#endif + /** * access_ok - Checks if a user space pointer is valid * @addr: User space pointer to start of block to check @@ -38,10 +70,10 @@ static inline bool pagefault_disabled(void); * Return: true (nonzero) if the memory block may be valid, false (zero) * if it is definitely invalid. */ -#define access_ok(addr, size) \ +#define access_ok(addr, size) \ ({ \ WARN_ON_IN_IRQ(); \ - likely(__access_ok(addr, size)); \ + likely(__access_ok(untagged_addr(addr), size)); \ }) #include @@ -127,7 +159,11 @@ extern int __get_user_bad(void); * Return: zero on success, or -EFAULT on error. * On error, the variable @x is set to zero. */ -#define get_user(x,ptr) ({ might_fault(); do_get_user_call(get_user,x,ptr); }) +#define get_user(x,ptr) \ +({ \ + might_fault(); \ + do_get_user_call(get_user,x,untagged_ptr(ptr)); \ +}) /** * __get_user - Get a simple variable from user space, with less checking. @@ -227,7 +263,10 @@ extern void __put_user_nocheck_8(void); * * Return: zero on success, or -EFAULT on error. */ -#define put_user(x, ptr) ({ might_fault(); do_put_user_call(put_user,x,ptr); }) +#define put_user(x, ptr) ({ \ + might_fault(); \ + do_put_user_call(put_user,x,untagged_ptr(ptr)); \ +}) /** * __put_user - Write a simple value into user space, with less checking. diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index 40d156a31676..ef6bde1d40d8 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -47,6 +47,7 @@ #include #include #include +#include #include "process.h" @@ -367,6 +368,8 @@ void arch_setup_new_exec(void) task_clear_spec_ssb_noexec(current); speculation_ctrl_update(read_thread_flags()); } + + mm_reset_untag_mask(current->mm); } #ifdef CONFIG_X86_IOPL_IOPERM From patchwork Tue Dec 27 03:08:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36755 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200918wrt; Mon, 26 Dec 2022 19:11:17 -0800 (PST) X-Google-Smtp-Source: AMrXdXvU/FWZtopi64o00lzg4BSsdyQMb4p/B4H0Zs2UNWzXT+ha4Ni76QSqg5i9O5y0emlPela4 X-Received: by 2002:a05:6a21:1507:b0:ac:1cf0:61e2 with SMTP id nq7-20020a056a21150700b000ac1cf061e2mr29334063pzb.3.1672110677023; Mon, 26 Dec 2022 19:11:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110677; cv=none; d=google.com; s=arc-20160816; b=JfjJnp+5dc5C3XIFHMczpQEtC+yLbBjIykF3v0iDvHezgQMCVVhljHYYMXAx5kexwn 9GUx5opxKV/3Zb8JW6DGBai+y4u3yQgDfQcAd/MQL2/jSGyY0MbPyKSTdaNXmpPLaleu ArrdGGY8sWuiOth09xDrotoDdKsIjrYni2Q7GvmdPyyLvtPbyTbz2WAvbyNCVRXxNuah V5tmNWdt/EbQZR906Y3zpU+3wYHPJqRvLNBUMpMlUFtiocs9mqTKADoK4hK76D483nRY xe8ZAi9AzUHs1KEplcb4Rb1RsbgWdcQ+CPWzEUKGp9gLFIQ+zOaUuOSTAXYrNkIP9oxz jynQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SxTKEkp8/tdII07qIAtITmj87G/OWchkwbQPUIEqdi0=; b=vpZxL0oXyVFHnZiOG2JEp6KF5auKT60C9i72yFT6u84clion5QRuZRGB2VfFxr6u8u RZepyFjXPQj4KuBFglJHDjm5a8DE9OserKk5h5YBwcseSHHxVbQ7DrmjfKc+vv4PN6dm NLQqWx5+gB5+9HIpJB7slGYWLiYSZCIgihwKMZiqF7hqWSwgBGODQ5yWB74HBdyW9gzH +/DzPNBaVhBme+HnDtYsHhYDvfrNjmCyhXUNh7MztGE086JuqLZsHHw336hPucDtphmj fRQisy1fXrFocTF2s4Z+Nj2sSpMEgj1KdiCsx4SeXYDMbf1HyNEXXDQ2eh89QeFQOjnA 8RMQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Qk7mfQoM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b7-20020a656687000000b00482c6f5971asi13426386pgw.280.2022.12.26.19.11.05; Mon, 26 Dec 2022 19:11:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Qk7mfQoM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231154AbiL0DJi (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229979AbiL0DJG (ORCPT ); Mon, 26 Dec 2022 22:09:06 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BA382AFB for ; Mon, 26 Dec 2022 19:08:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110535; x=1703646535; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PApE0uZLyXFEodnTh3dZSNMMuSzjiKqxGiIkJUnjdqE=; b=Qk7mfQoMMbPh30d7IY4eMDLF5nPNEyHDrtW3hi2/vTBkDuC3HXTnfbCD hHP9oxDXSA1z/ss1OKGEyijHxkZ6BuszS37TarqjldEtuY7PXD4efGR+A I4mxaktOZFCnue5w1oRXhDKbw7IOZdjXsLkPzM9QVB0gM1aPONFZVQeOb KJyUjX96rsY/aAtBX0hMF0Z/RcllSR2wb4AmEXPmWz1boefFWXWsNmXAv uWWbHwrZabcO5N39B9t9y1OCjz/HAu7TcyvSVvCBoLZ/+XMt+n/oA5aUr ywsB7rwlg4+odBQcYBQC12PdwfIQvmkFaRjnaMGr8l2+y0OCTXb7CClC4 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="321869789" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="321869789" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:52 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="652917547" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="652917547" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:47 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id B478C10BBB3; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 06/16] x86/mm: Provide arch_prctl() interface for LAM Date: Tue, 27 Dec 2022 06:08:19 +0300 Message-Id: <20221227030829.12508-7-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335125643341994?= X-GMAIL-MSGID: =?utf-8?q?1753335125643341994?= Add a couple of arch_prctl() handles: - ARCH_ENABLE_TAGGED_ADDR enabled LAM. The argument is required number of tag bits. It is rounded up to the nearest LAM mode that can provide it. For now only LAM_U57 is supported, with 6 tag bits. - ARCH_GET_UNTAG_MASK returns untag mask. It can indicates where tag bits located in the address. - ARCH_GET_MAX_TAG_BITS returns the maximum tag bits user can request. Zero if LAM is not supported. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/mmu.h | 2 ++ arch/x86/include/uapi/asm/prctl.h | 4 +++ arch/x86/kernel/process.c | 3 +++ arch/x86/kernel/process_64.c | 44 ++++++++++++++++++++++++++++++- 4 files changed, 52 insertions(+), 1 deletion(-) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index ed72fcd2292d..54e4a3e9b5c5 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -12,6 +12,8 @@ #define MM_CONTEXT_UPROBE_IA32 0 /* vsyscall page is accessible on this MM */ #define MM_CONTEXT_HAS_VSYSCALL 1 +/* Do not allow changing LAM mode */ +#define MM_CONTEXT_LOCK_LAM 2 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index 500b96e71f18..a31e27b95b19 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -20,4 +20,8 @@ #define ARCH_MAP_VDSO_32 0x2002 #define ARCH_MAP_VDSO_64 0x2003 +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index ef6bde1d40d8..cc0677f58f42 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -162,6 +162,9 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) savesegment(es, p->thread.es); savesegment(ds, p->thread.ds); + + if (p->mm && (clone_flags & (CLONE_VM | CLONE_VFORK)) == CLONE_VM) + set_bit(MM_CONTEXT_LOCK_LAM, &p->mm->context.flags); #else p->thread.sp0 = (unsigned long) (childregs + 1); savesegment(gs, p->thread.gs); diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 8b06034e8c70..fef127ed79b6 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -743,6 +743,39 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) } #endif +#define LAM_U57_BITS 6 + +static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) +{ + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return -ENODEV; + + if (test_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags)) + return -EBUSY; + + if (mmap_write_lock_killable(mm)) + return -EINTR; + + if (!nr_bits) { + mmap_write_unlock(mm); + return -EINVAL; + } else if (nr_bits <= LAM_U57_BITS) { + mm->context.lam_cr3_mask = X86_CR3_LAM_U57; + mm->context.untag_mask = ~GENMASK(62, 57); + } else { + mmap_write_unlock(mm); + return -EINVAL; + } + + write_cr3(__read_cr3() | mm->context.lam_cr3_mask); + set_tlbstate_cr3_lam_mask(mm->context.lam_cr3_mask); + set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags); + + mmap_write_unlock(mm); + + return 0; +} + long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) { int ret = 0; @@ -830,7 +863,16 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) case ARCH_MAP_VDSO_64: return prctl_map_vdso(&vdso_image_64, arg2); #endif - + case ARCH_GET_UNTAG_MASK: + return put_user(task->mm->context.untag_mask, + (unsigned long __user *)arg2); + case ARCH_ENABLE_TAGGED_ADDR: + return prctl_enable_tagged_addr(task->mm, arg2); + case ARCH_GET_MAX_TAG_BITS: + if (!cpu_feature_enabled(X86_FEATURE_LAM)) + return put_user(0, (unsigned long __user *)arg2); + else + return put_user(LAM_U57_BITS, (unsigned long __user *)arg2); default: ret = -EINVAL; break; From patchwork Tue Dec 27 03:08:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36748 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200528wrt; Mon, 26 Dec 2022 19:09:52 -0800 (PST) X-Google-Smtp-Source: AMrXdXuSXj6t7J871rfvv1vLtKP+jvsEA5Zx1L+AAjau5KPiHSo+lpIXiYbKjadLT4WFw+8SdaES X-Received: by 2002:a17:902:c942:b0:189:c93c:e5e6 with SMTP id i2-20020a170902c94200b00189c93ce5e6mr34119424pla.33.1672110592184; Mon, 26 Dec 2022 19:09:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110592; cv=none; d=google.com; s=arc-20160816; b=u2SQ9ve3dwYFXB3zKfNyTn1pu0TR9tomyNekDDpakUwE1jU7L6d1jUH7vyDl1B0k0X BbkFhPMRbxqGLgSEuGz9EC4zlJiI5Uhwh3tfzuE7QZurK2J2L6FY87yKfwPPp+DwFaYi WubuWiO7hd/7n5Iu4cuHcLKh8gj/R9jCleMn7PWMLrzrLKA/yRjyrZ1OS0fAAlOokqi6 11Kh6F9xzeohcPDc93bfSbiJ3hOoFPp/aXdQQUut7ZQIqsBwzHZhhtPcPlSxVGT6g0nr 3ochyrPXTQE25+EnGZevewEdPzNX1DP7Sqf++8uyje9EPOonmzO5NfJDzNr0Y3OhxUww Dk0g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JaHQeXh+L8VDr3v14jKHr2jtqcMM5uGvvy+DcgWw4zw=; b=oi/kH+wsh40LEjDOpUlEPQxpiTDoEoPRjjDU8XzYBjHBh8A/FNzElotYG3eGJEB4+q LQZ6uPi4NPBWmmHrPFyC19eQRr71JLmVC6Vbs+6ZPZV9tJYuF+4JOHOw41iK7RVdykb2 U+b62iJEASeLlqwYVswU2p+9xvEek5JeVD9cH19oJJnsZNO23PIcOFQlBq9g/CCJMXTR aE4xa7E9q5Pp2MbqAoU7hFxYRDxP2yk8mi5wFIRmb9BCYcglxSY8gn6JVVcHxJv8jX4z zUHwHWJefNcGW1+KHqwfmOIH4AhdTNQXRzDIXLyOwShkprCXSuxrUsVK0yoqRN6zK/Fw myMw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=LaH+MN7p; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 4-20020a631944000000b00439f026cf5dsi12714575pgz.612.2022.12.26.19.09.39; Mon, 26 Dec 2022 19:09:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=LaH+MN7p; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229969AbiL0DJK (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229772AbiL0DIw (ORCPT ); Mon, 26 Dec 2022 22:08:52 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 21BDF264C for ; Mon, 26 Dec 2022 19:08:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110531; x=1703646531; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cWbMXq+BulH9gIMaAKVyjGjPeJ6XOcZ2RXQodh7xORI=; b=LaH+MN7ps6xJB5ynzm204FXQ8IRzOinh24arYS6jyaMrQizlLPteoXD4 LFtF5e9/g//pMsHMacbRkebUUtC0sz0H4mbcsIWiHRC8ttF6mandkDjfl mhzRcaWkZKlHFFttE7OODZiveh1evk7v0ASIFabgKT2qbD5w7PnlU6HKl mJmqeZ5BljWscLXcCrw3pNwAGm0+mRRzewgTVYnx1ej7X31sS4F0gGTim Gpd5YRRrOsman9AlOmxSWx+oMsEU5T4bQWgjzsIMZ/R7PcO8pbKVyKF9C ukBnGfT6l/Re9OodXf853N0Gd2lFJUXf+hEuEqV3rv9Mdx6y0MYmTO+DL A==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="300994696" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="300994696" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="741605217" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="741605217" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:46 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id BEEAE10BBB4; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 07/16] x86/mm: Reduce untagged_addr() overhead until the first LAM user Date: Tue, 27 Dec 2022 06:08:20 +0300 Message-Id: <20221227030829.12508-8-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335036618641268?= X-GMAIL-MSGID: =?utf-8?q?1753335036618641268?= Use static key to reduce untagged_addr() overhead. The key only gets enabled when the first process enables LAM. Signed-off-by: Kirill A. Shutemov --- arch/x86/include/asm/uaccess.h | 9 +++++++-- arch/x86/kernel/process_64.c | 4 ++++ 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index cbb463e9344f..1d931c7f6741 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -23,6 +23,8 @@ static inline bool pagefault_disabled(void); #endif #ifdef CONFIG_X86_64 +DECLARE_STATIC_KEY_FALSE(tagged_addr_key); + /* * Mask out tag bits from the address. * @@ -31,8 +33,11 @@ static inline bool pagefault_disabled(void); */ #define __untagged_addr(mm, addr) ({ \ u64 __addr = (__force u64)(addr); \ - s64 sign = (s64)__addr >> 63; \ - __addr &= READ_ONCE((mm)->context.untag_mask) | sign; \ + if (static_branch_likely(&tagged_addr_key)) { \ + s64 sign = (s64)__addr >> 63; \ + u64 mask = READ_ONCE((mm)->context.untag_mask); \ + __addr &= mask | sign; \ + } \ (__force __typeof__(addr))__addr; \ }) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index fef127ed79b6..09e7f3d3fb5c 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -743,6 +743,9 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) } #endif +DEFINE_STATIC_KEY_FALSE(tagged_addr_key); +EXPORT_SYMBOL_GPL(tagged_addr_key); + #define LAM_U57_BITS 6 static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) @@ -773,6 +776,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) mmap_write_unlock(mm); + static_branch_enable(&tagged_addr_key); return 0; } From patchwork Tue Dec 27 03:08:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36753 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200704wrt; Mon, 26 Dec 2022 19:10:29 -0800 (PST) X-Google-Smtp-Source: AMrXdXuwAS/5hVKLSK5NaVLKQDbusobxUlvn4EtPe6GOo8kGqsNsfqiCWXxfOVC0KcOVZW2sjopw X-Received: by 2002:a17:90a:17a6:b0:225:e3e5:7558 with SMTP id q35-20020a17090a17a600b00225e3e57558mr8380440pja.29.1672110628697; Mon, 26 Dec 2022 19:10:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110628; cv=none; d=google.com; s=arc-20160816; b=iLYHMTlO2btJBB6olsYob5rIkJUFXPp8LV34XrUQDNkJcLoYSHEIVWTogIY+j6AzrY sAcL7LXHdaHtYEaca8jkwjIRm2n/nP4n161r76qAJ2umpdXlGFxlNfyVclz40PcXzR85 CziqcFXDNye3UX4mHtvgwUfnA1NqeQCu6LNqwDDsWyN/QDbBjxU6QGoeSSyuBsBHmu2M oNzXm/sZLslF/zOOuPTLqBz2R0IkhA+pUjBg2B3FUlVWOY+zXrYHI/hlc94u/XhrLgCX BVroywnE3uJGpcwmHPnFE3AsBoSd103/jGvbHz1DJW2e0oGnZD2CGGnKEaTSV22x0lrg sFYw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=IZoWeD42yWh9XTUsLlcRaLNhHxRpiWGm/Bt6fqyV4JE=; b=PBrHaq/nMeGG/78WQFhGvqy9vn0YVICAuUNI3yBzDHS1U4RLY+YazVpRxnrKqgcxgA H49xJ3z9zhF/pyxI0vV5XgsyJ36vqPqVo/6wtZuP/rdUx9tdiYMspXT1NCYcPDnRVAnI D8eosT+5nDi+muJ8RoAmxiowMCb18yaVPTdI8VQj3wJzxZHFwOnKrbyC9tMbDEsnEh+x 06baztqCZWoHu76HUfJBXa25LxA4acCoySP/GYGob0i2fATdvBDJmZuuQnS1cFvTcInA dMlO0NBddpWAI3RXAC21REkg10LRbJKsXLPO9mf3mgUvWZgW5VJb3Apfrim4VkiRwAdS SCJg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=NLusNa0t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o5-20020a17090ac70500b00213997a5fe6si12961274pjt.113.2022.12.26.19.10.16; Mon, 26 Dec 2022 19:10:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=NLusNa0t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230244AbiL0DJS (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229792AbiL0DIw (ORCPT ); Mon, 26 Dec 2022 22:08:52 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB71F10E9 for ; Mon, 26 Dec 2022 19:08:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110531; x=1703646531; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2hNvtI+I64q9AU5Zc+yHHwHftID1PyaOfc87+zaI4fg=; b=NLusNa0tTl3d92B5cSckeus1ibwrLRBvcNbmvLFUM1faxNowaGQ5j/y7 XTNR8j5Yxz709mRkS1hAMzfZGGDA1igY4NXSxI7H/nI/xvLS3beDELgFz C/ziUYQLiErLuHT1Ke1s2lWOnzAST/mv2k+tbCTkp2CDbccgmT6xg+qyS /vqIB3z4j2F7CPbekT7cTSKTHQknfX1uvVVEg74jTTlBMBduckXwIbTo4 qMv+jrC3jyw+1CwvgIVD8fVYbW6V4hqt69ONZLV7n/JXSFsdYT/qFYAQ9 TFV+w3oxIRUnc5bRzpe4UXEZ8o2NzSC/PGRymUm/kpsWikZtmB8//Vc0w Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="300994704" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="300994704" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="741605219" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="741605219" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:46 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id C95CE10BBB5; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" , Catalin Marinas Subject: [PATCHv13 08/16] mm: Expose untagging mask in /proc/$PID/status Date: Tue, 27 Dec 2022 06:08:21 +0300 Message-Id: <20221227030829.12508-9-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335074388302227?= X-GMAIL-MSGID: =?utf-8?q?1753335074388302227?= Add a line in /proc/$PID/status to report untag_mask. It can be used to find out LAM status of the process from the outside. It is useful for debuggers. Signed-off-by: Kirill A. Shutemov Acked-by: Catalin Marinas --- arch/arm64/include/asm/mmu_context.h | 6 ++++++ arch/sparc/include/asm/mmu_context_64.h | 6 ++++++ arch/x86/include/asm/mmu_context.h | 6 ++++++ fs/proc/array.c | 6 ++++++ include/linux/mmu_context.h | 7 +++++++ 5 files changed, 31 insertions(+) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 72dbd6400549..56911691bef0 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -288,6 +288,12 @@ void post_ttbr_update_workaround(void); unsigned long arm64_mm_context_get(struct mm_struct *mm); void arm64_mm_context_put(struct mm_struct *mm); +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL >> 8; +} + #include #endif /* !__ASSEMBLY__ */ diff --git a/arch/sparc/include/asm/mmu_context_64.h b/arch/sparc/include/asm/mmu_context_64.h index 7a8380c63aab..799e797c5cdd 100644 --- a/arch/sparc/include/asm/mmu_context_64.h +++ b/arch/sparc/include/asm/mmu_context_64.h @@ -185,6 +185,12 @@ static inline void finish_arch_post_lock_switch(void) } } +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL >> adi_nbits(); +} + #include #endif /* !(__ASSEMBLY__) */ diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 71581cb4811b..7f9f9978c811 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -103,6 +103,12 @@ static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) mm->context.untag_mask = oldmm->context.untag_mask; } +#define mm_untag_mask mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return mm->context.untag_mask; +} + static inline void mm_reset_untag_mask(struct mm_struct *mm) { mm->context.untag_mask = -1UL; diff --git a/fs/proc/array.c b/fs/proc/array.c index 49283b8103c7..d2a94eafe9a3 100644 --- a/fs/proc/array.c +++ b/fs/proc/array.c @@ -428,6 +428,11 @@ static inline void task_thp_status(struct seq_file *m, struct mm_struct *mm) seq_printf(m, "THP_enabled:\t%d\n", thp_enabled); } +static inline void task_untag_mask(struct seq_file *m, struct mm_struct *mm) +{ + seq_printf(m, "untag_mask:\t%#lx\n", mm_untag_mask(mm)); +} + int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, struct pid *pid, struct task_struct *task) { @@ -443,6 +448,7 @@ int proc_pid_status(struct seq_file *m, struct pid_namespace *ns, task_mem(m, mm); task_core_dumping(m, task); task_thp_status(m, mm); + task_untag_mask(m, mm); mmput(mm); } task_sig(m, task); diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index b9b970f7ab45..14b9c1fa05c4 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -28,4 +28,11 @@ static inline void leave_mm(int cpu) { } # define task_cpu_possible(cpu, p) cpumask_test_cpu((cpu), task_cpu_possible_mask(p)) #endif +#ifndef mm_untag_mask +static inline unsigned long mm_untag_mask(struct mm_struct *mm) +{ + return -1UL; +} +#endif + #endif From patchwork Tue Dec 27 03:08:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36752 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200626wrt; Mon, 26 Dec 2022 19:10:11 -0800 (PST) X-Google-Smtp-Source: AMrXdXtO6ezyGojM1yLp0yEBydoa4KnNMD9xKGQpJbrfEF7XWwAQ6cfouoWjmHbbT1hSnD9+/BIp X-Received: by 2002:a17:902:a98a:b0:189:cef2:88e3 with SMTP id bh10-20020a170902a98a00b00189cef288e3mr22133998plb.57.1672110610920; Mon, 26 Dec 2022 19:10:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110610; cv=none; d=google.com; s=arc-20160816; b=U2vcipiToGch+OG39Je5nWVA+LvmL28cXjNNpfkjIAV9uaxKHbBA3W4b4Y27F8e6xT XGg+taQSEsK3wWGRzHFfuby8BNE3gJcpwJtIKLUW7hl4YSmcPwyEChBI7MivbM4erbQi HP9s03EvZOLmHxn0X+E8te7FaqF2v09heGYknTYS9JK+D8SoCL7pw11RBUZZf/8+Pf0C L5rlLPWbefhHwm11tPBaqpRwNM2pI4BjKT6/AHm1sQGOy4yAgo4uy2he8avqXUFSiWrn 5SlG0rsRfRpu/VuillJR+LmDXGMm17EqfqgcWltxK1ReBa9+Fc1ipj38dBiFYZBINv9Q Wxbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=RH27/yMEaIKNSxbdZlEBBsW8S4Z6jdHWiIv/A1SgiNc=; b=OxOrlyyXh4Z2rPn1OC4oYXOV8Of3Zyh3fsfl/bB2HK7uOHlN3VV/FOcYSM1WeQPN1T e9BiU47YQMyGV2V+dL1OeZrS1jwhTohUZTiIT283krpZCXsj/bmjnNLFr6ojwkkpQxGo IxJPyrQalXyz4avNZAjAYYglkQqsCzZH96DuE81LBL3/3hsvIzu1J9jAeu85/JVkpRKp 8oaOFnonQS9IbitoooalmML9GvKj4NlYCGU94FoEGufAdgWXRAL0e/kgN2b3RIlYpidD XPULgLculFR45Wx/PqVWDwWd+fM8xV9iguGWFvVINpPZJPcJYs+G3TWgW/JP+Q6PxFVL MPhg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XxX7AET1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b3-20020a63cf43000000b00478eaccedbbsi12468032pgj.582.2022.12.26.19.09.58; Mon, 26 Dec 2022 19:10:10 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=XxX7AET1; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230448AbiL0DJd (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229954AbiL0DJC (ORCPT ); Mon, 26 Dec 2022 22:09:02 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2BF22647 for ; Mon, 26 Dec 2022 19:08:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110533; x=1703646533; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Qz3TmQ0EtndSBYuEcqzm7EswKf58sIKEztNONQoPvAY=; b=XxX7AET1R1uWNRRJFhhb5sGg950jh6x4vp2pzcpdWJTEFE8lXonnfENH Dy5nEdMzicJHZmmaMU9Bjs/Abvmg6keqixiwvwIlx0aLoe7J76BpvhCow Zi6gY/6W0tftPpRDYWnIvUkenKhMC+tTqihKtduMf6Ej1Puqqlx1+6h2T C29ADRCNYXpbFcudiAZbsW2Qqie0ZQgqRAki/i+IDTz8aVhf8vZs0qpk8 mZh1sutqdzHRQxvjxoGk4dmJOa3KkBwjB2sQOYq1aXdzX9xhpDyTgStSO QMisq/Bq0iGGGNZZsy4787kXxllj4EPVrG3PnpOa9lKIqt9lfF5lRjIA9 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="321869781" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="321869781" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:52 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="652917545" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="652917545" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:47 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id D474E10BBB6; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 09/16] iommu/sva: Replace pasid_valid() helper with mm_valid_pasid() Date: Tue, 27 Dec 2022 06:08:22 +0300 Message-Id: <20221227030829.12508-10-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335055911384730?= X-GMAIL-MSGID: =?utf-8?q?1753335055911384730?= Kernel has few users of pasid_valid() and all but one checks if the process has PASID allocated. The helper takes ioasid_t as the input. Replace the helper with mm_valid_pasid() that takes mm_struct as the argument. The only call that checks PASID that is not tied to mm_struct is open-codded now. This is preparatory patch. It helps avoid ifdeffery: no need to dereference mm->pasid in generic code to check if the process has PASID. Signed-off-by: Kirill A. Shutemov --- arch/x86/kernel/traps.c | 6 +++--- drivers/iommu/iommu-sva.c | 4 ++-- include/linux/ioasid.h | 9 --------- include/linux/sched/mm.h | 8 +++++++- 4 files changed, 12 insertions(+), 15 deletions(-) diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index d317dc3d06a3..8b83d8fbce71 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -671,15 +671,15 @@ static bool try_fixup_enqcmd_gp(void) if (!cpu_feature_enabled(X86_FEATURE_ENQCMD)) return false; - pasid = current->mm->pasid; - /* * If the mm has not been allocated a * PASID, the #GP can not be fixed up. */ - if (!pasid_valid(pasid)) + if (!mm_valid_pasid(current->mm)) return false; + pasid = current->mm->pasid; + /* * Did this thread already have its PASID activated? * If so, the #GP must be from something else. diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 24bf9b2b58aa..4ee2929f0d7a 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -34,14 +34,14 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) mutex_lock(&iommu_sva_lock); /* Is a PASID already associated with this mm? */ - if (pasid_valid(mm->pasid)) { + if (mm_valid_pasid(mm)) { if (mm->pasid < min || mm->pasid >= max) ret = -EOVERFLOW; goto out; } pasid = ioasid_alloc(&iommu_sva_pasid, min, max, mm); - if (!pasid_valid(pasid)) + if (pasid == INVALID_IOASID) ret = -ENOMEM; else mm_pasid_set(mm, pasid); diff --git a/include/linux/ioasid.h b/include/linux/ioasid.h index af1c9d62e642..836ae09e92c2 100644 --- a/include/linux/ioasid.h +++ b/include/linux/ioasid.h @@ -40,10 +40,6 @@ void *ioasid_find(struct ioasid_set *set, ioasid_t ioasid, int ioasid_register_allocator(struct ioasid_allocator_ops *allocator); void ioasid_unregister_allocator(struct ioasid_allocator_ops *allocator); int ioasid_set_data(ioasid_t ioasid, void *data); -static inline bool pasid_valid(ioasid_t ioasid) -{ - return ioasid != INVALID_IOASID; -} #else /* !CONFIG_IOASID */ static inline ioasid_t ioasid_alloc(struct ioasid_set *set, ioasid_t min, @@ -74,10 +70,5 @@ static inline int ioasid_set_data(ioasid_t ioasid, void *data) return -ENOTSUPP; } -static inline bool pasid_valid(ioasid_t ioasid) -{ - return false; -} - #endif /* CONFIG_IOASID */ #endif /* __LINUX_IOASID_H */ diff --git a/include/linux/sched/mm.h b/include/linux/sched/mm.h index 2a243616f222..b69fe7e8c0ac 100644 --- a/include/linux/sched/mm.h +++ b/include/linux/sched/mm.h @@ -457,6 +457,11 @@ static inline void mm_pasid_init(struct mm_struct *mm) mm->pasid = INVALID_IOASID; } +static inline bool mm_valid_pasid(struct mm_struct *mm) +{ + return mm->pasid != INVALID_IOASID; +} + /* Associate a PASID with an mm_struct: */ static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) { @@ -465,13 +470,14 @@ static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) static inline void mm_pasid_drop(struct mm_struct *mm) { - if (pasid_valid(mm->pasid)) { + if (mm_valid_pasid(mm)) { ioasid_free(mm->pasid); mm->pasid = INVALID_IOASID; } } #else static inline void mm_pasid_init(struct mm_struct *mm) {} +static inline bool mm_valid_pasid(struct mm_struct *mm) { return false; } static inline void mm_pasid_set(struct mm_struct *mm, u32 pasid) {} static inline void mm_pasid_drop(struct mm_struct *mm) {} #endif From patchwork Tue Dec 27 03:08:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36754 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200716wrt; Mon, 26 Dec 2022 19:10:32 -0800 (PST) X-Google-Smtp-Source: AMrXdXudMzTP0OH11JWfcxSWiPxKdiaepi/5B18YWuMZ73PVYUrfZFu0Bj2S0bwU27K0+zzrBu9N X-Received: by 2002:a17:90b:2682:b0:225:e761:ff0a with SMTP id pl2-20020a17090b268200b00225e761ff0amr7307552pjb.1.1672110631829; Mon, 26 Dec 2022 19:10:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110631; cv=none; d=google.com; s=arc-20160816; b=dkbiOyAtAWf5GDvHAAIjNmcsNKMEtDFFjFbRNaDVg3NPCjMVERrAO8P0yHFOTR5XPr VTJIXVNIVqlY5fWRVQxCHA9e7Bz/hmILFwvxpzgdQ079aBU6Tt/TGHcARjeolx8eTM5p v9fEPW9oGtw6XOY05q+WhA5MjIoh5zP7/za2oLjrra5dT5bHXswpwshM7uG4KpOnmvmq pn/4UReoOhFpVBgGy5PhqdHT6nkGbNBv13FmpIRruhlxEWPCapm+1/xX6rCe9gh8UaXf nH5p2UXmoG6XiBG/P5j7chwl+1UprnQeola+yYxsPdfupGgqqG10LwlOpIEJAY3rsAxU YWfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+IRLHyVHXI80YSVYaqVuD5iuaMqaAmx3HLnreDtNbNE=; b=Rtr/AsMe3AR/6NFyCDZ657wNKCWXQC3e6GUdcBDbr2Qsw4gF3OViifU0N4m21uBwrC dniDOlxbz3BumjoSGtW1+L6+jaFh1giOQhyT7KEgE11gQUTi+S7RerHq4cGVKYkoReFr Rv7p2qHtNaW7pV3axtKZSE8+2l2Lf3LXokiQeAMz0bB/8urpAno019AiZ8FSzOr6t7Om SQTsEhaqKe0kV3mSYygLBxjsyVx9T9YVb/gjMl8mESjxvHn8eOB3l5vYGfmsMoB9phfB BDK9HVOX7aIV8GT2yNLT+xEqzHfIPStNW5atHJSgXUTyjcV5fPFicoxqG4UgrEnz2Fld J4gQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=PRVgfvZa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p12-20020a17090a428c00b00219404caaeasi16111429pjg.164.2022.12.26.19.10.19; Mon, 26 Dec 2022 19:10:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=PRVgfvZa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230164AbiL0DJn (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38602 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229993AbiL0DJG (ORCPT ); Mon, 26 Dec 2022 22:09:06 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45FD12AFD for ; Mon, 26 Dec 2022 19:08:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110535; x=1703646535; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=NXgfwVMopqIsMq8iO8GLDE0ia9jK476IcYsVpRyLMg4=; b=PRVgfvZaTdpzBmV6zYB3d8CaSen6KVElpuCgazTXJ6NFn1nflMCrAe3t rlhx7N6omnjDkybvAjGxQQ++hvc9W6fsN3/u97Sbp5mTdmpylhMlaX5Hm 8XrEvJSaxVOBqNeDamGqDNR10Q/DxKJ/cJmPVMUwEJ/n4YWLKn9bwCtK+ PphZLrXrDiRQojpM3wLT9iwS/Tglu7D5VZfcbNUrwA3SrZUvhDnWpsrAR Ct7UNrgBVdt4hpliuFzbFdwahV2o0sy0eI2HsxdQXzVDFfSg+qQhkeLOQ OHN9KGR9xG//95dZTDENAMqIwc2uCgufoZNi/xCIbH07h5gwMkhj8EIBO w==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="321869797" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="321869797" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:53 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="652917550" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="652917550" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:47 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id DEAA810BBB7; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 10/16] x86/mm/iommu/sva: Make LAM and SVA mutually exclusive Date: Tue, 27 Dec 2022 06:08:23 +0300 Message-Id: <20221227030829.12508-11-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335077549682650?= X-GMAIL-MSGID: =?utf-8?q?1753335077549682650?= IOMMU and SVA-capable devices know nothing about LAM and only expect canonical addresses. An attempt to pass down tagged pointer will lead to address translation failure. By default do not allow to enable both LAM and use SVA in the same process. The new ARCH_FORCE_TAGGED_SVA arch_prctl() overrides the limitation. By using the arch_prctl() userspace takes responsibility to never pass tagged address to the device. Signed-off-by: Kirill A. Shutemov Reviewed-by: Ashok Raj Reviewed-by: Jacob Pan --- arch/x86/include/asm/mmu.h | 2 ++ arch/x86/include/asm/mmu_context.h | 6 ++++++ arch/x86/include/uapi/asm/prctl.h | 1 + arch/x86/kernel/process_64.c | 7 +++++++ drivers/iommu/iommu-sva.c | 4 ++++ include/linux/mmu_context.h | 7 +++++++ 6 files changed, 27 insertions(+) diff --git a/arch/x86/include/asm/mmu.h b/arch/x86/include/asm/mmu.h index 54e4a3e9b5c5..90d20679e4d7 100644 --- a/arch/x86/include/asm/mmu.h +++ b/arch/x86/include/asm/mmu.h @@ -14,6 +14,8 @@ #define MM_CONTEXT_HAS_VSYSCALL 1 /* Do not allow changing LAM mode */ #define MM_CONTEXT_LOCK_LAM 2 +/* Allow LAM and SVA coexisting */ +#define MM_CONTEXT_FORCE_TAGGED_SVA 3 /* * x86 has arch-specific MMU state beyond what lives in mm_struct. diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 7f9f9978c811..4bc95c35cbd3 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -114,6 +114,12 @@ static inline void mm_reset_untag_mask(struct mm_struct *mm) mm->context.untag_mask = -1UL; } +#define arch_pgtable_dma_compat arch_pgtable_dma_compat +static inline bool arch_pgtable_dma_compat(struct mm_struct *mm) +{ + return !mm_lam_cr3_mask(mm) || + test_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &mm->context.flags); +} #else static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) diff --git a/arch/x86/include/uapi/asm/prctl.h b/arch/x86/include/uapi/asm/prctl.h index a31e27b95b19..eb290d89cb32 100644 --- a/arch/x86/include/uapi/asm/prctl.h +++ b/arch/x86/include/uapi/asm/prctl.h @@ -23,5 +23,6 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 #define ARCH_GET_MAX_TAG_BITS 0x4003 +#define ARCH_FORCE_TAGGED_SVA 0x4004 #endif /* _ASM_X86_PRCTL_H */ diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 09e7f3d3fb5c..add85615d5ae 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -756,6 +756,10 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) if (test_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags)) return -EBUSY; + if (mm_valid_pasid(mm) && + !test_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &mm->context.flags)) + return -EINTR; + if (mmap_write_lock_killable(mm)) return -EINTR; @@ -872,6 +876,9 @@ long do_arch_prctl_64(struct task_struct *task, int option, unsigned long arg2) (unsigned long __user *)arg2); case ARCH_ENABLE_TAGGED_ADDR: return prctl_enable_tagged_addr(task->mm, arg2); + case ARCH_FORCE_TAGGED_SVA: + set_bit(MM_CONTEXT_FORCE_TAGGED_SVA, &task->mm->context.flags); + return 0; case ARCH_GET_MAX_TAG_BITS: if (!cpu_feature_enabled(X86_FEATURE_LAM)) return put_user(0, (unsigned long __user *)arg2); diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index 4ee2929f0d7a..dd76a1a09cf7 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -2,6 +2,7 @@ /* * Helpers for IOMMU drivers implementing SVA */ +#include #include #include #include @@ -32,6 +33,9 @@ int iommu_sva_alloc_pasid(struct mm_struct *mm, ioasid_t min, ioasid_t max) min == 0 || max < min) return -EINVAL; + if (!arch_pgtable_dma_compat(mm)) + return -EBUSY; + mutex_lock(&iommu_sva_lock); /* Is a PASID already associated with this mm? */ if (mm_valid_pasid(mm)) { diff --git a/include/linux/mmu_context.h b/include/linux/mmu_context.h index 14b9c1fa05c4..f2b7a3f04099 100644 --- a/include/linux/mmu_context.h +++ b/include/linux/mmu_context.h @@ -35,4 +35,11 @@ static inline unsigned long mm_untag_mask(struct mm_struct *mm) } #endif +#ifndef arch_pgtable_dma_compat +static inline bool arch_pgtable_dma_compat(struct mm_struct *mm) +{ + return true; +} +#endif + #endif From patchwork Tue Dec 27 03:08:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36750 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200579wrt; Mon, 26 Dec 2022 19:10:02 -0800 (PST) X-Google-Smtp-Source: AMrXdXvw4llLcMwPQVWzibC14aeijB5c4lRPpqdGBmM2goUlldyAZ5AdGxyFJxwpnRs1477DLlh9 X-Received: by 2002:a17:90a:77c5:b0:219:b330:7f6d with SMTP id e5-20020a17090a77c500b00219b3307f6dmr23527822pjs.24.1672110602513; Mon, 26 Dec 2022 19:10:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110602; cv=none; d=google.com; s=arc-20160816; b=DwQaaXJcHRmq/08VxWXlJMykK8M4vZOOWapfxdBZaGe892qDhgWE61bIM7D34ZmAT2 LnOQIiH6pBe+PvH0hAOJfSIPYzxK8cnyYyWZrMDYgehszRAIr6lXdrOO9yZWqVqp8ztf BOTcTNRdl4+YBI8QceCCymlaDO4txSP8PpLqT+CnrYtoswyQ8wxijMk7mMNZuuG0ll5/ 0KefsG/308BTJU9xwu3oHeK2gGUSg70706yWo7bZTMfLn/ts7U2CLfIthOSLOqUe2vUu Y0kdyphuDU4gJHknTJhttbrxovGuYLXVLEwR9bdtbRMuTQKQW+wcOuwlYE2OEWK0fIgf ak8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VP40NBrtZaTCiilSOIQrzWAq7tpojQEqiNiuMY8T3tU=; b=ZXssmv1x4Qi40GZYCsP6wPicqubEd1TNUu0iSZ60CLfiPJe+bYT+KN0rKbAvBTiavX BKvvFQS65rpF8BUamrt0aSwTBFViiXoTXQKTcnO/tAZG1y5Ozy8tMxNjrjo/Q0eVZY2x rrR/dBWWoxFceU3Ed1TR1XfibA251G6YRPoCWjQ68w+57MsGOCazE5Iuq8x6OgQdDYh8 ZqpdhbE96/pOUeoDrDNxLAaa0chbkTrMYGLcSMcmpBUgJR0whbnMBIcNcVmFbEzCWjDm PwSk74t+WIP7gEO+1mCQYvHsQn53bGkMfpL0fsoJvs3kwR+mEir8qnSaGrkRrQsy9DsD V8WQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=AK39fl2m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id nn16-20020a17090b38d000b0020038eb8b5asi14064683pjb.21.2022.12.26.19.09.50; Mon, 26 Dec 2022 19:10:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=AK39fl2m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230330AbiL0DJX (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229818AbiL0DIx (ORCPT ); Mon, 26 Dec 2022 22:08:53 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89EB1EA5 for ; Mon, 26 Dec 2022 19:08:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110532; x=1703646532; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=XGbLeIJ0IKKIjilK4bbz2t30FLF4Tx7fCu7ADrjpn30=; b=AK39fl2moS2D8Ny9JGgo7RI1zqnUcJvVyqfy7pBhEjznfyBpoKr1IIuU 7CkJu5d+tpKDpu7Y36k99OKtoZlCsE4XzwuY5snnWnZyOQZSNNKrB4ir7 c7NS5XwnpkvR8/bmQgDoZdAv14jop3p/aBJbkFcTBQlqiwwo69BwZV2d8 JPirGsCv+JkT7bIgu+Q/+l4P3qxNCA3hibQUrbQfcOc2dqt39sYjY3K8V VP6JIFcNNwzCfiquG/XK/9bEH5MCEcZXErL0TvivJv6faWb0V4YWbXBHH ee5YxrddQ5MSsXRvC725RdZscUZd6fHDANUnePTPIHkg+sumCzsoAVXvp Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="300994716" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="300994716" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:52 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="741605222" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="741605222" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:47 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id EA43610BBB8; Tue, 27 Dec 2022 06:08:36 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv13 11/16] selftests/x86/lam: Add malloc and tag-bits test cases for linear-address masking Date: Tue, 27 Dec 2022 06:08:24 +0300 Message-Id: <20221227030829.12508-12-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335047318612460?= X-GMAIL-MSGID: =?utf-8?q?1753335047318612460?= From: Weihong Zhang LAM is supported only in 64-bit mode and applies only addresses used for data accesses. In 64-bit mode, linear address have 64 bits. LAM is applied to 64-bit linear address and allow software to use high bits for metadata. LAM supports configurations that differ regarding which pointer bits are masked and can be used for metadata. LAM includes following mode: - LAM_U57, pointer bits in positions 62:57 are masked (LAM width 6), allows bits 62:57 of a user pointer to be used as metadata. There are some arch_prctls: ARCH_ENABLE_TAGGED_ADDR: enable LAM mode, mask high bits of a user pointer. ARCH_GET_UNTAG_MASK: get current untagged mask. ARCH_GET_MAX_TAG_BITS: the maximum tag bits user can request. zero if LAM is not supported. The LAM mode is for pre-process, a process has only one chance to set LAM mode. But there is no API to disable LAM mode. So all of test cases are run under child process. Functions of this test: MALLOC - LAM_U57 masks bits 57:62 of a user pointer. Process on user space can dereference such pointers. - Disable LAM, dereference a pointer with metadata above 48 bit or 57 bit lead to trigger SIGSEGV. TAG_BITS - Max tag bits of LAM_U57 is 6. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/Makefile | 2 +- tools/testing/selftests/x86/lam.c | 326 +++++++++++++++++++++++++++ 2 files changed, 327 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/x86/lam.c diff --git a/tools/testing/selftests/x86/Makefile b/tools/testing/selftests/x86/Makefile index 0388c4d60af0..c1a16a9d4f2f 100644 --- a/tools/testing/selftests/x86/Makefile +++ b/tools/testing/selftests/x86/Makefile @@ -18,7 +18,7 @@ TARGETS_C_32BIT_ONLY := entry_from_vm86 test_syscall_vdso unwind_vdso \ test_FCMOV test_FCOMI test_FISTTP \ vdso_restorer TARGETS_C_64BIT_ONLY := fsgsbase sysret_rip syscall_numbering \ - corrupt_xstate_header amx + corrupt_xstate_header amx lam # Some selftests require 32bit support enabled also on 64bit systems TARGETS_C_32BIT_NEEDED := ldt_gdt ptrace_syscall diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c new file mode 100644 index 000000000000..900a3a0fb709 --- /dev/null +++ b/tools/testing/selftests/x86/lam.c @@ -0,0 +1,326 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "../kselftest.h" + +#ifndef __x86_64__ +# error This test is 64-bit only +#endif + +/* LAM modes, these definitions were copied from kernel code */ +#define LAM_NONE 0 +#define LAM_U57_BITS 6 + +#define LAM_U57_MASK (0x3fULL << 57) +/* arch prctl for LAM */ +#define ARCH_GET_UNTAG_MASK 0x4001 +#define ARCH_ENABLE_TAGGED_ADDR 0x4002 +#define ARCH_GET_MAX_TAG_BITS 0x4003 + +/* Specified test function bits */ +#define FUNC_MALLOC 0x1 +#define FUNC_BITS 0x2 + +#define TEST_MASK 0x3 + +#define MALLOC_LEN 32 + +struct testcases { + unsigned int later; + int expected; /* 2: SIGSEGV Error; 1: other errors */ + unsigned long lam; + uint64_t addr; + int (*test_func)(struct testcases *test); + const char *msg; +}; + +int tests_cnt; +jmp_buf segv_env; + +static void segv_handler(int sig) +{ + ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); +} + +static inline int cpu_has_lam(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 1, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[0] & (1 << 26)); +} + +/* + * Set tagged address and read back untag mask. + * check if the untagged mask is expected. + * + * @return: + * 0: Set LAM mode successfully + * others: failed to set LAM + */ +static int set_lam(unsigned long lam) +{ + int ret = 0; + uint64_t ptr = 0; + + if (lam != LAM_U57_BITS && lam != LAM_NONE) + return -1; + + /* Skip check return */ + syscall(SYS_arch_prctl, ARCH_ENABLE_TAGGED_ADDR, lam); + + /* Get untagged mask */ + syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr); + + /* Check mask returned is expected */ + if (lam == LAM_U57_BITS) + ret = (ptr != ~(LAM_U57_MASK)); + else if (lam == LAM_NONE) + ret = (ptr != -1ULL); + + return ret; +} + +static unsigned long get_default_tag_bits(void) +{ + pid_t pid; + int lam = LAM_NONE; + int ret = 0; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + } else if (pid == 0) { + /* Set LAM mode in child process */ + if (set_lam(LAM_U57_BITS) == 0) + lam = LAM_U57_BITS; + else + lam = LAM_NONE; + exit(lam); + } else { + wait(&ret); + lam = WEXITSTATUS(ret); + } + + return lam; +} + +/* According to LAM mode, set metadata in high bits */ +static uint64_t set_metadata(uint64_t src, unsigned long lam) +{ + uint64_t metadata; + + srand(time(NULL)); + /* Get a random value as metadata */ + metadata = rand(); + + switch (lam) { + case LAM_U57_BITS: /* Set metadata in bits 62:57 */ + metadata = (src & ~(LAM_U57_MASK)) | ((metadata & 0x3f) << 57); + break; + default: + metadata = src; + break; + } + + return metadata; +} + +/* + * Set metadata in user pointer, compare new pointer with original pointer. + * both pointers should point to the same address. + * + * @return: + * 0: value on the pointer with metadate and value on original are same + * 1: not same. + */ +static int handle_lam_test(void *src, unsigned int lam) +{ + char *ptr; + + strcpy((char *)src, "USER POINTER"); + + ptr = (char *)set_metadata((uint64_t)src, lam); + if (src == ptr) + return 0; + + /* Copy a string into the pointer with metadata */ + strcpy((char *)ptr, "METADATA POINTER"); + + return (!!strcmp((char *)src, (char *)ptr)); +} + + +int handle_max_bits(struct testcases *test) +{ + unsigned long exp_bits = get_default_tag_bits(); + unsigned long bits = 0; + + if (exp_bits != LAM_NONE) + exp_bits = LAM_U57_BITS; + + /* Get LAM max tag bits */ + if (syscall(SYS_arch_prctl, ARCH_GET_MAX_TAG_BITS, &bits) == -1) + return 1; + + return (exp_bits != bits); +} + +/* + * Test lam feature through dereference pointer get from malloc. + * @return 0: Pass test. 1: Get failure during test 2: Get SIGSEGV + */ +static int handle_malloc(struct testcases *test) +{ + char *ptr = NULL; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) == -1) + return 1; + + ptr = (char *)malloc(MALLOC_LEN); + if (ptr == NULL) { + perror("malloc() failure\n"); + return 1; + } + + /* Set signal handler */ + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) == -1 && ret == 0) + ret = 1; + + free(ptr); + + return ret; +} + +static int fork_test(struct testcases *test) +{ + int ret, child_ret; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + ret = test->test_func(test); + exit(ret); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + } + + return ret; +} + +static void run_test(struct testcases *test, int count) +{ + int i, ret = 0; + + for (i = 0; i < count; i++) { + struct testcases *t = test + i; + + /* fork a process to run test case */ + ret = fork_test(t); + if (ret != 0) + ret = (t->expected == ret); + else + ret = !(t->expected); + + tests_cnt++; + ksft_test_result(ret, t->msg); + } +} + +static struct testcases malloc_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 2, + .lam = LAM_U57_BITS, + .test_func = handle_malloc, + .msg = "MALLOC:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + + +static struct testcases bits_cases[] = { + { + .test_func = handle_max_bits, + .msg = "BITS: Check default tag bits\n", + }, +}; + +static void cmd_help(void) +{ + printf("usage: lam [-h] [-t test list]\n"); + printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); + printf("\t\t0x1:malloc; 0x2:max_bits;\n"); + printf("\t-h: help\n"); +} + +int main(int argc, char **argv) +{ + int c = 0; + unsigned int tests = TEST_MASK; + + tests_cnt = 0; + + if (!cpu_has_lam()) { + ksft_print_msg("Unsupported LAM feature!\n"); + return -1; + } + + while ((c = getopt(argc, argv, "ht:")) != -1) { + switch (c) { + case 't': + tests = strtoul(optarg, NULL, 16); + if (!(tests & TEST_MASK)) { + ksft_print_msg("Invalid argument!\n"); + return -1; + } + break; + case 'h': + cmd_help(); + return 0; + default: + ksft_print_msg("Invalid argument\n"); + return -1; + } + } + + if (tests & FUNC_MALLOC) + run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); + + if (tests & FUNC_BITS) + run_test(bits_cases, ARRAY_SIZE(bits_cases)); + + ksft_set_plan(tests_cnt); + + return ksft_exit_pass(); +} From patchwork Tue Dec 27 03:08:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36756 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1201077wrt; Mon, 26 Dec 2022 19:11:50 -0800 (PST) X-Google-Smtp-Source: AMrXdXtYbWLawoo7D4JkZsi4DCQTk4QWAgvSDFRxSif8GIbXN9ohys8a6QEoxy3Vayq2YLMQr09N X-Received: by 2002:a05:6a21:7884:b0:af:98cd:7df1 with SMTP id bf4-20020a056a21788400b000af98cd7df1mr31599978pzc.24.1672110710137; Mon, 26 Dec 2022 19:11:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110710; cv=none; d=google.com; s=arc-20160816; b=jmNVBPXI/BQEcYz6i9MYKBplwlLrMAxdJqS3PfCfieHde+Eqi46FxuDFgSnw20E45R FfVj32ZOpO94S/op8tt7+kOC0OXSHShBsvNguG5IaptglKxoRKPFchwnviTv/H39OMF1 hTQqUCsk6Hnwuz9dG/aVbXwUsH55EUDZhWudwW07PxQvav8wL/XUherDAcHCOLz8qWIK TeW7z5vKBiW/ewtkDMAfOHEgpZK+M3eGcaxWAp9pmrytbloUAw35mHf0oXCIfofcj+co IJR0my0RBmhXt8pjtzglPyPOW3zlDyFIF9YSCICDAZvxHW0c+VFf6/ee5QW26UFAJRKU fYLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Qg4vaTMb+9PkFsCOYp+C/L7w4P1xSjElnkamq7/2Ef8=; b=X7fW/GCPIjdzIqQY/BmXxUk5cd8Xvi92d7lP5MjTcmM9H11uIJ/dpFQbyWD0XWGlHq 9sDONjV+uGv+e432fbpuj91kjlNFQyFbiajjK3d1lmNwEA95E+JgDgsbUS0rjIUfT1Ss WOHsY9LAChbdMataGqy/7VEf7YK7bhDKjXa4PxuYvCVbTspkzsYfTlbXXwibtAFBpVPj r0w62XvAqS0KZjT2V24PJKbdFFhiisuYwZkjthFsUpErVuV0FDTDlBnED0feZ5GG1k0t KWPcRipXap2MI6Nh15zAJDVnPoxgISpJXgB4pHVC3LSSpvzInmwve1B7DT54Ler57Ca7 s9Eg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="V/dUzOrX"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x30-20020a63171e000000b0047860cda19dsi11940159pgl.859.2022.12.26.19.11.38; Mon, 26 Dec 2022 19:11:50 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="V/dUzOrX"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230191AbiL0DJr (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230013AbiL0DJH (ORCPT ); Mon, 26 Dec 2022 22:09:07 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B1E13893 for ; Mon, 26 Dec 2022 19:08:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110536; x=1703646536; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=1pBfAOkQ7J1AFfoVfQmMQolBNFBcBfukgLGZulMuiy0=; b=V/dUzOrXfo5QVaT47wWKB4u3x5tV9EKVkP7CDN85GMfcKpnpbTynAelG L1JnzTCKVkh2yIQG0XbkiS2p93mLXE0y1oWnJTBqS/I8ulCQpjPD6UTEn FUoS13w6ag2ymqt1uDeRtOcmvGgm2j4lr8Ed56Z9b7T0+W7SIoerZFKpG OnicFkIq8siYCMe6W2PYwc4gJ92VEzoE5e2YT9UQt4TRWDIwCc8lihK6v EnUwITUizS9eBRxuyLFiwcYFvr51o8YmvY4M1gMcfARfKOVVg32U4YgBD anSo9464TU2f1R7+xONngECQ+PbBhi+LgTZvI61rjnHtLKs5o8YgNJ4b1 g==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="321869805" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="321869805" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:53 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="652917554" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="652917554" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:48 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 013ED10BBB9; Tue, 27 Dec 2022 06:08:37 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv13 12/16] selftests/x86/lam: Add mmap and SYSCALL test cases for linear-address masking Date: Tue, 27 Dec 2022 06:08:25 +0300 Message-Id: <20221227030829.12508-13-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335159873149928?= X-GMAIL-MSGID: =?utf-8?q?1753335159873149928?= From: Weihong Zhang Add mmap and SYSCALL test cases. SYSCALL test cases: - LAM supports set metadata in high bits 62:57 (LAM_U57) of a user pointer, pass the pointer to SYSCALL, SYSCALL can dereference the pointer and return correct result. - Disable LAM, pass a pointer with metadata in high bits to SYSCALL, SYSCALL returns -1 (EFAULT). MMAP test cases: - Enable LAM_U57, MMAP with low address (below bits 47), set metadata in high bits of the address, dereference the address should be allowed. - Enable LAM_U57, MMAP with high address (above bits 47), set metadata in high bits of the address, dereference the address should be allowed. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 144 +++++++++++++++++++++++++++++- 1 file changed, 140 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 900a3a0fb709..cdc6e40e00e0 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -29,11 +30,18 @@ /* Specified test function bits */ #define FUNC_MALLOC 0x1 #define FUNC_BITS 0x2 +#define FUNC_MMAP 0x4 +#define FUNC_SYSCALL 0x8 -#define TEST_MASK 0x3 +#define TEST_MASK 0xf + +#define LOW_ADDR (0x1UL << 30) +#define HIGH_ADDR (0x3UL << 48) #define MALLOC_LEN 32 +#define PAGE_SIZE (4 << 10) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -49,6 +57,7 @@ jmp_buf segv_env; static void segv_handler(int sig) { ksft_print_msg("Get segmentation fault(%d).", sig); + siglongjmp(segv_env, 1); } @@ -61,6 +70,16 @@ static inline int cpu_has_lam(void) return (cpuinfo[0] & (1 << 26)); } +/* Check 5-level page table feature in CPUID.(EAX=07H, ECX=00H):ECX.[bit 16] */ +static inline int cpu_has_la57(void) +{ + unsigned int cpuinfo[4]; + + __cpuid_count(0x7, 0, cpuinfo[0], cpuinfo[1], cpuinfo[2], cpuinfo[3]); + + return (cpuinfo[2] & (1 << 16)); +} + /* * Set tagged address and read back untag mask. * check if the untagged mask is expected. @@ -213,6 +232,68 @@ static int handle_malloc(struct testcases *test) return ret; } +static int handle_mmap(struct testcases *test) +{ + void *ptr; + unsigned int flags = MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + ptr = mmap((void *)test->addr, PAGE_SIZE, PROT_READ | PROT_WRITE, + flags, -1, 0); + if (ptr == MAP_FAILED) { + if (test->addr == HIGH_ADDR) + if (!cpu_has_la57()) + return 3; /* unsupport LA57 */ + return 1; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + ret = 1; + + if (ret == 0) { + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = handle_lam_test(ptr, test->lam); + } else { + ret = 2; + } + } + + munmap(ptr, PAGE_SIZE); + return ret; +} + +static int handle_syscall(struct testcases *test) +{ + struct utsname unme, *pu; + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + pu = (struct utsname *)set_metadata((uint64_t)&unme, test->lam); + ret = uname(pu); + if (ret < 0) + ret = 1; + } else { + ret = 2; + } + + if (test->later != 0 && test->lam != 0) + if (set_lam(test->lam) != -1 && ret == 0) + ret = 1; + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -241,13 +322,20 @@ static void run_test(struct testcases *test, int count) struct testcases *t = test + i; /* fork a process to run test case */ + tests_cnt++; ret = fork_test(t); + + /* return 3 is not support LA57, the case should be skipped */ + if (ret == 3) { + ksft_test_result_skip(t->msg); + continue; + } + if (ret != 0) ret = (t->expected == ret); else ret = !(t->expected); - tests_cnt++; ksft_test_result(ret, t->msg); } } @@ -268,7 +356,6 @@ static struct testcases malloc_cases[] = { }, }; - static struct testcases bits_cases[] = { { .test_func = handle_max_bits, @@ -276,11 +363,54 @@ static struct testcases bits_cases[] = { }, }; +static struct testcases syscall_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL: LAM_U57. syscall with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_syscall, + .msg = "SYSCALL:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + +static struct testcases mmap_cases[] = { + { + .later = 1, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First mmap high address, then set LAM_U57.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = HIGH_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then High address.\n", + }, + { + .later = 0, + .expected = 0, + .lam = LAM_U57_BITS, + .addr = LOW_ADDR, + .test_func = handle_mmap, + .msg = "MMAP: First LAM_U57, then Low address.\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits;\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n"); printf("\t-h: help\n"); } @@ -320,6 +450,12 @@ int main(int argc, char **argv) if (tests & FUNC_BITS) run_test(bits_cases, ARRAY_SIZE(bits_cases)); + if (tests & FUNC_MMAP) + run_test(mmap_cases, ARRAY_SIZE(mmap_cases)); + + if (tests & FUNC_SYSCALL) + run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Tue Dec 27 03:08:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36757 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1201121wrt; Mon, 26 Dec 2022 19:11:59 -0800 (PST) X-Google-Smtp-Source: AMrXdXvqa7MV40lKCghr7evfVQDKrqTgZUcb16fWycVhGc3eU5QLD25dd0HRBMgswhZWRk/NGBMh X-Received: by 2002:a17:903:445:b0:191:386:2314 with SMTP id iw5-20020a170903044500b0019103862314mr23168374plb.3.1672110719200; Mon, 26 Dec 2022 19:11:59 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110719; cv=none; d=google.com; s=arc-20160816; b=n7ZnDCqWxzWLpomwzwgTZT28NLoTl1DydGZ9timh2a39l2KKjRQruiBlKG6Q/8NgDy Z3uI4HXStPvlAcr1xHpPsRpmDJzqPg/ISzVxriTYQRs+O/1o850giJ0xaiseP6dxtLZi HTpvwjD7X29qoJ7syReo22g6tRxe18fXkq4wb7dUBEyoTA7uFam1kMjQ/qBQGiHRzGS7 Y9nLw7osE8hicQJbnQ3y83mgmfS7imWM7zg71sW+jxHB5h6SOIEO3edmXyjc3FUqqtny UNHQ/gikGMLKBZHjQ1K5zuAnKN0d1DysqqCCIyvm8UCaKHKxSBAfMYsbsbrJtjK+Wf62 vKMQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5qlHGcYI2a+UMNjhFqkkk9OrWv2Sl2EuCGTAp0aU/iM=; b=vGSQuPGe94/8WGXy5C0VqUFq/cTkT52ca2bZ1uQ0KEMtDcn6zyK+JtoP0cclo706NO WANKfNOWGVqcOqreJ/XsdU8GQzq2OX7sGKVM3coQxBur0P0HYe6kcmX/n8yJX1NdiHkp ogecmpURRUXqlSw5GywDzONjgiS+eLifYxf+0AkjUz2kEqjKWo1fwmu5SiLQQZaoj0NK l5bNd1Z12ra7TvrVF9S2aEN2wT/HuW0O5C25DrJ70gztQuvK5bHeF1uR05fPNylb8wE5 V1muQ4U3acfA7OJwluYkGvTCe7rhE/Rf+jPoXdibaKoNVu60R20cd84MR4ZVUrxpFEZk MBzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=LmuTOpg6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z12-20020a170902d54c00b0018919169bc4si12350698plf.430.2022.12.26.19.11.47; Mon, 26 Dec 2022 19:11:59 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=LmuTOpg6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231305AbiL0DJw (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230075AbiL0DJI (ORCPT ); Mon, 26 Dec 2022 22:09:08 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 634D95F7F for ; Mon, 26 Dec 2022 19:08:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110536; x=1703646536; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=E1p+Kr85nTDaBukWmiZhTMSRcyZXCm4ZQk3augfSpS0=; b=LmuTOpg64H21sRnScyazvpLPg4QE24uFf8y668rCKgViDg517C2n2TU5 Mp/2SzY9yP5fpNjc/+WoEr1a3FI2zY1YstdBu2GUr/g2O+0Nbpi1mAXuP dQJACppc3OqJuxB5Xi4DuJhBHsuHOaybqSd13lH/CA2/ghPxqWsS+DI2Z cHP9C0ctGi0AhbltkEirPx7P9Wxr3+6h9o1dfrln6fDWdRORWKrllsbVK CBPk41lsASZsj4b94ZPsldd6lMfHmGqpyziCdoisJbwKS6DnOr6txszBW n6bZMqQ8QIqjHeDY+gxgevXJpcyn3OB6iumSKTfbZG4l502Ot3hh4ZJZG w==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="321869813" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="321869813" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:54 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="652917557" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="652917557" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:48 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 0CD9C10BBBA; Tue, 27 Dec 2022 06:08:37 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv13 13/16] selftests/x86/lam: Add io_uring test cases for linear-address masking Date: Tue, 27 Dec 2022 06:08:26 +0300 Message-Id: <20221227030829.12508-14-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335169471419224?= X-GMAIL-MSGID: =?utf-8?q?1753335169471419224?= From: Weihong Zhang LAM should be supported in kernel thread, using io_uring to verify LAM feature. The test cases implement read a file through io_uring, the test cases choose an iovec array as receiving buffer, which used to receive data, according to LAM mode, set metadata in high bits of these buffer. io_uring can deal with these buffers that pointed to pointers with the metadata in high bits. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 341 +++++++++++++++++++++++++++++- 1 file changed, 339 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index cdc6e40e00e0..8ea1fcef4c9f 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -9,8 +9,12 @@ #include #include #include +#include +#include #include +#include +#include #include "../kselftest.h" #ifndef __x86_64__ @@ -32,8 +36,9 @@ #define FUNC_BITS 0x2 #define FUNC_MMAP 0x4 #define FUNC_SYSCALL 0x8 +#define FUNC_URING 0x10 -#define TEST_MASK 0xf +#define TEST_MASK 0x1f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -42,6 +47,13 @@ #define PAGE_SIZE (4 << 10) +#define barrier() ({ \ + __asm__ __volatile__("" : : : "memory"); \ +}) + +#define URING_QUEUE_SZ 1 +#define URING_BLOCK_SZ 2048 + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ @@ -51,6 +63,33 @@ struct testcases { const char *msg; }; +/* Used by CQ of uring, source file handler and file's size */ +struct file_io { + int file_fd; + off_t file_sz; + struct iovec iovecs[]; +}; + +struct io_uring_queue { + unsigned int *head; + unsigned int *tail; + unsigned int *ring_mask; + unsigned int *ring_entries; + unsigned int *flags; + unsigned int *array; + union { + struct io_uring_cqe *cqes; + struct io_uring_sqe *sqes; + } queue; + size_t ring_sz; +}; + +struct io_ring { + int ring_fd; + struct io_uring_queue sq_ring; + struct io_uring_queue cq_ring; +}; + int tests_cnt; jmp_buf segv_env; @@ -294,6 +333,285 @@ static int handle_syscall(struct testcases *test) return ret; } +int sys_uring_setup(unsigned int entries, struct io_uring_params *p) +{ + return (int)syscall(__NR_io_uring_setup, entries, p); +} + +int sys_uring_enter(int fd, unsigned int to, unsigned int min, unsigned int flags) +{ + return (int)syscall(__NR_io_uring_enter, fd, to, min, flags, NULL, 0); +} + +/* Init submission queue and completion queue */ +int mmap_io_uring(struct io_uring_params p, struct io_ring *s) +{ + struct io_uring_queue *sring = &s->sq_ring; + struct io_uring_queue *cring = &s->cq_ring; + + sring->ring_sz = p.sq_off.array + p.sq_entries * sizeof(unsigned int); + cring->ring_sz = p.cq_off.cqes + p.cq_entries * sizeof(struct io_uring_cqe); + + if (p.features & IORING_FEAT_SINGLE_MMAP) { + if (cring->ring_sz > sring->ring_sz) + sring->ring_sz = cring->ring_sz; + + cring->ring_sz = sring->ring_sz; + } + + void *sq_ptr = mmap(0, sring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_SQ_RING); + + if (sq_ptr == MAP_FAILED) { + perror("sub-queue!"); + return 1; + } + + void *cq_ptr = sq_ptr; + + if (!(p.features & IORING_FEAT_SINGLE_MMAP)) { + cq_ptr = mmap(0, cring->ring_sz, PROT_READ | PROT_WRITE, + MAP_SHARED | MAP_POPULATE, s->ring_fd, + IORING_OFF_CQ_RING); + if (cq_ptr == MAP_FAILED) { + perror("cpl-queue!"); + munmap(sq_ptr, sring->ring_sz); + return 1; + } + } + + sring->head = sq_ptr + p.sq_off.head; + sring->tail = sq_ptr + p.sq_off.tail; + sring->ring_mask = sq_ptr + p.sq_off.ring_mask; + sring->ring_entries = sq_ptr + p.sq_off.ring_entries; + sring->flags = sq_ptr + p.sq_off.flags; + sring->array = sq_ptr + p.sq_off.array; + + /* Map a queue as mem map */ + s->sq_ring.queue.sqes = mmap(0, p.sq_entries * sizeof(struct io_uring_sqe), + PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, + s->ring_fd, IORING_OFF_SQES); + if (s->sq_ring.queue.sqes == MAP_FAILED) { + munmap(sq_ptr, sring->ring_sz); + if (sq_ptr != cq_ptr) { + ksft_print_msg("failed to mmap uring queue!"); + munmap(cq_ptr, cring->ring_sz); + return 1; + } + } + + cring->head = cq_ptr + p.cq_off.head; + cring->tail = cq_ptr + p.cq_off.tail; + cring->ring_mask = cq_ptr + p.cq_off.ring_mask; + cring->ring_entries = cq_ptr + p.cq_off.ring_entries; + cring->queue.cqes = cq_ptr + p.cq_off.cqes; + + return 0; +} + +/* Init io_uring queues */ +int setup_io_uring(struct io_ring *s) +{ + struct io_uring_params para; + + memset(¶, 0, sizeof(para)); + s->ring_fd = sys_uring_setup(URING_QUEUE_SZ, ¶); + if (s->ring_fd < 0) + return 1; + + return mmap_io_uring(para, s); +} + +/* + * Get data from completion queue. the data buffer saved the file data + * return 0: success; others: error; + */ +int handle_uring_cq(struct io_ring *s) +{ + struct file_io *fi = NULL; + struct io_uring_queue *cring = &s->cq_ring; + struct io_uring_cqe *cqe; + unsigned int head; + off_t len = 0; + + head = *cring->head; + + do { + barrier(); + if (head == *cring->tail) + break; + /* Get the entry */ + cqe = &cring->queue.cqes[head & *s->cq_ring.ring_mask]; + fi = (struct file_io *)cqe->user_data; + if (cqe->res < 0) + break; + + int blocks = (int)(fi->file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + for (int i = 0; i < blocks; i++) + len += fi->iovecs[i].iov_len; + + head++; + } while (1); + + *cring->head = head; + barrier(); + + return (len != fi->file_sz); +} + +/* + * Submit squeue. specify via IORING_OP_READV. + * the buffer need to be set metadata according to LAM mode + */ +int handle_uring_sq(struct io_ring *ring, struct file_io *fi, unsigned long lam) +{ + int file_fd = fi->file_fd; + struct io_uring_queue *sring = &ring->sq_ring; + unsigned int index = 0, cur_block = 0, tail = 0, next_tail = 0; + struct io_uring_sqe *sqe; + + off_t remain = fi->file_sz; + int blocks = (int)(remain + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + while (remain) { + off_t bytes = remain; + void *buf; + + if (bytes > URING_BLOCK_SZ) + bytes = URING_BLOCK_SZ; + + fi->iovecs[cur_block].iov_len = bytes; + + if (posix_memalign(&buf, URING_BLOCK_SZ, URING_BLOCK_SZ)) + return 1; + + fi->iovecs[cur_block].iov_base = (void *)set_metadata((uint64_t)buf, lam); + remain -= bytes; + cur_block++; + } + + next_tail = *sring->tail; + tail = next_tail; + next_tail++; + + barrier(); + + index = tail & *ring->sq_ring.ring_mask; + + sqe = &ring->sq_ring.queue.sqes[index]; + sqe->fd = file_fd; + sqe->flags = 0; + sqe->opcode = IORING_OP_READV; + sqe->addr = (unsigned long)fi->iovecs; + sqe->len = blocks; + sqe->off = 0; + sqe->user_data = (uint64_t)fi; + + sring->array[index] = index; + tail = next_tail; + + if (*sring->tail != tail) { + *sring->tail = tail; + barrier(); + } + + if (sys_uring_enter(ring->ring_fd, 1, 1, IORING_ENTER_GETEVENTS) < 0) + return 1; + + return 0; +} + +/* + * Test LAM in async I/O and io_uring, read current binery through io_uring + * Set metadata in pointers to iovecs buffer. + */ +int do_uring(unsigned long lam) +{ + struct io_ring *ring; + struct file_io *fi; + struct stat st; + int ret = 1; + char path[PATH_MAX]; + + /* get current process path */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + return 1; + + int file_fd = open(path, O_RDONLY); + + if (file_fd < 0) + return 1; + + if (fstat(file_fd, &st) < 0) + return 1; + + off_t file_sz = st.st_size; + + int blocks = (int)(file_sz + URING_BLOCK_SZ - 1) / URING_BLOCK_SZ; + + fi = malloc(sizeof(*fi) + sizeof(struct iovec) * blocks); + if (!fi) + return 1; + + fi->file_sz = file_sz; + fi->file_fd = file_fd; + + ring = malloc(sizeof(*ring)); + if (!ring) + return 1; + + memset(ring, 0, sizeof(struct io_ring)); + + if (setup_io_uring(ring)) + goto out; + + if (handle_uring_sq(ring, fi, lam)) + goto out; + + ret = handle_uring_cq(ring); + +out: + free(ring); + + for (int i = 0; i < blocks; i++) { + if (fi->iovecs[i].iov_base) { + uint64_t addr = ((uint64_t)fi->iovecs[i].iov_base); + + switch (lam) { + case LAM_U57_BITS: /* Clear bits 62:57 */ + addr = (addr & ~(0x3fULL << 57)); + break; + } + free((void *)addr); + fi->iovecs[i].iov_base = NULL; + } + } + + free(fi); + + return ret; +} + +int handle_uring(struct testcases *test) +{ + int ret = 0; + + if (test->later == 0 && test->lam != 0) + if (set_lam(test->lam) != 0) + return 1; + + if (sigsetjmp(segv_env, 1) == 0) { + signal(SIGSEGV, segv_handler); + ret = do_uring(test->lam); + } else { + ret = 2; + } + + return ret; +} + static int fork_test(struct testcases *test) { int ret, child_ret; @@ -340,6 +658,22 @@ static void run_test(struct testcases *test, int count) } } +static struct testcases uring_cases[] = { + { + .later = 0, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING: LAM_U57. Dereferencing pointer with metadata\n", + }, + { + .later = 1, + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_uring, + .msg = "URING:[Negative] Disable LAM. Dereferencing pointer with metadata.\n", + }, +}; + static struct testcases malloc_cases[] = { { .later = 0, @@ -410,7 +744,7 @@ static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall.\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n"); printf("\t-h: help\n"); } @@ -456,6 +790,9 @@ int main(int argc, char **argv) if (tests & FUNC_SYSCALL) run_test(syscall_cases, ARRAY_SIZE(syscall_cases)); + if (tests & FUNC_URING) + run_test(uring_cases, ARRAY_SIZE(uring_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Tue Dec 27 03:08:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36751 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1200605wrt; Mon, 26 Dec 2022 19:10:08 -0800 (PST) X-Google-Smtp-Source: AMrXdXtNh/2rvdfV4iPcJkMWvQz4MV+gi09WLtquX3bulWHDJAqH/sh2uPvee61AJU9A8jZn3BC0 X-Received: by 2002:a17:902:a510:b0:192:48d1:f06c with SMTP id s16-20020a170902a51000b0019248d1f06cmr19386203plq.35.1672110607693; Mon, 26 Dec 2022 19:10:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672110607; cv=none; d=google.com; s=arc-20160816; b=elrOVgUeLyBDX41wX4qEEUUfkbvI6DcB/Mwtr6kvJzxrPYK10TAb14OERQxfrRm33B EW0lLJWkeaUURMtvLzAY+KQjR+OuJpIC55YCsDzy8Eca9YP7LYqFDDzUEN04RTl34cfs 6U6Skb8ACzCIN1pBW6T5WdGsloW90Kr8OtFIIxav/lGx2xMs5Z25r0gdO8X5qVGU6lSh ed9ox06osxDXSwRftNB+QE+FoI7glbiW1tWCcRTc6zvtaGyd0PmlciPFuaLO/+KoVfmy xHhFyIp8ZBALX9slcf3ylG0sxq90QJdfNgs9Ag3xOnx8jBWWur3f+5kza/Rz+63Es4Mt +0Qg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YeALEwNO/xMU2Q7tN0wfJtLKXIf9sYx85DUZKQ3UlP0=; b=JMfL7C2q0LPuS/jaXk2ekXosyiSh5QXgZqFmtpLmdjAhqMQ88rkLUlnlBYjvH7T/ef vgKYWgKgQ+JAXP4hbEXKxcFGAnud+jtUY/88cpi44nSZ8CbF1MqNhZksU8s7+XnCv6xO jcC9J5Ksq07oCuPRZsGpqyG6p8XyBXzQRiSQWZPxF5yDlh+GQSYgs+e/zxkhqHocwjCm sTgsBfa1z1qItSBlvslh9m2nZLsIYS8tmrm7ZUJFzoQ0b69SCuZhgQ4fR6cDit3UUnhQ Z644ysrDc+wvRzmGGdrU8vRkjlScF1oxbc8HGMjvZK9Lm11MRsRuDbeKuZloZDPlypAK SIkg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=bLMS71KF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t11-20020a63444b000000b00491f8d42b05si13120187pgk.369.2022.12.26.19.09.55; Mon, 26 Dec 2022 19:10:07 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=bLMS71KF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230403AbiL0DJ1 (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229836AbiL0DIy (ORCPT ); Mon, 26 Dec 2022 22:08:54 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3868210FB for ; Mon, 26 Dec 2022 19:08:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110533; x=1703646533; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Xl8eyXVMqbT7yHbfHamtG0/Bq0s3VLeHRKgBe7wTJ3s=; b=bLMS71KFAuYLBOJt+/oOXtF22SRymx2icMUUqRIfvMP5iXFcbtG19DgC YAA5s4Ah5YAN1KSv1y/yZ1m5hvlPwzB6yHOycf8tpkCtN4fLeZNL7eyyp yCSkWL29pVTS3+rMAjry9V4cK2D787C50OO0NX4HK+RLdkNlkqSsws4yP 3eSw6iPzCJntBchwOLBBUd0d4Azl+FQ1n1Maje/Edcw/PgYuL3tyTsiVE GLgPESx2cpetMJdOO4qDckumnZ3dH7a/AwuXoKz3KEoyWwqNLRowCz+w8 WWusykPY74j/GgU86ZcNflZ7hn7jpwk/IzL+rF9oCz6bd5FZ3am1BU9zM Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="300994726" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="300994726" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:52 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="741605226" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="741605226" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:48 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 17E9B10BBBB; Tue, 27 Dec 2022 06:08:37 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv13 14/16] selftests/x86/lam: Add inherit test cases for linear-address masking Date: Tue, 27 Dec 2022 06:08:27 +0300 Message-Id: <20221227030829.12508-15-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335052650337941?= X-GMAIL-MSGID: =?utf-8?q?1753335052650337941?= From: Weihong Zhang LAM is enabled per-thread and gets inherited on fork(2)/clone(2). exec() reverts LAM status to the default disabled state. There are two test scenarios: - Fork test cases: These cases were used to test the inheritance of LAM for per-thread, Child process generated by fork() should inherit LAM feature from parent process, Child process can get the LAM mode same as parent process. - Execve test cases: Processes generated by execve() are different from processes generated by fork(), these processes revert LAM status to disabled status. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 125 +++++++++++++++++++++++++++++- 1 file changed, 121 insertions(+), 4 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 8ea1fcef4c9f..cfc9073c0262 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -37,8 +37,9 @@ #define FUNC_MMAP 0x4 #define FUNC_SYSCALL 0x8 #define FUNC_URING 0x10 +#define FUNC_INHERITE 0x20 -#define TEST_MASK 0x1f +#define TEST_MASK 0x3f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -174,6 +175,28 @@ static unsigned long get_default_tag_bits(void) return lam; } +/* + * Set tagged address and read back untag mask. + * check if the untag mask is expected. + */ +static int get_lam(void) +{ + uint64_t ptr = 0; + int ret = -1; + /* Get untagged mask */ + if (syscall(SYS_arch_prctl, ARCH_GET_UNTAG_MASK, &ptr) == -1) + return -1; + + /* Check mask returned is expected */ + if (ptr == ~(LAM_U57_MASK)) + ret = LAM_U57_BITS; + else if (ptr == -1ULL) + ret = LAM_NONE; + + + return ret; +} + /* According to LAM mode, set metadata in high bits */ static uint64_t set_metadata(uint64_t src, unsigned long lam) { @@ -581,7 +604,7 @@ int do_uring(unsigned long lam) switch (lam) { case LAM_U57_BITS: /* Clear bits 62:57 */ - addr = (addr & ~(0x3fULL << 57)); + addr = (addr & ~(LAM_U57_MASK)); break; } free((void *)addr); @@ -632,6 +655,72 @@ static int fork_test(struct testcases *test) return ret; } +static int handle_execve(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + ret = 1; + } else if (pid == 0) { + char path[PATH_MAX]; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + /* Get current binary's path and the binary was run by execve */ + if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) + exit(-1); + + /* run binary to get LAM mode and return to parent process */ + if (execlp(path, path, "-t 0x0", NULL) < 0) { + perror("error on exec"); + exit(-1); + } + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + if (ret != LAM_NONE) + return 1; + } + + return 0; +} + +static int handle_inheritance(struct testcases *test) +{ + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + /* Set LAM mode in parent process */ + if (set_lam(lam) != 0) + return 1; + + pid = fork(); + if (pid < 0) { + perror("Fork failed."); + return 1; + } else if (pid == 0) { + /* Set LAM mode in parent process */ + int child_lam = get_lam(); + + exit(child_lam); + } else { + wait(&child_ret); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + } + + return 0; +} + static void run_test(struct testcases *test, int count) { int i, ret = 0; @@ -740,11 +829,26 @@ static struct testcases mmap_cases[] = { }, }; +static struct testcases inheritance_cases[] = { + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_inheritance, + .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", + }, + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_execve, + .msg = "EXECVE: LAM_U57, child process should get disabled LAM mode\n", + }, +}; + static void cmd_help(void) { printf("usage: lam [-h] [-t test list]\n"); printf("\t-t test list: run tests specified in the test list, default:0x%x\n", TEST_MASK); - printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring.\n"); + printf("\t\t0x1:malloc; 0x2:max_bits; 0x4:mmap; 0x8:syscall; 0x10:io_uring; 0x20:inherit;\n"); printf("\t-h: help\n"); } @@ -764,7 +868,7 @@ int main(int argc, char **argv) switch (c) { case 't': tests = strtoul(optarg, NULL, 16); - if (!(tests & TEST_MASK)) { + if (tests && !(tests & TEST_MASK)) { ksft_print_msg("Invalid argument!\n"); return -1; } @@ -778,6 +882,16 @@ int main(int argc, char **argv) } } + /* + * When tests is 0, it is not a real test case; + * the option used by test case(execve) to check the lam mode in + * process generated by execve, the process read back lam mode and + * check with lam mode in parent process. + */ + if (!tests) + return (get_lam()); + + /* Run test cases */ if (tests & FUNC_MALLOC) run_test(malloc_cases, ARRAY_SIZE(malloc_cases)); @@ -793,6 +907,9 @@ int main(int argc, char **argv) if (tests & FUNC_URING) run_test(uring_cases, ARRAY_SIZE(uring_cases)); + if (tests & FUNC_INHERITE) + run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Tue Dec 27 03:08:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36758 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1203467wrt; Mon, 26 Dec 2022 19:20:57 -0800 (PST) X-Google-Smtp-Source: AMrXdXsFbZuosCGNOkebFksaZwKjZPp2F0NFY7OiNAmD3fx7AYa4aD3EnGcm5Rk73QZE2YvIzq9Z X-Received: by 2002:a05:6402:2b93:b0:47e:5b70:de79 with SMTP id fj19-20020a0564022b9300b0047e5b70de79mr17241871edb.17.1672111257341; Mon, 26 Dec 2022 19:20:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672111257; cv=none; d=google.com; s=arc-20160816; b=R5WXNUiaMXFX1dgxcA0IIW4VYTTewLjDLicFK1NcOdPsnN6CVDxNahpz0vAHQSFoVk PhzKoe7T609N7OLZTSvjKTXml6JKad/XOyZ5oC6ZWcGghaHJr0naemEFDZ7O5M4Mp+xh k9QITk16tsMBiDjeZ0mvnOTeH0m4V+x4fDICwnVp4MoAI+TaR7b4S88CihqVUny5pGsD GdkhzkOWFJKJX/QsSBxry72e6VJ6K6z7njfhqFTtfprAf+etzXV7J6Z6FQ1C7W9aU983 ogGigchDgAuPHWs+FMFfK3xARMsu7PFQn6j56mrRJclhHLG6eM2fnJTIUSkLey8FaQkL z0RA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=r2/52RjNjSK66BVgK2EdYH1PKm+PIu5EDbn8bh67oQM=; b=NGsQQLttwYsd1LaHxA4eE0BIb6f6huVEN/79/GWFs04ANanmQmMRykR2buCPo9MZ2S nEC4ei0Po5svrWN3GrZIVRcFjLSOxvwVgYUvSIwnE0UxVtQHwa7xuzIv61llkNkpuGLU 64vX60rHs6pIH1mdby1aPbk0OJ6ClaIj2WR23N7nzKwTDP81nPo30nuB68gVQHGy4kBB PV+NAYYYh4MJNfoqxXedUo40qblpO1mC4N23v4GMmvAcCUux7z/AMnlRq66C8V1SblWP lgx2sApxl+B5HNWRnrQVqLINyeoM5dtUoh23Im6BougpWMqMkUo7rmslUGhOwuEElAfW uxFw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=htE4SFLT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y7-20020a50eb87000000b0046dd0c29688si9548518edr.267.2022.12.26.19.20.32; Mon, 26 Dec 2022 19:20:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=htE4SFLT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230342AbiL0DKR (ORCPT + 99 others); Mon, 26 Dec 2022 22:10:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230292AbiL0DJW (ORCPT ); Mon, 26 Dec 2022 22:09:22 -0500 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 33FFF626B for ; Mon, 26 Dec 2022 19:08:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110539; x=1703646539; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tSraI7mF3xf9ymHhx9/Cb8Xz0YxRRv+Q6MzcUQxecxI=; b=htE4SFLTVCA3rQYnHVbHigaNB+vCHMJ7LhkSMUWkQUzIg5515cTdStkX IwlOUO3QlVd+g3FtsRVVivsvH272UhORg5LIWKYETpIqLjurQK8N8d936 RDEhhLZwvzqMp9N/8jONZ/JAozZWb/p9kcT81dnLjJJ2NpF/Ii8lAgDJM D4F65C491K342LzDQkK8Qke5dpB7CX3bXU+Qr7VguLL5kcOK9mkifdRMo ax/n4YUGuoUll5DJ+9WmGziNo9bOmZa2qsMOo9jK458mE7DsNYbJ6qHZW 79rctb0jvg0ScKTvrgyHoQdt2dspH9XR244bMT6Tlm9/oQXBYYW9vT/9t g==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="321869832" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="321869832" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:58 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="652917562" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="652917562" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:53 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 237DF10BBBC; Tue, 27 Dec 2022 06:08:37 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Weihong Zhang , "Kirill A . Shutemov" Subject: [PATCHv13 15/16] selftests/x86/lam: Add ARCH_FORCE_TAGGED_SVA test cases for linear-address masking Date: Tue, 27 Dec 2022 06:08:28 +0300 Message-Id: <20221227030829.12508-16-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335733680790083?= X-GMAIL-MSGID: =?utf-8?q?1753335733680790083?= From: Weihong Zhang By default do not allow to enable both LAM and use SVA in the same process. The new ARCH_FORCE_TAGGED_SVA arch_prctl() overrides the limitation. Add new test cases for the new arch_prctl: Before using ARCH_FORCE_TAGGED_SVA, should not allow to enable LAM/SVA coexisting. the test cases should be negative. The test depands on idxd driver and iommu. before test, need add "intel_iommu=on,sm_on" in kernel command line and insmod idxd driver. Signed-off-by: Weihong Zhang Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 237 +++++++++++++++++++++++++++++- 1 file changed, 235 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index cfc9073c0262..52a876a7ccb8 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -30,6 +30,7 @@ #define ARCH_GET_UNTAG_MASK 0x4001 #define ARCH_ENABLE_TAGGED_ADDR 0x4002 #define ARCH_GET_MAX_TAG_BITS 0x4003 +#define ARCH_FORCE_TAGGED_SVA 0x4004 /* Specified test function bits */ #define FUNC_MALLOC 0x1 @@ -38,8 +39,9 @@ #define FUNC_SYSCALL 0x8 #define FUNC_URING 0x10 #define FUNC_INHERITE 0x20 +#define FUNC_PASID 0x40 -#define TEST_MASK 0x3f +#define TEST_MASK 0x7f #define LOW_ADDR (0x1UL << 30) #define HIGH_ADDR (0x3UL << 48) @@ -55,11 +57,19 @@ #define URING_QUEUE_SZ 1 #define URING_BLOCK_SZ 2048 +/* Pasid test define */ +#define LAM_CMD_BIT 0x1 +#define PAS_CMD_BIT 0x2 +#define SVA_CMD_BIT 0x4 + +#define PAS_CMD(cmd1, cmd2, cmd3) (((cmd3) << 8) | ((cmd2) << 4) | ((cmd1) << 0)) + struct testcases { unsigned int later; int expected; /* 2: SIGSEGV Error; 1: other errors */ unsigned long lam; uint64_t addr; + uint64_t cmd; int (*test_func)(struct testcases *test); const char *msg; }; @@ -556,7 +566,7 @@ int do_uring(unsigned long lam) struct file_io *fi; struct stat st; int ret = 1; - char path[PATH_MAX]; + char path[PATH_MAX] = {0}; /* get current process path */ if (readlink("/proc/self/exe", path, PATH_MAX) <= 0) @@ -852,6 +862,226 @@ static void cmd_help(void) printf("\t-h: help\n"); } +/* Check for file existence */ +uint8_t file_Exists(const char *fileName) +{ + struct stat buffer; + + uint8_t ret = (stat(fileName, &buffer) == 0); + + return ret; +} + +/* Sysfs idxd files */ +const char *dsa_configs[] = { + "echo 1 > /sys/bus/dsa/devices/dsa0/wq0.1/group_id", + "echo shared > /sys/bus/dsa/devices/dsa0/wq0.1/mode", + "echo 10 > /sys/bus/dsa/devices/dsa0/wq0.1/priority", + "echo 16 > /sys/bus/dsa/devices/dsa0/wq0.1/size", + "echo 15 > /sys/bus/dsa/devices/dsa0/wq0.1/threshold", + "echo user > /sys/bus/dsa/devices/dsa0/wq0.1/type", + "echo MyApp1 > /sys/bus/dsa/devices/dsa0/wq0.1/name", + "echo 1 > /sys/bus/dsa/devices/dsa0/engine0.1/group_id", + "echo dsa0 > /sys/bus/dsa/drivers/idxd/bind", + /* bind files and devices, generated a device file in /dev */ + "echo wq0.1 > /sys/bus/dsa/drivers/user/bind", +}; + +/* DSA device file */ +const char *dsaDeviceFile = "/dev/dsa/wq0.1"; +/* file for io*/ +const char *dsaPasidEnable = "/sys/bus/dsa/devices/dsa0/pasid_enabled"; + +/* + * DSA depends on kernel cmdline "intel_iommu=on,sm_on" + * return pasid_enabled (0: disable 1:enable) + */ +int Check_DSA_Kernel_Setting(void) +{ + char command[256] = ""; + char buf[256] = ""; + char *ptr; + int rv = -1; + + snprintf(command, sizeof(command) - 1, "cat %s", dsaPasidEnable); + + FILE *cmd = popen(command, "r"); + + if (cmd) { + while (fgets(buf, sizeof(buf) - 1, cmd) != NULL); + + pclose(cmd); + rv = strtol(buf, &ptr, 16); + } + + return rv; +} + +/* + * Config DSA's sysfs files as shared DSA's WQ. + * Generated a device file /dev/dsa/wq0.1 + * Return: 0 OK; 1 Failed; 3 Skip(SVA disabled). + */ +int Dsa_Init_Sysfs(void) +{ + uint len = ARRAY_SIZE(dsa_configs); + const char **p = dsa_configs; + + if (file_Exists(dsaDeviceFile) == 1) + return 0; + + /* check the idxd driver */ + if (file_Exists(dsaPasidEnable) != 1) { + printf("Please make sure idxd driver was loaded\n"); + return 3; + } + + /* Check SVA feature */ + if (Check_DSA_Kernel_Setting() != 1) { + printf("Please enable SVA.(Add intel_iommu=on,sm_on in kernel cmdline)\n"); + return 3; + } + + /* Check the idxd device file on /dev/dsa/ */ + for (int i = 0; i < len; i++) { + if (system(p[i])) + return 1; + } + + /* After config, /dev/dsa/wq0.1 should be generated */ + return (file_Exists(dsaDeviceFile) != 1); +} + +/* + * Open DSA device file, triger API: iommu_sva_alloc_pasid + */ +void *allocate_dsa_pasid(void) +{ + int fd; + void *wq; + + fd = open(dsaDeviceFile, O_RDWR); + if (fd < 0) { + perror("open"); + return MAP_FAILED; + } + + wq = mmap(NULL, 0x1000, PROT_WRITE, + MAP_SHARED | MAP_POPULATE, fd, 0); + if (wq == MAP_FAILED) + perror("mmap"); + + return wq; +} + +int set_force_svm(void) +{ + int ret = 0; + + ret = syscall(SYS_arch_prctl, ARCH_FORCE_TAGGED_SVA); + + return ret; +} + +int handle_pasid(struct testcases *test) +{ + uint tmp = test->cmd; + uint runed = 0x0; + int ret = 0; + void *wq = NULL; + + ret = Dsa_Init_Sysfs(); + if (ret != 0) + return ret; + + for (int i = 0; i < 3; i++) { + int err = 0; + + if (tmp & 0x1) { + /* run set lam mode*/ + if ((runed & 0x1) == 0) { + err = set_lam(LAM_U57_BITS); + runed = runed | 0x1; + } else + err = 1; + } else if (tmp & 0x4) { + /* run force svm */ + if ((runed & 0x4) == 0) { + err = set_force_svm(); + runed = runed | 0x4; + } else + err = 1; + } else if (tmp & 0x2) { + /* run allocate pasid */ + if ((runed & 0x2) == 0) { + runed = runed | 0x2; + wq = allocate_dsa_pasid(); + if (wq == MAP_FAILED) + err = 1; + } else + err = 1; + } + + ret = ret + err; + if (ret > 0) + break; + + tmp = tmp >> 4; + } + + if (wq != MAP_FAILED && wq != NULL) + if (munmap(wq, 0x1000)) + printf("munmap failed %d\n", errno); + + if (runed != 0x7) + ret = 1; + + return (ret != 0); +} + +/* + * Pasid test depends on idxd and SVA, kernel should enable iommu and sm. + * command line(intel_iommu=on,sm_on) + */ +static struct testcases pasid_cases[] = { + { + .expected = 1, + .cmd = PAS_CMD(LAM_CMD_BIT, PAS_CMD_BIT, SVA_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: [Negative] Execute LAM, PASID, SVA in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(LAM_CMD_BIT, SVA_CMD_BIT, PAS_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute LAM, SVA, PASID in sequence\n", + }, + { + .expected = 1, + .cmd = PAS_CMD(PAS_CMD_BIT, LAM_CMD_BIT, SVA_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: [Negative] Execute PASID, LAM, SVA in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(PAS_CMD_BIT, SVA_CMD_BIT, LAM_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute PASID, SVA, LAM in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(SVA_CMD_BIT, LAM_CMD_BIT, PAS_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute SVA, LAM, PASID in sequence\n", + }, + { + .expected = 0, + .cmd = PAS_CMD(SVA_CMD_BIT, PAS_CMD_BIT, LAM_CMD_BIT), + .test_func = handle_pasid, + .msg = "PASID: Execute SVA, PASID, LAM in sequence\n", + }, +}; + int main(int argc, char **argv) { int c = 0; @@ -910,6 +1140,9 @@ int main(int argc, char **argv) if (tests & FUNC_INHERITE) run_test(inheritance_cases, ARRAY_SIZE(inheritance_cases)); + if (tests & FUNC_PASID) + run_test(pasid_cases, ARRAY_SIZE(pasid_cases)); + ksft_set_plan(tests_cnt); return ksft_exit_pass(); From patchwork Tue Dec 27 03:08:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Kirill A. Shutemov" X-Patchwork-Id: 36759 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp1203843wrt; Mon, 26 Dec 2022 19:22:35 -0800 (PST) X-Google-Smtp-Source: AMrXdXuZX1A5Gpbe7yGIuhvujgLbZBAQ9ms6itujevRjUq0YCTU2ua6/Djv2RHpLFe17aUfVBFwk X-Received: by 2002:a17:907:c007:b0:7ad:f165:70c2 with SMTP id ss7-20020a170907c00700b007adf16570c2mr24515024ejc.27.1672111355364; Mon, 26 Dec 2022 19:22:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1672111355; cv=none; d=google.com; s=arc-20160816; b=mNtf/9tIxs24Hr7UiCPfaiPvtVEBNDn34bHXVtfsRblxaiWtARcN6W9SqKcwP5avLc VjbX+wjNtGhxezGHrmOAGv/rMwlPNe7DgTa3Fojh2hW4JrNeUL75P0IzrHSsutU7OQit TZbsl71x9udSxHcpgt1f+s2rkdaJNfu3gWZMYlxatZTdu4WUzVumxpwV0UUU7uEzcEBZ EuoLo8lhAUC3BK1b0MmmTN306vMVCyBL38ENMHyFcyCQmQ1Pgf5a88luNv58+ty/Ypk/ GVcvNOKIokNqf4HDP6Zh8Xnm543trdvSKXjG4GSrm68SAekytMWFt8h7fEeX6y9rwQTe y1BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=T2ttVOyXwwAKLgSrL6vJ8hGSYqOJwHORhvEqiDZwz3E=; b=Szh4yIwnffkkVXkdfq/+fY+KBKaIdxwScmuPf2Vp4N0qbXPf4D69+dyhyDgA6Y/dAA sWieC/6uLqEWiWYYgarTLsjqScPOoggGRDSnyTsgYOQ4Jm99a52GaRA9m6iL+PdMvFXC 0PYwIbln5r2uQQ8WLkQ+S9FMhEdd8zuF1+U61vcHa6z/kQZCHnPYw/S4XPnRBC5sWs5C eQIzjXuWHLnZ2kwGUd3gC2t0scbikdUUHWNJaIrCENCb6mPkxQlVZpB6Z0XmcVav+BTy 3z1/PJYJhFklfHDkbT2f5wFwoAVTSyc3+tlQojMP6XFLGRhIqQyE4fxEKO+J6z1AKkFZ LCtA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=HOGWHEVx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g10-20020a056402320a00b004829afcb6eesi7790281eda.95.2022.12.26.19.22.11; Mon, 26 Dec 2022 19:22:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=HOGWHEVx; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231318AbiL0DJ5 (ORCPT + 99 others); Mon, 26 Dec 2022 22:09:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38668 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230090AbiL0DJI (ORCPT ); Mon, 26 Dec 2022 22:09:08 -0500 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 706EB60EA for ; Mon, 26 Dec 2022 19:08:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672110537; x=1703646537; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=W/1WmzpYknIw+qctuMHiI9dXZs4L/e/XIgoQK8Su5Ss=; b=HOGWHEVxQOTyrWUwzz1njDmCUAOPz/vPcNSyg/RkjdTzKAFf1qZf+L8h QSdMZ4OtaCcw+YAehr5f+OlSfGbdfJIelji67bwQQzxtPs6fwalwLeJ5M PYFPQT6TafwGEmhP0riyU2t2mJS/9Vi1fqtheJVo23+NsRQAVKGE5led/ i+AQL39StC2fSoXrOT/cTZ9dfpUakATwAtQDTeiREue6Q2nsRQGQ4Gw5P sAVXTzX8j/PNvRSWQ+kdGPjktuas8GOqqX4hXQ001Tn+cjI7pxFP1JTM+ AjOwWsuBsjOSANu3qSSDSBvOUFyJnOhF1Qiq44Ec1HosKKR1ph87G+MSV w==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="300994737" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="300994737" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:57 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="741605234" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="741605234" Received: from ppogotov-mobl.ger.corp.intel.com (HELO box.shutemov.name) ([10.252.62.152]) by fmsmga003-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 19:08:52 -0800 Received: by box.shutemov.name (Postfix, from userid 1000) id 2DC6210BBBD; Tue, 27 Dec 2022 06:08:37 +0300 (+03) From: "Kirill A. Shutemov" To: Dave Hansen , Andy Lutomirski , Peter Zijlstra Cc: x86@kernel.org, Kostya Serebryany , Andrey Ryabinin , Andrey Konovalov , Alexander Potapenko , Taras Madan , Dmitry Vyukov , "H . J . Lu" , Andi Kleen , Rick Edgecombe , Bharata B Rao , Jacob Pan , Ashok Raj , Linus Torvalds , linux-mm@kvack.org, linux-kernel@vger.kernel.org, "Kirill A. Shutemov" Subject: [PATCHv13 16/16] selftests/x86/lam: Add test cases for LAM vs thread creation Date: Tue, 27 Dec 2022 06:08:29 +0300 Message-Id: <20221227030829.12508-17-kirill.shutemov@linux.intel.com> X-Mailer: git-send-email 2.38.2 In-Reply-To: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> References: <20221227030829.12508-1-kirill.shutemov@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1753335836414823685?= X-GMAIL-MSGID: =?utf-8?q?1753335836414823685?= LAM enabling is only allowed when the process has single thread. LAM mode is inherited into child thread. Trying to enable LAM after spawning a thread has to fail. Signed-off-by: Kirill A. Shutemov --- tools/testing/selftests/x86/lam.c | 92 +++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/tools/testing/selftests/x86/lam.c b/tools/testing/selftests/x86/lam.c index 52a876a7ccb8..93e6089164b6 100644 --- a/tools/testing/selftests/x86/lam.c +++ b/tools/testing/selftests/x86/lam.c @@ -1,4 +1,5 @@ // SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE #include #include #include @@ -12,6 +13,7 @@ #include #include #include +#include #include #include @@ -50,6 +52,8 @@ #define PAGE_SIZE (4 << 10) +#define STACK_SIZE 65536 + #define barrier() ({ \ __asm__ __volatile__("" : : : "memory"); \ }) @@ -731,6 +735,75 @@ static int handle_inheritance(struct testcases *test) return 0; } +static int thread_fn_get_lam(void *arg) +{ + return get_lam(); +} + +static int thread_fn_set_lam(void *arg) +{ + struct testcases *test = arg; + + return set_lam(test->lam); +} + +static int handle_thread(struct testcases *test) +{ + char stack[STACK_SIZE]; + int ret, child_ret; + int lam = 0; + pid_t pid; + + /* Set LAM mode in parent process */ + if (!test->later) { + lam = test->lam; + if (set_lam(lam) != 0) + return 1; + } + + pid = clone(thread_fn_get_lam, stack + STACK_SIZE, + SIGCHLD | CLONE_FILES | CLONE_FS | CLONE_VM, NULL); + if (pid < 0) { + perror("Clone failed."); + return 1; + } + + waitpid(pid, &child_ret, 0); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + + if (test->later) { + if (set_lam(test->lam) != 0) + return 1; + } + + return 0; +} + +static int handle_thread_enable(struct testcases *test) +{ + char stack[STACK_SIZE]; + int ret, child_ret; + int lam = test->lam; + pid_t pid; + + pid = clone(thread_fn_set_lam, stack + STACK_SIZE, + SIGCHLD | CLONE_FILES | CLONE_FS | CLONE_VM, test); + if (pid < 0) { + perror("Clone failed."); + return 1; + } + + waitpid(pid, &child_ret, 0); + ret = WEXITSTATUS(child_ret); + + if (lam != ret) + return 1; + + return 0; +} static void run_test(struct testcases *test, int count) { int i, ret = 0; @@ -846,6 +919,25 @@ static struct testcases inheritance_cases[] = { .test_func = handle_inheritance, .msg = "FORK: LAM_U57, child process should get LAM mode same as parent\n", }, + { + .expected = 0, + .lam = LAM_U57_BITS, + .test_func = handle_thread, + .msg = "THREAD: LAM_U57, child thread should get LAM mode same as parent\n", + }, + { + .expected = 1, + .lam = LAM_U57_BITS, + .test_func = handle_thread_enable, + .msg = "THREAD: [NEGATIVE] Enable LAM in child.\n", + }, + { + .expected = 1, + .later = 1, + .lam = LAM_U57_BITS, + .test_func = handle_thread, + .msg = "THREAD: [NEGATIVE] Enable LAM in parent after thread created.\n", + }, { .expected = 0, .lam = LAM_U57_BITS,