From patchwork Fri Dec 15 09:12:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 179130 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp9149166dys; Fri, 15 Dec 2023 01:35:46 -0800 (PST) X-Google-Smtp-Source: AGHT+IGiPHeG+y7mZa/hWaOmwKOlSGTkewaUUXXBjaTDr0LvUCqWXCimuUMZrLeW3L9O5IklUqNk X-Received: by 2002:ae9:e64d:0:b0:77f:ae76:aa with SMTP id x13-20020ae9e64d000000b0077fae7600aamr1087751qkl.104.1702632946596; Fri, 15 Dec 2023 01:35:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702632946; cv=none; d=google.com; s=arc-20160816; b=PkWDzE0TZIx3liVYliD7eOxW99dxBFiU9M4ImnyLsxBm7NlrBEb/gTG4BF7I1TVnaS GrkQOddoZdv3zhFr1BtQ5btWMXb3EInV3w6s6K5/2tflJ2Hn0r76sHc7efpMEgxBFL1j milJGVeBWFNdRe4N0+dgkv8atDaCzMKulP5rGtkySxi6lWOLdzYIXEfgMG/ULreU+bin dRzJj8FJ9phlP078va0FvC2PM3amp5/GmXIM2r4pPFrcuqHPZWrvmUTtlPzwPdCXD4WU tHnfV81BuvMFzYMPUEF6yQagXdAc+Pc9qh5nRd95SZR9h4FS+eSnKu9XrRtcQ6kqDnXD w9RQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:list-unsubscribe:list-subscribe:list-id:precedence :references:subject:cc:to:from:date:user-agent:message-id :dkim-signature; bh=pbCkSkpL0VTDsMcIMxCWVNworlFDlzLYPEKR3Py08RA=; fh=vd912/km9qZuP2V7TBjqUWoA7VSy9ThzbPxJdH/Qh1E=; b=KKQeyyLW40HVwa3qt71w3bLf0TfEOE/mbS6AS9oQ+v7OtZnqhX17Fqe3We9KNVvndw 41a/JJjflnRB7oMs0Yuc5kRrRMw2LEI0g/N63LWeSYjJCjSeJyqw/+QyAfbDfFuFjJTB oH/6RtB+9lBGa8bHeTzBelHAcN1zzXBpZevYVFJWO70YV6nzMilVLpAFLeEN/JQmsdXJ xeVMwpmy1AezC85oevcGDY5QX76AYHqLQ4Ysm0YvnRwgcDmOYhuwdNnzncgJINuxQe6Z 3iJx6ZfAmqNQUytWY0u8Q8/RnOeRSyseSiICrRqDrtLaWaBMiHkIJXpLT1h2QEonastp W95Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=DJUGDSMs; spf=pass (google.com: domain of linux-kernel+bounces-685-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-685-ouuuleilei=gmail.com@vger.kernel.org" Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [2604:1380:45d1:ec00::1]) by mx.google.com with ESMTPS id dv4-20020a05620a1b8400b0077956418783si17814949qkb.652.2023.12.15.01.35.46 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Dec 2023 01:35:46 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-685-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) client-ip=2604:1380:45d1:ec00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=DJUGDSMs; spf=pass (google.com: domain of linux-kernel+bounces-685-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45d1:ec00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-685-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 518691C22811 for ; Fri, 15 Dec 2023 09:35:46 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id F29E124B5F; Fri, 15 Dec 2023 09:33:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="DJUGDSMs" X-Original-To: linux-kernel@vger.kernel.org Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9794171A6; Fri, 15 Dec 2023 09:33:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=pbCkSkpL0VTDsMcIMxCWVNworlFDlzLYPEKR3Py08RA=; b=DJUGDSMsYcY1FuLTMqpTLZuuD7 hLcMb7eOFpemVUB+Svi/65P+67gUSbeKh1GOBa7cr2FYt8/qyXeoCT/pi9uAhu9eJwsOkfFe9UNZA E+tGfdbdcJMPCvbJI7kQhXFFPwJ/zB9n+CEo8T0lj3ii//1wGhtPUl50MbNeh94rHVUSccRW0YYK0 U3qkPi7dJl+StxkZqBM6AttokeY+LDewP6v1hQKFnOmnEqTogDs7l+ALwC9Yv1WLjHiAofFrqYNBs J4UwojSpM8p8N5gRGZbTqyJFVtkENll8kEtpzjHrxv2v395vu0J14T0nFZ4f2uZJEZWyLZsr8TYXI 2mV7ppVg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1rE4ZD-00FSOT-HF; Fri, 15 Dec 2023 09:33:12 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id BF53730049D; Fri, 15 Dec 2023 10:33:11 +0100 (CET) Message-Id: <20231215092707.231038174@infradead.org> User-Agent: quilt/0.65 Date: Fri, 15 Dec 2023 10:12:17 +0100 From: Peter Zijlstra To: Alexei Starovoitov Cc: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, davem@davemloft.net, dsahern@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Arnd Bergmann , samitolvanen@google.com, keescook@chromium.org, nathan@kernel.org, ndesaulniers@google.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-arch@vger.kernel.org, llvm@lists.linux.dev, jpoimboe@kernel.org, joao@overdrivepizza.com, mark.rutland@arm.com, peterz@infradead.org Subject: [PATCH v3 1/7] cfi: Flip headers References: <20231215091216.135791411@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785340044388562512 X-GMAIL-MSGID: 1785340044388562512 Normal include order is that linux/foo.h should include asm/foo.h, CFI has it the wrong way around. Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Sami Tolvanen --- arch/riscv/include/asm/cfi.h | 3 ++- arch/riscv/kernel/cfi.c | 2 +- arch/x86/include/asm/cfi.h | 3 ++- arch/x86/kernel/cfi.c | 4 ++-- include/asm-generic/Kbuild | 1 + include/asm-generic/cfi.h | 5 +++++ include/linux/cfi.h | 1 + 7 files changed, 14 insertions(+), 5 deletions(-) --- a/arch/riscv/include/asm/cfi.h +++ b/arch/riscv/include/asm/cfi.h @@ -7,8 +7,9 @@ * * Copyright (C) 2023 Google LLC */ +#include -#include +struct pt_regs; #ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure(struct pt_regs *regs); --- a/arch/riscv/kernel/cfi.c +++ b/arch/riscv/kernel/cfi.c @@ -4,7 +4,7 @@ * * Copyright (C) 2023 Google LLC */ -#include +#include #include /* --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -7,8 +7,9 @@ * * Copyright (C) 2022 Google LLC */ +#include -#include +struct pt_regs; #ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure(struct pt_regs *regs); --- a/arch/x86/kernel/cfi.c +++ b/arch/x86/kernel/cfi.c @@ -4,10 +4,10 @@ * * Copyright (C) 2022 Google LLC */ -#include +#include +#include #include #include -#include /* * Returns the target address and the expected type when regs->ip points --- a/include/asm-generic/Kbuild +++ b/include/asm-generic/Kbuild @@ -11,6 +11,7 @@ mandatory-y += bitops.h mandatory-y += bug.h mandatory-y += bugs.h mandatory-y += cacheflush.h +mandatory-y += cfi.h mandatory-y += checksum.h mandatory-y += compat.h mandatory-y += current.h --- /dev/null +++ b/include/asm-generic/cfi.h @@ -0,0 +1,5 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ASM_GENERIC_CFI_H +#define __ASM_GENERIC_CFI_H + +#endif /* __ASM_GENERIC_CFI_H */ --- a/include/linux/cfi.h +++ b/include/linux/cfi.h @@ -9,6 +9,7 @@ #include #include +#include #ifdef CONFIG_CFI_CLANG enum bug_trap_type report_cfi_failure(struct pt_regs *regs, unsigned long addr, From patchwork Fri Dec 15 09:12:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 179131 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp9149304dys; Fri, 15 Dec 2023 01:36:08 -0800 (PST) X-Google-Smtp-Source: AGHT+IF7p4TReVxtOBDdPs1nS3z5MVmxpSslhUfXLnD8CHgbwKDObrjcw0HMhtpXoqq7H4NMDteA X-Received: by 2002:a05:6808:1385:b0:3b8:b063:8255 with SMTP id c5-20020a056808138500b003b8b0638255mr14662001oiw.87.1702632968568; Fri, 15 Dec 2023 01:36:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702632968; cv=none; d=google.com; s=arc-20160816; b=NdBbJ99mmWNtMIfoQgVUdyltQyz7yBzfuBbbA1L0XXjXwGQILLXn697Vb0ZIKBqsqY NyWiOAS02gnGnODHud/KhWzGwvGxQHZESZeOdZ9aLHl4VCGyCXFEjdNYPXh8iLz1lOKZ CS5y+RGO3ZxnRLtN4wN8f3OuSoQHyGgzhNx815zyecHNjWftZgmYeM+KRFWqtqbQmiYn ikKA4krDd/19pRq2j5DD72inr7GXj1MCiIQzMo8ss36zcGBW7UpgcUIAeQUAroyMl7fe +AUQYo+T3rkBvghg0n0S60BoVUYT0lCg4jHykEGyPE65zvP8ZPtFdo57vdJrZN8v+sT6 h2fg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:list-unsubscribe:list-subscribe:list-id:precedence :references:subject:cc:to:from:date:user-agent:message-id :dkim-signature; bh=O7bFFVYmt8/SO0BRBI22kgLL6Er6Cuo7T0F2P0zePs4=; fh=vd912/km9qZuP2V7TBjqUWoA7VSy9ThzbPxJdH/Qh1E=; b=ZPiwNEo5iAXqrGiyilBshXMCnVYEsDaaByAk4vlUOl98GZbz9DjTphkJ6mM1wF4jyH p1VIwx7cBHgOf6LAUD60UTydTm0uuLYC64Go4FIC0Iwn6gu3jW9SeECx2he2qEDFWlF4 0T/gOU3N8eFOdfFsv4uXygn0PjOzk1YFf0HnlplKkkCp+1DvuEHyJ4GpTY1VzKN2jHSz tDwRufCYTDtu9CRiaV9vC52uCzK9ASEpqqsnud7byHUwEa1LO8t/MrxHzMQOOrj9EUc3 itUKcvsvKxo+iQz4JSL5Gr2lQkwLmeTypCrPP6WHwFkbvZ2/2F9yr5XEntlzFLbG5nb7 4hkQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=cG62lmBe; spf=pass (google.com: domain of linux-kernel+bounces-686-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-686-ouuuleilei=gmail.com@vger.kernel.org" Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id fj40-20020a056a003a2800b006ce7bda5f18si12982373pfb.301.2023.12.15.01.36.08 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Dec 2023 01:36:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-686-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=cG62lmBe; spf=pass (google.com: domain of linux-kernel+bounces-686-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-686-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 50C3B2847E8 for ; Fri, 15 Dec 2023 09:36:08 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 89E022556A; Fri, 15 Dec 2023 09:33:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="cG62lmBe" X-Original-To: linux-kernel@vger.kernel.org Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2BB2B17986; Fri, 15 Dec 2023 09:33:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=O7bFFVYmt8/SO0BRBI22kgLL6Er6Cuo7T0F2P0zePs4=; b=cG62lmBegHlJs/hBEpaFyDjcaE 7zXfDyrph/ky8S3+o1xIyBuTuzFYLrJj6nkpeUDKPsrklImHVjuaIH+mOcRPnQ9GTUKJCipC1HqWs RvUR9j5OA0GJRd6mkO9PRJ5T8xJRQFnkzZnRu/sKf6hQJ6bde1pSBCg6/H+DG4iostV9rrHHcikZt kS3LrPS9jvvzTUpaApPYaiF7flQtfD1KLbbyi7d7mDyWB9pgeqtupfM1pmm02DJ39fX1xi21nYCZP vo2YEm3Stez0AP0uC7kZ/kS9fCd/vESx3kSfihlGYgQcmz8tfw7crZEXSaN75Nq5YWB0nZHVOm/v+ 4E95eKwg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1rE4ZD-00FSOU-HK; Fri, 15 Dec 2023 09:33:12 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id C5D8B3006CE; Fri, 15 Dec 2023 10:33:11 +0100 (CET) Message-Id: <20231215092707.345270396@infradead.org> User-Agent: quilt/0.65 Date: Fri, 15 Dec 2023 10:12:18 +0100 From: Peter Zijlstra To: Alexei Starovoitov Cc: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, davem@davemloft.net, dsahern@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Arnd Bergmann , samitolvanen@google.com, keescook@chromium.org, nathan@kernel.org, ndesaulniers@google.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-arch@vger.kernel.org, llvm@lists.linux.dev, jpoimboe@kernel.org, joao@overdrivepizza.com, mark.rutland@arm.com, peterz@infradead.org Subject: [PATCH v3 2/7] x86/cfi,bpf: Fix BPF JIT call References: <20231215091216.135791411@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785340067512713863 X-GMAIL-MSGID: 1785340067512713863 The current BPF call convention is __nocfi, except when it calls !JIT things, then it calls regular C functions. It so happens that with FineIBT the __nocfi and C calling conventions are incompatible. Specifically __nocfi will call at func+0, while FineIBT will have endbr-poison there, which is not a valid indirect target. Causing #CP. Notably this only triggers on IBT enabled hardware, which is probably why this hasn't been reported (also, most people will have JIT on anyway). Implement proper CFI prologues for the BPF JIT codegen and drop __nocfi for x86. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/cfi.h | 110 ++++++++++++++++++++++++++++++++++++++++++ arch/x86/kernel/alternative.c | 47 ++++++++++++++--- arch/x86/net/bpf_jit_comp.c | 82 +++++++++++++++++++++++++++++-- include/linux/bpf.h | 12 +++- include/linux/cfi.h | 7 ++ kernel/bpf/core.c | 25 +++++++++ 6 files changed, 269 insertions(+), 14 deletions(-) --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -9,15 +9,125 @@ */ #include +/* + * An overview of the various calling conventions... + * + * Traditional: + * + * foo: + * ... code here ... + * ret + * + * direct caller: + * call foo + * + * indirect caller: + * lea foo(%rip), %r11 + * ... + * call *%r11 + * + * + * IBT: + * + * foo: + * endbr64 + * ... code here ... + * ret + * + * direct caller: + * call foo / call foo+4 + * + * indirect caller: + * lea foo(%rip), %r11 + * ... + * call *%r11 + * + * + * kCFI: + * + * __cfi_foo: + * movl $0x12345678, %eax + * # 11 nops when CONFIG_CALL_PADDING + * foo: + * endbr64 # when IBT + * ... code here ... + * ret + * + * direct call: + * call foo # / call foo+4 when IBT + * + * indirect call: + * lea foo(%rip), %r11 + * ... + * movl $(-0x12345678), %r10d + * addl -4(%r11), %r10d # -15 when CONFIG_CALL_PADDING + * jz 1f + * ud2 + * 1:call *%r11 + * + * + * FineIBT (builds as kCFI + CALL_PADDING + IBT + RETPOLINE and runtime patches into): + * + * __cfi_foo: + * endbr64 + * subl 0x12345678, %r10d + * jz foo + * ud2 + * nop + * foo: + * osp nop3 # was endbr64 + * ... code here ... + * ret + * + * direct caller: + * call foo / call foo+4 + * + * indirect caller: + * lea foo(%rip), %r11 + * ... + * movl $0x12345678, %r10d + * subl $16, %r11 + * nop4 + * call *%r11 + * + */ +enum cfi_mode { + CFI_DEFAULT, /* FineIBT if hardware has IBT, otherwise kCFI */ + CFI_OFF, /* Taditional / IBT depending on .config */ + CFI_KCFI, /* Optionally CALL_PADDING, IBT, RETPOLINE */ + CFI_FINEIBT, /* see arch/x86/kernel/alternative.c */ +}; + +extern enum cfi_mode cfi_mode; + struct pt_regs; #ifdef CONFIG_CFI_CLANG enum bug_trap_type handle_cfi_failure(struct pt_regs *regs); +#define __bpfcall +extern u32 cfi_bpf_hash; + +static inline int cfi_get_offset(void) +{ + switch (cfi_mode) { + case CFI_FINEIBT: + return 16; + case CFI_KCFI: + if (IS_ENABLED(CONFIG_CALL_PADDING)) + return 16; + return 5; + default: + return 0; + } +} +#define cfi_get_offset cfi_get_offset + #else static inline enum bug_trap_type handle_cfi_failure(struct pt_regs *regs) { return BUG_TRAP_TYPE_NONE; } +#define cfi_bpf_hash 0U #endif /* CONFIG_CFI_CLANG */ #endif /* _ASM_X86_CFI_H */ --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -30,6 +30,7 @@ #include #include #include +#include int __read_mostly alternatives_patched; @@ -832,15 +833,43 @@ void __init_or_module apply_seal_endbr(s #endif /* CONFIG_X86_KERNEL_IBT */ #ifdef CONFIG_FINEIBT +#define __CFI_DEFAULT CFI_DEFAULT +#elif defined(CONFIG_CFI_CLANG) +#define __CFI_DEFAULT CFI_KCFI +#else +#define __CFI_DEFAULT CFI_OFF +#endif -enum cfi_mode { - CFI_DEFAULT, - CFI_OFF, - CFI_KCFI, - CFI_FINEIBT, -}; +enum cfi_mode cfi_mode __ro_after_init = __CFI_DEFAULT; + +#ifdef CONFIG_CFI_CLANG +struct bpf_insn; + +/* Must match bpf_func_t / DEFINE_BPF_PROG_RUN() */ +extern unsigned int __bpf_prog_runX(const void *ctx, + const struct bpf_insn *insn); + +/* + * Force a reference to the external symbol so the compiler generates + * __kcfi_typid. + */ +__ADDRESSABLE(__bpf_prog_runX); + +/* u32 __ro_after_init cfi_bpf_hash = __kcfi_typeid___bpf_prog_runX; */ +asm ( +" .pushsection .data..ro_after_init,\"aw\",@progbits \n" +" .type cfi_bpf_hash,@object \n" +" .globl cfi_bpf_hash \n" +" .p2align 2, 0x0 \n" +"cfi_bpf_hash: \n" +" .long __kcfi_typeid___bpf_prog_runX \n" +" .size cfi_bpf_hash, 4 \n" +" .popsection \n" +); +#endif + +#ifdef CONFIG_FINEIBT -static enum cfi_mode cfi_mode __ro_after_init = CFI_DEFAULT; static bool cfi_rand __ro_after_init = true; static u32 cfi_seed __ro_after_init; @@ -1149,8 +1178,10 @@ static void __apply_fineibt(s32 *start_r goto err; if (cfi_rand) { - if (builtin) + if (builtin) { cfi_seed = get_random_u32(); + cfi_bpf_hash = cfi_rehash(cfi_bpf_hash); + } ret = cfi_rand_preamble(start_cfi, end_cfi); if (ret) --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -17,6 +17,7 @@ #include #include #include +#include static bool all_callee_regs_used[4] = {true, true, true, true}; @@ -51,9 +52,11 @@ static u8 *emit_code(u8 *ptr, u32 bytes, do { EMIT4(b1, b2, b3, b4); EMIT(off, 4); } while (0) #ifdef CONFIG_X86_KERNEL_IBT -#define EMIT_ENDBR() EMIT(gen_endbr(), 4) +#define EMIT_ENDBR() EMIT(gen_endbr(), 4) +#define EMIT_ENDBR_POISON() EMIT(gen_endbr_poison(), 4) #else #define EMIT_ENDBR() +#define EMIT_ENDBR_POISON() #endif static bool is_imm8(int value) @@ -305,6 +308,69 @@ static void pop_callee_regs(u8 **pprog, } /* + * Emit the various CFI preambles, see asm/cfi.h and the comments about FineIBT + * in arch/x86/kernel/alternative.c + */ + +static void emit_fineibt(u8 **pprog) +{ + u8 *prog = *pprog; + + EMIT_ENDBR(); + EMIT3_off32(0x41, 0x81, 0xea, cfi_bpf_hash); /* subl $hash, %r10d */ + EMIT2(0x74, 0x07); /* jz.d8 +7 */ + EMIT2(0x0f, 0x0b); /* ud2 */ + EMIT1(0x90); /* nop */ + EMIT_ENDBR_POISON(); + + *pprog = prog; +} + +static void emit_kcfi(u8 **pprog) +{ + u8 *prog = *pprog; + + EMIT1_off32(0xb8, cfi_bpf_hash); /* movl $hash, %eax */ +#ifdef CONFIG_CALL_PADDING + EMIT1(0x90); + EMIT1(0x90); + EMIT1(0x90); + EMIT1(0x90); + EMIT1(0x90); + EMIT1(0x90); + EMIT1(0x90); + EMIT1(0x90); + EMIT1(0x90); + EMIT1(0x90); + EMIT1(0x90); +#endif + EMIT_ENDBR(); + + *pprog = prog; +} + +static void emit_cfi(u8 **pprog) +{ + u8 *prog = *pprog; + + switch (cfi_mode) { + case CFI_FINEIBT: + emit_fineibt(&prog); + break; + + case CFI_KCFI: + emit_kcfi(&prog); + break; + + default: + EMIT_ENDBR(); + break; + } + + *pprog = prog; +} + +/* * Emit x86-64 prologue code for BPF program. * bpf_tail_call helper will skip the first X86_TAIL_CALL_OFFSET bytes * while jumping to another program @@ -315,10 +381,10 @@ static void emit_prologue(u8 **pprog, u3 { u8 *prog = *pprog; + emit_cfi(&prog); /* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later */ - EMIT_ENDBR(); memcpy(prog, x86_nops[5], X86_PATCH_SIZE); prog += X86_PATCH_SIZE; if (!ebpf_from_cbpf) { @@ -3013,9 +3079,16 @@ struct bpf_prog *bpf_int_jit_compile(str jit_data->header = header; jit_data->rw_header = rw_header; } - prog->bpf_func = (void *)image; + /* + * ctx.prog_offset is used when CFI preambles put code *before* + * the function. See emit_cfi(). For FineIBT specifically this code + * can also be executed and bpf_prog_kallsyms_add() will + * generate an additional symbol to cover this, hence also + * decrement proglen. + */ + prog->bpf_func = (void *)image + cfi_get_offset(); prog->jited = 1; - prog->jited_len = proglen; + prog->jited_len = proglen - cfi_get_offset(); } else { prog = orig_prog; } @@ -3070,6 +3143,7 @@ void bpf_jit_free(struct bpf_prog *prog) kvfree(jit_data->addrs); kfree(jit_data); } + prog->bpf_func = (void *)prog->bpf_func - cfi_get_offset(); hdr = bpf_jit_binary_pack_hdr(prog); bpf_jit_binary_pack_free(hdr, NULL); WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog)); --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -29,6 +29,7 @@ #include #include #include +#include struct bpf_verifier_env; struct bpf_verifier_log; @@ -1211,7 +1212,11 @@ struct bpf_dispatcher { #endif }; -static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func( +#ifndef __bpfcall +#define __bpfcall __nocfi +#endif + +static __always_inline __bpfcall unsigned int bpf_dispatcher_nop_func( const void *ctx, const struct bpf_insn *insnsi, bpf_func_t bpf_func) @@ -1303,7 +1308,7 @@ int arch_prepare_bpf_dispatcher(void *im #define DEFINE_BPF_DISPATCHER(name) \ __BPF_DISPATCHER_SC(name); \ - noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ + noinline __bpfcall unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \ bpf_func_t bpf_func) \ @@ -1453,6 +1458,9 @@ struct bpf_prog_aux { struct bpf_kfunc_desc_tab *kfunc_tab; struct bpf_kfunc_btf_tab *kfunc_btf_tab; u32 size_poke_tab; +#ifdef CONFIG_FINEIBT + struct bpf_ksym ksym_prefix; +#endif struct bpf_ksym ksym; const struct bpf_prog_ops *ops; struct bpf_map **used_maps; --- a/include/linux/cfi.h +++ b/include/linux/cfi.h @@ -11,6 +11,13 @@ #include #include +#ifndef cfi_get_offset +static inline int cfi_get_offset(void) +{ + return 0; +} +#endif + #ifdef CONFIG_CFI_CLANG enum bug_trap_type report_cfi_failure(struct pt_regs *regs, unsigned long addr, unsigned long *target, u32 type); --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -121,6 +121,9 @@ struct bpf_prog *bpf_prog_alloc_no_stats #endif INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode); +#ifdef CONFIG_FINEIBT + INIT_LIST_HEAD_RCU(&fp->aux->ksym_prefix.lnode); +#endif mutex_init(&fp->aux->used_maps_mutex); mutex_init(&fp->aux->dst_mutex); @@ -683,6 +686,23 @@ void bpf_prog_kallsyms_add(struct bpf_pr fp->aux->ksym.prog = true; bpf_ksym_add(&fp->aux->ksym); + +#ifdef CONFIG_FINEIBT + /* + * When FineIBT, code in the __cfi_foo() symbols can get executed + * and hence unwinder needs help. + */ + if (cfi_mode != CFI_FINEIBT) + return; + + snprintf(fp->aux->ksym_prefix.name, KSYM_NAME_LEN, + "__cfi_%s", fp->aux->ksym.name); + + fp->aux->ksym_prefix.start = (unsigned long) fp->bpf_func - 16; + fp->aux->ksym_prefix.end = (unsigned long) fp->bpf_func; + + bpf_ksym_add(&fp->aux->ksym_prefix); +#endif } void bpf_prog_kallsyms_del(struct bpf_prog *fp) @@ -691,6 +711,11 @@ void bpf_prog_kallsyms_del(struct bpf_pr return; bpf_ksym_del(&fp->aux->ksym); +#ifdef CONFIG_FINEIBT + if (cfi_mode != CFI_FINEIBT) + return; + bpf_ksym_del(&fp->aux->ksym_prefix); +#endif } static struct bpf_ksym *bpf_ksym_find(unsigned long addr) From patchwork Fri Dec 15 09:12:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 179132 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp9149361dys; Fri, 15 Dec 2023 01:36:15 -0800 (PST) X-Google-Smtp-Source: AGHT+IGHPJysDSd30T6I6Pe7IHegEJ78TFUqFG4vYv4JZGQIiCJbYAooiq5L9UrB/RH3odM9VL68 X-Received: by 2002:a05:6a00:21ca:b0:6cd:cd77:afae with SMTP id t10-20020a056a0021ca00b006cdcd77afaemr14151112pfj.3.1702632975373; Fri, 15 Dec 2023 01:36:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702632975; cv=none; d=google.com; s=arc-20160816; b=BgrMSVTNiiqgfSvZJtGbY7is40/wnhvEuEJx1rB9mm5MTDl2HkIu7Us0pTXnlhWL5W TnYI9MOV9BsPosJ3NCEVnj9chpqBArSyX035uZ3K7uYFb3UVa6bL8q7dYN3d4rHLUyz2 zTk/PsVYJqMatDS8bAKPz/Tyqb74xmW956Z4FVvqzlPJ3Ns3DwnbYDOKjLIWAVxT4Xaa 05Nd9lZxx3bnYtG8XtRWTnJtB6Yz8tz1xCj3djkBHmUWdIFs+A/bMNVy97igh8K9PD3E jGSC+q+ewoma+7VEsxVRC9tj5ILh7dK56c8N1dGsKjKJUPwHuTlln2C3sXViI2r5yb0J E7bQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:list-unsubscribe:list-subscribe:list-id:precedence :references:subject:cc:to:from:date:user-agent:message-id :dkim-signature; bh=z79WJTlUkpGdsrYGZxRofGTWpjl5gntweNg1HQWKrSY=; fh=vd912/km9qZuP2V7TBjqUWoA7VSy9ThzbPxJdH/Qh1E=; b=RoVsTG421ncUkMJ7B1fftw9m3jGjPjXpbuPl+MuQIeuGMla8kJetfngsnuAuYHF+Rh ASArDHdx/zoJMPfcz0pIVHBwUmBn7Fbl4+RNTxjtzE2UMTYGUsb6WXWoNmDuIU7ngF3d b/raYgUbwLmJh0EglX340CEJPbT4q0x4rxTkm9mPxbZP+lIhErXERCy+hJMUXhtbNVjs 0/RB3PuudoJKpx+qzw6PKENwNXorGBy3LRi1FryEe+gva17aXmZ3ghqwwF4OnH0Q14QU SJddosNBHOCN9nI4wR59aIhp8C3keUS518+PsCJguJgEsUsbC3jaTD9xF2mNAWpd9Dsx +VzQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=BKTY9C+Y; spf=pass (google.com: domain of linux-kernel+bounces-687-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-687-ouuuleilei=gmail.com@vger.kernel.org" Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [139.178.88.99]) by mx.google.com with ESMTPS id t5-20020a62d145000000b006ce72ca697asi12774461pfl.251.2023.12.15.01.36.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Dec 2023 01:36:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-687-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) client-ip=139.178.88.99; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=BKTY9C+Y; spf=pass (google.com: domain of linux-kernel+bounces-687-ouuuleilei=gmail.com@vger.kernel.org designates 139.178.88.99 as permitted sender) smtp.mailfrom="linux-kernel+bounces-687-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 2267928495F for ; Fri, 15 Dec 2023 09:36:15 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id B1F8C25753; Fri, 15 Dec 2023 09:33:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="BKTY9C+Y" X-Original-To: linux-kernel@vger.kernel.org Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C97E6171BB; Fri, 15 Dec 2023 09:33:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=z79WJTlUkpGdsrYGZxRofGTWpjl5gntweNg1HQWKrSY=; b=BKTY9C+YqD87P54owc5F7b/Qn0 UQ7eqYLs8JXjLvGcrgMXgzA1mB5IRjRx7oambwG+fguM5VRL8jgy8NuT/pVOaMbNQ28W9Zsu5gXkv GxAlZtOMKilcoqkIzW1weuxThEKpLS2idiB9L7vznPnbIwA3laTCWDkli8FoBfPoV/NLMoR0yRid0 oXnsQTNdNPK90SrKYQaH1kXrZyqo9Zj8IrtQH2KoOCitrlUzD9Tb6wpg6kkOWUGgdJqarNsDQkrL7 37FbfJe680BaRWDccu5O5vQRHIK1Bd2TsCV1DKwJ7UDzae8DkOkZQJumJ6D7ZYt0HW05qAWNdOz1H yHv/QUfg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1rE4ZD-00FSOV-HC; Fri, 15 Dec 2023 09:33:12 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id CB29B3006DD; Fri, 15 Dec 2023 10:33:11 +0100 (CET) Message-Id: <20231215092707.451956710@infradead.org> User-Agent: quilt/0.65 Date: Fri, 15 Dec 2023 10:12:19 +0100 From: Peter Zijlstra To: Alexei Starovoitov Cc: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, davem@davemloft.net, dsahern@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Arnd Bergmann , samitolvanen@google.com, keescook@chromium.org, nathan@kernel.org, ndesaulniers@google.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-arch@vger.kernel.org, llvm@lists.linux.dev, jpoimboe@kernel.org, joao@overdrivepizza.com, mark.rutland@arm.com, peterz@infradead.org Subject: [PATCH v3 3/7] x86/cfi,bpf: Fix bpf_callback_t CFI References: <20231215091216.135791411@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785340074570732760 X-GMAIL-MSGID: 1785340074570732760 Where the main BPF program is expected to match bpf_func_t, sub-programs are expected to match bpf_callback_t. This fixes things like: tools/testing/selftests/bpf/progs/bloom_filter_bench.c: bpf_for_each_map_elem(&array_map, bloom_callback, &data, 0); Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/cfi.h | 2 ++ arch/x86/kernel/alternative.c | 18 ++++++++++++++++++ arch/x86/net/bpf_jit_comp.c | 18 ++++++++++-------- 3 files changed, 30 insertions(+), 8 deletions(-) --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -106,6 +106,7 @@ struct pt_regs; enum bug_trap_type handle_cfi_failure(struct pt_regs *regs); #define __bpfcall extern u32 cfi_bpf_hash; +extern u32 cfi_bpf_subprog_hash; static inline int cfi_get_offset(void) { @@ -128,6 +129,7 @@ static inline enum bug_trap_type handle_ return BUG_TRAP_TYPE_NONE; } #define cfi_bpf_hash 0U +#define cfi_bpf_subprog_hash 0U #endif /* CONFIG_CFI_CLANG */ #endif /* _ASM_X86_CFI_H */ --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -866,6 +866,23 @@ asm ( " .size cfi_bpf_hash, 4 \n" " .popsection \n" ); + +/* Must match bpf_callback_t */ +extern u64 __bpf_callback_fn(u64, u64, u64, u64, u64); + +__ADDRESSABLE(__bpf_callback_fn); + +/* u32 __ro_after_init cfi_bpf_subprog_hash = __kcfi_typeid___bpf_callback_fn; */ +asm ( +" .pushsection .data..ro_after_init,\"aw\",@progbits \n" +" .type cfi_bpf_subprog_hash,@object \n" +" .globl cfi_bpf_subprog_hash \n" +" .p2align 2, 0x0 \n" +"cfi_bpf_subprog_hash: \n" +" .long __kcfi_typeid___bpf_callback_fn \n" +" .size cfi_bpf_subprog_hash, 4 \n" +" .popsection \n" +); #endif #ifdef CONFIG_FINEIBT @@ -1181,6 +1198,7 @@ static void __apply_fineibt(s32 *start_r if (builtin) { cfi_seed = get_random_u32(); cfi_bpf_hash = cfi_rehash(cfi_bpf_hash); + cfi_bpf_subprog_hash = cfi_rehash(cfi_bpf_subprog_hash); } ret = cfi_rand_preamble(start_cfi, end_cfi); --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -312,12 +312,13 @@ static void pop_callee_regs(u8 **pprog, * in arch/x86/kernel/alternative.c */ -static void emit_fineibt(u8 **pprog) +static void emit_fineibt(u8 **pprog, bool is_subprog) { + u32 hash = is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash; u8 *prog = *pprog; EMIT_ENDBR(); - EMIT3_off32(0x41, 0x81, 0xea, cfi_bpf_hash); /* subl $hash, %r10d */ + EMIT3_off32(0x41, 0x81, 0xea, hash); /* subl $hash, %r10d */ EMIT2(0x74, 0x07); /* jz.d8 +7 */ EMIT2(0x0f, 0x0b); /* ud2 */ EMIT1(0x90); /* nop */ @@ -326,11 +327,12 @@ static void emit_fineibt(u8 **pprog) *pprog = prog; } -static void emit_kcfi(u8 **pprog) +static void emit_kcfi(u8 **pprog, bool is_subprog) { + u32 hash = is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash; u8 *prog = *pprog; - EMIT1_off32(0xb8, cfi_bpf_hash); /* movl $hash, %eax */ + EMIT1_off32(0xb8, hash); /* movl $hash, %eax */ #ifdef CONFIG_CALL_PADDING EMIT1(0x90); EMIT1(0x90); @@ -349,17 +351,17 @@ static void emit_kcfi(u8 **pprog) *pprog = prog; } -static void emit_cfi(u8 **pprog) +static void emit_cfi(u8 **pprog, bool is_subprog) { u8 *prog = *pprog; switch (cfi_mode) { case CFI_FINEIBT: - emit_fineibt(&prog); + emit_fineibt(&prog, is_subprog); break; case CFI_KCFI: - emit_kcfi(&prog); + emit_kcfi(&prog, is_subprog); break; default: @@ -381,7 +383,7 @@ static void emit_prologue(u8 **pprog, u3 { u8 *prog = *pprog; - emit_cfi(&prog); + emit_cfi(&prog, is_subprog); /* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later */ From patchwork Fri Dec 15 09:12:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 179134 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp9149838dys; Fri, 15 Dec 2023 01:37:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IEjdK+FSiFiWoKy0kyEt1kkBqQ6umOaXK42PEwq0GQFWO8OdM+tmqlHx/NasrLi3WMHiBOs X-Received: by 2002:a17:906:2192:b0:a1d:da25:1ad5 with SMTP id 18-20020a170906219200b00a1dda251ad5mr3229613eju.162.1702633043223; Fri, 15 Dec 2023 01:37:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702633043; cv=none; d=google.com; s=arc-20160816; b=JbHDSHiNhdgHqDC21ycU6A6/4jZByxErtc9tEPVvd8iPyiXgcwJ93mIiCMbpNYL+eR oKOYM/k9jx9oVxY1zMukQ2i7HHo6x8hx9LyOLqsc3Ln/0PBDxpmYThCkkezeADVPXDuX rO2UawhBW7m1+pkg+8ykMq7A2V8NeAaepnWS61GSZQjD5tJ0EPtI+j4CjiUy4O70DN6G 6cu4/t+zHZw1ZUy8ZlTi/+1M6/39RQKFjRo3rf7J5D8M1MCtLEy3PVTLfQjaE3fNGaVT fbvdcvzbNp8R5AhDxZI+mIdV1bTUt3Mv1KAU1/S5dEvfyMDd+/ZY+enOr7ynG13Hei7M bupw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:list-unsubscribe:list-subscribe:list-id:precedence :references:subject:cc:to:from:date:user-agent:message-id :dkim-signature; bh=MCUZcifVBbh4jTlbDrOicZht7hCfL66Ib8nwra146uo=; fh=vd912/km9qZuP2V7TBjqUWoA7VSy9ThzbPxJdH/Qh1E=; b=huUA3A73A9UEm7rfV2d2SLHauBkC8iHJo0h9GgOVKCNspf7hl/KxFzebaZxRuYIlIV LBWAxYl0PKniqolH4/WmASbyHlF2IcJpUhuMGyYsc22yULu49MfJgXAzhV2vnBUW5BNX NVA4kvkYFou7KqQdBuS7g5HcUZNi1ubZ1Yvy8F9g+hJtlMxan8LASmXceN8tAEazPmpq 1hymczZgtBfEJJyYRbCTVmGsu7673Fu08MO1P4V12ohdj4R0rEuNAowiLbnSQ5WXtKm5 L2WNRBITfxEBkyt+PLYq6UNIRqAdY5Odx3LbjFuqgLUPAuTL+kYh8uW1JPt9fRcOHxHD Zb3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=jFpUSyeM; spf=pass (google.com: domain of linux-kernel+bounces-691-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-691-ouuuleilei=gmail.com@vger.kernel.org" Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id j18-20020a170906831200b00a1da4f8e510si6978529ejx.446.2023.12.15.01.37.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Dec 2023 01:37:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-691-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=jFpUSyeM; spf=pass (google.com: domain of linux-kernel+bounces-691-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-691-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id AACC91F23683 for ; Fri, 15 Dec 2023 09:37:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 75A8118C24; Fri, 15 Dec 2023 09:34:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="jFpUSyeM" X-Original-To: linux-kernel@vger.kernel.org Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0252288DD; Fri, 15 Dec 2023 09:33:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=MCUZcifVBbh4jTlbDrOicZht7hCfL66Ib8nwra146uo=; b=jFpUSyeMwgLTWbV2PALOxZGQbm wiV8p/KxPLfG27bxfj82u2/yxJHqajdUlKgK6kcMqwSE8YJ9kT0sRDKZtDqIVofQrjpldY7cjj27M JKKxHWCIpnCSY4tRV8kZwsxVxwvxwsvaCLKynjvG+sUxGOvtLPO3M5jHmNkym6nXTzC6UPPZ9xifs PFnw/WrzeCidbA6/2PJrHyG5oX8tDm8AKwtPd+dt5w8tED75xicumn1hIxsB+ai7hGoVTGx+J6tzB 72FZozFY5yGacMn/luTcaXmCmUGLYVkHCj8qcEL0I83ycPRoEpfToUAK1oLfzuzjMfBYSAvCPl+Zd Gk/zCnIQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1rE4ZE-009rFy-0d; Fri, 15 Dec 2023 09:33:12 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D18BE300940; Fri, 15 Dec 2023 10:33:11 +0100 (CET) Message-Id: <20231215092707.566977112@infradead.org> User-Agent: quilt/0.65 Date: Fri, 15 Dec 2023 10:12:20 +0100 From: Peter Zijlstra To: Alexei Starovoitov Cc: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, davem@davemloft.net, dsahern@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Arnd Bergmann , samitolvanen@google.com, keescook@chromium.org, nathan@kernel.org, ndesaulniers@google.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-arch@vger.kernel.org, llvm@lists.linux.dev, jpoimboe@kernel.org, joao@overdrivepizza.com, mark.rutland@arm.com, peterz@infradead.org Subject: [PATCH v3 4/7] x86/cfi,bpf: Fix bpf_struct_ops CFI References: <20231215091216.135791411@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785340145686058063 X-GMAIL-MSGID: 1785340145686058063 BPF struct_ops uses __arch_prepare_bpf_trampoline() to write trampolines for indirect function calls. These tramplines much have matching CFI. In order to obtain the correct CFI hash for the various methods, add a matching structure that contains stub functions, the compiler will generate correct CFI which we can pilfer for the trampolines. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/cfi.h | 6 +++ arch/x86/kernel/alternative.c | 22 +++++++++++++ arch/x86/net/bpf_jit_comp.c | 66 +++++++++++++++++++++++++-------------- include/linux/bpf.h | 13 +++++++ kernel/bpf/bpf_struct_ops.c | 16 ++++----- net/bpf/bpf_dummy_struct_ops.c | 31 +++++++++++++++++- net/ipv4/bpf_tcp_ca.c | 69 +++++++++++++++++++++++++++++++++++++++++ 7 files changed, 191 insertions(+), 32 deletions(-) --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -123,6 +123,8 @@ static inline int cfi_get_offset(void) } #define cfi_get_offset cfi_get_offset +extern u32 cfi_get_func_hash(void *func); + #else static inline enum bug_trap_type handle_cfi_failure(struct pt_regs *regs) { @@ -130,6 +132,10 @@ static inline enum bug_trap_type handle_ } #define cfi_bpf_hash 0U #define cfi_bpf_subprog_hash 0U +static inline u32 cfi_get_func_hash(void *func) +{ + return 0; +} #endif /* CONFIG_CFI_CLANG */ #endif /* _ASM_X86_CFI_H */ --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -883,6 +883,28 @@ asm ( " .size cfi_bpf_subprog_hash, 4 \n" " .popsection \n" ); + +u32 cfi_get_func_hash(void *func) +{ + u32 hash; + + func -= cfi_get_offset(); + switch (cfi_mode) { + case CFI_FINEIBT: + func += 7; + break; + case CFI_KCFI: + func += 1; + break; + default: + return 0; + } + + if (get_kernel_nofault(hash, func)) + return 0; + + return hash; +} #endif #ifdef CONFIG_FINEIBT --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -312,9 +312,8 @@ static void pop_callee_regs(u8 **pprog, * in arch/x86/kernel/alternative.c */ -static void emit_fineibt(u8 **pprog, bool is_subprog) +static void emit_fineibt(u8 **pprog, u32 hash) { - u32 hash = is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash; u8 *prog = *pprog; EMIT_ENDBR(); @@ -327,9 +326,8 @@ static void emit_fineibt(u8 **pprog, boo *pprog = prog; } -static void emit_kcfi(u8 **pprog, bool is_subprog) +static void emit_kcfi(u8 **pprog, u32 hash) { - u32 hash = is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash; u8 *prog = *pprog; EMIT1_off32(0xb8, hash); /* movl $hash, %eax */ @@ -351,17 +349,17 @@ static void emit_kcfi(u8 **pprog, bool i *pprog = prog; } -static void emit_cfi(u8 **pprog, bool is_subprog) +static void emit_cfi(u8 **pprog, u32 hash) { u8 *prog = *pprog; switch (cfi_mode) { case CFI_FINEIBT: - emit_fineibt(&prog, is_subprog); + emit_fineibt(&prog, hash); break; case CFI_KCFI: - emit_kcfi(&prog, is_subprog); + emit_kcfi(&prog, hash); break; default: @@ -383,7 +381,7 @@ static void emit_prologue(u8 **pprog, u3 { u8 *prog = *pprog; - emit_cfi(&prog, is_subprog); + emit_cfi(&prog, is_subprog ? cfi_bpf_subprog_hash : cfi_bpf_hash); /* BPF trampoline can be made to work without these nops, * but let's waste 5 bytes for now and optimize later */ @@ -2510,10 +2508,19 @@ static int __arch_prepare_bpf_trampoline u8 *prog; bool save_ret; + /* + * F_INDIRECT is only compatible with F_RET_FENTRY_RET, it is + * explicitly incompatible with F_CALL_ORIG | F_SKIP_FRAME | F_IP_ARG + * because @func_addr. + */ + WARN_ON_ONCE((flags & BPF_TRAMP_F_INDIRECT) && + (flags & ~(BPF_TRAMP_F_INDIRECT | BPF_TRAMP_F_RET_FENTRY_RET))); + /* extra registers for struct arguments */ - for (i = 0; i < m->nr_args; i++) + for (i = 0; i < m->nr_args; i++) { if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG) nr_regs += (m->arg_size[i] + 7) / 8 - 1; + } /* x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6 * are passed through regs, the remains are through stack. @@ -2596,20 +2603,27 @@ static int __arch_prepare_bpf_trampoline prog = rw_image; - EMIT_ENDBR(); - /* - * This is the direct-call trampoline, as such it needs accounting - * for the __fentry__ call. - */ - x86_call_depth_emit_accounting(&prog, NULL); + if (flags & BPF_TRAMP_F_INDIRECT) { + /* + * Indirect call for bpf_struct_ops + */ + emit_cfi(&prog, cfi_get_func_hash(func_addr)); + } else { + /* + * Direct-call fentry stub, as such it needs accounting for the + * __fentry__ call. + */ + x86_call_depth_emit_accounting(&prog, NULL); + } EMIT1(0x55); /* push rbp */ EMIT3(0x48, 0x89, 0xE5); /* mov rbp, rsp */ - if (!is_imm8(stack_size)) + if (!is_imm8(stack_size)) { /* sub rsp, stack_size */ EMIT3_off32(0x48, 0x81, 0xEC, stack_size); - else + } else { /* sub rsp, stack_size */ EMIT4(0x48, 0x83, 0xEC, stack_size); + } if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) EMIT1(0x50); /* push rax */ /* mov QWORD PTR [rbp - rbx_off], rbx */ @@ -2643,10 +2657,11 @@ static int __arch_prepare_bpf_trampoline } } - if (fentry->nr_links) + if (fentry->nr_links) { if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image)) return -EINVAL; + } if (fmod_ret->nr_links) { branches = kcalloc(fmod_ret->nr_links, sizeof(u8 *), @@ -2665,11 +2680,12 @@ static int __arch_prepare_bpf_trampoline restore_regs(m, &prog, regs_off); save_args(m, &prog, arg_stack_off, true); - if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) + if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) { /* Before calling the original function, restore the * tail_call_cnt from stack to rax. */ RESTORE_TAIL_CALL_CNT(stack_size); + } if (flags & BPF_TRAMP_F_ORIG_STACK) { emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, 8); @@ -2698,17 +2714,19 @@ static int __arch_prepare_bpf_trampoline /* Update the branches saved in invoke_bpf_mod_ret with the * aligned address of do_fexit. */ - for (i = 0; i < fmod_ret->nr_links; i++) + for (i = 0; i < fmod_ret->nr_links; i++) { emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image), image + (branches[i] - (u8 *)rw_image), X86_JNE); + } } - if (fexit->nr_links) + if (fexit->nr_links) { if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, false, image, rw_image)) { ret = -EINVAL; goto cleanup; } + } if (flags & BPF_TRAMP_F_RESTORE_REGS) restore_regs(m, &prog, regs_off); @@ -2725,11 +2743,12 @@ static int __arch_prepare_bpf_trampoline ret = -EINVAL; goto cleanup; } - } else if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) + } else if (flags & BPF_TRAMP_F_TAIL_CALL_CTX) { /* Before running the original function, restore the * tail_call_cnt from stack to rax. */ RESTORE_TAIL_CALL_CNT(stack_size); + } /* restore return value of orig_call or fentry prog back into RAX */ if (save_ret) @@ -2737,9 +2756,10 @@ static int __arch_prepare_bpf_trampoline emit_ldx(&prog, BPF_DW, BPF_REG_6, BPF_REG_FP, -rbx_off); EMIT1(0xC9); /* leave */ - if (flags & BPF_TRAMP_F_SKIP_FRAME) + if (flags & BPF_TRAMP_F_SKIP_FRAME) { /* skip our return address and return to parent */ EMIT4(0x48, 0x83, 0xC4, 8); /* add rsp, 8 */ + } emit_return(&prog, image + (prog - (u8 *)rw_image)); /* Make sure the trampoline generation logic doesn't overflow */ if (WARN_ON_ONCE(prog > (u8 *)rw_image_end - BPF_INSN_SAFETY)) { --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1060,6 +1060,17 @@ struct btf_func_model { */ #define BPF_TRAMP_F_TAIL_CALL_CTX BIT(7) +/* + * Indicate the trampoline should be suitable to receive indirect calls; + * without this indirectly calling the generated code can result in #UD/#CP, + * depending on the CFI options. + * + * Used by bpf_struct_ops. + * + * Incompatible with FENTRY usage, overloads @func_addr argument. + */ +#define BPF_TRAMP_F_INDIRECT BIT(8) + /* Each call __bpf_prog_enter + call bpf_func + call __bpf_prog_exit is ~50 * bytes on x86. */ @@ -1695,6 +1706,7 @@ struct bpf_struct_ops { struct btf_func_model func_models[BPF_STRUCT_OPS_MAX_NR_MEMBERS]; u32 type_id; u32 value_id; + void *cfi_stubs; }; #if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL) @@ -1708,6 +1720,7 @@ int bpf_struct_ops_map_sys_lookup_elem(s int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, struct bpf_tramp_link *link, const struct btf_func_model *model, + void *stub_func, void *image, void *image_end); static inline bool bpf_try_module_get(const void *data, struct module *owner) { --- a/kernel/bpf/bpf_struct_ops.c +++ b/kernel/bpf/bpf_struct_ops.c @@ -352,17 +352,16 @@ const struct bpf_link_ops bpf_struct_ops int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, struct bpf_tramp_link *link, const struct btf_func_model *model, - void *image, void *image_end) + void *stub_func, void *image, void *image_end) { - u32 flags; + u32 flags = BPF_TRAMP_F_INDIRECT; int size; tlinks[BPF_TRAMP_FENTRY].links[0] = link; tlinks[BPF_TRAMP_FENTRY].nr_links = 1; - /* BPF_TRAMP_F_RET_FENTRY_RET is only used by bpf_struct_ops, - * and it must be used alone. - */ - flags = model->ret_size > 0 ? BPF_TRAMP_F_RET_FENTRY_RET : 0; + + if (model->ret_size > 0) + flags |= BPF_TRAMP_F_RET_FENTRY_RET; size = arch_bpf_trampoline_size(model, flags, tlinks, NULL); if (size < 0) @@ -370,7 +369,7 @@ int bpf_struct_ops_prepare_trampoline(st if (size > (unsigned long)image_end - (unsigned long)image) return -E2BIG; return arch_prepare_bpf_trampoline(NULL, image, image_end, - model, flags, tlinks, NULL); + model, flags, tlinks, stub_func); } static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, @@ -504,11 +503,12 @@ static long bpf_struct_ops_map_update_el err = bpf_struct_ops_prepare_trampoline(tlinks, link, &st_ops->func_models[i], + *(void **)(st_ops->cfi_stubs + moff), image, image_end); if (err < 0) goto reset_unlock; - *(void **)(kdata + moff) = image; + *(void **)(kdata + moff) = image + cfi_get_offset(); image += err; /* put prog_id to udata */ --- a/net/bpf/bpf_dummy_struct_ops.c +++ b/net/bpf/bpf_dummy_struct_ops.c @@ -12,6 +12,11 @@ extern struct bpf_struct_ops bpf_bpf_dum /* A common type for test_N with return value in bpf_dummy_ops */ typedef int (*dummy_ops_test_ret_fn)(struct bpf_dummy_ops_state *state, ...); +static int dummy_ops_test_ret_function(struct bpf_dummy_ops_state *state, ...) +{ + return 0; +} + struct bpf_dummy_ops_test_args { u64 args[MAX_BPF_FUNC_ARGS]; struct bpf_dummy_ops_state state; @@ -62,7 +67,7 @@ static int dummy_ops_copy_args(struct bp static int dummy_ops_call_op(void *image, struct bpf_dummy_ops_test_args *args) { - dummy_ops_test_ret_fn test = (void *)image; + dummy_ops_test_ret_fn test = (void *)image + cfi_get_offset(); struct bpf_dummy_ops_state *state = NULL; /* state needs to be NULL if args[0] is 0 */ @@ -119,6 +124,7 @@ int bpf_struct_ops_test_run(struct bpf_p op_idx = prog->expected_attach_type; err = bpf_struct_ops_prepare_trampoline(tlinks, link, &st_ops->func_models[op_idx], + &dummy_ops_test_ret_function, image, image + PAGE_SIZE); if (err < 0) goto out; @@ -219,6 +225,28 @@ static void bpf_dummy_unreg(void *kdata) { } +static int bpf_dummy_test_1(struct bpf_dummy_ops_state *cb) +{ + return 0; +} + +static int bpf_dummy_test_2(struct bpf_dummy_ops_state *cb, int a1, unsigned short a2, + char a3, unsigned long a4) +{ + return 0; +} + +static int bpf_dummy_test_sleepable(struct bpf_dummy_ops_state *cb) +{ + return 0; +} + +static struct bpf_dummy_ops __bpf_bpf_dummy_ops = { + .test_1 = bpf_dummy_test_1, + .test_2 = bpf_dummy_test_2, + .test_sleepable = bpf_dummy_test_sleepable, +}; + struct bpf_struct_ops bpf_bpf_dummy_ops = { .verifier_ops = &bpf_dummy_verifier_ops, .init = bpf_dummy_init, @@ -227,4 +255,5 @@ struct bpf_struct_ops bpf_bpf_dummy_ops .reg = bpf_dummy_reg, .unreg = bpf_dummy_unreg, .name = "bpf_dummy_ops", + .cfi_stubs = &__bpf_bpf_dummy_ops, }; --- a/net/ipv4/bpf_tcp_ca.c +++ b/net/ipv4/bpf_tcp_ca.c @@ -271,6 +271,74 @@ static int bpf_tcp_ca_validate(void *kda return tcp_validate_congestion_control(kdata); } +static u32 bpf_tcp_ca_ssthresh(struct sock *sk) +{ + return 0; +} + +static void bpf_tcp_ca_cong_avoid(struct sock *sk, u32 ack, u32 acked) +{ +} + +static void bpf_tcp_ca_set_state(struct sock *sk, u8 new_state) +{ +} + +static void bpf_tcp_ca_cwnd_event(struct sock *sk, enum tcp_ca_event ev) +{ +} + +static void bpf_tcp_ca_in_ack_event(struct sock *sk, u32 flags) +{ +} + +static void bpf_tcp_ca_pkts_acked(struct sock *sk, const struct ack_sample *sample) +{ +} + +static u32 bpf_tcp_ca_min_tso_segs(struct sock *sk) +{ + return 0; +} + +static void bpf_tcp_ca_cong_control(struct sock *sk, const struct rate_sample *rs) +{ +} + +static u32 bpf_tcp_ca_undo_cwnd(struct sock *sk) +{ + return 0; +} + +static u32 bpf_tcp_ca_sndbuf_expand(struct sock *sk) +{ + return 0; +} + +static void __bpf_tcp_ca_init(struct sock *sk) +{ +} + +static void __bpf_tcp_ca_release(struct sock *sk) +{ +} + +static struct tcp_congestion_ops __bpf_ops_tcp_congestion_ops = { + .ssthresh = bpf_tcp_ca_ssthresh, + .cong_avoid = bpf_tcp_ca_cong_avoid, + .set_state = bpf_tcp_ca_set_state, + .cwnd_event = bpf_tcp_ca_cwnd_event, + .in_ack_event = bpf_tcp_ca_in_ack_event, + .pkts_acked = bpf_tcp_ca_pkts_acked, + .min_tso_segs = bpf_tcp_ca_min_tso_segs, + .cong_control = bpf_tcp_ca_cong_control, + .undo_cwnd = bpf_tcp_ca_undo_cwnd, + .sndbuf_expand = bpf_tcp_ca_sndbuf_expand, + + .init = __bpf_tcp_ca_init, + .release = __bpf_tcp_ca_release, +}; + struct bpf_struct_ops bpf_tcp_congestion_ops = { .verifier_ops = &bpf_tcp_ca_verifier_ops, .reg = bpf_tcp_ca_reg, @@ -281,6 +349,7 @@ struct bpf_struct_ops bpf_tcp_congestion .init = bpf_tcp_ca_init, .validate = bpf_tcp_ca_validate, .name = "tcp_congestion_ops", + .cfi_stubs = &__bpf_ops_tcp_congestion_ops, }; static int __init bpf_tcp_ca_kfunc_init(void) From patchwork Fri Dec 15 09:12:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 179133 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp9149797dys; Fri, 15 Dec 2023 01:37:15 -0800 (PST) X-Google-Smtp-Source: AGHT+IHrJnqVpkpJY/mQuNzROc6Th1tdkmZZC7bTLlK2RO2CLbhI/7k69065tX6mUPtf2pIcyQF7 X-Received: by 2002:a17:90a:f013:b0:28b:3e7:9d81 with SMTP id bt19-20020a17090af01300b0028b03e79d81mr1440588pjb.95.1702633035492; Fri, 15 Dec 2023 01:37:15 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702633035; cv=none; d=google.com; s=arc-20160816; b=yZYooyXeiSGTI0b/n96Ugx3OcnzfXZ5drJNadCuB7K7CKDBiXm/g4ih4ZphwnBKhgv lJ+372XtFnnngpzIzUAzXD9pMAM6uiyrjS7KrlXPaFwZHtu4+e9I47plIrEJMpXxLVEg NfROtTaLqSv3tCmKabx6Z85P1m8ZC7oWt1Uokn/zNyZaCqNciOoWWQIDjqFqU2UxGB15 pSl/7We14Rd8gNh8ADtOFbrceT4ahSqxylQINiFWPUwc78qlVCAOV1HNJNT320NcW3Lr 4+Hemwox2xwnJy5lN6u4so1VzDpsWen0TCuu/uFzU+B4NVqRhFTcRFvv2Q3rQV3HTlUk KpPg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:list-unsubscribe:list-subscribe:list-id:precedence :references:subject:cc:to:from:date:user-agent:message-id :dkim-signature; bh=b4R8oBfA9zqLdMspEmf7BM9tcAtmYn2SA6ZbJ+T1P5c=; fh=vd912/km9qZuP2V7TBjqUWoA7VSy9ThzbPxJdH/Qh1E=; b=ZlGxpDwBRTqv3q7kFGgxIGv/KzNaC0bkYI2nQijwTTah+YIo1NgTd/COBTIVwOgQwb 4L42cfttAeRwgd4vG6MuXpk0bYlVVZsVF6K+NdGun0FloRNVYTp8MH+CBqdjWm9B9JTI ouyyHjnHpBbpRzzvNS9y6ROs9fd09BHEopkkabsEYF6nz6/bEOCgq6OZ/NizmaQm8hw2 +4NQRrILWdXGegmBef+MDXV20nGIkjwB+hXfETinFtu6Mpc6b2833J/dmEVUyny/MDTr 7NShJLdUsxgK7IwZFX3sPOOYbZDr5jFtdLD+6cTIDztOqn9Vovetep6xGSh//uqwXyjj qR1A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=Lq2PgWht; spf=pass (google.com: domain of linux-kernel+bounces-688-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-688-ouuuleilei=gmail.com@vger.kernel.org" Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id y7-20020a17090a86c700b0028672033a5csi1520026pjv.124.2023.12.15.01.37.15 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Dec 2023 01:37:15 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-688-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=Lq2PgWht; spf=pass (google.com: domain of linux-kernel+bounces-688-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-688-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id D7FB0B22C97 for ; Fri, 15 Dec 2023 09:36:29 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 302C32C681; Fri, 15 Dec 2023 09:33:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Lq2PgWht" X-Original-To: linux-kernel@vger.kernel.org Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 061F5179A2; Fri, 15 Dec 2023 09:33:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=b4R8oBfA9zqLdMspEmf7BM9tcAtmYn2SA6ZbJ+T1P5c=; b=Lq2PgWhtX3V8toHwd4gXthsfPV 51X7SJQC7Nru1W8AxzAo9FgQf8X7B//HhiVa4Wni0K2G+n5NSLp0J6KuWaVPHCg5LqaWnnTct6an7 VSn/94veX/IqLryss/BBYEDgrH3uHxiAng0tbAwUZR/2wWcS1xNJe4HBpFsWFWk2lHDRHwf+hSYIV jW3O6OYVrI2YIUtYDEzNZCpeDl/mhsRZv8X+fwN3HuvgHv0Zv4s2ZWXFlNfony2rirfUxjBaebt0e 2AMaCofhDa/dAdzVSTw7hG62RjkawKG+01OP4YeCDE2L2mGeW2/sgrNjJVf24DXfwrfjR9XaYP+Bk aMms2tmQ==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1rE4ZF-009rG1-0w; Fri, 15 Dec 2023 09:33:14 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id D6BA730098F; Fri, 15 Dec 2023 10:33:11 +0100 (CET) Message-Id: <20231215092707.669401084@infradead.org> User-Agent: quilt/0.65 Date: Fri, 15 Dec 2023 10:12:21 +0100 From: Peter Zijlstra To: Alexei Starovoitov Cc: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, davem@davemloft.net, dsahern@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Arnd Bergmann , samitolvanen@google.com, keescook@chromium.org, nathan@kernel.org, ndesaulniers@google.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-arch@vger.kernel.org, llvm@lists.linux.dev, jpoimboe@kernel.org, joao@overdrivepizza.com, mark.rutland@arm.com, peterz@infradead.org Subject: [PATCH v3 5/7] cfi: Add CFI_NOSEAL() References: <20231215091216.135791411@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785340137967640072 X-GMAIL-MSGID: 1785340137967640072 Add a CFI_NOSEAL() helper to mark functions that need to retain their CFI information, despite not otherwise leaking their address. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/cfi.h | 5 +++++ include/linux/cfi.h | 4 ++++ 2 files changed, 9 insertions(+) --- a/arch/x86/include/asm/cfi.h +++ b/arch/x86/include/asm/cfi.h @@ -8,6 +8,7 @@ * Copyright (C) 2022 Google LLC */ #include +#include /* * An overview of the various calling conventions... @@ -138,4 +139,8 @@ static inline u32 cfi_get_func_hash(void } #endif /* CONFIG_CFI_CLANG */ +#if HAS_KERNEL_IBT == 1 +#define CFI_NOSEAL(x) asm(IBT_NOSEAL(__stringify(x))) +#endif + #endif /* _ASM_X86_CFI_H */ --- a/include/linux/cfi.h +++ b/include/linux/cfi.h @@ -46,4 +46,8 @@ static inline void module_cfi_finalize(c #endif /* CONFIG_ARCH_USES_CFI_TRAPS */ #endif /* CONFIG_MODULES */ +#ifndef CFI_NOSEAL +#define CFI_NOSEAL(x) +#endif + #endif /* _LINUX_CFI_H */ From patchwork Fri Dec 15 09:12:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 179129 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp9149121dys; Fri, 15 Dec 2023 01:35:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IHYvReAEeEDDe/yyjb/12KuA+dYVnR9lMTVeRt8G95U9H25dugTmVj9c2ugtvHraZvV4JgV X-Received: by 2002:a17:907:7e84:b0:a1b:8297:f761 with SMTP id qb4-20020a1709077e8400b00a1b8297f761mr8416466ejc.66.1702632940387; Fri, 15 Dec 2023 01:35:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702632940; cv=none; d=google.com; s=arc-20160816; b=eX0rlFlc9lf/m5HMCVFP9c1FqODO182u1wkKOB+8JTPFw7nkE66ih3H0+iwKTk6zNY Cw/W1bGMadV5QDue3uaPCKHbfbwKgDs8y6kAo9aiRhHnZtlQDnOVLSC51ZUFJPrC7IC5 ++LKhCJ2ZuZcqjTH098u+TsWRtDsJ13mmr/kIy0Ns9Z/YIjF96Ssrn7+vOLskol5CEnO 8/wNScbbzkOgtVHUm7MgVOpJcqETv9SYi8e8N3qEEanFUojfcET+m/usjLppyCbm9BsZ 9qrHhhOrnyUL3jpc+Ul+hhmI0MOB338hig/bkrv16ZE186cGLHrhDwgtDiHCfHRqFSTr S51g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:list-unsubscribe:list-subscribe:list-id:precedence :references:subject:cc:to:from:date:user-agent:message-id :dkim-signature; bh=78VUp4ekMeaCQuFyW0sC3Rbbj/THiYo4c7UXzXOpUYU=; fh=vd912/km9qZuP2V7TBjqUWoA7VSy9ThzbPxJdH/Qh1E=; b=NfVRKxRvzPlv64Rfyf3P+RACFuSUJj3J5fzEw6iVrDuVQZBUavbO4ljnEZFcGXOcQY tDFGmRzPBQMhNzy3Pt7Noz8ApFuUBpYVXkzzXWx4ePgq8hoSY8kMhFMfzDT8+h/6/Eht weDGUjxFnv84ABoRc+MjErpiNEA7mNqWsGLxvxCdkyriuvt3aV3ksTLAGHlaLZXNGK4I XRBgI1XyloVaIslXv6Zoqp7NNnAoXJh5C/qeug1DzaXAvLpffcitvdapsFyUq0sIIhCZ 81l6Z7Rmt2zEJJzs1lEkJ5ZskC/PnVCGrzlbl5Q/zjhWFonMAEXhhXRroACnlzNHVU7I TvhA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=bnPrR5+C; spf=pass (google.com: domain of linux-kernel+bounces-684-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-684-ouuuleilei=gmail.com@vger.kernel.org" Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id ga17-20020a170906b85100b00a23163e56easi714901ejb.109.2023.12.15.01.35.40 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Dec 2023 01:35:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-684-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=bnPrR5+C; spf=pass (google.com: domain of linux-kernel+bounces-684-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-684-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id ED4421F235EA for ; Fri, 15 Dec 2023 09:35:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id D6D5C24B3F; Fri, 15 Dec 2023 09:33:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="bnPrR5+C" X-Original-To: linux-kernel@vger.kernel.org Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C976814290; Fri, 15 Dec 2023 09:33:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=78VUp4ekMeaCQuFyW0sC3Rbbj/THiYo4c7UXzXOpUYU=; b=bnPrR5+CXr7BBoaFWXx1hwtAwH 2F2PEBm7YfZ8Z+ek5E6sU2NggQXV8mH21iJOx8IuUUKojL8nl8piu1SVhyFdQehQqPX2euFT8vltW gihZz2bb9gv+VspJ5CUh/WjJz42XFPueBSyn/WRVvdtzjuELfVeUcwPUpxitCH1rPHzI5cnsWn6R6 OA71iWY86Q5Duna4NQFs5qEO9r76fyhbsZt9MW8nobXMxoFO5xvrRTH57KxULwxHYWII82RLcrG8u 0xuFVu5NdLh7F6MnKASZQN+xq+AblZov9qntQ2dc8J39R9EsSjSbj7uUsPN6ddau0FzX8ZUSPYnLe HmXCrtSg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1rE4ZE-00FSOc-4Z; Fri, 15 Dec 2023 09:33:13 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id DB5FA301157; Fri, 15 Dec 2023 10:33:11 +0100 (CET) Message-Id: <20231215092707.799451071@infradead.org> User-Agent: quilt/0.65 Date: Fri, 15 Dec 2023 10:12:22 +0100 From: Peter Zijlstra To: Alexei Starovoitov Cc: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, davem@davemloft.net, dsahern@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Arnd Bergmann , samitolvanen@google.com, keescook@chromium.org, nathan@kernel.org, ndesaulniers@google.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-arch@vger.kernel.org, llvm@lists.linux.dev, jpoimboe@kernel.org, joao@overdrivepizza.com, mark.rutland@arm.com, peterz@infradead.org Subject: [PATCH v3 6/7] bpf: Fix dtor CFI References: <20231215091216.135791411@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785340038016521385 X-GMAIL-MSGID: 1785340038016521385 Ensure the various dtor functions match their prototype and retain their CFI signatures, since they don't have their address taken, they are prone to not getting CFI, making them impossible to call indirectly. Signed-off-by: Peter Zijlstra (Intel) --- kernel/bpf/cpumask.c | 8 +++++++- kernel/bpf/helpers.c | 16 ++++++++++++++-- net/bpf/test_run.c | 15 +++++++++++++-- 3 files changed, 34 insertions(+), 5 deletions(-) --- a/kernel/bpf/cpumask.c +++ b/kernel/bpf/cpumask.c @@ -96,6 +96,12 @@ __bpf_kfunc void bpf_cpumask_release(str migrate_enable(); } +__bpf_kfunc void bpf_cpumask_release_dtor(void *cpumask) +{ + bpf_cpumask_release(cpumask); +} +CFI_NOSEAL(bpf_cpumask_release_dtor); + /** * bpf_cpumask_first() - Get the index of the first nonzero bit in the cpumask. * @cpumask: The cpumask being queried. @@ -441,7 +447,7 @@ static const struct btf_kfunc_id_set cpu BTF_ID_LIST(cpumask_dtor_ids) BTF_ID(struct, bpf_cpumask) -BTF_ID(func, bpf_cpumask_release) +BTF_ID(func, bpf_cpumask_release_dtor) static int __init cpumask_kfunc_init(void) { --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2150,6 +2150,12 @@ __bpf_kfunc void bpf_task_release(struct put_task_struct_rcu_user(p); } +__bpf_kfunc void bpf_task_release_dtor(void *p) +{ + put_task_struct_rcu_user(p); +} +CFI_NOSEAL(bpf_task_release_dtor); + #ifdef CONFIG_CGROUPS /** * bpf_cgroup_acquire - Acquire a reference to a cgroup. A cgroup acquired by @@ -2174,6 +2180,12 @@ __bpf_kfunc void bpf_cgroup_release(stru cgroup_put(cgrp); } +__bpf_kfunc void bpf_cgroup_release_dtor(void *cgrp) +{ + cgroup_put(cgrp); +} +CFI_NOSEAL(bpf_cgroup_release_dtor); + /** * bpf_cgroup_ancestor - Perform a lookup on an entry in a cgroup's ancestor * array. A cgroup returned by this kfunc which is not subsequently stored in a @@ -2570,10 +2582,10 @@ static const struct btf_kfunc_id_set gen BTF_ID_LIST(generic_dtor_ids) BTF_ID(struct, task_struct) -BTF_ID(func, bpf_task_release) +BTF_ID(func, bpf_task_release_dtor) #ifdef CONFIG_CGROUPS BTF_ID(struct, cgroup) -BTF_ID(func, bpf_cgroup_release) +BTF_ID(func, bpf_cgroup_release_dtor) #endif BTF_SET8_START(common_btf_ids) --- a/net/bpf/test_run.c +++ b/net/bpf/test_run.c @@ -600,10 +600,21 @@ __bpf_kfunc void bpf_kfunc_call_test_rel refcount_dec(&p->cnt); } +__bpf_kfunc void bpf_kfunc_call_test_release_dtor(void *p) +{ + bpf_kfunc_call_test_release(p); +} +CFI_NOSEAL(bpf_kfunc_call_test_release_dtor); + __bpf_kfunc void bpf_kfunc_call_memb_release(struct prog_test_member *p) { } +__bpf_kfunc void bpf_kfunc_call_memb_release_dtor(void *p) +{ +} +CFI_NOSEAL(bpf_kfunc_call_memb_release_dtor); + __bpf_kfunc_end_defs(); BTF_SET8_START(bpf_test_modify_return_ids) @@ -1671,9 +1682,9 @@ static const struct btf_kfunc_id_set bpf BTF_ID_LIST(bpf_prog_test_dtor_kfunc_ids) BTF_ID(struct, prog_test_ref_kfunc) -BTF_ID(func, bpf_kfunc_call_test_release) +BTF_ID(func, bpf_kfunc_call_test_release_dtor) BTF_ID(struct, prog_test_member) -BTF_ID(func, bpf_kfunc_call_memb_release) +BTF_ID(func, bpf_kfunc_call_memb_release_dtor) static int __init bpf_prog_test_run_init(void) { From patchwork Fri Dec 15 09:12:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 179135 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:3b04:b0:fb:cd0c:d3e with SMTP id c4csp9150068dys; Fri, 15 Dec 2023 01:38:01 -0800 (PST) X-Google-Smtp-Source: AGHT+IGY3st+0pbmjf2bUl87xcTme1J4xZUh8nr/xXI8doR85zQDDbY4qs8Pv17S0cH963AzVcJl X-Received: by 2002:a17:90b:250e:b0:28a:c84d:1366 with SMTP id ns14-20020a17090b250e00b0028ac84d1366mr2553458pjb.71.1702633080826; Fri, 15 Dec 2023 01:38:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1702633080; cv=none; d=google.com; s=arc-20160816; b=Itr1nqK6ooOa+iEgSynyHzXGN1wafdE88+2RN1hxAZZTRxQ1pQQmidLWgK5ZyTmU3P 1GefhMjXoc68v80ko3WOrZ1vGcfiYuYU4krXABF76bPG/YILfSAtKzgJXuDEFcvzP+jE U0PRihx1CQaq3mbIfZ3L1+VQXOgh2MZELq0gId7eZXThuoJsC3KZY9l/RSxScYKs4HUb AgWn6lp3O5jZUpPTkOlDZKDXBJ8kFqi5LVyt207RJ6P3UVxTpNybbQ8BU1EdMaJ9pIIY cDJzUMgLPnCxoC+7iwz2w4QXvFb3nIn/FEgTBlW94hXrOWsf5k2alRYT2L1vfT3dpTHB sD8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=mime-version:list-unsubscribe:list-subscribe:list-id:precedence :references:subject:cc:to:from:date:user-agent:message-id :dkim-signature; bh=cG37Lky4vNGjI+/dn78V8dged+rcVq6oxgHwY0WVgrM=; fh=vd912/km9qZuP2V7TBjqUWoA7VSy9ThzbPxJdH/Qh1E=; b=hl4PJml5PMfG91dSOYlRT+3+/PA4SD/1TTKVyBRzlbMaC6+e4vCuCbGSvdnJfAcb1F 8reZHewyyLLtGfstRfXXGHFM8wVVVbM/FRKZTwlVvmdfcMJvcKF5HxdNRoi5EPv97HrP xc/CwDg0u1a2bHMlrPHlhBB8RSuxoAohHRZAMOgEa9X6VWx8DYWVp4Rtty0K2VTa8nPr ER9dvmp3W2hG9pIVTp8vzqgYWqWeu4WJEVakCCMF+rDiWICIWDZLA2AAieVPDruJCmUW 9CLOTuZ/y1btpsyOs9o0loIoeA6EgrCd613nIbhCZpobKGhXFaKmjnnvPlu6XwtlqMyF 35GA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b="HnOlO/cK"; spf=pass (google.com: domain of linux-kernel+bounces-690-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-690-ouuuleilei=gmail.com@vger.kernel.org" Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [2604:1380:40f1:3f00::1]) by mx.google.com with ESMTPS id x14-20020a17090a46ce00b002867eb143afsi14141071pjg.133.2023.12.15.01.38.00 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 15 Dec 2023 01:38:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-690-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) client-ip=2604:1380:40f1:3f00::1; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b="HnOlO/cK"; spf=pass (google.com: domain of linux-kernel+bounces-690-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:40f1:3f00::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-690-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 35529B22DD1 for ; Fri, 15 Dec 2023 09:36:59 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 2349E2D623; Fri, 15 Dec 2023 09:33:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="HnOlO/cK" X-Original-To: linux-kernel@vger.kernel.org Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 061A917996; Fri, 15 Dec 2023 09:33:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-Id:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=cG37Lky4vNGjI+/dn78V8dged+rcVq6oxgHwY0WVgrM=; b=HnOlO/cKUyALtWFkVS+r+9xPPj W8PtzBF/Ag2XAodU9cpkAPlJOKG6sCB7AjvimCTrEb5e15JI6mbhdhAuLXKjD4qhF67REnTRkcT5R Q9F9ePvPqYS8n6SN/4cevBgqVZ0Sct1sdx5cs2oMDt6nEZLZlkzQ/Monzno8X29cLfBZhkWyB/OKW S4Ywj+Bu2R6eV5V9m0DrxorpYikjPcmvTjIgwrJqLxZBLVz1t2BziAQ5XgunVD/XEU7gGG20fFUPk eOYHdeaPSr2R1cqz/Gsr8qy5oAKCQpncNzmQuBU0ZaX4w6tudaj+eth8AK45GC4Snht/g5h+uQpB2 Ex/lSrZw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.96 #2 (Red Hat Linux)) id 1rE4ZF-009rG2-0y; Fri, 15 Dec 2023 09:33:14 +0000 Received: by noisy.programming.kicks-ass.net (Postfix, from userid 0) id EB92A3006F6; Fri, 15 Dec 2023 10:33:11 +0100 (CET) Message-Id: <20231215092707.910319166@infradead.org> User-Agent: quilt/0.65 Date: Fri, 15 Dec 2023 10:12:23 +0100 From: Peter Zijlstra To: Alexei Starovoitov Cc: paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, davem@davemloft.net, dsahern@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, john.fastabend@gmail.com, kpsingh@kernel.org, sdf@google.com, haoluo@google.com, jolsa@kernel.org, Arnd Bergmann , samitolvanen@google.com, keescook@chromium.org, nathan@kernel.org, ndesaulniers@google.com, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, linux-arch@vger.kernel.org, llvm@lists.linux.dev, jpoimboe@kernel.org, joao@overdrivepizza.com, mark.rutland@arm.com, peterz@infradead.org Subject: [PATCH v3 7/7] x86/cfi,bpf: Fix bpf_exception_cb() signature References: <20231215091216.135791411@infradead.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1785340185358306629 X-GMAIL-MSGID: 1785340185358306629 From: Alexei Starovoitov As per the earlier patches, BPF sub-programs have bpf_callback_t signature and CFI expects callers to have matching signature. This is violated by bpf_prog_aux::bpf_exception_cb(). [peterz: Changelog] Reported-by: Peter Zijlstra Signed-off-by: Alexei Starovoitov Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/CAADnVQ+Z7UcXXBBhMubhcMM=R-dExk-uHtfOLtoLxQ1XxEpqEA@mail.gmail.com --- include/linux/bpf.h | 2 +- kernel/bpf/helpers.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1484,7 +1484,7 @@ struct bpf_prog_aux { int cgroup_atype; /* enum cgroup_bpf_attach_type */ struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE]; char name[BPF_OBJ_NAME_LEN]; - unsigned int (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp); + u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64); #ifdef CONFIG_SECURITY void *security; #endif --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2537,7 +2537,7 @@ __bpf_kfunc void bpf_throw(u64 cookie) * which skips compiler generated instrumentation to do the same. */ kasan_unpoison_task_stack_below((void *)(long)ctx.sp); - ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp); + ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0); WARN(1, "A call to BPF exception callback should never return\n"); }