From patchwork Thu Nov 3 12:00:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 14839 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp493792wru; Thu, 3 Nov 2022 05:22:48 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5qheAFQkFEOhRkzf4Bls3ZnAOVKUwKB6UocNppCeUu+X38QqK8GCtwv3kGeMtMqjz1q/P3 X-Received: by 2002:a17:907:ea7:b0:7ad:c3b7:ca52 with SMTP id ho39-20020a1709070ea700b007adc3b7ca52mr23662601ejc.626.1667478168184; Thu, 03 Nov 2022 05:22:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667478168; cv=none; d=google.com; s=arc-20160816; b=PA0RaVYkAHNtI4uTdan3/geF/kSlGxE1NH6R2LV5zFWAnbchoN1kVVEOzUpl6iLKKG +T8GYCRAg+nh844nG8ezina3fwg6iGaR2fAG8ios5zjNBJjGvTTAtrtnw/iFfQS6/Im9 0KN5ERSjtXL2tCFSJma+i4hafuf8LJp1gT82w7wnbb5HDivQGW9URA44rTseYo8rixdk enUTaQxXKdnWId0qpkvwmso5VcZQ/ZpeAA8P8qirf0N4vEOhzSDtvkURMlp+P4TL6NVs 4rQg2nX5LhmTvc1Bs4mmIww1Soa/4XBaXQg03S4uWX+qCeT44PmW5NZbXVZ5eurH2q8+ /TaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=D+IbIcIImMPdQpJwzXPlYzbuRJccnY/ode4zkKbsw8o=; b=TMmrzrBEVnTJtIAjpvJWKtSonpBEW4rbLzydemYlSWRrLH7y5MBByh/p0ra4JSISIz 554JlGbaiNY5p9NabdIGOIOx89mI/YCkvmS4CsZdTjFttyvI1YkeycSfJe5xTe6wxBha fVywl4hNj4IcwP1KNSbwlrHmbj8rGO1coaiWFoyW/cAMRfnCATr8hwYIEpOFpQ4YNon/ zg46sjVXR8WzPCM/llJ/gULOsbuN36VHyraPxSQnHu8Iup80epeGiaJj/qK15XlVKaxx OP/sw3xVr1VOseUyU/mVMRITg0A9zUnwVF72khRCyfcXEdZUU9QH+HzsWkSwrBhfB6n5 RdMA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=djh2Mvwa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g14-20020a1709065d0e00b0078e11a20640si207157ejt.2.2022.11.03.05.22.23; Thu, 03 Nov 2022 05:22:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=djh2Mvwa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231591AbiKCMJk (ORCPT + 99 others); Thu, 3 Nov 2022 08:09:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231543AbiKCMJd (ORCPT ); Thu, 3 Nov 2022 08:09:33 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9653110DE; Thu, 3 Nov 2022 05:09:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=D+IbIcIImMPdQpJwzXPlYzbuRJccnY/ode4zkKbsw8o=; b=djh2MvwagoWZ2wxNVF79AMiAZ3 cjSbuepOvsmsIVV3NzIPip4mJqeBySJZAlQgNs+So6jNLViKVTkIQalaosQNw//ZhVUZ1woL+FKfm sokEw4B7RcevgVDj0rLBX/Al2Xou+0iGvWvFFZsAcytESkIgoe0Yv9NgSyvwGx1R+vgsJtJ3QBY8o y03y4OwSLufc3l8LHzX/NmP4aUq0Xs1rErlQnMV3ws1oAXEHtRRT2wJ8WrJD4g0EtG2F8MjNhexlJ ptt5dLq1Zr3tZj5zjlKvFehj3UvM8rn8ct/aP4uzdVGL4tIgs/zqZPhzub4/01cMKq66YChAsmqn6 D2Sj1+yw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oqZ1d-008fpA-8v; Thu, 03 Nov 2022 12:08:51 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 311BE300411; Thu, 3 Nov 2022 13:08:44 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 855C720AB9A14; Thu, 3 Nov 2022 13:08:44 +0100 (CET) Message-ID: <20221103120647.728830733@infradead.org> User-Agent: quilt/0.66 Date: Thu, 03 Nov 2022 13:00:13 +0100 From: Peter Zijlstra To: olsajiri@gmail.com, ast@kernel.org, daniel@iogearbox.net Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, peterz@infradead.org, bjorn@kernel.org, toke@redhat.com, David.Laight@aculab.com, rostedt@goodmis.org Subject: [PATCH 1/2] bpf: Revert ("Fix dispatcher patchable function entry to 5 bytes nop") References: <20221103120012.717020618@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748477587635227498?= X-GMAIL-MSGID: =?utf-8?q?1748477587635227498?= Because __attribute__((patchable_function_entry)) is only available since GCC-8 this solution fails to build on the minimum required GCC version. Undo these changes so we might try again -- without cluttering up the patches with too many changes. This is an almost complete revert of: dbe69b299884 ("bpf: Fix dispatcher patchable function entry to 5 bytes nop") ceea991a019c ("bpf: Move bpf_dispatcher function out of ftrace locations") (notably the arch/x86/Kconfig hunk is kept). Reported-by: David Laight Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/439d8dc735bb4858875377df67f1b29a@AcuMS.aculab.com --- arch/x86/net/bpf_jit_comp.c | 13 ------------- include/linux/bpf.h | 21 +-------------------- kernel/bpf/dispatcher.c | 6 ------ 3 files changed, 1 insertion(+), 39 deletions(-) --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -11,7 +11,6 @@ #include #include #include -#include #include #include #include @@ -389,18 +388,6 @@ static int __bpf_arch_text_poke(void *ip return ret; } -int __init bpf_arch_init_dispatcher_early(void *ip) -{ - const u8 *nop_insn = x86_nops[5]; - - if (is_endbr(*(u32 *)ip)) - ip += ENDBR_INSN_SIZE; - - if (memcmp(ip, nop_insn, X86_PATCH_SIZE)) - text_poke_early(ip, nop_insn, X86_PATCH_SIZE); - return 0; -} - int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type t, void *old_addr, void *new_addr) { --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -27,7 +27,6 @@ #include #include #include -#include struct bpf_verifier_env; struct bpf_verifier_log; @@ -971,8 +970,6 @@ struct bpf_trampoline *bpf_trampoline_ge struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs); -int __init bpf_arch_init_dispatcher_early(void *ip); - #define BPF_DISPATCHER_INIT(_name) { \ .mutex = __MUTEX_INITIALIZER(_name.mutex), \ .func = &_name##_func, \ @@ -986,21 +983,7 @@ int __init bpf_arch_init_dispatcher_earl }, \ } -#define BPF_DISPATCHER_INIT_CALL(_name) \ - static int __init _name##_init(void) \ - { \ - return bpf_arch_init_dispatcher_early(_name##_func); \ - } \ - early_initcall(_name##_init) - -#ifdef CONFIG_X86_64 -#define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5))) -#else -#define BPF_DISPATCHER_ATTRIBUTES -#endif - #define DEFINE_BPF_DISPATCHER(name) \ - notrace BPF_DISPATCHER_ATTRIBUTES \ noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \ @@ -1010,9 +993,7 @@ int __init bpf_arch_init_dispatcher_earl } \ EXPORT_SYMBOL(bpf_dispatcher_##name##_func); \ struct bpf_dispatcher bpf_dispatcher_##name = \ - BPF_DISPATCHER_INIT(bpf_dispatcher_##name); \ - BPF_DISPATCHER_INIT_CALL(bpf_dispatcher_##name); - + BPF_DISPATCHER_INIT(bpf_dispatcher_##name); #define DECLARE_BPF_DISPATCHER(name) \ unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ --- a/kernel/bpf/dispatcher.c +++ b/kernel/bpf/dispatcher.c @@ -4,7 +4,6 @@ #include #include #include -#include /* The BPF dispatcher is a multiway branch code generator. The * dispatcher is a mechanism to avoid the performance penalty of an @@ -90,11 +89,6 @@ int __weak arch_prepare_bpf_dispatcher(v { return -ENOTSUPP; } - -int __weak __init bpf_arch_init_dispatcher_early(void *ip) -{ - return -ENOTSUPP; -} static int bpf_dispatcher_prepare(struct bpf_dispatcher *d, void *image, void *buf) { From patchwork Thu Nov 3 12:00:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 14838 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp489761wru; Thu, 3 Nov 2022 05:14:34 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7B1LSG/ssbWUnPyADFbWrsLPWCpMavMnSewNVZN9yRXanSngsGcGo3HVanCYzqZVioJLPv X-Received: by 2002:a17:907:3e96:b0:7ad:e35a:be36 with SMTP id hs22-20020a1709073e9600b007ade35abe36mr17477975ejc.678.1667477674328; Thu, 03 Nov 2022 05:14:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667477674; cv=none; d=google.com; s=arc-20160816; b=NenUalag5ewciIraRWmAo5yfGIyq/8Ez0m/fumbs8tXasgm4bXAXbNnDRkSaNFHgjl kUkf7rsdQcch0OVnlI+TJKFEsCl8Xy0i7PqORHtwPn4aAAjmWLqKlktGmzw53nrSUf3J PvcxC1Rd87uN3PqcNBIELRuXTDiaUitUjIjJ4sRzH65ofO3sx9ochEDyVFSca0+dohyn m+kF2amKpufwQ6aXOOWD7p0Zws1KEUAe8XNP3Iiq0H+fTlIl3DKE7mP0W4gNtnPfiDRm P9UT1PwwhBM9NZYrnL66Toa1SfTjIzhiG5CM8xbLs0bYLP3RYvIMi5GG5GtjHkuNoyE4 urUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=KYBwDtUVZtl4ICiIQ6x4pKdkkIuJAYSCVKXUwRe3sYI=; b=sUo2FiOVgwB3Gli6ZFpJzClJFfXWhfDhukRMkKoB+rjs+KHN/ENSppgiF5mh6TI69L AZ9SDD3wYhe3u2Vhk9IYGjTU+I7tN9EhIQJdfooN5542rFjfYGVVqX2Q1xLzmznXNQ2R VdQjs+guHp0MXx2Io79oSLj2KgHB1C3mF665spfeWT9XOK+0vbI4HrLHg42j4qygNdpP G2E1oZthYVOql0pkbadlkNw9BVeFluGjKOr8DdpDJBub/z9jQ2swTsqg41IJA1u+0kKi gS/5SmYXtm6WrwE76nKlWP5HNzXqmISM+15np+0UMbwaToDCjMLHeTLUcZYDlSvlDXz4 2+aw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=BOizHypu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id by12-20020a0564021b0c00b004627d582888si944831edb.24.2022.11.03.05.14.09; Thu, 03 Nov 2022 05:14:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=BOizHypu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231516AbiKCMJI (ORCPT + 99 others); Thu, 3 Nov 2022 08:09:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231491AbiKCMJE (ORCPT ); Thu, 3 Nov 2022 08:09:04 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 106E0FD1; Thu, 3 Nov 2022 05:09:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=KYBwDtUVZtl4ICiIQ6x4pKdkkIuJAYSCVKXUwRe3sYI=; b=BOizHypuaKlhX11+jt8hI9YbjV TnOrcNkasSXlMm56qZYZ77K12wtHC4bw8CITLCqQ8ozcCjpPNCDZO9eg7MEWrv8I0ZKKq6M3m3xMC VNb7JYaUOK301Ld6LUh0nBdn6fmnsTqeGUjDDdOWxZKfWn5wwI6bouGTlAzwCHrJGsvipnV3S53HJ Tmzbl2WZXW9CERPwtbRH8gMcFWHszVe8QKGtyVc4eGkaZ/29RHNqP52jeNKjmdqaPqbCaCYweX9JJ VklcerTX+cmiCLzSxXXJPHvrFHZutqH0R0+Wtf6LmeaFThkHi/zTQUkJxsFjXIPMNMa8LP6CWxsQe ttZ+Hb3Q==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1oqZ1g-006RMW-9b; Thu, 03 Nov 2022 12:08:52 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 2F8BD3003E1; Thu, 3 Nov 2022 13:08:44 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 8886120AB9A25; Thu, 3 Nov 2022 13:08:44 +0100 (CET) Message-ID: <20221103120647.796772565@infradead.org> User-Agent: quilt/0.66 Date: Thu, 03 Nov 2022 13:00:14 +0100 From: Peter Zijlstra To: olsajiri@gmail.com, ast@kernel.org, daniel@iogearbox.net Cc: linux-kernel@vger.kernel.org, netdev@vger.kernel.org, bpf@vger.kernel.org, peterz@infradead.org, bjorn@kernel.org, toke@redhat.com, David.Laight@aculab.com, rostedt@goodmis.org Subject: [PATCH 2/2] bpf: Convert BPF_DISPATCHER to use static_call() (not ftrace) References: <20221103120012.717020618@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748477069557447105?= X-GMAIL-MSGID: =?utf-8?q?1748477069557447105?= The dispatcher function is currently abusing the ftrace __fentry__ call location for its own purposes -- this obviously gives trouble when the dispatcher and ftrace are both in use. A previous solution tried using __attribute__((patchable_function_entry())) which works, except it is GCC-8+ only, breaking the build on the earlier still supported compilers. Instead use static_call() -- which has its own annotations and does not conflict with ftrace -- to rewrite the dispatch function. By using: return static_call()(ctx, insni, bpf_func) you get a perfect forwarding tail call as function body (iow a single jmp instruction). By having the default static_call() target be bpf_dispatcher_nop_func() it retains the default behaviour (an indirect call to the argument function). Only once a dispatcher program is attached is the target rewritten to directly call the JIT'ed image. Signed-off-by: Peter Zijlstra (Intel) Link: https://lkml.kernel.org/r/Y1/oBlK0yFk5c/Im@hirez.programming.kicks-ass.net Acked-by: Steven Rostedt (Google) --- include/linux/bpf.h | 39 ++++++++++++++++++++++++++++++++++++++- kernel/bpf/dispatcher.c | 22 ++++++++-------------- 2 files changed, 46 insertions(+), 15 deletions(-) --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -27,6 +27,7 @@ #include #include #include +#include struct bpf_verifier_env; struct bpf_verifier_log; @@ -953,6 +954,10 @@ struct bpf_dispatcher { void *rw_image; u32 image_off; struct bpf_ksym ksym; +#ifdef CONFIG_HAVE_STATIC_CALL + struct static_call_key *sc_key; + void *sc_tramp; +#endif }; static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func( @@ -970,6 +975,34 @@ struct bpf_trampoline *bpf_trampoline_ge struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs); + +/* + * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn + * indirection with a direct call to the bpf program. If the architecture does + * not have STATIC_CALL, avoid a double-indirection. + */ +#ifdef CONFIG_HAVE_STATIC_CALL + +#define __BPF_DISPATCHER_SC_INIT(_name) \ + .sc_key = &STATIC_CALL_KEY(_name), \ + .sc_tramp = STATIC_CALL_TRAMP_ADDR(_name), + +#define __BPF_DISPATCHER_SC(name) \ + DEFINE_STATIC_CALL(bpf_dispatcher_##name##_call, bpf_dispatcher_nop_func) + +#define __BPF_DISPATCHER_CALL(name) \ + static_call(bpf_dispatcher_##name##_call)(ctx, insnsi, bpf_func) + +#define __BPF_DISPATCHER_UPDATE(_d, _new) \ + __static_call_update((_d)->sc_key, (_d)->sc_tramp, (_new)) + +#else +#define __BPF_DISPATCHER_SC_INIT(name) +#define __BPF_DISPATCHER_SC(name) +#define __BPF_DISPATCHER_CALL(name) bpf_func(ctx, insnsi) +#define __BPF_DISPATCHER_UPDATE(_d, _new) +#endif + #define BPF_DISPATCHER_INIT(_name) { \ .mutex = __MUTEX_INITIALIZER(_name.mutex), \ .func = &_name##_func, \ @@ -981,25 +1014,29 @@ int arch_prepare_bpf_dispatcher(void *im .name = #_name, \ .lnode = LIST_HEAD_INIT(_name.ksym.lnode), \ }, \ + __BPF_DISPATCHER_SC_INIT(_name##_call) \ } #define DEFINE_BPF_DISPATCHER(name) \ + __BPF_DISPATCHER_SC(name); \ noinline __nocfi unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \ bpf_func_t bpf_func) \ { \ - return bpf_func(ctx, insnsi); \ + return __BPF_DISPATCHER_CALL(name); \ } \ EXPORT_SYMBOL(bpf_dispatcher_##name##_func); \ struct bpf_dispatcher bpf_dispatcher_##name = \ BPF_DISPATCHER_INIT(bpf_dispatcher_##name); + #define DECLARE_BPF_DISPATCHER(name) \ unsigned int bpf_dispatcher_##name##_func( \ const void *ctx, \ const struct bpf_insn *insnsi, \ bpf_func_t bpf_func); \ extern struct bpf_dispatcher bpf_dispatcher_##name; + #define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_##name##_func #define BPF_DISPATCHER_PTR(name) (&bpf_dispatcher_##name) void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from, --- a/kernel/bpf/dispatcher.c +++ b/kernel/bpf/dispatcher.c @@ -4,6 +4,7 @@ #include #include #include +#include /* The BPF dispatcher is a multiway branch code generator. The * dispatcher is a mechanism to avoid the performance penalty of an @@ -104,17 +105,11 @@ static int bpf_dispatcher_prepare(struct static void bpf_dispatcher_update(struct bpf_dispatcher *d, int prev_num_progs) { - void *old, *new, *tmp; - u32 noff; - int err; - - if (!prev_num_progs) { - old = NULL; - noff = 0; - } else { - old = d->image + d->image_off; + void *new, *tmp; + u32 noff = 0; + + if (prev_num_progs) noff = d->image_off ^ (PAGE_SIZE / 2); - } new = d->num_progs ? d->image + noff : NULL; tmp = d->num_progs ? d->rw_image + noff : NULL; @@ -128,11 +123,10 @@ static void bpf_dispatcher_update(struct return; } - err = bpf_arch_text_poke(d->func, BPF_MOD_JUMP, old, new); - if (err || !new) - return; + __BPF_DISPATCHER_UPDATE(d, new ?: &bpf_dispatcher_nop_func); - d->image_off = noff; + if (new) + d->image_off = noff; } void bpf_dispatcher_change_prog(struct bpf_dispatcher *d, struct bpf_prog *from,