From patchwork Mon Jul 10 07:41:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= X-Patchwork-Id: 117684 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp4857018vqx; Mon, 10 Jul 2023 00:58:42 -0700 (PDT) X-Google-Smtp-Source: APBJJlHeO6lkqwKVtMW/Xma9CoO4Tx5xTYIF6uEBrt7Tui9Y5TyZYoCIkALx3Eua9rvoVmVHK3XC X-Received: by 2002:a17:906:7056:b0:982:487c:7508 with SMTP id r22-20020a170906705600b00982487c7508mr11049008ejj.38.1688975922647; Mon, 10 Jul 2023 00:58:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688975922; cv=none; d=google.com; s=arc-20160816; b=R4LO40AmH5hQ3XdA2Mtiq1+LgNcb2iX8TtHKiy/aG+nMG1lzgxlxs6bbj/CE090Pjb uLRYv6MyOes9v6h85pXYD+gX8YPHULgE8ecJZ0Mm/viZ/JmV1GPWuzIOI6N6vkJq2GI7 4Jfcshkkg+wLfsCr+xHsWtke0YKEseRGNnEFvkVXj52vpouqM8YzdDB0s5Hgqo1sYQAH 07QaWHJNDcbBgHVlzPxpKILpHfqGCJXv4hhIwujYOAi7+DQNodvZjKgYX+1IFjm2k5cI c4oWqyeO0kpR/LZXpnhA/G7KlSVBeCR+0Bc9e1qmi/3jPKBX74y5wH8DGNSwgda1K5wY 4Jog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=fY/5T2gDYv5okGXmRT23p7vi0lB2tsrh5jdY4e7R1Z0=; fh=OACJ/kPQ/wu/iw9DlzHkS1Ut/3Ipypmt0jsSaQrqLAA=; b=DzKY4oc1VxJpgCoYaJ3faG/LYBqN+zZTWkDZxT9HSlpS36i+GM3iWBo4nrNtkT4fZO dWHSegURCh3KLvuJGnpbK1HYwKxI9qF7rSri8eCfP2Y91vw00FKxmc6bDScWXPZfcgEH kUb66tUxBPjBzfi7ztgeoomKYuPYsCzp3DoB/DzvPp416YibRVE//gxSHDWNAQttxgVO fHg/GB3PVM7QQaGwH8M7TtNIT+s5qxgu0XGxPRqyoeHCHLe2ZEMi1wxzbYbQ5Ajjfg9Q OmKniz1euOg3he4gI24bpRHQ/UyiVgx27rgfQR6azCNX0UZ0o5cmnyzVlTnAB6+5Sj0K P98g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=s7vvHK1t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m9-20020a1709060d8900b00982762e2cc4si8473175eji.775.2023.07.10.00.58.19; Mon, 10 Jul 2023 00:58:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=s7vvHK1t; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229967AbjGJHlv (ORCPT + 99 others); Mon, 10 Jul 2023 03:41:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229917AbjGJHlu (ORCPT ); Mon, 10 Jul 2023 03:41:50 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6268595 for ; Mon, 10 Jul 2023 00:41:49 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id F3C2460EB7 for ; Mon, 10 Jul 2023 07:41:48 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D931FC433C7; Mon, 10 Jul 2023 07:41:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1688974908; bh=V7owo16SHNJvQvnUS1layBkJE0wLL8p0Pv8o3MHbPao=; h=From:To:Cc:Subject:Date:From; b=s7vvHK1tGfEJLv0E7+EHWLxJNiTB8OxA+fERalMvF2nRgkiJ05hq8iuIDlHDU+AjV 8bOxeSTv5jXElYsObZOxWu27bDTW8/s557pExcCVAwRGluBtAiIey4g0UYdflhlKwC MdFvi8/dkOoocv+f9+V8hPD3bW44gWvYsLnN4XT1yaNFUNt+9+kDlakxPG6eQN5DKS 5CA6Toq7l5GCPiQOcWxGwxpA3QQQvSvH0hc/xnkgrLmLaUZN1JrHWU/DnR9blh5BuO sE5auzbeMqIaRMvw6dk6owIIR70iQATJr9csMXqbEhLQ9u2w3LUAfkLuHdQ4LSF197 pTMWzAOQy2IhQ== From: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= To: Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , bpf@vger.kernel.org, netdev@vger.kernel.org Cc: =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Martin KaFai Lau , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Pu Lehui , Luke Nelson , Xi Wang , linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux@rivosinc.com Subject: [PATCH bpf] riscv, bpf: Fix inconsistent JIT image generation Date: Mon, 10 Jul 2023 09:41:31 +0200 Message-Id: <20230710074131.19596-1-bjorn@kernel.org> X-Mailer: git-send-email 2.39.2 MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1771019616678785298 X-GMAIL-MSGID: 1771019616678785298 From: Björn Töpel In order to generate the prologue and epilogue, the BPF JIT needs to know which registers that are clobbered. Therefore, the during pre-final passes, the prologue is generated after the body of the program body-prologue-epilogue. Then, in the final pass, a proper prologue-body-epilogue JITted image is generated. This scheme has worked most of the time. However, for some large programs with many jumps, e.g. the test_kmod.sh BPF selftest with hardening enabled (blinding constants), this has shown to be incorrect. For the final pass, when the proper prologue-body-epilogue is generated, the image has not converged. This will lead to that the final image will have incorrect jump offsets. The following is an excerpt from an incorrect image: | ... | 3b8: 00c50663 beq a0,a2,3c4 <.text+0x3c4> | 3bc: 0020e317 auipc t1,0x20e | 3c0: 49630067 jalr zero,1174(t1) # 20e852 <.text+0x20e852> | ... | 20e84c: 8796 c.mv a5,t0 | 20e84e: 6422 c.ldsp s0,8(sp) # Epilogue start | 20e850: 6141 c.addi16sp sp,16 | 20e852: 853e c.mv a0,a5 # Incorrect jump target | 20e854: 8082 c.jr ra The image has shrunk, and the epilogue offset is incorrect in the final pass. Correct the problem by always generating proper prologue-body-epilogue outputs, which means that the first pass will only generate the body to track what registers that are touched. Fixes: 2353ecc6f91f ("bpf, riscv: add BPF JIT for RV64G") Signed-off-by: Björn Töpel Acked-by: Palmer Dabbelt Reviewed-by: Palmer Dabbelt --- arch/riscv/net/bpf_jit.h | 6 +++--- arch/riscv/net/bpf_jit_core.c | 19 +++++++++++++------ 2 files changed, 16 insertions(+), 9 deletions(-) base-commit: 496720b7cfb6574a8f6f4d434f23e3d1e6cfaeb9 diff --git a/arch/riscv/net/bpf_jit.h b/arch/riscv/net/bpf_jit.h index bf9802a63061..2717f5490428 100644 --- a/arch/riscv/net/bpf_jit.h +++ b/arch/riscv/net/bpf_jit.h @@ -69,7 +69,7 @@ struct rv_jit_context { struct bpf_prog *prog; u16 *insns; /* RV insns */ int ninsns; - int body_len; + int prologue_len; int epilogue_offset; int *offset; /* BPF to RV */ int nexentries; @@ -216,8 +216,8 @@ static inline int rv_offset(int insn, int off, struct rv_jit_context *ctx) int from, to; off++; /* BPF branch is from PC+1, RV is from PC */ - from = (insn > 0) ? ctx->offset[insn - 1] : 0; - to = (insn + off > 0) ? ctx->offset[insn + off - 1] : 0; + from = (insn > 0) ? ctx->offset[insn - 1] : ctx->prologue_len; + to = (insn + off > 0) ? ctx->offset[insn + off - 1] : ctx->prologue_len; return ninsns_rvoff(to - from); } diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c index 737baf8715da..7a26a3e1c73c 100644 --- a/arch/riscv/net/bpf_jit_core.c +++ b/arch/riscv/net/bpf_jit_core.c @@ -44,7 +44,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) unsigned int prog_size = 0, extable_size = 0; bool tmp_blinded = false, extra_pass = false; struct bpf_prog *tmp, *orig_prog = prog; - int pass = 0, prev_ninsns = 0, prologue_len, i; + int pass = 0, prev_ninsns = 0, i; struct rv_jit_data *jit_data; struct rv_jit_context *ctx; @@ -83,6 +83,12 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog = orig_prog; goto out_offset; } + + if (build_body(ctx, extra_pass, NULL)) { + prog = orig_prog; + goto out_offset; + } + for (i = 0; i < prog->len; i++) { prev_ninsns += 32; ctx->offset[i] = prev_ninsns; @@ -91,12 +97,15 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) for (i = 0; i < NR_JIT_ITERATIONS; i++) { pass++; ctx->ninsns = 0; + + bpf_jit_build_prologue(ctx); + ctx->prologue_len = ctx->ninsns; + if (build_body(ctx, extra_pass, ctx->offset)) { prog = orig_prog; goto out_offset; } - ctx->body_len = ctx->ninsns; - bpf_jit_build_prologue(ctx); + ctx->epilogue_offset = ctx->ninsns; bpf_jit_build_epilogue(ctx); @@ -162,10 +171,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) if (!prog->is_func || extra_pass) { bpf_jit_binary_lock_ro(jit_data->header); - prologue_len = ctx->epilogue_offset - ctx->body_len; for (i = 0; i < prog->len; i++) - ctx->offset[i] = ninsns_rvoff(prologue_len + - ctx->offset[i]); + ctx->offset[i] = ninsns_rvoff(ctx->offset[i]); bpf_prog_fill_jited_linfo(prog, ctx->offset); out_offset: kfree(ctx->offset);