From patchwork Thu Oct 27 09:28:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11694 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp129956wru; Thu, 27 Oct 2022 02:41:09 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5VNMFp4UTWwLESJJvSwqFRfLvLC0AJ9EGRnExsnxmdaSxcNwY578fJeaOhelg0ferz5iFn X-Received: by 2002:a17:907:3e01:b0:730:a690:a211 with SMTP id hp1-20020a1709073e0100b00730a690a211mr40793219ejc.596.1666863669070; Thu, 27 Oct 2022 02:41:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666863669; cv=none; d=google.com; s=arc-20160816; b=j2/MWdG5agib7wxnGy2vqc/Yii0+eRMVnNLYliOIg1DAGv2MC3XCxCq6nDeqrbywRJ LXDRD+MR1lCVMvx4NwP0sZVnKvIwigKyAV2QUp03O8kG9mmUeTXmZ9MimUYC2AaFCfAr fZ/q48k5UrPadjQ4wAf1B8IF6vRnu6esvSJgPryYDqQiE72NdjJCkx7+afLNt3jXT7+R DFbB2tVhV+yl3PlgRZv/TRIvwa2bgZek6ofhxgPCGT38yUkfABZBHpkNAp35uXe+w9P8 DrGh0t8smXLeJA3oos1doLFb1DLD1WlcHdcTS0hzUmYxPRe8BPruivHLDG1tr8fobo99 iBaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=DMAH9kfM4tEFqzW7U3XOFl7UPt7+R8Cga6pBO8g46JQ=; b=y98pVlkgjO2fzcK3yCsg62fKx2gh0NJo2+M8KOrP03DKWFtGxDaCdHuCgf0Rf4F8Yn 00l3haSh3ljmzCAYfQ/dxtYJvo+UY04okRGi4vwcaIVRkd0X4l1exOWEVIGNN2om94io zldDYjW+oMTwHqVU7fk2EPsKjki/jY+Ggl2dQ6xGgHSpuyedaRSEeaqsX+hBU4hsDODA 7g5BBfB/mzYn86VNWDbO7zpFsXxzIZmPejmQbR+njykS1lLLtzjA/S8+4kLYsyQHjGOi ug1yEUp3KWfhWcRrCEFrNMNpHJSkzJQIB/VgAc6dxUvk4SPqF339PrBx8B5ObHf+BDS6 mazw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=l4jn8iSI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sd32-20020a1709076e2000b0072fb108db55si993473ejc.895.2022.10.27.02.40.44; Thu, 27 Oct 2022 02:41:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=l4jn8iSI; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234897AbiJ0JaI (ORCPT + 99 others); Thu, 27 Oct 2022 05:30:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235033AbiJ0J3o (ORCPT ); Thu, 27 Oct 2022 05:29:44 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE4D4BCAE for ; Thu, 27 Oct 2022 02:29:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=DMAH9kfM4tEFqzW7U3XOFl7UPt7+R8Cga6pBO8g46JQ=; b=l4jn8iSIh7prip20Uz5pPewEvV gAa7AXNkok0jcvCOfRqC2iNwDfWiUIY7shqcgfyhSxBRXGqeuKTwaj75p4xhMsBYlOw5xW/74qv8+ fLeV3bUWua5x+jY7YDCaEG2fbLoTVFltJBlrLwREJCLDKP+K/MrWWu2k9O35zWUBTAoL07S9oCysA RirVDRX8+oY9nAzFYsEVZrIh1vQRj3LC5jkDDOdPGrXtxpBMR5VSK8Okrv4OJSRw7TFJtnrLjz3yy EBdx8JeyY2xWLRRA3YwAstCh3chPREbeufnOJDcL8QiwNwFi0M/4Qem/5pOTb9405V5M0lyTjZP76 zfyzZ4Eg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1onzCi-0005S9-EN; Thu, 27 Oct 2022 09:29:36 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 8DB39300411; Thu, 27 Oct 2022 11:29:30 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 6C0DD2C450475; Thu, 27 Oct 2022 11:29:30 +0200 (CEST) Message-ID: <20221027092842.568039454@infradead.org> User-Agent: quilt/0.66 Date: Thu, 27 Oct 2022 11:28:13 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, Kees Cook , Sami Tolvanen , Joao Moreira , Josh Poimboeuf , Mark Rutland Subject: [PATCH 1/4] objtool: Add --cfi to generate the .cfi_sites section References: <20221027092812.185993858@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747833238358900213?= X-GMAIL-MSGID: =?utf-8?q?1747833238358900213?= Add the location of all __cfi_##name symbols (as generated by kCFI) to a section such that we might re-write things at kernel boot. Notably; boot time re-hashing and FineIBT are the intended use of this. Signed-off-by: Peter Zijlstra (Intel) --- tools/objtool/builtin-check.c | 1 tools/objtool/check.c | 69 ++++++++++++++++++++++++++++++++ tools/objtool/include/objtool/builtin.h | 1 3 files changed, 71 insertions(+) --- a/tools/objtool/builtin-check.c +++ b/tools/objtool/builtin-check.c @@ -79,6 +79,7 @@ const struct option check_options[] = { OPT_BOOLEAN('s', "stackval", &opts.stackval, "validate frame pointer rules"), OPT_BOOLEAN('t', "static-call", &opts.static_call, "annotate static calls"), OPT_BOOLEAN('u', "uaccess", &opts.uaccess, "validate uaccess rules for SMAP"), + OPT_BOOLEAN(0 , "cfi", &opts.cfi, "annotate kernel control flow integrity (kCFI) function preambles"), OPT_CALLBACK_OPTARG(0, "dump", NULL, NULL, "orc", "dump metadata", parse_dump), OPT_GROUP("Options:"), --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -861,6 +861,68 @@ static int create_ibt_endbr_seal_section return 0; } +static int create_cfi_sections(struct objtool_file *file) +{ + struct section *sec, *s; + struct symbol *sym; + unsigned int *loc; + int idx; + + sec = find_section_by_name(file->elf, ".cfi_sites"); + if (sec) { + INIT_LIST_HEAD(&file->call_list); + WARN("file already has .cfi_sites section, skipping"); + return 0; + } + + idx = 0; + for_each_sec(file, s) { + if (!s->text) + continue; + + list_for_each_entry(sym, &s->symbol_list, list) { + if (sym->type != STT_FUNC) + continue; + + if (strncmp(sym->name, "__cfi_", 6)) + continue; + + idx++; + } + } + + sec = elf_create_section(file->elf, ".cfi_sites", 0, sizeof(unsigned int), idx); + if (!sec) + return -1; + + idx = 0; + for_each_sec(file, s) { + if (!s->text) + continue; + + list_for_each_entry(sym, &s->symbol_list, list) { + if (sym->type != STT_FUNC) + continue; + + if (strncmp(sym->name, "__cfi_", 6)) + continue; + + loc = (unsigned int *)sec->data->d_buf + idx; + memset(loc, 0, sizeof(unsigned int)); + + if (elf_add_reloc_to_insn(file->elf, sec, + idx * sizeof(unsigned int), + R_X86_64_PC32, + s, sym->offset)) + return -1; + + idx++; + } + } + + return 0; +} + static int create_mcount_loc_sections(struct objtool_file *file) { struct section *sec; @@ -4397,6 +4459,13 @@ int check(struct objtool_file *file) if (ret < 0) goto out; warnings += ret; + } + + if (opts.cfi) { + ret = create_cfi_sections(file); + if (ret < 0) + goto out; + warnings += ret; } if (opts.rethunk) { --- a/tools/objtool/include/objtool/builtin.h +++ b/tools/objtool/include/objtool/builtin.h @@ -26,6 +26,7 @@ struct opts { bool stackval; bool static_call; bool uaccess; + bool cfi; /* options: */ bool backtrace; From patchwork Thu Oct 27 09:28:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11695 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp129975wru; Thu, 27 Oct 2022 02:41:11 -0700 (PDT) X-Google-Smtp-Source: AMsMyM66ca7ZqHvQx8/nbuwqhecT5nc2su3P68+/dmMjl4lo7LSdUarNvyvcgh7p/lvChagO/g/z X-Received: by 2002:a17:907:6e93:b0:78d:dff1:71e3 with SMTP id sh19-20020a1709076e9300b0078ddff171e3mr39489114ejc.94.1666863671449; Thu, 27 Oct 2022 02:41:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666863671; cv=none; d=google.com; s=arc-20160816; b=qnocWXmKRsBby0HPBxx04BGTeAlEDqu+sf+iASauhJW6TbwXVgeWXYKsbnF6v4K6D0 GpQSO0ljmtC6lWPLsoY2ma1e91Vf6jcqpXr48TDKlNiMZvisC6Di+Gb5GcDIdF70d9e1 jAzAUJwimfq4+PwEzIQqD179rjJniL62J338TlVTSrgkV1v9DyxPCZmGVXEJylag2RUG p1DS0NN0k8+4turwiqL7OQDwn7iOw/2SiD7AZ3V4M7XW1GkdpQFi9+mshZ9TVt6KaLDP f39bvyAbQnt0j25eNtKwP3rdpb6tcjjnCSKpdRIun+H6EiYjFxAK0spy5auoAQU4fcOI FoQQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=m3y/33VRA8YmYNWMBo8ucrhq6F3FIEDO+TdLyfnXiGY=; b=M+X8JIsgrZD9IfjXtV68jQOSAkaupGu6ecbvIWYB2sDwX+gqQLGc5j2AelmIU8Cs9P oe7xZauPpVJF4PjV/kUqnUw9xQunx0YWW+9Ate9EOqOLHRBrLIFr2CXlFfh7TrCdzi63 NpdVIU0nZigBsuAn17KSXnXZv3OTbg4G708M6Nk2h3xJLTR1bxz2P4oZ9yYtyoEx1C2o hW1hSQVnIsd/RWt+RYkXiP1CM23V6gFOiguGHOLgUNpNqmm1jlRFaDyOg5jXpW1aqmK1 Gx3WXvamRDMEG+NvuoXSnxoa5+5o/j2CsEXjDGXhXFM8KRHVfTImiK4s06fnLX6KoBVf cHjA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=Uo+3pNiN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id sc29-20020a1709078a1d00b007418a1e877dsi433183ejc.580.2022.10.27.02.40.46; Thu, 27 Oct 2022 02:41:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=Uo+3pNiN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235292AbiJ0JaN (ORCPT + 99 others); Thu, 27 Oct 2022 05:30:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235112AbiJ0J3o (ORCPT ); Thu, 27 Oct 2022 05:29:44 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE53CBE04 for ; Thu, 27 Oct 2022 02:29:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=m3y/33VRA8YmYNWMBo8ucrhq6F3FIEDO+TdLyfnXiGY=; b=Uo+3pNiNZMTO6marCHPFB5wWQd 3CTnuygzl0nDmtDGduUzOHAuoVgM7/c5Ho4xTkkuknDsjoBqV+ah9Tf8BN5v1dvgMpAtMgO5Eeatm KI3lNjZqGnFriRY50OTsscWKSMj7ujfWaHfX5OOor5q10dpzfRh2l/Zadx6jNID2YNQdcAFYbRwin UynN/NMGfzGZXVbPHDtOvGjI80xuRcx5ydO049QYNvjsPnFWC6iEbJTg9gB9/qKUy7EI/+mszfU0s h/ID0obxM82mORZ6VPhbNtMwqlxcaIgLz7jMC3rNVzW9WbNX0VXXUvvhcNHszNOldZKGJCX5mlwJu yYr5EXOg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1onzCd-006p4M-3I; Thu, 27 Oct 2022 09:29:31 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 8A439300322; Thu, 27 Oct 2022 11:29:30 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 716B12C450474; Thu, 27 Oct 2022 11:29:30 +0200 (CEST) Message-ID: <20221027092842.634714496@infradead.org> User-Agent: quilt/0.66 Date: Thu, 27 Oct 2022 11:28:14 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, Kees Cook , Sami Tolvanen , Joao Moreira , Josh Poimboeuf , Mark Rutland Subject: [PATCH 2/4] x86/ibt: Implement FineIBT References: <20221027092812.185993858@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747833219457383404?= X-GMAIL-MSGID: =?utf-8?q?1747833241182992644?= Implement an alternative CFI scheme that merges both the fine-grained nature of kCFI but also takes full advantage of the coarse grained hardware CFI as provided by IBT. To contrast: kCFI is a pure software CFI scheme and relies on being able to read text -- specifically the instruction *before* the target symbol, and does the hash validation *before* doing the call (otherwise control flow is compromised already). FineIBT is a software and hardware hybrid scheme; by ensuring every branch target starts with a hash validation it is possible to place the hash validation after the branch. This has several advantages: o the (hash) load is avoided; no memop; no RX requirement. o IBT WAIT-FOR-ENDBR state is a speculation stop; by placing the hash validation in the immediate instruction after the branch target there is a minimal speculation window and the whole is a viable defence against SpectreBHB. o Kees feels obliged to mention it is slightly more vulnerable when the attacker can write code. Obviously this patch relies on kCFI, but additionally it also relies on the padding from the call-depth-tracking patches. It uses this padding to place the hash-validation while the call-sites are re-written to modify the indirect target to be 16 bytes in front of the original target, thus hitting this new preamble. Notably, there is no hardware that needs call-depth-tracking (Skylake) and supports IBT (Tigerlake and onwards). Suggested-by: Joao Moreira (Intel) Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/um/kernel/um_arch.c | 5 arch/x86/Kconfig | 12 + arch/x86/Makefile | 2 arch/x86/include/asm/alternative.h | 2 arch/x86/include/asm/linkage.h | 6 arch/x86/kernel/alternative.c | 253 +++++++++++++++++++++++++++++++++++-- arch/x86/kernel/cpu/common.c | 1 arch/x86/kernel/module.c | 20 ++ arch/x86/kernel/vmlinux.lds.S | 9 + include/linux/bpf.h | 2 scripts/Makefile.lib | 1 11 files changed, 293 insertions(+), 20 deletions(-) --- a/arch/um/kernel/um_arch.c +++ b/arch/um/kernel/um_arch.c @@ -444,6 +444,11 @@ void apply_returns(s32 *start, s32 *end) { } +void apply_fineibt(s32 *start_retpoline, s32 *end_retpoline, + s32 *start_cfi, s32 *end_cfi) +{ +} + void apply_alternatives(struct alt_instr *start, struct alt_instr *end) { } --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -2463,13 +2463,23 @@ config FUNCTION_PADDING_BYTES default FUNCTION_PADDING_CFI if CFI_CLANG default FUNCTION_ALIGNMENT +config CALL_PADDING + def_bool n + depends on CC_HAS_ENTRY_PADDING && OBJTOOL + select FUNCTION_ALIGNMENT_16B + +config FINEIBT + def_bool y + depends on X86_KERNEL_IBT && CFI_CLANG && RETPOLINE + select CALL_PADDING + config HAVE_CALL_THUNKS def_bool y depends on CC_HAS_ENTRY_PADDING && RETHUNK && OBJTOOL config CALL_THUNKS def_bool n - select FUNCTION_ALIGNMENT_16B + select CALL_PADDING menuconfig SPECULATION_MITIGATIONS bool "Mitigations for speculative execution vulnerabilities" --- a/arch/x86/Makefile +++ b/arch/x86/Makefile @@ -208,7 +208,7 @@ ifdef CONFIG_SLS KBUILD_CFLAGS += -mharden-sls=all endif -ifdef CONFIG_CALL_THUNKS +ifdef CONFIG_CALL_PADDING PADDING_CFLAGS := -fpatchable-function-entry=$(CONFIG_FUNCTION_PADDING_BYTES),$(CONFIG_FUNCTION_PADDING_BYTES) KBUILD_CFLAGS += $(PADDING_CFLAGS) export PADDING_CFLAGS --- a/arch/x86/include/asm/alternative.h +++ b/arch/x86/include/asm/alternative.h @@ -78,6 +78,8 @@ extern void apply_alternatives(struct al extern void apply_retpolines(s32 *start, s32 *end); extern void apply_returns(s32 *start, s32 *end); extern void apply_ibt_endbr(s32 *start, s32 *end); +extern void apply_fineibt(s32 *start_retpoline, s32 *end_retpoine, + s32 *start_cfi, s32 *end_cfi); struct module; struct paravirt_patch_site; --- a/arch/x86/include/asm/linkage.h +++ b/arch/x86/include/asm/linkage.h @@ -15,7 +15,7 @@ #define __ALIGN .balign CONFIG_FUNCTION_ALIGNMENT, 0x90; #define __ALIGN_STR __stringify(__ALIGN) -#if defined(CONFIG_CALL_THUNKS) && !defined(__DISABLE_EXPORTS) && !defined(BUILD_VDSO) +#if defined(CONFIG_CALL_PADDING) && !defined(__DISABLE_EXPORTS) && !defined(BUILD_VDSO) #define FUNCTION_PADDING .skip CONFIG_FUNCTION_ALIGNMENT, 0x90; #else #define FUNCTION_PADDING @@ -57,7 +57,7 @@ #endif /* __ASSEMBLY__ */ /* - * Depending on -fpatchable-function-entry=N,N usage (CONFIG_CALL_THUNKS) the + * Depending on -fpatchable-function-entry=N,N usage (CONFIG_CALL_PADDING) the * CFI symbol layout changes. * * Without CALL_THUNKS: @@ -81,7 +81,7 @@ * In both cases the whole thing is FUNCTION_ALIGNMENT aligned and sized. */ -#ifdef CONFIG_CALL_THUNKS +#ifdef CONFIG_CALL_PADDING #define CFI_PRE_PADDING #define CFI_POST_PADDING .skip CONFIG_FUNCTION_PADDING_BYTES, 0x90; #else --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -116,6 +116,7 @@ static void __init_or_module add_nops(vo extern s32 __retpoline_sites[], __retpoline_sites_end[]; extern s32 __return_sites[], __return_sites_end[]; +extern s32 __cfi_sites[], __cfi_sites_end[]; extern s32 __ibt_endbr_seal[], __ibt_endbr_seal_end[]; extern struct alt_instr __alt_instructions[], __alt_instructions_end[]; extern s32 __smp_locks[], __smp_locks_end[]; @@ -656,6 +657,28 @@ void __init_or_module noinline apply_ret #ifdef CONFIG_X86_KERNEL_IBT +static void poison_endbr(void *addr, bool warn) +{ + u32 endbr, poison = gen_endbr_poison(); + + if (WARN_ON_ONCE(get_kernel_nofault(endbr, addr))) + return; + + if (!is_endbr(endbr)) { + WARN_ON_ONCE(warn); + return; + } + + DPRINTK("ENDBR at: %pS (%px)", addr, addr); + + /* + * When we have IBT, the lack of ENDBR will trigger #CP + */ + DUMP_BYTES(((u8*)addr), 4, "%px: orig: ", addr); + DUMP_BYTES(((u8*)&poison), 4, "%px: repl: ", addr); + text_poke_early(addr, &poison, 4); +} + /* * Generated by: objtool --ibt */ @@ -664,31 +687,232 @@ void __init_or_module noinline apply_ibt s32 *s; for (s = start; s < end; s++) { - u32 endbr, poison = gen_endbr_poison(); void *addr = (void *)s + *s; - if (WARN_ON_ONCE(get_kernel_nofault(endbr, addr))) - continue; + poison_endbr(addr, true); + if (IS_ENABLED(CONFIG_FINEIBT)) + poison_endbr(addr - 16, false); + } +} + +#else + +void __init_or_module noinline apply_ibt_endbr(s32 *start, s32 *end) { } + +#endif /* CONFIG_X86_KERNEL_IBT */ + +#ifdef CONFIG_FINEIBT +/* + * kCFI FineIBT + * + * __cfi_\func: __cfi_\func: + * movl $0x12345678,%eax // 5 endbr64 // 4 + * nop subl $0x12345678,%r10d // 7 + * nop jz 1f // 2 + * nop ud2 // 2 + * nop 1: nop // 1 + * nop + * nop + * nop + * nop + * nop + * nop + * nop + * + * + * caller: caller: + * movl $(-0x12345678),%r10d // 6 movl $0x12345678,%r10d // 6 + * addl $-15(%r11),%r10d // 4 sub $16,%r11 // 4 + * je 1f // 2 nop4 // 4 + * ud2 // 2 + * 1: call __x86_indirect_thunk_r11 // 5 call *%r11; nop2; // 5 + * + */ + +asm( ".pushsection .rodata \n" + "fineibt_preamble_start: \n" + " endbr64 \n" + " subl $0x12345678, %r10d \n" + " je fineibt_preamble_end \n" + " ud2 \n" + " nop \n" + "fineibt_preamble_end: \n" + ".popsection\n" +); + +extern u8 fineibt_preamble_start[]; +extern u8 fineibt_preamble_end[]; + +#define fineibt_preamble_size (fineibt_preamble_end - fineibt_preamble_start) +#define fineibt_preamble_hash 7 + +asm( ".pushsection .rodata \n" + "fineibt_caller_start: \n" + " movl $0x12345678, %r10d \n" + " sub $16, %r11 \n" + ASM_NOP4 + "fineibt_caller_end: \n" + ".popsection \n" +); + +extern u8 fineibt_caller_start[]; +extern u8 fineibt_caller_end[]; + +#define fineibt_caller_size (fineibt_caller_end - fineibt_caller_start) +#define fineibt_caller_hash 2 + +#define fineibt_caller_jmp (fineibt_caller_size - 2) + +static u32 decode_preamble_hash(void *addr) +{ + u8 *p = addr; + + /* b8 78 56 34 12 mov $0x12345678,%eax */ + if (p[0] == 0xb8) + return *(u32 *)(addr + 1); + + return 0; /* invalid hash value */ +} + +static u32 decode_caller_hash(void *addr) +{ + u8 *p = addr; + + /* 41 ba 78 56 34 12 mov $0x12345678,%r10d */ + if (p[0] == 0x41 && p[1] == 0xba) + return -*(u32 *)(addr + 2); + + /* e8 0c 78 56 34 12 jmp.d8 +12 */ + if (p[0] == JMP8_INSN_OPCODE && p[1] == fineibt_caller_jmp) + return -*(u32 *)(addr + 2); + + return 0; /* invalid hash value */ +} + +/* .retpoline_sites */ +static int cfi_disable_callers(s32 *start, s32 *end) +{ + /* + * Disable kCFI by patching in a JMP.d8, this leaves the hash immediate + * in tact for later usage. Also see decode_caller_hash() and + * cfi_rewrite_callers(). + */ + const u8 jmp[] = { JMP8_INSN_OPCODE, fineibt_caller_jmp }; + s32 *s; - if (WARN_ON_ONCE(!is_endbr(endbr))) + for (s = start; s < end; s++) { + void *addr = (void *)s + *s; + u32 hash; + + addr -= fineibt_caller_size; + hash = decode_caller_hash(addr); + if (!hash) /* nocfi callers */ continue; - DPRINTK("ENDBR at: %pS (%px)", addr, addr); + text_poke_early(addr, jmp, 2); + } - /* - * When we have IBT, the lack of ENDBR will trigger #CP - */ - DUMP_BYTES(((u8*)addr), 4, "%px: orig: ", addr); - DUMP_BYTES(((u8*)&poison), 4, "%px: repl: ", addr); - text_poke_early(addr, &poison, 4); + return 0; +} + +/* .cfi_sites */ +static int cfi_rewrite_preamble(s32 *start, s32 *end) +{ + s32 *s; + + for (s = start; s < end; s++) { + void *addr = (void *)s + *s; + u32 hash; + + hash = decode_preamble_hash(addr); + if (WARN(!hash, "no CFI hash found at: %pS %px %*ph\n", + addr, addr, 5, addr)) + return -EINVAL; + + text_poke_early(addr, fineibt_preamble_start, fineibt_preamble_size); + WARN_ON(*(u32 *)(addr + fineibt_preamble_hash) != 0x12345678); + text_poke_early(addr + fineibt_preamble_hash, &hash, 4); } + + return 0; +} + +/* .retpoline_sites */ +static int cfi_rewrite_callers(s32 *start, s32 *end) +{ + s32 *s; + + for (s = start; s < end; s++) { + void *addr = (void *)s + *s; + u32 hash; + + addr -= fineibt_caller_size; + hash = decode_caller_hash(addr); + if (hash) { + text_poke_early(addr, fineibt_caller_start, fineibt_caller_size); + WARN_ON(*(u32 *)(addr + fineibt_caller_hash) != 0x12345678); + text_poke_early(addr + fineibt_caller_hash, &hash, 4); + } + /* rely on apply_retpolines() */ + } + + return 0; +} + +static void __apply_fineibt(s32 *start_retpoline, s32 *end_retpoline, + s32 *start_cfi, s32 *end_cfi, bool builtin) +{ + int ret; + + if (WARN_ONCE(fineibt_preamble_size != 16, + "FineIBT preamble wrong size: %ld", fineibt_preamble_size)) + return; + + if (!HAS_KERNEL_IBT || !cpu_feature_enabled(X86_FEATURE_IBT)) + return; + + /* + * Rewrite the callers to not use the __cfi_ stubs, such that we might + * rewrite them. This disables all CFI. If this succeeds but any of the + * later stages fails, we're without CFI. + */ + ret = cfi_disable_callers(start_retpoline, end_retpoline); + if (ret) + goto err; + + ret = cfi_rewrite_preamble(start_cfi, end_cfi); + if (ret) + goto err; + + ret = cfi_rewrite_callers(start_retpoline, end_retpoline); + if (ret) + goto err; + + if (builtin) + pr_info("Using FineIBT CFI\n"); + + return; + +err: + pr_err("Something went horribly wrong trying to rewrite the CFI implementation.\n"); } #else -void __init_or_module noinline apply_ibt_endbr(s32 *start, s32 *end) { } +static void __apply_fineibt(s32 *start_retpoline, s32 *end_retpoline, + s32 *start_cfi, s32 *end_cfi, bool builtin) +{ +} -#endif /* CONFIG_X86_KERNEL_IBT */ +#endif + +void apply_fineibt(s32 *start_retpoline, s32 *end_retpoline, + s32 *start_cfi, s32 *end_cfi) +{ + return __apply_fineibt(start_retpoline, end_retpoline, + start_cfi, end_cfi, + /* .builtin = */ false); +} #ifdef CONFIG_SMP static void alternatives_smp_lock(const s32 *start, const s32 *end, @@ -996,6 +1220,9 @@ void __init alternative_instructions(voi */ apply_paravirt(__parainstructions, __parainstructions_end); + __apply_fineibt(__retpoline_sites, __retpoline_sites_end, + __cfi_sites, __cfi_sites_end, true); + /* * Rewrite the retpolines, must be done before alternatives since * those can rewrite the retpoline thunks. --- a/arch/x86/kernel/cpu/common.c +++ b/arch/x86/kernel/cpu/common.c @@ -609,6 +609,7 @@ static __always_inline void setup_cet(st if (!ibt_selftest()) { pr_err("IBT selftest: Failed!\n"); + wrmsrl(MSR_IA32_S_CET, 0); setup_clear_cpu_cap(X86_FEATURE_IBT); return; } --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -255,7 +255,7 @@ int module_finalize(const Elf_Ehdr *hdr, const Elf_Shdr *s, *text = NULL, *alt = NULL, *locks = NULL, *para = NULL, *orc = NULL, *orc_ip = NULL, *retpolines = NULL, *returns = NULL, *ibt_endbr = NULL, - *calls = NULL; + *calls = NULL, *cfi = NULL; char *secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset; for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) { @@ -277,6 +277,8 @@ int module_finalize(const Elf_Ehdr *hdr, returns = s; if (!strcmp(".call_sites", secstrings + s->sh_name)) calls = s; + if (!strcmp(".cfi_sites", secstrings + s->sh_name)) + cfi = s; if (!strcmp(".ibt_endbr_seal", secstrings + s->sh_name)) ibt_endbr = s; } @@ -289,6 +291,22 @@ int module_finalize(const Elf_Ehdr *hdr, void *pseg = (void *)para->sh_addr; apply_paravirt(pseg, pseg + para->sh_size); } + if (retpolines || cfi) { + void *rseg = NULL, *cseg = NULL; + unsigned int rsize = 0, csize = 0; + + if (retpolines) { + rseg = (void *)retpolines->sh_addr; + rsize = retpolines->sh_size; + } + + if (cfi) { + cseg = (void *)cfi->sh_addr; + csize = cfi->sh_size; + } + + apply_fineibt(rseg, rseg + rsize, cseg, cseg + csize); + } if (retpolines) { void *rseg = (void *)retpolines->sh_addr; apply_retpolines(rseg, rseg + retpolines->sh_size); --- a/arch/x86/kernel/vmlinux.lds.S +++ b/arch/x86/kernel/vmlinux.lds.S @@ -309,6 +309,15 @@ SECTIONS } #endif +#ifdef CONFIG_FINEIBT + . = ALIGN(8); + .cfi_sites : AT(ADDR(.cfi_sites) - LOAD_OFFSET) { + __cfi_sites = .; + *(.cfi_sites) + __cfi_sites_end = .; + } +#endif + /* * struct alt_inst entries. From the header (alternative.h): * "Alternative instructions for different CPU types or capabilities" --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -984,7 +984,7 @@ int arch_prepare_bpf_dispatcher(void *im } #ifdef CONFIG_X86_64 -#ifdef CONFIG_CALL_THUNKS +#ifdef CONFIG_CALL_PADDING #define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5+CONFIG_FUNCTION_PADDING_BYTES,CONFIG_FUNCTION_PADDING_BYTES))) #else #define BPF_DISPATCHER_ATTRIBUTES __attribute__((patchable_function_entry(5))) --- a/scripts/Makefile.lib +++ b/scripts/Makefile.lib @@ -256,6 +256,7 @@ objtool-args-$(CONFIG_HAVE_JUMP_LABEL_HA objtool-args-$(CONFIG_HAVE_NOINSTR_HACK) += --hacks=noinstr objtool-args-$(CONFIG_CALL_DEPTH_TRACKING) += --hacks=skylake objtool-args-$(CONFIG_X86_KERNEL_IBT) += --ibt +objtool-args-$(CONFIG_FINEIBT) += --cfi objtool-args-$(CONFIG_FTRACE_MCOUNT_USE_OBJTOOL) += --mcount objtool-args-$(CONFIG_UNWINDER_ORC) += --orc objtool-args-$(CONFIG_RETPOLINE) += --retpoline From patchwork Thu Oct 27 09:28:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11696 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp130475wru; Thu, 27 Oct 2022 02:42:44 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5S/FpoOV+Kf5DkHHlWZhFqcGS/1RJNExR35kLmF3cc555O0zj0g8Pk4yXYPVqPP5VsZug4 X-Received: by 2002:a17:907:97d5:b0:7ac:5f72:6c1a with SMTP id js21-20020a17090797d500b007ac5f726c1amr11610773ejc.126.1666863763997; Thu, 27 Oct 2022 02:42:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666863763; cv=none; d=google.com; s=arc-20160816; b=hPXZ2a1d83J2JDmaywYiNQipwE0vMnZ/lMleek2meBz9LbfE5vnxCHOG1PRbQ9+Xo0 aLyQKlr+kpXAut5ZRmeQ6waL3eZTEuaZikxdStqSZ5/738W4teN/2mtEK+I7p4AvfKSn 3q0KYz7CDSdQzm20FDE2xSe5fGzqidRXmbR5PHptLa//uknerh4surNgSZ42ucFJZ99A U8jFrNDFjrjOh4u0FqJ5oTLshBmUBaf/pdw9O3ig5TJJFsXxc1+/BT9Ocm4wafaQADtd 7l58UJjCM4vgcH63GMV3aFsbZVqVG+VVzH0Rav8sRvrbIpqs5eb71kgBygf5d7OD0iWp ANdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=vAQwSFuGwqx04mJjqCuHpC9VhBTtgDgSbSkN0/TPFU4=; b=Ogh3oGkrh77qx5OkMB/DfgMQMtrFeCI2S0h6aK6SO7/JtYKxOHUFI5MKsvn/bmbU/j H7tTpBld9ASxtDEGgA/heccOUdp0kzEoxw7oSd/gV3gsmfwLM4+yV9885XTmNwGwCvE3 iN65tMLWtUa0bHqq4Eqm0GxEAnOdSxzkx5ZgdXLnNyFcnkVlytZQkZtvyPzMTH7/dbDC fiXrZdjQgbKZKd1Tf8668LLX4EntWrbUnElHaeatuSnncnQcnpI9O68kvn5i2myp7gtM TFG4xwrU2c3ARADnZLhkXKuakSI3KFXglFiN07REsaCebKNLoPSZ0+JVVXTLWFu5KQhs R5XQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=XEGecbCN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n10-20020a50934a000000b0045da52f303csi1165975eda.213.2022.10.27.02.42.20; Thu, 27 Oct 2022 02:42:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=casper.20170209 header.b=XEGecbCN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235165AbiJ0JaE (ORCPT + 99 others); Thu, 27 Oct 2022 05:30:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235015AbiJ0J3o (ORCPT ); Thu, 27 Oct 2022 05:29:44 -0400 Received: from casper.infradead.org (casper.infradead.org [IPv6:2001:8b0:10b:1236::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD571B866 for ; Thu, 27 Oct 2022 02:29:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=vAQwSFuGwqx04mJjqCuHpC9VhBTtgDgSbSkN0/TPFU4=; b=XEGecbCNpEF5d5h1y/I1BeKWho 4ZO1769+2IMfHIyXLdUZLhEvbFYxTVQ5rDM6brI+McCX+xabG5iQ0MPM5K9W7CTXwlZUXBTz7Mjek eFmnsPRBm5sdLukARARiYoiwOV2ZOSBWn1Tk38brlIH23u5Uq5yEri7ZonWzFObhkERMbauvt0Ywu c+y+Q2aCZqZqHq8V5MLD48qilK1t98oe+Ro6AD8ntwOp9GHxk90+1PgKHFcZYiF/CSwleHzKpA0TR Pr9oVyknfSfUD6QDGfg4GMCSmfIQtXQyjcw30V4k+ROmO2n0sfu51A3nUEAqxERT9f14O7+8QyZBI yOpQ2ITg==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1onzCi-0005SA-EL; Thu, 27 Oct 2022 09:29:36 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 8F61930041D; Thu, 27 Oct 2022 11:29:30 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 7724E2C450395; Thu, 27 Oct 2022 11:29:30 +0200 (CEST) Message-ID: <20221027092842.699804264@infradead.org> User-Agent: quilt/0.66 Date: Thu, 27 Oct 2022 11:28:15 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, Kees Cook , Sami Tolvanen , Joao Moreira , Josh Poimboeuf , Mark Rutland Subject: [PATCH 3/4] x86/cfi: Boot time selection of CFI scheme References: <20221027092812.185993858@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747833338425665581?= X-GMAIL-MSGID: =?utf-8?q?1747833338425665581?= Add the "cfi=" boot parameter to allow people to select a CFI scheme at boot time. Mostly useful for development / debugging. Requested-by: Kees Cook Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/kernel/alternative.c | 103 +++++++++++++++++++++++++++++++++--------- 1 file changed, 83 insertions(+), 20 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -702,6 +702,47 @@ void __init_or_module noinline apply_ibt #endif /* CONFIG_X86_KERNEL_IBT */ #ifdef CONFIG_FINEIBT + +enum cfi_mode { + CFI_DEFAULT, + CFI_OFF, + CFI_KCFI, + CFI_FINEIBT, +}; + +static enum cfi_mode cfi_mode __ro_after_init = CFI_DEFAULT; + +static __init int cfi_parse_cmdline(char *str) +{ + if (!str) + return -EINVAL; + + while (str) { + char *next = strchr(str, ','); + if (next) { + *next = 0; + next++; + } + + if (!strcmp(str, "auto")) { + cfi_mode = CFI_DEFAULT; + } else if (!strcmp(str, "off")) { + cfi_mode = CFI_OFF; + } else if (!strcmp(str, "kcfi")) { + cfi_mode = CFI_KCFI; + } else if (!strcmp(str, "fineibt")) { + cfi_mode = CFI_FINEIBT; + } else { + pr_err("Ignoring unknown cfi option (%s).", str); + } + + str = next; + } + + return 0; +} +early_param("cfi", cfi_parse_cmdline); + /* * kCFI FineIBT * @@ -868,30 +909,52 @@ static void __apply_fineibt(s32 *start_r "FineIBT preamble wrong size: %ld", fineibt_preamble_size)) return; - if (!HAS_KERNEL_IBT || !cpu_feature_enabled(X86_FEATURE_IBT)) + if (cfi_mode == CFI_DEFAULT) { + cfi_mode = CFI_KCFI; + if (HAS_KERNEL_IBT && cpu_feature_enabled(X86_FEATURE_IBT)) + cfi_mode = CFI_FINEIBT; + } + + switch (cfi_mode) { + case CFI_OFF: + ret = cfi_disable_callers(start_retpoline, end_retpoline); + if (ret) + goto err; + + if (builtin) + pr_info("Disabling CFI\n"); return; - /* - * Rewrite the callers to not use the __cfi_ stubs, such that we might - * rewrite them. This disables all CFI. If this succeeds but any of the - * later stages fails, we're without CFI. - */ - ret = cfi_disable_callers(start_retpoline, end_retpoline); - if (ret) - goto err; - - ret = cfi_rewrite_preamble(start_cfi, end_cfi); - if (ret) - goto err; - - ret = cfi_rewrite_callers(start_retpoline, end_retpoline); - if (ret) - goto err; + case CFI_KCFI: + if (builtin) + pr_info("Using kCFI\n"); + return; - if (builtin) - pr_info("Using FineIBT CFI\n"); + case CFI_FINEIBT: + /* + * Rewrite the callers to not use the __cfi_ stubs, such that we might + * rewrite them. This disables all CFI. If this succeeds but any of the + * later stages fails, we're without CFI. + */ + ret = cfi_disable_callers(start_retpoline, end_retpoline); + if (ret) + goto err; + + ret = cfi_rewrite_preamble(start_cfi, end_cfi); + if (ret) + goto err; + + ret = cfi_rewrite_callers(start_retpoline, end_retpoline); + if (ret) + goto err; - return; + if (builtin) + pr_info("Using FineIBT CFI\n"); + return; + + default: + break; + } err: pr_err("Something went horribly wrong trying to rewrite the CFI implementation.\n"); From patchwork Thu Oct 27 09:28:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11697 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp130497wru; Thu, 27 Oct 2022 02:42:47 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6NblFJ9VGNHv4pXGiB3vuhvKyVoL2ec593QLK5mL/+YSx4hk8qqj5+/cm+4jkwqPin4yRi X-Received: by 2002:a17:907:72d6:b0:79b:8cce:112a with SMTP id du22-20020a17090772d600b0079b8cce112amr27640937ejc.118.1666863767067; Thu, 27 Oct 2022 02:42:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666863767; cv=none; d=google.com; s=arc-20160816; b=wet+au2JrKeUPov6RqMWma3QV76R1iqDVD8BkU3r8a00jTuhfryQcslmbQIO0GWVEr itBtuRn/OEStva4zrifZ7ZDjfaslH5vLMXsUi3Ng2iN+vc6xoycJqNPVzmQhkDXqMJiq IU8SkaXNmkilBN8bViBTWBT0NPKKvCh5karZJHpoY4qkOAKO6etnnuJaGDVqPxU4Zzic jCpKPJ3DP+ujy2qtTxPjQnce9kSJh8C7OpWUscLt1DtEC7gh1vV3HoXD166Eqd/gHY30 9bnIHk4cbzsdapZZS1lYI2Gi8qVUYuS7XdQXDWBKubLmMPEOLBhnlxOxbbirhJ+WV+73 saGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:references:subject:cc:to:from:date :user-agent:message-id:dkim-signature; bh=PZvHoRnYLkmlxJjyek8hLDrtaYqtEqTyI6XZhXt7/Jk=; b=hJlA0JMY6R2MGf8OzL69l4q0UrczfFrQxETIQ5Pp0iQZh/5Zbr8Ig25nncgcxD60QD Qw4gMBbZS/PNJT6SWozjviXpoDIAHs8Fc7Nm2OCHV1ixGUkaCAatjn9wHnyy8GeD7Zz/ uPRNXjC7rNGwzw8DkoQav3aBXSPpzyy0xC8AMUIFQQnoFxm2/6jHICXb4L1oMNkdurI5 43KXixcfUHm6tKOSMBlzGdlbvnNTpTvrVpRhCgfDl5YC3tuwpMHuXztJbR3P3MHSkudU WJXpnM+/nqdqQKtNNt4h/9+45HGR7SMfsu9VouFxn+1w4JqbQQNvoPTu6QyVJM7ZH5zB di3Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=NRT8TsHh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m2-20020a170906720200b007919388d2c6si906220ejk.357.2022.10.27.02.42.20; Thu, 27 Oct 2022 02:42:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=desiato.20200630 header.b=NRT8TsHh; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234908AbiJ0J35 (ORCPT + 99 others); Thu, 27 Oct 2022 05:29:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234985AbiJ0J3o (ORCPT ); Thu, 27 Oct 2022 05:29:44 -0400 Received: from desiato.infradead.org (desiato.infradead.org [IPv6:2001:8b0:10b:1:d65d:64ff:fe57:4e05]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 852D8B48A for ; Thu, 27 Oct 2022 02:29:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=PZvHoRnYLkmlxJjyek8hLDrtaYqtEqTyI6XZhXt7/Jk=; b=NRT8TsHhHmJXeBEeGB/xo3YHig aiHGaiZnT1HEwqIalZqrgZTFO+iFDJhwS42V7jGAXUUeViKhGEmcUQJLJkb7ND8qGa9OqKOjqb9Gl GDyJwpULNDuLYIar7LqL0046DKDnWsaVvGFpHBGT5w/EfbTPo1PP7RXtRPwsy7jrh389saYNc2STx jyVSG3Y2M1if+AskXZ+NRvUHNxIU/M4SFRDn9d+/cgCZa3if70TRLGYe+0xDzjaPCMj++6jf5GPrV l9vuKE9uff9hyR5B2CHDuzweXYD4M56albzMFTW4LuvC8XDNoqY4BKbUlq0Y/mtcDebsR7RbpHcD3 AOAqe+cw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by desiato.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1onzCd-006p4L-2m; Thu, 27 Oct 2022 09:29:31 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 910FB300454; Thu, 27 Oct 2022 11:29:30 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 7C0652C450476; Thu, 27 Oct 2022 11:29:30 +0200 (CEST) Message-ID: <20221027092842.765195516@infradead.org> User-Agent: quilt/0.66 Date: Thu, 27 Oct 2022 11:28:16 +0200 From: Peter Zijlstra To: x86@kernel.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, Kees Cook , Sami Tolvanen , Joao Moreira , Josh Poimboeuf , Mark Rutland Subject: [PATCH 4/4] x86/cfi: Add boot time hash randomization References: <20221027092812.185993858@infradead.org> MIME-Version: 1.0 X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747833341601337866?= X-GMAIL-MSGID: =?utf-8?q?1747833341601337866?= In order to avoid known hashes (from knowing the boot image), randomize the CFI hashes with a per-boot random seed. Suggested-by: Kees Cook Signed-off-by: Peter Zijlstra (Intel) Reviewed-by: Kees Cook --- arch/x86/kernel/alternative.c | 120 +++++++++++++++++++++++++++++++++++++----- 1 file changed, 108 insertions(+), 12 deletions(-) --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -711,6 +711,24 @@ enum cfi_mode { }; static enum cfi_mode cfi_mode __ro_after_init = CFI_DEFAULT; +static bool cfi_rand __ro_after_init = true; +static u32 cfi_seed __ro_after_init; + +/* + * Re-hash the CFI hash with a boot-time seed while making sure the result is + * not a valid ENDBR instruction. + */ +static u32 cfi_rehash(u32 hash) +{ + hash ^= cfi_seed; + while (unlikely(is_endbr(hash) || is_endbr(-hash))) { + bool lsb = hash & 1; + hash >>= 1; + if (lsb) + hash ^= 0x80200003; + } + return hash; +} static __init int cfi_parse_cmdline(char *str) { @@ -728,10 +746,13 @@ static __init int cfi_parse_cmdline(char cfi_mode = CFI_DEFAULT; } else if (!strcmp(str, "off")) { cfi_mode = CFI_OFF; + cfi_rand = false; } else if (!strcmp(str, "kcfi")) { cfi_mode = CFI_KCFI; } else if (!strcmp(str, "fineibt")) { cfi_mode = CFI_FINEIBT; + } else if (!strcmp(str, "norand")) { + cfi_rand = false; } else { pr_err("Ignoring unknown cfi option (%s).", str); } @@ -856,7 +877,50 @@ static int cfi_disable_callers(s32 *star return 0; } +static int cfi_enable_callers(s32 *start, s32 *end) +{ + /* + * Re-enable kCFI, undo what cfi_disable_callers() did. + */ + const u8 mov[] = { 0x41, 0xba }; + s32 *s; + + for (s = start; s < end; s++) { + void *addr = (void *)s + *s; + u32 hash; + + addr -= fineibt_caller_size; + hash = decode_caller_hash(addr); + if (!hash) /* nocfi callers */ + continue; + + text_poke_early(addr, mov, 2); + } + + return 0; +} + /* .cfi_sites */ +static int cfi_rand_preamble(s32 *start, s32 *end) +{ + s32 *s; + + for (s = start; s < end; s++) { + void *addr = (void *)s + *s; + u32 hash; + + hash = decode_preamble_hash(addr); + if (WARN(!hash, "no CFI hash found at: %pS %px %*ph\n", + addr, addr, 5, addr)) + return -EINVAL; + + hash = cfi_rehash(hash); + text_poke_early(addr + 1, &hash, 4); + } + + return 0; +} + static int cfi_rewrite_preamble(s32 *start, s32 *end) { s32 *s; @@ -879,6 +943,25 @@ static int cfi_rewrite_preamble(s32 *sta } /* .retpoline_sites */ +static int cfi_rand_callers(s32 *start, s32 *end) +{ + s32 *s; + + for (s = start; s < end; s++) { + void *addr = (void *)s + *s; + u32 hash; + + addr -= fineibt_caller_size; + hash = decode_caller_hash(addr); + if (hash) { + hash = -cfi_rehash(hash); + text_poke_early(addr + 2, &hash, 4); + } + } + + return 0; +} + static int cfi_rewrite_callers(s32 *start, s32 *end) { s32 *s; @@ -915,31 +998,44 @@ static void __apply_fineibt(s32 *start_r cfi_mode = CFI_FINEIBT; } - switch (cfi_mode) { - case CFI_OFF: - ret = cfi_disable_callers(start_retpoline, end_retpoline); + /* + * Rewrite the callers to not use the __cfi_ stubs, such that we might + * rewrite them. This disables all CFI. If this succeeds but any of the + * later stages fails, we're without CFI. + */ + ret = cfi_disable_callers(start_retpoline, end_retpoline); + if (ret) + goto err; + + if (cfi_rand) { + if (builtin) + cfi_seed = get_random_u32(); + + ret = cfi_rand_preamble(start_cfi, end_cfi); if (ret) goto err; + ret = cfi_rand_callers(start_retpoline, end_retpoline); + if (ret) + goto err; + } + + switch (cfi_mode) { + case CFI_OFF: if (builtin) pr_info("Disabling CFI\n"); return; case CFI_KCFI: + ret = cfi_enable_callers(start_retpoline, end_retpoline); + if (ret) + goto err; + if (builtin) pr_info("Using kCFI\n"); return; case CFI_FINEIBT: - /* - * Rewrite the callers to not use the __cfi_ stubs, such that we might - * rewrite them. This disables all CFI. If this succeeds but any of the - * later stages fails, we're without CFI. - */ - ret = cfi_disable_callers(start_retpoline, end_retpoline); - if (ret) - goto err; - ret = cfi_rewrite_preamble(start_cfi, end_cfi); if (ret) goto err;