From patchwork Wed Jul 5 18:12:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116294 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2060701vqx; Wed, 5 Jul 2023 11:34:05 -0700 (PDT) X-Google-Smtp-Source: APBJJlFk1Uvq8Ur0g+RcAUjEC9s9MTohhmCT7i1+Wp0/jofLdikzbMIuA+uzOAif5dW1Gn7wgQ6i X-Received: by 2002:a05:6a20:42a6:b0:125:29e5:ae53 with SMTP id o38-20020a056a2042a600b0012529e5ae53mr21668297pzj.62.1688582045258; Wed, 05 Jul 2023 11:34:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688582045; cv=none; d=google.com; s=arc-20160816; b=eprlgVRPdcSVXbCYYWMivv1/LyEBx/nwyGC6bRHW9YcjtGrOOs7o+1HUNrd4VxkHKm hX7zQneSIVHaarpqLinwqT9DvADL/88t0MN6znrQxaMHG3lpVnOgoTsyN47e+6eOh9iC VeVGZaIxC1vox2RyhH86k0c3Zh7Rdg/qCH56qWV2E/WldgxwUSB4leSxxwhuezoqWrdh goxLpUTI64Jq0fribHK77AkIPjQlWnTRqSG8NmrGUVW5aPiNORbPvfwT2nolq5PGZJr7 DUzeYxnohiGo2jbfZNL76OjcCc3r8AiOAmEEuWqeqJrV+3fcSvlHy9oARov9yzGjMwgX h09w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=cr2h91iK1BhRzl41+qr9b8AIAdoW4uuXRZNrsLrfkns=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=H04+IENjZEw8xuqfkd2wf9xxYh5uU2ZkUyaZTPv8AWP1hAh3LY0g8zzUl8miLUKt46 deMgzwCOpGOGKxbBDOCg65RJjpL9EdZYajVebLNLmLvtyfbkhTdIQ2Ojvthhg+O++sDM H/WZglwmGPRaEVPjUCOXhM4Svj6Y9AaMz3nQzzInTfi9IMt/kIGpHI55F/BORgB+T9ZT 54V2E8ejH6DvR6hkfIWKMBffhqgaVBg02GQAukPNkQHlo93wTfvRxHZgDjzMjEqgDFPk Z7gQfMmkbMCg7x6PVPeo5VTa8Yy3ydEoW7DqN54gfsjBjhWJDz6h4o6U/2MkLxQwyw1I F3Kw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=D03arb0V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k14-20020a056a00168e00b00656f1d69ec6si24676275pfc.292.2023.07.05.11.33.51; Wed, 05 Jul 2023 11:34:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=D03arb0V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232464AbjGESRN (ORCPT + 99 others); Wed, 5 Jul 2023 14:17:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231913AbjGESRL (ORCPT ); Wed, 5 Jul 2023 14:17:11 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C3DA01994 for ; Wed, 5 Jul 2023 11:15:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688580951; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cr2h91iK1BhRzl41+qr9b8AIAdoW4uuXRZNrsLrfkns=; b=D03arb0VUq+IKd2+0zIjSGXczU8xicqva5koEyrs/4XMtdVrnlWWf60qMtTXoutca84FeW z0bQHqmtL9z++sq5R1auswYGNB5sOLzR5F68Myoi0/cxwQ5bdrTk3davXr35ivViLGLhV6 NQSnWnizYK9zBz/duOdpef1fI7YcA9g= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-638-Nhv5uZDOOuONs0L1KaIvxQ-1; Wed, 05 Jul 2023 14:15:46 -0400 X-MC-Unique: Nhv5uZDOOuONs0L1KaIvxQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5FD79800C7F; Wed, 5 Jul 2023 18:15:44 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 0565A18EB4; Wed, 5 Jul 2023 18:15:37 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 01/14] tracing/filters: Dynamically allocate filter_pred.regex Date: Wed, 5 Jul 2023 19:12:43 +0100 Message-Id: <20230705181256.3539027-2-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770606607012029277?= X-GMAIL-MSGID: =?utf-8?q?1770606607012029277?= Every predicate allocation includes a MAX_FILTER_STR_VAL (256) char array in the regex field, even if the predicate function does not use the field. A later commit will introduce a dynamically allocated cpumask to struct filter_pred, which will require a dedicated freeing function. Bite the bullet and make filter_pred.regex dynamically allocated. While at it, reorder the fields of filter_pred to fill in the byte holes. The struct now fits on a single cacheline. No change in behaviour intended. The kfree()'s were patched via Coccinelle: @@ struct filter_pred *pred; @@ -kfree(pred); +free_predicate(pred); Signed-off-by: Valentin Schneider --- kernel/trace/trace_events_filter.c | 62 ++++++++++++++++++------------ 1 file changed, 37 insertions(+), 25 deletions(-) diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index 1dad64267878c..d999a218fe833 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -70,15 +70,15 @@ enum filter_pred_fn { }; struct filter_pred { - enum filter_pred_fn fn_num; - u64 val; - u64 val2; - struct regex regex; + struct regex *regex; unsigned short *ops; struct ftrace_event_field *field; - int offset; + u64 val; + u64 val2; + enum filter_pred_fn fn_num; + int offset; int not; - int op; + int op; }; /* @@ -186,6 +186,12 @@ enum { PROCESS_OR = 4, }; +static void free_predicate(struct filter_pred *pred) +{ + kfree(pred->regex); + kfree(pred); +} + /* * Without going into a formal proof, this explains the method that is used in * parsing the logical expressions. @@ -623,7 +629,7 @@ predicate_parse(const char *str, int nr_parens, int nr_preds, kfree(inverts); if (prog_stack) { for (i = 0; prog_stack[i].pred; i++) - kfree(prog_stack[i].pred); + free_predicate(prog_stack[i].pred); kfree(prog_stack); } return ERR_PTR(ret); @@ -750,7 +756,7 @@ static int filter_pred_string(struct filter_pred *pred, void *event) char *addr = (char *)(event + pred->offset); int cmp, match; - cmp = pred->regex.match(addr, &pred->regex, pred->regex.field_len); + cmp = pred->regex->match(addr, pred->regex, pred->regex->field_len); match = cmp ^ pred->not; @@ -763,7 +769,7 @@ static __always_inline int filter_pchar(struct filter_pred *pred, char *str) int len; len = strlen(str) + 1; /* including tailing '\0' */ - cmp = pred->regex.match(str, &pred->regex, len); + cmp = pred->regex->match(str, pred->regex, len); match = cmp ^ pred->not; @@ -813,7 +819,7 @@ static int filter_pred_strloc(struct filter_pred *pred, void *event) char *addr = (char *)(event + str_loc); int cmp, match; - cmp = pred->regex.match(addr, &pred->regex, str_len); + cmp = pred->regex->match(addr, pred->regex, str_len); match = cmp ^ pred->not; @@ -836,7 +842,7 @@ static int filter_pred_strrelloc(struct filter_pred *pred, void *event) char *addr = (char *)(&item[1]) + str_loc; int cmp, match; - cmp = pred->regex.match(addr, &pred->regex, str_len); + cmp = pred->regex->match(addr, pred->regex, str_len); match = cmp ^ pred->not; @@ -874,7 +880,7 @@ static int filter_pred_comm(struct filter_pred *pred, void *event) { int cmp; - cmp = pred->regex.match(current->comm, &pred->regex, + cmp = pred->regex->match(current->comm, pred->regex, TASK_COMM_LEN); return cmp ^ pred->not; } @@ -1004,7 +1010,7 @@ enum regex_type filter_parse_regex(char *buff, int len, char **search, int *not) static void filter_build_regex(struct filter_pred *pred) { - struct regex *r = &pred->regex; + struct regex *r = pred->regex; char *search; enum regex_type type = MATCH_FULL; @@ -1169,7 +1175,7 @@ static void free_prog(struct event_filter *filter) return; for (i = 0; prog[i].pred; i++) - kfree(prog[i].pred); + free_predicate(prog[i].pred); kfree(prog); } @@ -1553,9 +1559,12 @@ static int parse_pred(const char *str, void *data, goto err_free; } - pred->regex.len = len; - strncpy(pred->regex.pattern, str + s, len); - pred->regex.pattern[len] = 0; + pred->regex = kzalloc(sizeof(*pred->regex), GFP_KERNEL); + if (!pred->regex) + goto err_mem; + pred->regex->len = len; + strncpy(pred->regex->pattern, str + s, len); + pred->regex->pattern[len] = 0; /* This is either a string, or an integer */ } else if (str[i] == '\'' || str[i] == '"') { @@ -1597,9 +1606,12 @@ static int parse_pred(const char *str, void *data, goto err_free; } - pred->regex.len = len; - strncpy(pred->regex.pattern, str + s, len); - pred->regex.pattern[len] = 0; + pred->regex = kzalloc(sizeof(*pred->regex), GFP_KERNEL); + if (!pred->regex) + goto err_mem; + pred->regex->len = len; + strncpy(pred->regex->pattern, str + s, len); + pred->regex->pattern[len] = 0; filter_build_regex(pred); @@ -1608,7 +1620,7 @@ static int parse_pred(const char *str, void *data, } else if (field->filter_type == FILTER_STATIC_STRING) { pred->fn_num = FILTER_PRED_FN_STRING; - pred->regex.field_len = field->size; + pred->regex->field_len = field->size; } else if (field->filter_type == FILTER_DYN_STRING) { pred->fn_num = FILTER_PRED_FN_STRLOC; @@ -1691,10 +1703,10 @@ static int parse_pred(const char *str, void *data, return i; err_free: - kfree(pred); + free_predicate(pred); return -EINVAL; err_mem: - kfree(pred); + free_predicate(pred); return -ENOMEM; } @@ -2287,8 +2299,8 @@ static int ftrace_function_set_filter_pred(struct filter_pred *pred, return ret; return __ftrace_function_set_filter(pred->op == OP_EQ, - pred->regex.pattern, - pred->regex.len, + pred->regex->pattern, + pred->regex->len, data); } From patchwork Wed Jul 5 18:12:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116278 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2052500vqx; Wed, 5 Jul 2023 11:19:17 -0700 (PDT) X-Google-Smtp-Source: APBJJlHb01KMsqOhFmQ2dmaMxq2VTa1GHVao5q7Zzm66weD8Ph44mXezFX8UMww1IvDlFoJxn93s X-Received: by 2002:a05:6a00:998:b0:673:8dfb:af32 with SMTP id u24-20020a056a00099800b006738dfbaf32mr17042250pfg.26.1688581157428; Wed, 05 Jul 2023 11:19:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581157; cv=none; d=google.com; s=arc-20160816; b=lIvikkUw68po7ssR4Wz0fGSNtpWpRattUm0BGZWpfz1fNJ9lE4frzMjIi+sGaeq2PW 5tGb26htJ2B4M5VTF2J8v3Pr7GnnY1/Hc1FYTYM5KLTFuyV+xpaWAa4m7AeiyFH8EHw3 6j/HA2AcWPX1m+oFzOLnU/LDbKYQXv1oqFFLmb5XXvp2DqB8hMudTBqjL/3/lClDtr28 3IbMoRS1ng1Dv7Rd1yYCvrRs9pQzgp53Wp8U3eOuEIAi8XLmyVhy0syI2NzLaPu6ALDP sK9A/6Ol3DhgZxO0M1SrWjuOvcStwP1T2v9ddhTgaiNPxFL7r6T/wQHirzhMeRoGrKQo klHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=EWVIdw42kvaUEVB9fypVLDqQXfw7FfQjjY+1yjJwfjg=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=uf8G28Y2SGBTym2YEt9yR2W+nAToYfttt+Z8d2G4Bj25VeomO5Ct6cKGv2UHOu3k52 6M6uMF6LMSqqVtXQHrWxP2pCFmMDyViVp0UaB/+l4J9JcuuHUY8zYR5mGo0O2asnOc3M 6ZnLsImmV2GAOBz94DI647SlhMTrLDRHqAiBusAUY1j2ExzEL7QWPbMk0FN1f+Wznwpo jeOSq2yXUxyXBVu4uojSPfW9VIkDPv65b29MWbgoQsQiJhRmrCSQ0rH2at4bhWbsC+Cq EcGaVmrmxZVN2EKr7h3/54t+kVVh4TFcnJO24CiZqnHfMQZlnItZuG5pF1/8dOESDvN7 3Eog== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bFfchqu6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id eg27-20020a056a00801b00b006768627bc5asi20654299pfb.199.2023.07.05.11.19.04; Wed, 05 Jul 2023 11:19:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=bFfchqu6; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233379AbjGESRV (ORCPT + 99 others); Wed, 5 Jul 2023 14:17:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233370AbjGESRR (ORCPT ); Wed, 5 Jul 2023 14:17:17 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CE4A19A2 for ; Wed, 5 Jul 2023 11:15:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688580956; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EWVIdw42kvaUEVB9fypVLDqQXfw7FfQjjY+1yjJwfjg=; b=bFfchqu69SFo73UtcKoy8CUpBTqPxT2LLOrno9MjethJY8xsmccJLzKLtnCSlrvN9w8jy8 VjNsUWEu7LYQLEH5x90Qu7N8u88H8bCYlbp0/msP/OC9pQSsPCM0/Zl7mGcYrMulniLX0a zFAiHPPO9PLKz3fmILTbOQjXOX4S2I0= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-615-gN3SJfgUNXa5eGtNW0oHxQ-1; Wed, 05 Jul 2023 14:15:51 -0400 X-MC-Unique: gN3SJfgUNXa5eGtNW0oHxQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 87DBC2808E6A; Wed, 5 Jul 2023 18:15:49 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8CD04F5CFA; Wed, 5 Jul 2023 18:15:44 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 02/14] tracing/filters: Enable filtering a cpumask field by another cpumask Date: Wed, 5 Jul 2023 19:12:44 +0100 Message-Id: <20230705181256.3539027-3-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770605675932569039?= X-GMAIL-MSGID: =?utf-8?q?1770605675932569039?= The recently introduced ipi_send_cpumask trace event contains a cpumask field, but it currently cannot be used in filter expressions. Make event filtering aware of cpumask fields, and allow these to be filtered by a user-provided cpumask. The user-provided cpumask is to be given in cpulist format and wrapped as: "MASK{$cpulist}". The use of curly braces instead of parentheses is to prevent predicate_parse() from parsing the contents of MASK{...} as a full-fledged predicate subexpression. This enables e.g.: $ trace-cmd record -e 'ipi_send_cpumask' -f 'cpu & MASK{2,4,6,8-32}' Signed-off-by: Valentin Schneider --- include/linux/trace_events.h | 1 + kernel/trace/trace_events_filter.c | 85 +++++++++++++++++++++++++++++- 2 files changed, 84 insertions(+), 2 deletions(-) diff --git a/include/linux/trace_events.h b/include/linux/trace_events.h index 7c4a0b72334eb..974ef37a06c83 100644 --- a/include/linux/trace_events.h +++ b/include/linux/trace_events.h @@ -804,6 +804,7 @@ enum { FILTER_RDYN_STRING, FILTER_PTR_STRING, FILTER_TRACE_FN, + FILTER_CPUMASK, FILTER_COMM, FILTER_CPU, FILTER_STACKTRACE, diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index d999a218fe833..8af00caa363f7 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -64,6 +64,7 @@ enum filter_pred_fn { FILTER_PRED_FN_PCHAR_USER, FILTER_PRED_FN_PCHAR, FILTER_PRED_FN_CPU, + FILTER_PRED_FN_CPUMASK, FILTER_PRED_FN_FUNCTION, FILTER_PRED_FN_, FILTER_PRED_TEST_VISITED, @@ -71,6 +72,7 @@ enum filter_pred_fn { struct filter_pred { struct regex *regex; + struct cpumask *mask; unsigned short *ops; struct ftrace_event_field *field; u64 val; @@ -94,6 +96,8 @@ struct filter_pred { C(TOO_MANY_OPEN, "Too many '('"), \ C(TOO_MANY_CLOSE, "Too few '('"), \ C(MISSING_QUOTE, "Missing matching quote"), \ + C(MISSING_BRACE_OPEN, "Missing '{'"), \ + C(MISSING_BRACE_CLOSE, "Missing '}'"), \ C(OPERAND_TOO_LONG, "Operand too long"), \ C(EXPECT_STRING, "Expecting string field"), \ C(EXPECT_DIGIT, "Expecting numeric field"), \ @@ -103,6 +107,7 @@ struct filter_pred { C(BAD_SUBSYS_FILTER, "Couldn't find or set field in one of a subsystem's events"), \ C(TOO_MANY_PREDS, "Too many terms in predicate expression"), \ C(INVALID_FILTER, "Meaningless filter expression"), \ + C(INVALID_CPULIST, "Invalid cpulist"), \ C(IP_FIELD_ONLY, "Only 'ip' field is supported for function trace"), \ C(INVALID_VALUE, "Invalid value (did you forget quotes)?"), \ C(NO_FUNCTION, "Function not found"), \ @@ -189,6 +194,7 @@ enum { static void free_predicate(struct filter_pred *pred) { kfree(pred->regex); + kfree(pred->mask); kfree(pred); } @@ -875,6 +881,26 @@ static int filter_pred_cpu(struct filter_pred *pred, void *event) } } +/* Filter predicate for cpumasks. */ +static int filter_pred_cpumask(struct filter_pred *pred, void *event) +{ + u32 item = *(u32 *)(event + pred->offset); + int loc = item & 0xffff; + const struct cpumask *mask = (event + loc); + const struct cpumask *cmp = pred->mask; + + switch (pred->op) { + case OP_EQ: + return cpumask_equal(mask, cmp); + case OP_NE: + return !cpumask_equal(mask, cmp); + case OP_BAND: + return cpumask_intersects(mask, cmp); + default: + return 0; + } +} + /* Filter predicate for COMM. */ static int filter_pred_comm(struct filter_pred *pred, void *event) { @@ -1242,8 +1268,12 @@ static void filter_free_subsystem_filters(struct trace_subsystem_dir *dir, int filter_assign_type(const char *type) { - if (strstr(type, "__data_loc") && strstr(type, "char")) - return FILTER_DYN_STRING; + if (strstr(type, "__data_loc")) { + if (strstr(type, "char")) + return FILTER_DYN_STRING; + if (strstr(type, "cpumask_t")) + return FILTER_CPUMASK; + } if (strstr(type, "__rel_loc") && strstr(type, "char")) return FILTER_RDYN_STRING; @@ -1355,6 +1385,8 @@ static int filter_pred_fn_call(struct filter_pred *pred, void *event) return filter_pred_pchar(pred, event); case FILTER_PRED_FN_CPU: return filter_pred_cpu(pred, event); + case FILTER_PRED_FN_CPUMASK: + return filter_pred_cpumask(pred, event); case FILTER_PRED_FN_FUNCTION: return filter_pred_function(pred, event); case FILTER_PRED_TEST_VISITED: @@ -1566,6 +1598,55 @@ static int parse_pred(const char *str, void *data, strncpy(pred->regex->pattern, str + s, len); pred->regex->pattern[len] = 0; + } else if (!strncmp(str + i, "MASK", 4)) { + unsigned int maskstart; + char *tmp; + + if (field->filter_type != FILTER_CPUMASK) { + parse_error(pe, FILT_ERR_ILLEGAL_FIELD_OP, pos + i); + goto err_free; + } + + /* Skip MASK */ + i += 4; + if (str[i++] != '{') { + parse_error(pe, FILT_ERR_MISSING_BRACE_OPEN, pos + i); + goto err_free; + } + maskstart = i; + + /* Walk the cpulist until closing } */ + for (; str[i] && str[i] != '}'; i++); + if (str[i] != '}') { + parse_error(pe, FILT_ERR_MISSING_BRACE_CLOSE, pos + i); + goto err_free; + } + + if (maskstart == i) { + parse_error(pe, FILT_ERR_INVALID_CPULIST, pos + i); + goto err_free; + } + + /* Copy the cpulist between { and } */ + tmp = kmalloc(i - maskstart + 1, GFP_KERNEL); + strncpy(tmp, str + maskstart, i - maskstart); + tmp[i - maskstart] = '\0'; + + pred->mask = kzalloc(cpumask_size(), GFP_KERNEL); + if (!pred->mask) + goto err_mem; + + /* Now parse it */ + if (cpulist_parse(tmp, pred->mask)) { + parse_error(pe, FILT_ERR_INVALID_CPULIST, pos + i); + goto err_free; + } + + /* Move along */ + i++; + if (field->filter_type == FILTER_CPUMASK) + pred->fn_num = FILTER_PRED_FN_CPUMASK; + /* This is either a string, or an integer */ } else if (str[i] == '\'' || str[i] == '"') { char q = str[i]; From patchwork Wed Jul 5 18:12:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116280 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2052506vqx; Wed, 5 Jul 2023 11:19:18 -0700 (PDT) X-Google-Smtp-Source: APBJJlHTW+N1FKbl0tN13tpkQL5PORpTsgdcWpbr83NUZFTan6sOlx4MN4/WsxHdDVmt7GntrDCK X-Received: by 2002:a17:90a:694f:b0:262:ec74:bb33 with SMTP id j15-20020a17090a694f00b00262ec74bb33mr13703494pjm.46.1688581157880; Wed, 05 Jul 2023 11:19:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581157; cv=none; d=google.com; s=arc-20160816; b=oa28OFNhXoj9gksZXFgQz02rK68jEX9sJb/hYYmFXPMD4Y65+8GuT1k3VS7jGPlDWJ +rZpoZhem89twVl401Jwz9UKC7Duz1C8uzTifQIGtqGremKkAQ2BSxd7M66ZEt9LUUEr OOpU19zAalTcLqUv9htuPFJFvth9qfqypZKfpNp5LPWl9fuAVIMgMTL0sWIvpRfAhfAH E1XFpVTcyOpoR5CoD1aaTZfy/NnC14WITSHt8aUec16qCILzUoDx71riulqPx0VYR0Rm iNl0JEZMQmZGMUT6HvKuMYdoMYcRECgX2ZAlV53mbzVX9Ymv1uol8xNyEsy5SiUXJ6X1 ZRFA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=BTFUs8gxr9moRkZBUGlN4k/3c1CssODo/4E7AM+Boo0=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=x6qD5BAD28DUxcpLIKizg5ceih8l0Jlr6sn5LOMk2H96ezdICKz4zqasKDCmyc9NFl 7CMKmC85AMhfjN8VC29Gf7o32+1M7jfVu9/lg8S6LicqFde4Gm0DiK4YzvlYMUJTx05Q eXkcoteyOuY3/wipAWS5Kg8M5JCbdoURvV4NofPE4dIwx96/kJxY1XniJhRySE8pbqJo bxInQNQ0Y3qc6z+eBRvDBtEFpXQhw9DZpeddg0IiK8SHw5TmHdZx9pBD8lNlskq+S99R lHPq/ZKsqm+KJjduvs6QdTbFtSFLxjSGSvv7wDtQmO0Kq4RZngWjZGkd1VUyGofUliYx G4MQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="GfMTPF/C"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t22-20020a17090aba9600b00263e423753dsi2196646pjr.42.2023.07.05.11.19.04; Wed, 05 Jul 2023 11:19:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="GfMTPF/C"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233382AbjGESRT (ORCPT + 99 others); Wed, 5 Jul 2023 14:17:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233364AbjGESRP (ORCPT ); Wed, 5 Jul 2023 14:17:15 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54E0119A9 for ; Wed, 5 Jul 2023 11:16:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688580960; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BTFUs8gxr9moRkZBUGlN4k/3c1CssODo/4E7AM+Boo0=; b=GfMTPF/Cfwf/3te7Jr2UlHrhSdKVEW6XQQEdIe24hCb1/q+T3pq2ClGsS5G1AlJZ3pLyUg 3WnAAqtklf4XHG1VxvYPECWdUrAd7wBXV0BaAuu0I004Btr1uWrnfbkqzoMaP5CrzVfqDF kPrj5RzejCd/arMjSurM3aO2luSrlSg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-187-6Xa3y2PROnCrrdM8iKBVQA-1; Wed, 05 Jul 2023 14:15:56 -0400 X-MC-Unique: 6Xa3y2PROnCrrdM8iKBVQA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9D2EE185A78F; Wed, 5 Jul 2023 18:15:54 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D149AF5CFA; Wed, 5 Jul 2023 18:15:49 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 03/14] tracing/filters: Enable filtering a scalar field by a cpumask Date: Wed, 5 Jul 2023 19:12:45 +0100 Message-Id: <20230705181256.3539027-4-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770605676112494432?= X-GMAIL-MSGID: =?utf-8?q?1770605676112494432?= Several events use a scalar field to denote a CPU: o sched_wakeup.target_cpu o sched_migrate_task.orig_cpu,dest_cpu o sched_move_numa.src_cpu,dst_cpu o ipi_send_cpu.cpu o ... Filtering these currently requires using arithmetic comparison functions, which can be tedious when dealing with interleaved SMT or NUMA CPU ids. Allow these to be filtered by a user-provided cpumask, which enables e.g.: $ trace-cmd record -e 'sched_wakeup' -f 'target_cpu & MASK{2,4,6,8-32}' Signed-off-by: Valentin Schneider --- NOTE: I went with an implicit cpumask conversion of the event field, as AFAICT predicate_parse() does not support parsing the application of a function to a field (e.g. 'MASK(target_cpu) & MASK{2,4,6,8-32}') --- kernel/trace/trace_events_filter.c | 92 ++++++++++++++++++++++++++---- 1 file changed, 80 insertions(+), 12 deletions(-) diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index 8af00caa363f7..99e111c237a93 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -46,15 +46,19 @@ static const char * ops[] = { OPS }; enum filter_pred_fn { FILTER_PRED_FN_NOP, FILTER_PRED_FN_64, + FILTER_PRED_FN_64_CPUMASK, FILTER_PRED_FN_S64, FILTER_PRED_FN_U64, FILTER_PRED_FN_32, + FILTER_PRED_FN_32_CPUMASK, FILTER_PRED_FN_S32, FILTER_PRED_FN_U32, FILTER_PRED_FN_16, + FILTER_PRED_FN_16_CPUMASK, FILTER_PRED_FN_S16, FILTER_PRED_FN_U16, FILTER_PRED_FN_8, + FILTER_PRED_FN_8_CPUMASK, FILTER_PRED_FN_S8, FILTER_PRED_FN_U8, FILTER_PRED_FN_COMM, @@ -641,6 +645,37 @@ predicate_parse(const char *str, int nr_parens, int nr_preds, return ERR_PTR(ret); } +static inline int +do_filter_cpumask(int op, const struct cpumask *mask, const struct cpumask *cmp) +{ + switch (op) { + case OP_EQ: + return cpumask_equal(mask, cmp); + case OP_NE: + return !cpumask_equal(mask, cmp); + case OP_BAND: + return cpumask_intersects(mask, cmp); + default: + return 0; + } +} + +/* Optimisation of do_filter_cpumask() for scalar values */ +static inline int +do_filter_cpumask_scalar(int op, unsigned int cpu, const struct cpumask *mask) +{ + switch (op) { + case OP_EQ: + return cpumask_equal(mask, cpumask_of(cpu)); + case OP_NE: + return !cpumask_equal(mask, cpumask_of(cpu)); + case OP_BAND: + return cpumask_test_cpu(cpu, mask); + default: + return 0; + } +} + enum pred_cmp_types { PRED_CMP_TYPE_NOP, PRED_CMP_TYPE_LT, @@ -684,6 +719,18 @@ static int filter_pred_##type(struct filter_pred *pred, void *event) \ } \ } +#define DEFINE_CPUMASK_COMPARISON_PRED(size) \ +static int filter_pred_##size##_cpumask(struct filter_pred *pred, void *event) \ +{ \ + u##size *addr = (u##size *)(event + pred->offset); \ + unsigned int cpu = *addr; \ + \ + if (cpu >= nr_cpu_ids) \ + return 0; \ + \ + return do_filter_cpumask_scalar(pred->op, cpu, pred->mask); \ +} + #define DEFINE_EQUALITY_PRED(size) \ static int filter_pred_##size(struct filter_pred *pred, void *event) \ { \ @@ -705,6 +752,11 @@ DEFINE_COMPARISON_PRED(u16); DEFINE_COMPARISON_PRED(s8); DEFINE_COMPARISON_PRED(u8); +DEFINE_CPUMASK_COMPARISON_PRED(64); +DEFINE_CPUMASK_COMPARISON_PRED(32); +DEFINE_CPUMASK_COMPARISON_PRED(16); +DEFINE_CPUMASK_COMPARISON_PRED(8); + DEFINE_EQUALITY_PRED(64); DEFINE_EQUALITY_PRED(32); DEFINE_EQUALITY_PRED(16); @@ -889,16 +941,7 @@ static int filter_pred_cpumask(struct filter_pred *pred, void *event) const struct cpumask *mask = (event + loc); const struct cpumask *cmp = pred->mask; - switch (pred->op) { - case OP_EQ: - return cpumask_equal(mask, cmp); - case OP_NE: - return !cpumask_equal(mask, cmp); - case OP_BAND: - return cpumask_intersects(mask, cmp); - default: - return 0; - } + return do_filter_cpumask(pred->op, mask, cmp); } /* Filter predicate for COMM. */ @@ -1349,24 +1392,32 @@ static int filter_pred_fn_call(struct filter_pred *pred, void *event) switch (pred->fn_num) { case FILTER_PRED_FN_64: return filter_pred_64(pred, event); + case FILTER_PRED_FN_64_CPUMASK: + return filter_pred_64_cpumask(pred, event); case FILTER_PRED_FN_S64: return filter_pred_s64(pred, event); case FILTER_PRED_FN_U64: return filter_pred_u64(pred, event); case FILTER_PRED_FN_32: return filter_pred_32(pred, event); + case FILTER_PRED_FN_32_CPUMASK: + return filter_pred_32_cpumask(pred, event); case FILTER_PRED_FN_S32: return filter_pred_s32(pred, event); case FILTER_PRED_FN_U32: return filter_pred_u32(pred, event); case FILTER_PRED_FN_16: return filter_pred_16(pred, event); + case FILTER_PRED_FN_16_CPUMASK: + return filter_pred_16_cpumask(pred, event); case FILTER_PRED_FN_S16: return filter_pred_s16(pred, event); case FILTER_PRED_FN_U16: return filter_pred_u16(pred, event); case FILTER_PRED_FN_8: return filter_pred_8(pred, event); + case FILTER_PRED_FN_8_CPUMASK: + return filter_pred_8_cpumask(pred, event); case FILTER_PRED_FN_S8: return filter_pred_s8(pred, event); case FILTER_PRED_FN_U8: @@ -1602,7 +1653,8 @@ static int parse_pred(const char *str, void *data, unsigned int maskstart; char *tmp; - if (field->filter_type != FILTER_CPUMASK) { + if (field->filter_type != FILTER_CPUMASK && + field->filter_type != FILTER_OTHER) { parse_error(pe, FILT_ERR_ILLEGAL_FIELD_OP, pos + i); goto err_free; } @@ -1644,8 +1696,24 @@ static int parse_pred(const char *str, void *data, /* Move along */ i++; - if (field->filter_type == FILTER_CPUMASK) + if (field->filter_type == FILTER_CPUMASK) { pred->fn_num = FILTER_PRED_FN_CPUMASK; + } else { + switch (field->size) { + case 8: + pred->fn_num = FILTER_PRED_FN_64_CPUMASK; + break; + case 4: + pred->fn_num = FILTER_PRED_FN_32_CPUMASK; + break; + case 2: + pred->fn_num = FILTER_PRED_FN_16_CPUMASK; + break; + case 1: + pred->fn_num = FILTER_PRED_FN_8_CPUMASK; + break; + } + } /* This is either a string, or an integer */ } else if (str[i] == '\'' || str[i] == '"') { From patchwork Wed Jul 5 18:12:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116279 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2052505vqx; Wed, 5 Jul 2023 11:19:18 -0700 (PDT) X-Google-Smtp-Source: APBJJlF7rDtAsE+XwGHwQPMNfTaPs6hr/qakekLUeNzxkyjTKYNDDExgntL9l3mTxxKt/RYN1HUx X-Received: by 2002:a05:6871:5211:b0:1b3:e5fc:6aed with SMTP id ht17-20020a056871521100b001b3e5fc6aedmr3166729oac.29.1688581157902; Wed, 05 Jul 2023 11:19:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581157; cv=none; d=google.com; s=arc-20160816; b=UYwr9imvm2GrCgVW/lj75xKZg5dmwW9qODEeozfrGV2E0BJ8eZVmX/CCeskDvknBEO mheuzArTDC0NuYEbYzTYN7ZObzWLStleFrewW5gIkWgWs7WTA+RXWcwPbBiIVxwvIi9j JsuDVy+n/ZtyDfgqFY6vzsAod8kW3J60tRvrkff1qxqMp2c4LFN3zcVcEF8tqeVMGGcC rZkCgt3c31KF+rdY3aBqk2hrbt6hMs04mRydZm/85oTOsJoHB62zz5OLC7BdmJ+6f035 /oRs6uQwbFgOBlHa8W77caWrmNGClPsSIHB04alwXzEBrSZvk59M2s1AkuzmmTuduOwN f2OQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HhCGDFDeE6330wjTb7+eVdOM5sDQ6eAc/Pd0Q124xLM=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=SdX2NRkOFVzHIlc/1mQEyABhr3JD58ifCgx6+3LLGPcORx5QH5R7kirvRqbBmGfhBK PifgBOql7B5ggujP+2y8M8q4BgnBVnaMJFrSDJ0LB3qOIA83u+eBNZgdsVF6Rn5FgSte XyWsI4jGqW+WmLKExujMixXnKmAa6ZG/WkTHYmnys35JnBthpX8RRtUesTdq4EIhJi0k sRfFFObsv1Dw95yV6StIx2Y6uOoYhZg9Vxfd4BNcAguCmGarCmKSYOyf2dVhTGIYBFl9 Ou0okzW1O5qsfd/z6X6ydj0ywPMK1lyiHPMRGfhhEHsy46dG+399LF+akzNroEMgFfe6 DYdQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JXLEHIrX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c1-20020a63ea01000000b0054ff38f6448si23546301pgi.238.2023.07.05.11.19.04; Wed, 05 Jul 2023 11:19:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=JXLEHIrX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232018AbjGESRd (ORCPT + 99 others); Wed, 5 Jul 2023 14:17:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233364AbjGESRX (ORCPT ); Wed, 5 Jul 2023 14:17:23 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E4681996 for ; Wed, 5 Jul 2023 11:16:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688580990; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HhCGDFDeE6330wjTb7+eVdOM5sDQ6eAc/Pd0Q124xLM=; b=JXLEHIrXZ8qSc83jvJFjrhnw4PSubmsefZZkcKlG3RA6GBOiwTPvWvGq//pY60d7HUoo3w G/ZWUW56KM5lSQ5VVzp5DoYJ0clhliPBL4yIj86B5w3eLyESLpIJnQgSE/GErxFIpUc6TG 4cIjxWoNWix2dhXpSrx0Hf1+nwcfVec= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-630-9fom_J9fNdKBOQiCT3xrlw-1; Wed, 05 Jul 2023 14:16:26 -0400 X-MC-Unique: 9fom_J9fNdKBOQiCT3xrlw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 930C32808E6D; Wed, 5 Jul 2023 18:16:24 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E602BF5CFB; Wed, 5 Jul 2023 18:15:54 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 04/14] tracing/filters: Enable filtering the CPU common field by a cpumask Date: Wed, 5 Jul 2023 19:12:46 +0100 Message-Id: <20230705181256.3539027-5-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770605676114313347?= X-GMAIL-MSGID: =?utf-8?q?1770605676114313347?= The tracing_cpumask lets us specify which CPUs are traced in a buffer instance, but doesn't let us do this on a per-event basis (unless one creates an instance per event). A previous commit added filtering scalar fields by a user-given cpumask, make this work with the CPU common field as well. This enables doing things like $ trace-cmd record -e 'sched_switch' -f 'CPU & MASK{12-52}' \ -e 'sched_wakeup' -f 'target_cpu & MASK{12-52}' Signed-off-by: Valentin Schneider --- kernel/trace/trace_events_filter.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index 99e111c237a93..b3d2612d4670a 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -68,6 +68,7 @@ enum filter_pred_fn { FILTER_PRED_FN_PCHAR_USER, FILTER_PRED_FN_PCHAR, FILTER_PRED_FN_CPU, + FILTER_PRED_FN_CPU_CPUMASK, FILTER_PRED_FN_CPUMASK, FILTER_PRED_FN_FUNCTION, FILTER_PRED_FN_, @@ -933,6 +934,13 @@ static int filter_pred_cpu(struct filter_pred *pred, void *event) } } +static int filter_pred_cpu_cpumask(struct filter_pred *pred, void *event) +{ + int cpu = raw_smp_processor_id(); + + return do_filter_cpumask_scalar(pred->op, cpu, pred->mask); +} + /* Filter predicate for cpumasks. */ static int filter_pred_cpumask(struct filter_pred *pred, void *event) { @@ -1436,6 +1444,8 @@ static int filter_pred_fn_call(struct filter_pred *pred, void *event) return filter_pred_pchar(pred, event); case FILTER_PRED_FN_CPU: return filter_pred_cpu(pred, event); + case FILTER_PRED_FN_CPU_CPUMASK: + return filter_pred_cpu_cpumask(pred, event); case FILTER_PRED_FN_CPUMASK: return filter_pred_cpumask(pred, event); case FILTER_PRED_FN_FUNCTION: @@ -1654,6 +1664,7 @@ static int parse_pred(const char *str, void *data, char *tmp; if (field->filter_type != FILTER_CPUMASK && + field->filter_type != FILTER_CPU && field->filter_type != FILTER_OTHER) { parse_error(pe, FILT_ERR_ILLEGAL_FIELD_OP, pos + i); goto err_free; @@ -1698,6 +1709,8 @@ static int parse_pred(const char *str, void *data, i++; if (field->filter_type == FILTER_CPUMASK) { pred->fn_num = FILTER_PRED_FN_CPUMASK; + } else if (field->filter_type == FILTER_CPU) { + pred->fn_num = FILTER_PRED_FN_CPU_CPUMASK; } else { switch (field->size) { case 8: From patchwork Wed Jul 5 18:12:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116281 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2052544vqx; Wed, 5 Jul 2023 11:19:21 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4zrFnWbXf0rrUYI5WUNa7fR4AVQF2qoLu+S1ZcsFDhPo3b0mdWAe5TBwoWh2dT1cdJOk/p X-Received: by 2002:a05:6808:138f:b0:3a3:65d6:d76 with SMTP id c15-20020a056808138f00b003a365d60d76mr22060096oiw.42.1688581160699; Wed, 05 Jul 2023 11:19:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581160; cv=none; d=google.com; s=arc-20160816; b=n78M2ftK/f1iMNKuRPQy5pPL7robObrE7P6eHncelVUliAwgftG+jP1DBLJUNflifA S308Fi9bQ76bII6fOx0VB1bzyYiZ+QP/NYHSLl9Srcru979mc26LJij3VlZl/FLf9iyJ xiJQuso3Mr769o/+TYXQGScXiIMj8/9yfiebMcIY0jHq0rIUZOoZxcKCwdT2g5Vq7B5n GvIthuzFIIhwodIw+AFDNKel7D1xrFXo9DnUVaBQj/I68hfFN6k3RoLSdfcRAZBA83+W TPP8KDDrsqBMIwGsKl7LT5rISbT74BE+ggjxRfhr2Pee/J4/ZG+2i0cSszosOkhYizEs WeNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=snCuxFAiT0B/bnlHGtaiRqsJwVD+zCZujmUlSZJWpwM=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=XQ913Ba+Bxrl6VD2DcO5pCizy0L+qFzQHWOEbs1ioBYabGGNFKRMP8jOc/qHkQqVF7 bOagQCHzLLrnCQmkju0K8TYMBVQdAlpd5v6XeD4n0489EWlipxVgKqTzIvx5y/INbCT8 Sf90/Pihg9uOGL6lFMczmis2Llvg5ZpNu0W/BiHGcltFFi7K1uySMSkfXRBN8r2lHiKO N6N4WgMCDUFoI3cHBk8PWqpzZea7jfi35+1GJ7E9wU8bHC8gpQlP3rej/9kzOCeziS9j n0zGm3M/LpQe+5/ePu/cHeqEkrWovTFM1FT7qe4F0DVo+W854ZPglG2la4M+rWHEnEh3 SlzA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gbFZdI+C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b15-20020a630c0f000000b0051b9a71329fsi24899691pgl.360.2023.07.05.11.19.05; Wed, 05 Jul 2023 11:19:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=gbFZdI+C; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233380AbjGESRZ (ORCPT + 99 others); Wed, 5 Jul 2023 14:17:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233353AbjGESRU (ORCPT ); Wed, 5 Jul 2023 14:17:20 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D14C19AF for ; Wed, 5 Jul 2023 11:16:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688580996; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=snCuxFAiT0B/bnlHGtaiRqsJwVD+zCZujmUlSZJWpwM=; b=gbFZdI+CtiuzRmPWqKrw9wu3m900gR3Z0Q6ZTozWwRTClBZcjFW7ayEchlbBZ0z4VqxbQZ XjB1zBuO4FZiRDxVEk5RcZiJJR5S7Esde691DFrJCAT7sJ4QxQF0lMycbxKSfmlddbKE17 P5LzQabDCMxO6MPvxq9rjln1lRKkk4c= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-668-hNYP9ME2PT2zK_PDEefSuQ-1; Wed, 05 Jul 2023 14:16:33 -0400 X-MC-Unique: hNYP9ME2PT2zK_PDEefSuQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1D8FE3815EF2; Wed, 5 Jul 2023 18:16:31 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id CDDF218EB4; Wed, 5 Jul 2023 18:16:24 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 05/14] tracing/filters: Document cpumask filtering Date: Wed, 5 Jul 2023 19:12:47 +0100 Message-Id: <20230705181256.3539027-6-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770605679240025719?= X-GMAIL-MSGID: =?utf-8?q?1770605679240025719?= Cpumask, scalar and CPU fields can now be filtered by a user-provided cpumask, document the syntax. Signed-off-by: Valentin Schneider --- Documentation/trace/events.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/Documentation/trace/events.rst b/Documentation/trace/events.rst index f5fcb8e1218f6..e9bc9f23891a0 100644 --- a/Documentation/trace/events.rst +++ b/Documentation/trace/events.rst @@ -219,6 +219,20 @@ the function "security_prepare_creds" and less than the end of that function. The ".function" postfix can only be attached to values of size long, and can only be compared with "==" or "!=". +Cpumask fields or scalar fields that encode a CPU number can be filtered using +a user-provided cpumask in cpulist format. The format is as follows:: + + MASK{$cpulist} + +Operators available to cpumask filtering are: + +& (intersection), ==, != + +For example, this will filter events that have their .target_cpu field present +in the given cpumask:: + + target_cpu & MASK{17-42} + 5.2 Setting filters ------------------- From patchwork Wed Jul 5 18:12:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116293 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2056390vqx; Wed, 5 Jul 2023 11:26:19 -0700 (PDT) X-Google-Smtp-Source: APBJJlGPtSb5iR6xOOlguyrhBIbUVW9i9wQxrUcWUX5tQU0nfOdkJtoGPJ/fJkvsrc8KVYfwAVgd X-Received: by 2002:a17:90b:3ec5:b0:263:70d1:ba84 with SMTP id rm5-20020a17090b3ec500b0026370d1ba84mr4315507pjb.18.1688581579349; Wed, 05 Jul 2023 11:26:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581579; cv=none; d=google.com; s=arc-20160816; b=Cpt4HjRnJmL0iSMfZdn/eB+vrXTRsh8x5al8VHRkQ891Tx4OxchBNNu6iC2IaXMRyZ w7Bx4WU3t2uKs3R7pnpiumEb6GO5Wfcdvdu3blbcZLIQ4eRgXH7MxssZidp1dIUFlNW3 HyQCXHyM96iGIx64ggWvWqVCbYlbWWAqpNHLgPLiWpad1+8kH4VkuUFC0ZsKrqYtI8dJ tFn8fuh8w4+WhKgy4N1Pco3fDVZ/2bh6oOEX+KNw1CGOgX+FXfhCiKk5oQmI3HQjLfC7 CN18SDLb+zC93T+Nsg3ajnlZh8zV3LZyGm/vOrdmPElB3/FXpfF7c0iOF8Qu5VwugU1r Ji1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=touy8a7//BeI9DXwQuTy3DIG1jDKoaRDRv3dbDYYd7A=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=O8O6F0NybdSssByIPCdu/Q6vbBhFUokAxtL1miKmAPGefnNiYXJ4u9hahmzX69QdVl AswCMLZ7bYhHSaw4NkHBahuuM0lsOw/Hb9YMsDDfulyNRbIciVtOi6pZ570PgTvGFCVL h1ZvfOSLfoNt3lCmqjJEIdY2X3kCzJty22DRRiUe46oQ3jRYgw84VVMenb9Zgu2L4A0V VaFVcSrIrWrMkvwHFxukiatNCI3P3T+/na+m5ZHtaUcHz9HCmq5sdxMRsmPIKNckggEq 1zX18o4B0FI6fHKkJyRWiI7qxfTXGbUBIwyS/cf+75PKPKoDbLaB6h0unO1Z/cHoD+9D jotw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="bRf0/gE2"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id co5-20020a17090afe8500b002635333de96si2079545pjb.97.2023.07.05.11.26.04; Wed, 05 Jul 2023 11:26:19 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="bRf0/gE2"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233420AbjGESRj (ORCPT + 99 others); Wed, 5 Jul 2023 14:17:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233403AbjGESR2 (ORCPT ); Wed, 5 Jul 2023 14:17:28 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7C1A1985 for ; Wed, 5 Jul 2023 11:16:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688581001; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=touy8a7//BeI9DXwQuTy3DIG1jDKoaRDRv3dbDYYd7A=; b=bRf0/gE2Af75MWSNZa6GZtLmlRk10CBo3aTBBwDMg+36xwGoqleWJ21QUOgO4G/vqMyUPv UGQrPQzp06C56zVhhuBl2lHQpx7rWd4L3DUIyQYyY/P8CJ7GXLIPZuu45X5Usm6cOLCIAb Po/q/AyzuRQNuk0us1FXSYe87qmaRAI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-281-rTU3Fnz6O_mjXhT8Or3jrg-1; Wed, 05 Jul 2023 14:16:38 -0400 X-MC-Unique: rTU3Fnz6O_mjXhT8Or3jrg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6BA218037BA; Wed, 5 Jul 2023 18:16:36 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 508F218EB4; Wed, 5 Jul 2023 18:16:31 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 06/14] objtool: Flesh out warning related to pv_ops[] calls Date: Wed, 5 Jul 2023 19:12:48 +0100 Message-Id: <20230705181256.3539027-7-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770606118141061008?= X-GMAIL-MSGID: =?utf-8?q?1770606118141061008?= I had to look into objtool itself to understand what this warning was about; make it more explicit. Signed-off-by: Valentin Schneider --- tools/objtool/check.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/objtool/check.c b/tools/objtool/check.c index 0fcf99c914000..fe62232f218f9 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -3404,7 +3404,7 @@ static bool pv_call_dest(struct objtool_file *file, struct instruction *insn) list_for_each_entry(target, &file->pv_ops[idx].targets, pv_target) { if (!target->sec->noinstr) { - WARN("pv_ops[%d]: %s", idx, target->name); + WARN("pv_ops[%d]: indirect call to %s() leaves .noinstr.text section", idx, target->name); file->pv_ops[idx].clean = false; } } From patchwork Wed Jul 5 18:12:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116289 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2055701vqx; Wed, 5 Jul 2023 11:25:09 -0700 (PDT) X-Google-Smtp-Source: APBJJlHDzRbcxflDhKXH/oXqtIozs2sZ8a2bDMFOcMO1fU1raVxc7SKmGvSn17uygh3ztWXpIT94 X-Received: by 2002:a17:90b:3a8e:b0:263:4815:cb9a with SMTP id om14-20020a17090b3a8e00b002634815cb9amr16438028pjb.41.1688581508928; Wed, 05 Jul 2023 11:25:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581508; cv=none; d=google.com; s=arc-20160816; b=KCIcFcgZ2Hdz5RSAAhbhRBbmDyUuorlsx922+b1PWkFtFs+8520YGlycF/UB5aSSEH q0J7PQ/Mi5QH6SmnoDbQbCX8PM5rBBbikZD4Nv3EIcjFm671tKpXhKMS8W4CdC0ptEzJ edxX2JXZYAOJWdifhgWVS/S8Fq6OjtvHYjTXq3nEZUSd5nRXG7hFKzyCOp4i3KpmSBJX IhfKdceY0ScY2q5dNhaR7dTGuUuZeNkgRi8NtkLTZPlMzj05r4nfgvnuslQWxhi8/PwB /o5dSLFBfKxCjlLscwNSa0nh+b/yQGZTzixorPTh47xrpHbCO9JDnHsztUmecOiSfFbe 7pQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=J8rXW514UBch39kVc1ygiPYqeRYBiljQKIWLWtG0jLw=; fh=3UhQn3CgZPqYoVL3wUdj68RjB671pvt3bCy1XlPpML0=; b=I1AOrmcCZFK3HbnW0H0JaLQdqYJDRjK2mNgkXQmiBko5FZ3b/ng5vliFfl6t/hKGVJ TH9wbl7uVJUZNGZszgasUj44YQmjwo2nuKjhPwSJhjTj/2SksDvc9OkUPkT6RXYT4KSb DxXxbw9ZbtQbQrkvLI9tOc1HYo1z3idbLt0QOnBYxcA45cBi3PjGCxGpXaE1UGt6JPlv kn5A1/u8gn7bJHE7PxPftDqswFtM42RfyEQtmU+GqnF5ohrE5PpCr8kQPnBVdjsF3g6/ ObUQAKDTWvGJolXad47CXJhrdGCLuwpzKGIVJXICa4RCk6UYSgyGbB7iafowgIl1sjAt WOtw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=FX1r+ppN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x7-20020a17090a970700b0025690b76c95si2093174pjo.10.2023.07.05.11.24.52; Wed, 05 Jul 2023 11:25:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=FX1r+ppN; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233411AbjGESSB (ORCPT + 99 others); Wed, 5 Jul 2023 14:18:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233369AbjGESRl (ORCPT ); Wed, 5 Jul 2023 14:17:41 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC4B31BC1 for ; Wed, 5 Jul 2023 11:16:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688581005; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J8rXW514UBch39kVc1ygiPYqeRYBiljQKIWLWtG0jLw=; b=FX1r+ppNlRDeMGoNbaLNkLujoNP9R7FxFlVx8mczJNXxItZigQWKipehItWUWOwnlVMil8 E/yG3Ir2qaKxA3RqHwesRb9y7ebiVJJMprsjuymGtQt8DhMEm4UWGJ5s3dBoKpZI3437tr JT8Xuw7LNh4CrauuyrfdV6fNH1OY0qA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-101-9oeIA5OgNLqp_60MHHVNcQ-1; Wed, 05 Jul 2023 14:16:43 -0400 X-MC-Unique: 9oeIA5OgNLqp_60MHHVNcQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A8779810BB4; Wed, 5 Jul 2023 18:16:41 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id A081B18EB4; Wed, 5 Jul 2023 18:16:36 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Josh Poimboeuf , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 07/14] objtool: Warn about non __ro_after_init static key usage in .noinstr Date: Wed, 5 Jul 2023 19:12:49 +0100 Message-Id: <20230705181256.3539027-8-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770606043997355033?= X-GMAIL-MSGID: =?utf-8?q?1770606043997355033?= Later commits will depend on having no runtime-mutable text in early entry code. (ab)use the .noinstr section as a marker of early entry code and warn about static keys used in it that can be flipped at runtime. Suggested-by: Josh Poimboeuf Signed-off-by: Valentin Schneider --- tools/objtool/check.c | 20 ++++++++++++++++++++ tools/objtool/include/objtool/check.h | 1 + tools/objtool/include/objtool/special.h | 2 ++ tools/objtool/special.c | 3 +++ 4 files changed, 26 insertions(+) diff --git a/tools/objtool/check.c b/tools/objtool/check.c index fe62232f218f9..7f8d210ec88c3 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -2032,6 +2032,9 @@ static int add_special_section_alts(struct objtool_file *file) alt->next = orig_insn->alts; orig_insn->alts = alt; + if (special_alt->key_sym) + orig_insn->key_sym = special_alt->key_sym; + list_del(&special_alt->list); free(special_alt); } @@ -3520,6 +3523,20 @@ static int validate_return(struct symbol *func, struct instruction *insn, struct return 0; } +static int validate_static_key(struct instruction *insn, struct insn_state *state) +{ + if (state->noinstr && state->instr <= 0) { + if ((strcmp(insn->key_sym->sec->name, ".data..ro_after_init"))) { + WARN_INSN(insn, + "Non __ro_after_init static key \"%s\" in .noinstr section", + insn->key_sym->name); + return 1; + } + } + + return 0; +} + static struct instruction *next_insn_to_validate(struct objtool_file *file, struct instruction *insn) { @@ -3670,6 +3687,9 @@ static int validate_branch(struct objtool_file *file, struct symbol *func, if (handle_insn_ops(insn, next_insn, &state)) return 1; + if (insn->key_sym) + validate_static_key(insn, &state); + switch (insn->type) { case INSN_RETURN: diff --git a/tools/objtool/include/objtool/check.h b/tools/objtool/include/objtool/check.h index daa46f1f0965a..35dd21f8f41e1 100644 --- a/tools/objtool/include/objtool/check.h +++ b/tools/objtool/include/objtool/check.h @@ -77,6 +77,7 @@ struct instruction { struct symbol *sym; struct stack_op *stack_ops; struct cfi_state *cfi; + struct symbol *key_sym; }; static inline struct symbol *insn_func(struct instruction *insn) diff --git a/tools/objtool/include/objtool/special.h b/tools/objtool/include/objtool/special.h index 86d4af9c5aa9d..0e61f34fe3a28 100644 --- a/tools/objtool/include/objtool/special.h +++ b/tools/objtool/include/objtool/special.h @@ -27,6 +27,8 @@ struct special_alt { struct section *new_sec; unsigned long new_off; + struct symbol *key_sym; + unsigned int orig_len, new_len; /* group only */ }; diff --git a/tools/objtool/special.c b/tools/objtool/special.c index baa85c31526b3..830e6abf173a2 100644 --- a/tools/objtool/special.c +++ b/tools/objtool/special.c @@ -127,6 +127,9 @@ static int get_alt_entry(struct elf *elf, const struct special_entry *entry, return -1; } alt->key_addend = key_reloc->addend; + + reloc_to_sec_off(key_reloc, &sec, &offset); + alt->key_sym = find_symbol_by_offset(sec, offset & ~2); } return 0; From patchwork Wed Jul 5 18:12:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116296 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2066665vqx; Wed, 5 Jul 2023 11:46:00 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6VYSLdC0nMkZtLidJlH+MqOH2lRdWnwEOi9RlyOC3lO0NK+PMHqztoy5T5+WgIHLOieSVh X-Received: by 2002:a05:6a20:1b24:b0:104:35ec:c25c with SMTP id ch36-20020a056a201b2400b0010435ecc25cmr12507108pzb.41.1688582760181; Wed, 05 Jul 2023 11:46:00 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688582760; cv=none; d=google.com; s=arc-20160816; b=p5OTDucrEj/k4epJieDDEf9VrVZEef4VvHzJIUhvLRr7Ne/3CYq23H4B4522UIaEMr dwyFEfWpyIdu+ah95wt/bDHHJ6YI0dQj9U0lvfo3h1cB2eeKF/rSteVNwBC4F6NImZus TPSs57r6XDFrJuRfYJ5Y+HLINUsxdX2KNi14a6Rj9qX+x/NrSR+6txkW3cqfaIvFGwO9 6zU8iOQygt2g3ouhqHFxWvEiQzW1UUF7iop46C40OU06lUyoBY78nZJo8YUseTddnSAz D+OJKMLdGyqZAhARHhBV4LfZFZyhGriICOJLmBR7yYaNffG32DNq8jn39meqzbrJ3Tp8 2jKQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Rmsxmag8ZkUExVz48NdRE4bjGysGk4VVErYcJqwPIM4=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=SxZ9Rd5o31FfP6tf7MbYp9pWXsA4N4Gj8o6pMqFrL8j3gbA5tgvQZDJp0xRkfBChaU p/S9LYc++1+Xxu0mu+DKSWoUZWX3XRUkIC1HIpkpVW/iGLYEhs49+LuqBFIu2Td+fmz2 BMGq/ANpn1syL421gbeAOn4lUadMCv1U55iaAeiMRcuV5rrhO15WdWvZFAl7LK+1AjfF 7yUc6js7M/vJzWWSWygh8sa7e0/9Geti1Nws59UxlrfDbBOx5Poa35foqWkBXYV1nI1i /itr1x6/aePQWekyxOrEMUSHE7+QMVRA9o6KzHItehqKW49y4RrmnEIDO45HQTKQqN9F zhew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=jVug4lix; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w4-20020a170902e88400b001b89b7ce902si7137760plg.305.2023.07.05.11.45.43; Wed, 05 Jul 2023 11:46:00 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=jVug4lix; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233438AbjGESSE (ORCPT + 99 others); Wed, 5 Jul 2023 14:18:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231799AbjGESRr (ORCPT ); Wed, 5 Jul 2023 14:17:47 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EAEB19B9 for ; Wed, 5 Jul 2023 11:16:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688581010; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Rmsxmag8ZkUExVz48NdRE4bjGysGk4VVErYcJqwPIM4=; b=jVug4lixrd8sNTBzab87XhRmBUWCeutjFXjzcwdzf4m5Zw9+HX4i39LypG1+d2tmkks3cr 6diKooABmh6I+YRJpH/d8tRWQ/6HMD8lYeDrdsi2CkPTDFnQy8i3QWu8PJt4CAoK3Gvy/m f15sVMxI63qVqzBRCG0PGRjHoCIbOJY= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-534-K-_lME6ZNbOFxO4KCoVoVw-1; Wed, 05 Jul 2023 14:16:49 -0400 X-MC-Unique: K-_lME6ZNbOFxO4KCoVoVw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 980F53815EF4; Wed, 5 Jul 2023 18:16:46 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D5B65F5CFA; Wed, 5 Jul 2023 18:16:41 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 08/14] BROKEN: context_tracking: Make context_tracking_key __ro_after_init Date: Wed, 5 Jul 2023 19:12:50 +0100 Message-Id: <20230705181256.3539027-9-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770607356334719411?= X-GMAIL-MSGID: =?utf-8?q?1770607356334719411?= objtool now warns about it: vmlinux.o: warning: objtool: enter_from_user_mode+0x4e: Non __ro_after_init static key "context_tracking_key" in .noinstr section vmlinux.o: warning: objtool: enter_from_user_mode+0x50: Non __ro_after_init static key "context_tracking_key" in .noinstr section vmlinux.o: warning: objtool: syscall_enter_from_user_mode+0x60: Non __ro_after_init static key "context_tracking_key" in .noinstr section vmlinux.o: warning: objtool: syscall_enter_from_user_mode+0x62: Non __ro_after_init static key "context_tracking_key" in .noinstr section [...] The key can only be enabled (and not disabled) in the __init function ct_cpu_tracker_user(), so mark it as __ro_after_init. BROKEN: the struct static_key lives in a read-only mapping after mark_rodata_ro(), which falls apart when the KVM module is loaded after init and a write to the struct happens due to e.g. guest_state_exit_irqoff() relying on the static key: jump_label_add_module() `\ static_key_set_mod() static_key_set_linked() Signed-off-by: Valentin Schneider --- kernel/context_tracking.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index a09f1c19336ae..4e6cb14272fcb 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -432,7 +432,7 @@ static __always_inline void ct_kernel_enter(bool user, int offset) { } #define CREATE_TRACE_POINTS #include -DEFINE_STATIC_KEY_FALSE(context_tracking_key); +DEFINE_STATIC_KEY_FALSE_RO(context_tracking_key); EXPORT_SYMBOL_GPL(context_tracking_key); static noinstr bool context_tracking_recursion_enter(void) From patchwork Wed Jul 5 18:12:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116290 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2056265vqx; Wed, 5 Jul 2023 11:26:08 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ53C0p5+7UoFS2z9UXrtoDaC8WJcJtF5wbRHbk6Gn+EmyvtE7nCSYu1gmdwdFxNyWXdpLbi X-Received: by 2002:a05:6870:ee14:b0:1b0:7661:dd18 with SMTP id ga20-20020a056870ee1400b001b07661dd18mr22158952oab.23.1688581568477; Wed, 05 Jul 2023 11:26:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581568; cv=none; d=google.com; s=arc-20160816; b=ZvXhpgCcwXmT4TaNqci1u92eJD7UqZuLGiE7G7hyGK8BMgXNSHB44xPmyJbY5CLm1B KqORdN5omXJX/eBmDxWICtsUhdFuIyF5LJGupLQK9jGyo0eY0BEz368ou6iWI7plnyhS 2uEhzGHeu+Xt+ZwT4p1PfPW9XkU0PKNDpmnMXjr8I2uD50MnOxws4DYQv2k12m0u0Mbo Rufs6Ws0WG0XDpMP9HEHszjCTf2u09VM205O+SAJISStkl1sftB+zYwjX75wdiWLXXM6 IFaB8R5Aq6twhy+2VS5yz0EbdEb91c4oA+JXX7ZxSp3lamo0pxDRk+6vpRQ7UMI664K9 SYtg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=izi9UHkrSeZ4AHa49u6aA8W9HneeYs8L2V/Y70LmMpM=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=i+7LUGPK4r0tzKfeUASlzz/TwJGUMvQ5jvcacfU2Ntyvdak7yznYELNAAtfvJxYYSI xcgP4fkv2iM8FmdlXv0RFreF3wZlKFnlP0+qcfovtJylj7grZyWTlQSi7J+Dn0oCqIKu mTyO1xFaEPmL/2+I3Hu9+i/kFwEzRDrKbkF/A+WLLdPNkvOl4ciCcPRZkLaaKO4KaVH+ BHYQ0m4QlJHD8PZ0X+H2rfNUar4YiG/E1tq58ebnDqSGn8kPtfuQNs4yB5t8XpDIRKbb tGIufYQIfCg2xEe8wszc/mQPU5x4Zm2t3G/vLZqSyBc2AyaRPKq7Ye/3Eh1PXGf3KuWF N8GA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=aK8mmgOZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h190-20020a636cc7000000b0055792e57c89si22334258pgc.450.2023.07.05.11.25.53; Wed, 05 Jul 2023 11:26:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=aK8mmgOZ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231975AbjGESSO (ORCPT + 99 others); Wed, 5 Jul 2023 14:18:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233434AbjGESSD (ORCPT ); Wed, 5 Jul 2023 14:18:03 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F20F1BF4 for ; Wed, 5 Jul 2023 11:16:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688581018; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=izi9UHkrSeZ4AHa49u6aA8W9HneeYs8L2V/Y70LmMpM=; b=aK8mmgOZbP2nnH9iHE0N2/mOuS2xZWRC8g4jlmqb4m4D+Jm60Or+2a2FTt5HTB+imuEQvh zAso7CV3piJEPUqBnZVjP1Yzuxw2y6tAp0TUv0xyB3Q4IKzPOXvzb74tPX2kCeEcZ4H4pQ 0vHl5ZNTdoTSW0/xVjV8I2J44mYVuoU= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-197-m-BHSRQXMT2rSYl1-eA5hQ-1; Wed, 05 Jul 2023 14:16:54 -0400 X-MC-Unique: m-BHSRQXMT2rSYl1-eA5hQ-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B7B9F3815EF2; Wed, 5 Jul 2023 18:16:52 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D1D60F5CFA; Wed, 5 Jul 2023 18:16:46 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 09/14] x86/kvm: Make kvm_async_pf_enabled __ro_after_init Date: Wed, 5 Jul 2023 19:12:51 +0100 Message-Id: <20230705181256.3539027-10-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770606106429422894?= X-GMAIL-MSGID: =?utf-8?q?1770606106429422894?= objtool now warns about it: vmlinux.o: warning: objtool: exc_page_fault+0x2a: Non __ro_after_init static key "kvm_async_pf_enabled" in .noinstr section The key can only be enabled (and not disabled) in the __init function kvm_guest_init(), so mark it as __ro_after_init. Signed-off-by: Valentin Schneider --- arch/x86/kernel/kvm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/kvm.c b/arch/x86/kernel/kvm.c index 1cceac5984daa..319460090a836 100644 --- a/arch/x86/kernel/kvm.c +++ b/arch/x86/kernel/kvm.c @@ -44,7 +44,7 @@ #include #include -DEFINE_STATIC_KEY_FALSE(kvm_async_pf_enabled); +DEFINE_STATIC_KEY_FALSE_RO(kvm_async_pf_enabled); static int kvmapf = 1; From patchwork Wed Jul 5 18:12:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116282 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2052788vqx; Wed, 5 Jul 2023 11:19:48 -0700 (PDT) X-Google-Smtp-Source: APBJJlH+gjKJ77HJWotHaCOtBLOk43RpKcTcbvnNLRDpLUy06R4rSazaS+z9ugfYpJMaCyA941Bv X-Received: by 2002:a17:902:e5c5:b0:1b8:1b84:93e5 with SMTP id u5-20020a170902e5c500b001b81b8493e5mr15857270plf.1.1688581187856; Wed, 05 Jul 2023 11:19:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581187; cv=none; d=google.com; s=arc-20160816; b=u2WVGd3DQBnEvIjsdXATvQgFgoZZsln5tkIO48XdwlLYGC1VrNOQovmhw1m7nMBjP0 h6281x9XGql/g/j+FszJMP4xaWFKko6QKLPTU8UBCQnjgNYICGo1erEfF4Gk1heFeN9q gxZmFRl9Q/le+PX6zS5Xn3BDIyUxTPUxPj9mOwnM3K3M9H5gT55CEmBcK3fLWPgWokzI uc+/XSPqEnxAtxLZCQ98R//5VqYlvTgHCGhCZ1FTrPTeCz8GiZ8ijQg0qPRz1hm3K1wx te5hxqOiwfa56ULtiOI2M8rJfZ1/jw026vVTJXlyM4kO5lbASbxbZFaz38bm9CSzoQq6 ZVnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QCdKg/xbfvFvfuEOm23BNeuog3oDcFdsQXIARij7I9Y=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=I+5wcmJAd5eLks/s/vv1P+NVv9ttj0SzkgEsUKs49A397ckLgRcq+ejqVytxTt5WCK TMGHQvEj4pi8DkUodfCg4kbqDANvvInw3gh0Ijfk17V5ekxpvIYY51qv6czwdYN97IHr OsBBI2svE+WA4AX6jvUU2IqIPYwX0AelZZXr8V/Gis2iIc+897uNNq+Bt5OihQmZaw9n 6BdLi0gA1nqswULBdjaOJhOaTOehI5Z+pyO2p9kixOW5zQoYpXlyf9IpeTPcpp62030W m4H/FsLHg9+r9j+wJ/dDlsBHSI9JP2TEu2zTP1RSOR+E8Afq7qnAI2xbDlQQPQlQnK/c YHjA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="K9s9wXO/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c2-20020a170902d90200b001b86c54d963si11894643plz.543.2023.07.05.11.19.34; Wed, 05 Jul 2023 11:19:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="K9s9wXO/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233467AbjGESSS (ORCPT + 99 others); Wed, 5 Jul 2023 14:18:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233466AbjGESSJ (ORCPT ); Wed, 5 Jul 2023 14:18:09 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19FFD1735 for ; Wed, 5 Jul 2023 11:17:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688581022; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QCdKg/xbfvFvfuEOm23BNeuog3oDcFdsQXIARij7I9Y=; b=K9s9wXO/tpOk9RQuuO9ysZ+jlgrY6U7yt0MSwxSqQn7qvWBCxEMZkeBmZX89iA63PnUDMJ X0phXmReSW0CyjOZrgj/hAs7pyjkMihQN2aCn/scj9LV/ZCMal2q7Lyi3Pq627YlhrtvxD y1M9DbF08tSL/vVRMuVFENFsyVB53q8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-108-UhoNqooBN1GAazRibK82Gg-1; Wed, 05 Jul 2023 14:16:59 -0400 X-MC-Unique: UhoNqooBN1GAazRibK82Gg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E89FD3C108C9; Wed, 5 Jul 2023 18:16:57 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id E7BEE18EB4; Wed, 5 Jul 2023 18:16:52 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 10/14] x86/sev-es: Make sev_es_enable_key __ro_after_init Date: Wed, 5 Jul 2023 19:12:52 +0100 Message-Id: <20230705181256.3539027-11-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770605707727619828?= X-GMAIL-MSGID: =?utf-8?q?1770605707727619828?= objtool now warns about it: vmlinux.o: warning: objtool: exc_nmi+0xbf: Non __ro_after_init static key "sev_es_enable_key" in .noinstr section vmlinux.o: warning: objtool: exc_nmi+0x4d: Non __ro_after_init static key "sev_es_enable_key" in .noinstr section vmlinux.o: warning: objtool: exc_nmi+0xc: Non __ro_after_init static key "sev_es_enable_key" in .noinstr section The key can only be enabled (and not disabled) in the __init function sev_es_init_vc_handling(), so mark it as __ro_after_init. Signed-off-by: Valentin Schneider --- arch/x86/kernel/sev.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kernel/sev.c b/arch/x86/kernel/sev.c index b031244d6d2df..4178e18383232 100644 --- a/arch/x86/kernel/sev.c +++ b/arch/x86/kernel/sev.c @@ -113,7 +113,7 @@ struct ghcb_state { }; static DEFINE_PER_CPU(struct sev_es_runtime_data*, runtime_data); -DEFINE_STATIC_KEY_FALSE(sev_es_enable_key); +DEFINE_STATIC_KEY_FALSE_RO(sev_es_enable_key); static DEFINE_PER_CPU(struct sev_es_save_area *, sev_vmsa); From patchwork Wed Jul 5 18:12:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116291 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2056332vqx; Wed, 5 Jul 2023 11:26:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7wevzXSLdyzsrCtPpznHK1uD4Xoea6Tedp7Q44EUBQo2RV4947lW2/O5LLMo8Hol86N7vx X-Received: by 2002:a05:6a21:3392:b0:121:fd18:4b51 with SMTP id yy18-20020a056a21339200b00121fd184b51mr23156269pzb.36.1688581574156; Wed, 05 Jul 2023 11:26:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581574; cv=none; d=google.com; s=arc-20160816; b=FN69M7upyPhuIO2r9D0y5r7Xbi3oFLwZ84eWpswXFWLPm3tJJtKi8J0mpvSwbqWbs3 vIM1sgwa1BVYwSMnfFh068oNr3u60Apx6SwgbQsy+5lJEo0Zbc6A1bXp2Zanwq1ZI4j4 FbPo01CdW+PtuZquk2bZDb2H8iJq+tgs6ph4IrVS2Mmy85l0uV3ju0hFbkSA5bCBo0fW mgJGRxr4QCQAEfT6iOS8SfF8IqzKChI20GZJ/aUnT0X3dHkcgmPnjIXmhqLNQNGDlHGK ig5ZfChpsLFd7CYyjamHeMe1OqmekWQcKYOdkmdegB//Ase6sbYn/ZYhcmXCkSUbd/xc Z6Wg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=C/ucjOAEaHS5XCwJ7CRYxNKU/jQaRurVwoCVOxjEFxI=; fh=eSEthQQOYg3kVcTDsAMYwIEg8ix5xID06ug4Qa8z/Rc=; b=FOrOx+IG4M2BnfxT6INnBpeV0f51jqgXUmRf93OQYthvkbjJ+18sS//3JolIli9EdL FFqEtjou5UR6aMvd5KPzpesBs4j2cFx8XqGU4CmDTZcoUvgrnTgpaIG965txnwSTRSo6 K/0UjJfg1V4njSWuK9VOQJNQ9acuq2YZsCe/GcZ/KqceUBDfxbfdIXbjADWSLk47eNdc NhWNVbIMF1sU4KzAekcZdzqtke3XmrZJr4xNM7td9FRs3PODOG8ZVk76x2psxVHJVPEy fPhEcM3LI29neznxq16/TTbcrrzulgZ5dmc//7WrvjG3F5yZ4rckdvMjk4S6GRJM9zCE Nf4g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=doHzIz5v; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 79-20020a630252000000b0055baeced15esi7378317pgc.549.2023.07.05.11.25.59; Wed, 05 Jul 2023 11:26:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=doHzIz5v; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233418AbjGESSn (ORCPT + 99 others); Wed, 5 Jul 2023 14:18:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233466AbjGESSX (ORCPT ); Wed, 5 Jul 2023 14:18:23 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36E051BE1 for ; Wed, 5 Jul 2023 11:17:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688581027; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=C/ucjOAEaHS5XCwJ7CRYxNKU/jQaRurVwoCVOxjEFxI=; b=doHzIz5vR47Xf9CIejudGcE01BiZCnGhpNJNAR/2YeImA+FDuyrUkME6Y4V6VtoOWCKfev T1mVqRUtCVQ41+t5KEGKwu0wr5punMiJXA/DStBf5Uy9To7idBVlyLUWztzlCxZJKg1hur ixgD4y7qA6OujgA7b0p1BibXKbfrVd8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-550-LTvzUOljPxOE1cOXn7g4IA-1; Wed, 05 Jul 2023 14:17:05 -0400 X-MC-Unique: LTvzUOljPxOE1cOXn7g4IA-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5994C881B28; Wed, 5 Jul 2023 18:17:03 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 26F1E18EB4; Wed, 5 Jul 2023 18:16:58 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Nicolas Saenz Julienne , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 11/14] context-tracking: Introduce work deferral infrastructure Date: Wed, 5 Jul 2023 19:12:53 +0100 Message-Id: <20230705181256.3539027-12-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770606112510783476?= X-GMAIL-MSGID: =?utf-8?q?1770606112510783476?= smp_call_function() & friends have the unfortunate habit of sending IPIs to isolated, NOHZ_FULL, in-userspace CPUs, as they blindly target all online CPUs. Some callsites can be bent into doing the right, such as done by commit: cc9e303c91f5 ("x86/cpu: Disable frequency requests via aperfmperf IPI for nohz_full CPUs") Unfortunately, not all SMP callbacks can be omitted in this fashion. However, some of them only affect execution in kernelspace, which means they don't have to be executed *immediately* if the target CPU is in userspace: stashing the callback and executing it upon the next kernel entry would suffice. x86 kernel instruction patching or kernel TLB invalidation are prime examples of it. Add a field in struct context_tracking used as a bitmask to track deferred callbacks to execute upon kernel entry. The LSB of that field is used as a flag to prevent queueing deferred work when the CPU leaves userspace. Later commits introduce the bit:callback mappings. Note: A previous approach by PeterZ [1] used an extra bit in context_tracking.state to flag the presence of deferred callbacks to execute, and the actual callbacks were stored in a separate atomic variable. This meant that the atomic read of context_tracking.state was sufficient to determine whether there are any deferred callbacks to execute. Unfortunately, it presents a race window. Consider the work setting function as: preempt_disable(); seq = atomic_read(&ct->seq); if (__context_tracking_seq_in_user(seq)) { /* ctrl-dep */ atomic_or(work, &ct->work); ret = atomic_try_cmpxchg(&ct->seq, &seq, seq|CT_SEQ_WORK); } preempt_enable(); return ret; Then the following can happen: CPUx CPUy CT_SEQ_WORK \in context_tracking.state atomic_or(WORK_N, &ct->work); ct_kernel_enter() ct_state_inc(); atomic_try_cmpxchg(&ct->seq, &seq, seq|CT_SEQ_WORK); The cmpxchg() would fail, ultimately causing an IPI for WORK_N to be sent. Unfortunately, the work bit would remain set, and it can't be sanely cleared in case another CPU set it concurrently - this would ultimately lead to a double execution of the callback, one as a deferred callback and one in the IPI. As not all IPI callbacks are idempotent, this is undesirable. Link: https://lore.kernel.org/all/20210929151723.162004989@infradead.org/ Signed-off-by: Nicolas Saenz Julienne Signed-off-by: Valentin Schneider --- arch/Kconfig | 9 +++ arch/x86/Kconfig | 1 + arch/x86/include/asm/context_tracking_work.h | 14 +++++ include/linux/context_tracking.h | 1 + include/linux/context_tracking_state.h | 1 + include/linux/context_tracking_work.h | 28 +++++++++ kernel/context_tracking.c | 63 ++++++++++++++++++++ kernel/time/Kconfig | 5 ++ 8 files changed, 122 insertions(+) create mode 100644 arch/x86/include/asm/context_tracking_work.h create mode 100644 include/linux/context_tracking_work.h diff --git a/arch/Kconfig b/arch/Kconfig index 205fd23e0cada..e453e9fb864be 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -851,6 +851,15 @@ config HAVE_CONTEXT_TRACKING_USER_OFFSTACK - No use of instrumentation, unless instrumentation_begin() got called. +config HAVE_CONTEXT_TRACKING_WORK + bool + help + Architecture supports deferring work while not in kernel context. + This is especially useful on setups with isolated CPUs that might + want to avoid being interrupted to perform housekeeping tasks (for + ex. TLB invalidation or icache invalidation). The housekeeping + operations are performed upon re-entering the kernel. + config HAVE_TIF_NOHZ bool help diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 53bab123a8ee4..490c773105c0c 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -197,6 +197,7 @@ config X86 select HAVE_CMPXCHG_LOCAL select HAVE_CONTEXT_TRACKING_USER if X86_64 select HAVE_CONTEXT_TRACKING_USER_OFFSTACK if HAVE_CONTEXT_TRACKING_USER + select HAVE_CONTEXT_TRACKING_WORK if X86_64 select HAVE_C_RECORDMCOUNT select HAVE_OBJTOOL_MCOUNT if HAVE_OBJTOOL select HAVE_OBJTOOL_NOP_MCOUNT if HAVE_OBJTOOL_MCOUNT diff --git a/arch/x86/include/asm/context_tracking_work.h b/arch/x86/include/asm/context_tracking_work.h new file mode 100644 index 0000000000000..5bc29e6b2ed38 --- /dev/null +++ b/arch/x86/include/asm/context_tracking_work.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _ASM_X86_CONTEXT_TRACKING_WORK_H +#define _ASM_X86_CONTEXT_TRACKING_WORK_H + +static __always_inline void arch_context_tracking_work(int work) +{ + switch (work) { + case CONTEXT_WORK_n: + // Do work... + break; + } +} + +#endif diff --git a/include/linux/context_tracking.h b/include/linux/context_tracking.h index d3cbb6c16babf..80d571ddfc3a4 100644 --- a/include/linux/context_tracking.h +++ b/include/linux/context_tracking.h @@ -5,6 +5,7 @@ #include #include #include +#include #include #include diff --git a/include/linux/context_tracking_state.h b/include/linux/context_tracking_state.h index fdd537ea513ff..5af06ed26f858 100644 --- a/include/linux/context_tracking_state.h +++ b/include/linux/context_tracking_state.h @@ -36,6 +36,7 @@ struct context_tracking { int recursion; #endif #ifdef CONFIG_CONTEXT_TRACKING + atomic_t work; atomic_t state; #endif #ifdef CONFIG_CONTEXT_TRACKING_IDLE diff --git a/include/linux/context_tracking_work.h b/include/linux/context_tracking_work.h new file mode 100644 index 0000000000000..0b06c3dab58c7 --- /dev/null +++ b/include/linux/context_tracking_work.h @@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_CONTEXT_TRACKING_WORK_H +#define _LINUX_CONTEXT_TRACKING_WORK_H + +#include + +enum { + CONTEXT_WORK_DISABLED_OFFSET, + CONTEXT_WORK_n_OFFSET, + CONTEXT_WORK_MAX_OFFSET +}; + +enum ct_work { + CONTEXT_WORK_DISABLED = BIT(CONTEXT_WORK_DISABLED_OFFSET), + CONTEXT_WORK_n = BIT(CONTEXT_WORK_n_OFFSET), + CONTEXT_WORK_MAX = BIT(CONTEXT_WORK_MAX_OFFSET) +}; + +#include + +#ifdef CONFIG_CONTEXT_TRACKING_WORK +extern bool ct_set_cpu_work(unsigned int cpu, unsigned int work); +#else +static inline bool +ct_set_cpu_work(unsigned int cpu, unsigned int work) { return false; } +#endif + +#endif diff --git a/kernel/context_tracking.c b/kernel/context_tracking.c index 4e6cb14272fcb..b6aee3d0c0528 100644 --- a/kernel/context_tracking.c +++ b/kernel/context_tracking.c @@ -32,6 +32,9 @@ DEFINE_PER_CPU(struct context_tracking, context_tracking) = { .dynticks_nmi_nesting = DYNTICK_IRQ_NONIDLE, #endif .state = ATOMIC_INIT(RCU_DYNTICKS_IDX), +#ifdef CONFIG_CONTEXT_TRACKING_WORK + .work = ATOMIC_INIT(CONTEXT_WORK_DISABLED), +#endif }; EXPORT_SYMBOL_GPL(context_tracking); @@ -72,6 +75,57 @@ static __always_inline void rcu_dynticks_task_trace_exit(void) #endif /* #ifdef CONFIG_TASKS_TRACE_RCU */ } +#ifdef CONFIG_CONTEXT_TRACKING_WORK +static __always_inline unsigned int ct_work_fetch(struct context_tracking *ct) +{ + return arch_atomic_fetch_or(CONTEXT_WORK_DISABLED, &ct->work); +} +static __always_inline void ct_work_clear(struct context_tracking *ct) +{ + arch_atomic_set(&ct->work, 0); +} + +static noinstr void ct_work_flush(unsigned long work) +{ + int bit; + + /* DISABLED is never set while there are deferred works */ + WARN_ON_ONCE(work & CONTEXT_WORK_DISABLED); + + /* + * arch_context_tracking_work() must be noinstr, non-blocking, + * and NMI safe. + */ + for_each_set_bit(bit, &work, CONTEXT_WORK_MAX) + arch_context_tracking_work(BIT(bit)); +} + +bool ct_set_cpu_work(unsigned int cpu, unsigned int work) +{ + struct context_tracking *ct = per_cpu_ptr(&context_tracking, cpu); + unsigned int old_work; + bool ret = false; + + preempt_disable(); + + old_work = atomic_read(&ct->work); + /* + * Try setting the work until either + * - the target CPU no longer accepts any more deferred work + * - the work has been set + */ + while (!(old_work & CONTEXT_WORK_DISABLED) && !ret) + ret = atomic_try_cmpxchg(&ct->work, &old_work, old_work | work); + + preempt_enable(); + return ret; +} +#else +static __always_inline void ct_work_flush(unsigned long work) { } +static __always_inline unsigned int ct_work_fetch(struct context_tracking *ct) { return 0; } +static __always_inline void ct_work_clear(struct context_tracking *ct) { } +#endif + /* * Record entry into an extended quiescent state. This is only to be * called when not already in an extended quiescent state, that is, @@ -89,6 +143,10 @@ static noinstr void ct_kernel_exit_state(int offset) */ rcu_dynticks_task_trace_enter(); // Before ->dynticks update! seq = ct_state_inc(offset); + + /* Let this CPU allow deferred callbacks again */ + ct_work_clear(this_cpu_ptr(&context_tracking)); + // RCU is no longer watching. Better be in extended quiescent state! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && (seq & RCU_DYNTICKS_IDX)); } @@ -100,14 +158,19 @@ static noinstr void ct_kernel_exit_state(int offset) */ static noinstr void ct_kernel_enter_state(int offset) { + struct context_tracking *ct = this_cpu_ptr(&context_tracking); int seq; + unsigned int work; + work = ct_work_fetch(ct); /* * CPUs seeing atomic_add_return() must see prior idle sojourns, * and we also must force ordering with the next RCU read-side * critical section. */ seq = ct_state_inc(offset); + if (work) + ct_work_flush(work); // RCU is now watching. Better not be in an extended quiescent state! rcu_dynticks_task_trace_exit(); // After ->dynticks update! WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !(seq & RCU_DYNTICKS_IDX)); diff --git a/kernel/time/Kconfig b/kernel/time/Kconfig index bae8f11070bef..fdb266f2d774b 100644 --- a/kernel/time/Kconfig +++ b/kernel/time/Kconfig @@ -181,6 +181,11 @@ config CONTEXT_TRACKING_USER_FORCE Say N otherwise, this option brings an overhead that you don't want in production. +config CONTEXT_TRACKING_WORK + bool + depends on HAVE_CONTEXT_TRACKING_WORK && CONTEXT_TRACKING_USER + default y + config NO_HZ bool "Old Idle dynticks config" help From patchwork Wed Jul 5 18:12:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116283 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2052915vqx; Wed, 5 Jul 2023 11:20:01 -0700 (PDT) X-Google-Smtp-Source: APBJJlEATW/y8nY3te6O6+lVI9z1Sfw8ZkOK2Xrx8tXykgc+kNGrUPQWYdAVtxZ1kuYMAnSGjeZr X-Received: by 2002:a05:6870:f988:b0:1a2:8e53:c418 with SMTP id hv8-20020a056870f98800b001a28e53c418mr13096323oab.57.1688581201058; Wed, 05 Jul 2023 11:20:01 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581201; cv=none; d=google.com; s=arc-20160816; b=W0Ku3gnxzALBtMuha93jTZjxQgLg6d7AIpsYPupJmCLvHVf4eK6V7M0rOGJvI6iphV yPCS4XmfpT3bu2/HCTNqKAK2yBsKmy2Dc8YIv5lzu8tQGr08p4y+Aab/Dh/vpsZl88EM iv09C8LcTUYs3udF55TzJOSjgvu118a2diMauUxtvI4efASx44n+V9o413+4YCtPYJbu 7vGSgwvRlMKb5k1uQ6FYNqxcnP3fkE0zl1F4k8SyA9W8IGHoaQMfUOoiNMiuMBCB8a2/ 8x2g1e7WxvCh8pBUzKnlFYJozK+pcRqbmxq/hHbutXX+I5ld7abVfpjONmCeLqJH2+Yx ncfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=BI0V1APYLLZg+y9tuxzi74MUjtHjkVAo7OmZFgUYdFQ=; fh=VvU0LbOIaDT5AzPdwlcjDGgCdEYD/msdSvohhGzKJZY=; b=ipyQcpNukYZvAh7d+zlrRfHMe8LF8xL9qAIjC7AiESjJg8LH6iNc3c4QpvgISJhh9v celcTJ6uBDNNEMcKWcF5fJKVmIy45IXfw8bLaYui6kwjdIajvo8HU6VE5NZVFQkIPi93 HSeaWkOYLwUvpikToBhLRyKM3iwciMMwfxVShecmMPrEecC9Bhuqzc75OE14ARVpO6Wn 9fmAr2qygoFy66iCsKs+vo5Z9QsMLwHjU8XCLaAPRaZsUW3b+CWTtW0xqYXsMq5KIZY8 /IphzS7AiD1MnzSHj6JCOkDNODNGb/BjtFzO1uwq00hB9nHZmoHwME0jExk5gJlCYJ9Z pATA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=G63iFwyO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o20-20020a17090a5b1400b00262ebe643a2si2097985pji.186.2023.07.05.11.19.48; Wed, 05 Jul 2023 11:20:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=G63iFwyO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230268AbjGESTQ (ORCPT + 99 others); Wed, 5 Jul 2023 14:19:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232348AbjGESSb (ORCPT ); Wed, 5 Jul 2023 14:18:31 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 879A81BD3 for ; Wed, 5 Jul 2023 11:17:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688581036; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=BI0V1APYLLZg+y9tuxzi74MUjtHjkVAo7OmZFgUYdFQ=; b=G63iFwyOQ0kv36uXUPRQHMm6loR5MYflMD9fHzMp2+LUAPPcm6/QGpxGRighZP2DOBvMe4 9pPSVrhs+dHbCnIHx8+BdrOCmVypSsZ0ZT4T0cjonbGgFEUNkjzDRPEvd5nZ2IitjG+guF ybIRjwQArnsPKIubQQIDpAvfjVSB+ak= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-654--O94TDN7PcGOu2Irqnh9Aw-1; Wed, 05 Jul 2023 14:17:11 -0400 X-MC-Unique: -O94TDN7PcGOu2Irqnh9Aw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9A18E856F67; Wed, 5 Jul 2023 18:17:08 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8918B18EB4; Wed, 5 Jul 2023 18:17:03 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Peter Zijlstra , Nicolas Saenz Julienne , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 12/14] context_tracking,x86: Defer kernel text patching IPIs Date: Wed, 5 Jul 2023 19:12:54 +0100 Message-Id: <20230705181256.3539027-13-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770605721666387779?= X-GMAIL-MSGID: =?utf-8?q?1770605721666387779?= text_poke_bp_batch() sends IPIs to all online CPUs to synchronize them vs the newly patched instruction. CPUs that are executing in userspace do not need this synchronization to happen immediately, and this is actually harmful interference for NOHZ_FULL CPUs. As the synchronization IPIs are sent using a blocking call, returning from text_poke_bp_batch() implies all CPUs will observe the patched instruction(s), and this should be preserved even if the IPI is deferred. In other words, to safely defer this synchronization, any kernel instruction leading to the execution of the deferred instruction sync (ct_work_flush()) must *not* be mutable (patchable) at runtime. This means we must pay attention to mutable instructions in the early entry code: - alternatives - static keys - all sorts of probes (kprobes/ftrace/bpf/???) The early entry code leading to ct_work_flush() is noinstr, which gets rid of the probes. Alternatives are safe, because it's boot-time patching (before SMP is even brought up) which is before any IPI deferral can happen. This leaves us with static keys. Any static key used in early entry code should be only forever-enabled at boot time, IOW __ro_after_init (pretty much like alternatives). Objtool is now able to point at static keys that don't respect this, and all static keys used in early entry code have now been verified as behaving like so. Leverage the new context_tracking infrastructure to defer sync_core() IPIs to a target CPU's next kernel entry. Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Nicolas Saenz Julienne Signed-off-by: Valentin Schneider --- arch/x86/include/asm/context_tracking_work.h | 6 +++-- arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 24 ++++++++++++++++---- arch/x86/kernel/kprobes/core.c | 4 ++-- arch/x86/kernel/kprobes/opt.c | 4 ++-- arch/x86/kernel/module.c | 2 +- include/linux/context_tracking_work.h | 4 ++-- 7 files changed, 32 insertions(+), 13 deletions(-) diff --git a/arch/x86/include/asm/context_tracking_work.h b/arch/x86/include/asm/context_tracking_work.h index 5bc29e6b2ed38..2c66687ce00e2 100644 --- a/arch/x86/include/asm/context_tracking_work.h +++ b/arch/x86/include/asm/context_tracking_work.h @@ -2,11 +2,13 @@ #ifndef _ASM_X86_CONTEXT_TRACKING_WORK_H #define _ASM_X86_CONTEXT_TRACKING_WORK_H +#include + static __always_inline void arch_context_tracking_work(int work) { switch (work) { - case CONTEXT_WORK_n: - // Do work... + case CONTEXT_WORK_SYNC: + sync_core(); break; } } diff --git a/arch/x86/include/asm/text-patching.h b/arch/x86/include/asm/text-patching.h index 29832c338cdc5..b6939e965e69d 100644 --- a/arch/x86/include/asm/text-patching.h +++ b/arch/x86/include/asm/text-patching.h @@ -43,6 +43,7 @@ extern void text_poke_early(void *addr, const void *opcode, size_t len); */ extern void *text_poke(void *addr, const void *opcode, size_t len); extern void text_poke_sync(void); +extern void text_poke_sync_deferrable(void); extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len); extern void *text_poke_copy(void *addr, const void *opcode, size_t len); extern void *text_poke_copy_locked(void *addr, const void *opcode, size_t len, bool core_ok); diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index f615e0cb6d932..7770aef2204f3 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -1765,9 +1766,24 @@ static void do_sync_core(void *info) sync_core(); } +static bool do_sync_core_defer_cond(int cpu, void *info) +{ + return !ct_set_cpu_work(cpu, CONTEXT_WORK_SYNC); +} + +static void __text_poke_sync(smp_cond_func_t cond_func) +{ + on_each_cpu_cond(cond_func, do_sync_core, NULL, 1); +} + void text_poke_sync(void) { - on_each_cpu(do_sync_core, NULL, 1); + __text_poke_sync(NULL); +} + +void text_poke_sync_deferrable(void) +{ + __text_poke_sync(do_sync_core_defer_cond); } /* @@ -1967,7 +1983,7 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries text_poke(text_poke_addr(&tp[i]), &int3, INT3_INSN_SIZE); } - text_poke_sync(); + text_poke_sync_deferrable(); /* * Second step: update all but the first byte of the patched range. @@ -2029,7 +2045,7 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries * not necessary and we'd be safe even without it. But * better safe than sorry (plus there's not only Intel). */ - text_poke_sync(); + text_poke_sync_deferrable(); } /* @@ -2050,7 +2066,7 @@ static void text_poke_bp_batch(struct text_poke_loc *tp, unsigned int nr_entries } if (do_sync) - text_poke_sync(); + text_poke_sync_deferrable(); /* * Remove and wait for refs to be zero. diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c index f7f6042eb7e6c..a38c914753397 100644 --- a/arch/x86/kernel/kprobes/core.c +++ b/arch/x86/kernel/kprobes/core.c @@ -735,7 +735,7 @@ void arch_arm_kprobe(struct kprobe *p) u8 int3 = INT3_INSN_OPCODE; text_poke(p->addr, &int3, 1); - text_poke_sync(); + text_poke_sync_deferrable(); perf_event_text_poke(p->addr, &p->opcode, 1, &int3, 1); } @@ -745,7 +745,7 @@ void arch_disarm_kprobe(struct kprobe *p) perf_event_text_poke(p->addr, &int3, 1, &p->opcode, 1); text_poke(p->addr, &p->opcode, 1); - text_poke_sync(); + text_poke_sync_deferrable(); } void arch_remove_kprobe(struct kprobe *p) diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c index 57b0037d0a996..88451a744ceda 100644 --- a/arch/x86/kernel/kprobes/opt.c +++ b/arch/x86/kernel/kprobes/opt.c @@ -521,11 +521,11 @@ void arch_unoptimize_kprobe(struct optimized_kprobe *op) JMP32_INSN_SIZE - INT3_INSN_SIZE); text_poke(addr, new, INT3_INSN_SIZE); - text_poke_sync(); + text_poke_sync_deferrable(); text_poke(addr + INT3_INSN_SIZE, new + INT3_INSN_SIZE, JMP32_INSN_SIZE - INT3_INSN_SIZE); - text_poke_sync(); + text_poke_sync_deferrable(); perf_event_text_poke(op->kp.addr, old, JMP32_INSN_SIZE, new, JMP32_INSN_SIZE); } diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index b05f62ee2344b..8b4542dc51b6d 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -242,7 +242,7 @@ static int write_relocate_add(Elf64_Shdr *sechdrs, write, apply); if (!early) { - text_poke_sync(); + text_poke_sync_deferrable(); mutex_unlock(&text_mutex); } diff --git a/include/linux/context_tracking_work.h b/include/linux/context_tracking_work.h index 0b06c3dab58c7..b0c7463048b60 100644 --- a/include/linux/context_tracking_work.h +++ b/include/linux/context_tracking_work.h @@ -6,13 +6,13 @@ enum { CONTEXT_WORK_DISABLED_OFFSET, - CONTEXT_WORK_n_OFFSET, + CONTEXT_WORK_SYNC_OFFSET, CONTEXT_WORK_MAX_OFFSET }; enum ct_work { CONTEXT_WORK_DISABLED = BIT(CONTEXT_WORK_DISABLED_OFFSET), - CONTEXT_WORK_n = BIT(CONTEXT_WORK_n_OFFSET), + CONTEXT_WORK_SYNC = BIT(CONTEXT_WORK_SYNC_OFFSET), CONTEXT_WORK_MAX = BIT(CONTEXT_WORK_MAX_OFFSET) }; From patchwork Wed Jul 5 18:12:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116284 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2053071vqx; Wed, 5 Jul 2023 11:20:17 -0700 (PDT) X-Google-Smtp-Source: APBJJlGt/y5nbuqirZhF7tbwiwgpv/L9+5zlW0xCDf5BoDsl+y9SxaJqd/9FgncgjVUuuL7NENbK X-Received: by 2002:a92:d185:0:b0:346:e96:7cbd with SMTP id z5-20020a92d185000000b003460e967cbdmr10090542ilz.19.1688581217250; Wed, 05 Jul 2023 11:20:17 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581217; cv=none; d=google.com; s=arc-20160816; b=kKlB/ZUJo6742YYrmWVg5vXm6EHMs6XeBAhmc5D6wwBTs1HKbc8jmMgYoiKtLhAOxL 7RTQoIXmdia7qn2k0CdyaYc88mpfQ0UbBp+/Ma778IawSTy8fYGOhddaujTbm/8BjHIC sVQVywbMfth4hZ9nFawutd4YCC0nQwdsWa0ci1vy3YT4VtrWOlACNXa3wZawDBGFtVGk 95xfYXjVzds0CcGFS1zzZLPkKvlt5DK+FWM4dKoN+dQSJZPPhAGgw2iAQyWXA7Ic4Alv YKZMNqT0ZLUbL7QgPg3VWq2z2b+vWjNpKQFnkO1nrxmEfDw33zEYwPPlTd80uc6lGWxG YE3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=O0XyYRqaKclSpF/kJo4Izh2sg6d7OQ7Oy14ALgDyLr0=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=QL2IdzIvs4oZ5Xc9SUhmUjt7EifXxP/dRAHnQKAGVkvJV93dfA10d2NLHrK8WHLQ5Q OH94DOaGPLBGlkIHJTtVc/aVquAJHd155f3fj8RFqY03hkjFS0hEuhXbQMVEGk0J+tmY 8/4C6/s+hQSwiimnFqq45dhL8e1T8v3DwC/jZKt6xLha+x8nBF+Atr0bFT6LJet8/TPp RyZC3seQ0vIATG4OBL+86uy8piRvHCfDKgHSJ8hX8C4vAMn+YKrkodJzG32j5Jq0RMfH ZO5Ue0wRg8s8q5NW3BnlzilBboyLPdE+Phn2QPBMZPuTi2+3LDAb0GdAkJWHL0hUay/E ps5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=jE1QJBXJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i24-20020a63d458000000b0054ff0e193fesi24379289pgj.225.2023.07.05.11.20.01; Wed, 05 Jul 2023 11:20:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=jE1QJBXJ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232926AbjGEST1 (ORCPT + 99 others); Wed, 5 Jul 2023 14:19:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44152 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233508AbjGESSj (ORCPT ); Wed, 5 Jul 2023 14:18:39 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 814381FC4 for ; Wed, 5 Jul 2023 11:17:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688581041; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O0XyYRqaKclSpF/kJo4Izh2sg6d7OQ7Oy14ALgDyLr0=; b=jE1QJBXJxYWRKXUM7JGPZzZaHZW35aDfenAFgmKF+/qwKucoKMZHOKiNAeHfgwhSXpanVZ cw/wCMtY+ATasc1AgdO0aD+0cHuU5YmN8IQW3pba74XTYfFL2tSD3bkpsxz330rxyH2Lwo BGmGVARbFVwV+vgR/TDKMix5EDYyhXg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-528-mFoTFqj8OM6TsS7Ok5gs8A-1; Wed, 05 Jul 2023 14:17:17 -0400 X-MC-Unique: mFoTFqj8OM6TsS7Ok5gs8A-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 44FAB185A792; Wed, 5 Jul 2023 18:17:15 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C6AE518EB4; Wed, 5 Jul 2023 18:17:08 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 13/14] context_tracking,x86: Add infrastructure to defer kernel TLBI Date: Wed, 5 Jul 2023 19:12:55 +0100 Message-Id: <20230705181256.3539027-14-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770605738009778107?= X-GMAIL-MSGID: =?utf-8?q?1770605738009778107?= Kernel TLB invalidation IPIs are a common source of interference on NOHZ_FULL CPUs. Given NOHZ_FULL CPUs executing in userspace are not accessing any kernel addresses, these invalidations do not need to happen immediately, and can be deferred until the next user->kernel transition. Rather than make __flush_tlb_all() noinstr, add a minimal noinstr variant that doesn't try to leverage INVPCID. FIXME: not fully noinstr compliant XXX: same issue as with ins patching, when do we access data that should be invalidated? Signed-off-by: Valentin Schneider --- arch/x86/include/asm/context_tracking_work.h | 4 ++++ arch/x86/include/asm/tlbflush.h | 1 + arch/x86/mm/tlb.c | 17 +++++++++++++++++ include/linux/context_tracking_work.h | 2 ++ 4 files changed, 24 insertions(+) diff --git a/arch/x86/include/asm/context_tracking_work.h b/arch/x86/include/asm/context_tracking_work.h index 2c66687ce00e2..9d4f021b5a45b 100644 --- a/arch/x86/include/asm/context_tracking_work.h +++ b/arch/x86/include/asm/context_tracking_work.h @@ -3,6 +3,7 @@ #define _ASM_X86_CONTEXT_TRACKING_WORK_H #include +#include static __always_inline void arch_context_tracking_work(int work) { @@ -10,6 +11,9 @@ static __always_inline void arch_context_tracking_work(int work) case CONTEXT_WORK_SYNC: sync_core(); break; + case CONTEXT_WORK_TLBI: + __flush_tlb_all_noinstr(); + break; } } diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 75bfaa4210303..9064aa0027d05 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -15,6 +15,7 @@ #include void __flush_tlb_all(void); +void noinstr __flush_tlb_all_noinstr(void); #define TLB_FLUSH_ALL -1UL #define TLB_GENERATION_INVALID 0 diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 267acf27480af..631df9189ded4 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -1237,6 +1237,23 @@ void __flush_tlb_all(void) } EXPORT_SYMBOL_GPL(__flush_tlb_all); +void noinstr __flush_tlb_all_noinstr(void) +{ + /* + * This is for invocation in early entry code that cannot be + * instrumented. A RMW to CR4 works for most cases, but relies on + * being able to flip either of the PGE or PCIDE bits. Flipping CR4.PCID + * would require also resetting CR3.PCID, so just try with CR4.PGE, else + * do the CR3 write. + * + * TODO: paravirt + */ + if (cpu_feature_enabled(X86_FEATURE_PGE)) + __native_tlb_flush_global(this_cpu_read(cpu_tlbstate.cr4)); + else + flush_tlb_local(); +} + void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch) { struct flush_tlb_info *info; diff --git a/include/linux/context_tracking_work.h b/include/linux/context_tracking_work.h index b0c7463048b60..c084a60d2fec5 100644 --- a/include/linux/context_tracking_work.h +++ b/include/linux/context_tracking_work.h @@ -7,12 +7,14 @@ enum { CONTEXT_WORK_DISABLED_OFFSET, CONTEXT_WORK_SYNC_OFFSET, + CONTEXT_WORK_TLBI_OFFSET, CONTEXT_WORK_MAX_OFFSET }; enum ct_work { CONTEXT_WORK_DISABLED = BIT(CONTEXT_WORK_DISABLED_OFFSET), CONTEXT_WORK_SYNC = BIT(CONTEXT_WORK_SYNC_OFFSET), + CONTEXT_WORK_TLBI = BIT(CONTEXT_WORK_TLBI_OFFSET), CONTEXT_WORK_MAX = BIT(CONTEXT_WORK_MAX_OFFSET) }; From patchwork Wed Jul 5 18:12:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Valentin Schneider X-Patchwork-Id: 116285 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9f45:0:b0:3ea:f831:8777 with SMTP id v5csp2053102vqx; Wed, 5 Jul 2023 11:20:21 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4YxmsxMnVftCXOUz91KFKKBh+wejE78Yk7X+yAmXhMZbCDVCxu8wK6YBfAoloBpzu7oEXs X-Received: by 2002:a05:6871:4405:b0:1b0:5fc0:e2b5 with SMTP id nd5-20020a056871440500b001b05fc0e2b5mr19393444oab.53.1688581220759; Wed, 05 Jul 2023 11:20:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1688581220; cv=none; d=google.com; s=arc-20160816; b=rCQD3ojvBevmXHbepCTk7/jf7pSzEJp0+gaPLbQnzSqx2i6/Z8Qri0/UZrc+AVd/1s jDK1faeh/FU4VtlRs9rySUnPCFjAyd3Uv0FtFm1xndO196OYKs9ZXpPP2q+Cldujl9Oq 0kJ3iL0glXYa/iiuufV9gFDgLaYz15MTqdQ7JQ2JzJQZKs2uOx8kjUPKR3QR5Z0iQd0C 4N2WVxDiGpmnWbiuuPvOiwb+g67YxbNwdTr5/b5nphYWlMafWsgisQxvSHP2qndRW44J 22qabcdFJe/yHxDYxljGBTFRL1tw9UJOkK/QLPlvuwq4k1b1TljAxvYyHXu6QQlHm5yI DjKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4KAtt5Wg9gvWlallwwTPlSO2JwaIw83isELp5cjnhmg=; fh=dVCMRuKhK2aqTEuWethYlqkP4G2sINJQEE+rq5xoO1c=; b=dk/mkimenSgKo4gVy1aBDDeIZZNvTUOE77hxIoU0o8ryqsqSRE1vYYf1UF7WJMGP3C vxHrL5i/+Z/2Uxf9LNml5svXx8+AwoYPhD9Q40scY2abft2GwpolrUYm5hlD4c4/N8nE IZy52H+lAHRULvHDQBI3O0B7dvx+fbpStxwuCfBNG0MIjzXBqFD4tszJorBMVuPScrHy /JoQVER+S4AiBhLpy/VT46zreKhn9TEeGuUO0x1p3wXFu0rZJLrrWvMCOPsQuknA2eg2 wQH5KppHQxfNIGFd0oSRyL2i/7RYj4Zl/JcLY5eU/exLoH/BcUL2GpGEAxic5Iy2xUHV 8ozw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=FiDx++pe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mi2-20020a17090b4b4200b0024e35ef410fsi2426118pjb.131.2023.07.05.11.20.08; Wed, 05 Jul 2023 11:20:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=FiDx++pe; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232095AbjGESTc (ORCPT + 99 others); Wed, 5 Jul 2023 14:19:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233400AbjGESSn (ORCPT ); Wed, 5 Jul 2023 14:18:43 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EF9BD1728 for ; Wed, 5 Jul 2023 11:17:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688581045; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4KAtt5Wg9gvWlallwwTPlSO2JwaIw83isELp5cjnhmg=; b=FiDx++petWFoVlFU4wdVWFNOAZEWd3qrJ4nbsyTuguUgGFpVzqx6Lf4Y+U/+1RyGYDUWZX PSfSZmAfOl+H3L1c9jAQEbK6hsmo0uyS/YFeBvHGUSruThn5ifPkeJHzlqUqqoLKzfWAQU AyARbK+dPRqZtvLyxxvABIq+q+b9PkI= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-376-XnGbn6mUOsqBjFOroOxxLg-1; Wed, 05 Jul 2023 14:17:22 -0400 X-MC-Unique: XnGbn6mUOsqBjFOroOxxLg-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 48B262808E60; Wed, 5 Jul 2023 18:17:20 +0000 (UTC) Received: from vschneid.remote.csb (unknown [10.42.28.164]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7229918EB4; Wed, 5 Jul 2023 18:17:15 +0000 (UTC) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org Cc: Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Peter Zijlstra , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , =?utf-8?q?Thomas_Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: [RFC PATCH 14/14] x86/mm, mm/vmalloc: Defer flush_tlb_kernel_range() targeting NOHZ_FULL CPUs Date: Wed, 5 Jul 2023 19:12:56 +0100 Message-Id: <20230705181256.3539027-15-vschneid@redhat.com> In-Reply-To: <20230705181256.3539027-1-vschneid@redhat.com> References: <20230705181256.3539027-1-vschneid@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.5 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE, T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1770605741873280317?= X-GMAIL-MSGID: =?utf-8?q?1770605741873280317?= vunmap()'s issued from housekeeping CPUs are a relatively common source of interference for isolated NOHZ_FULL CPUs, as they are hit by the flush_tlb_kernel_range() IPIs. Given that CPUs executing in userspace do not access data in the vmalloc range, these IPIs could be deferred until their next kernel entry. This does require a guarantee that nothing in the vmalloc range can be accessed in early entry code. vmalloc'd kernel stacks (VMAP_STACK) are AFAICT a safe exception, as a task running in userspace needs to enter kernelspace to execute do_exit() before its stack can be vfree'd. XXX: Validation that nothing in the vmalloc range is accessed in .noinstr or somesuch? Blindly deferring any and all flush of the kernel mappings is a risky move, so introduce a variant of flush_tlb_kernel_range() that explicitly allows deferral. Use it for vunmap flushes. Note that while flush_tlb_kernel_range() may end up issuing a full flush (including user mappings), this only happens when reaching a invalidation range threshold where it is cheaper to do a full flush than to individually invalidate each page in the range via INVLPG. IOW, it doesn't *require* invalidating user mappings, and thus remains safe to defer until a later kernel entry. Signed-off-by: Valentin Schneider --- arch/x86/include/asm/tlbflush.h | 1 + arch/x86/mm/tlb.c | 23 ++++++++++++++++++++--- mm/vmalloc.c | 15 ++++++++++----- 3 files changed, 31 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 9064aa0027d05..2b3605c09649c 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -255,6 +255,7 @@ extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables); extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); +extern void flush_tlb_kernel_range_deferrable(unsigned long start, unsigned long end); static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) { diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 631df9189ded4..bb18b35e61b4a 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -1045,6 +1046,11 @@ static void do_flush_tlb_all(void *info) __flush_tlb_all(); } +static bool do_kernel_flush_defer_cond(int cpu, void *info) +{ + return !ct_set_cpu_work(cpu, CONTEXT_WORK_TLBI); +} + void flush_tlb_all(void) { count_vm_tlb_event(NR_TLB_REMOTE_FLUSH); @@ -1061,12 +1067,13 @@ static void do_kernel_range_flush(void *info) flush_tlb_one_kernel(addr); } -void flush_tlb_kernel_range(unsigned long start, unsigned long end) +static inline void +__flush_tlb_kernel_range(smp_cond_func_t cond_func, unsigned long start, unsigned long end) { /* Balance as user space task's flush, a bit conservative */ if (end == TLB_FLUSH_ALL || (end - start) > tlb_single_page_flush_ceiling << PAGE_SHIFT) { - on_each_cpu(do_flush_tlb_all, NULL, 1); + on_each_cpu_cond(cond_func, do_flush_tlb_all, NULL, 1); } else { struct flush_tlb_info *info; @@ -1074,13 +1081,23 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) info = get_flush_tlb_info(NULL, start, end, 0, false, TLB_GENERATION_INVALID); - on_each_cpu(do_kernel_range_flush, info, 1); + on_each_cpu_cond(cond_func, do_kernel_range_flush, info, 1); put_flush_tlb_info(); preempt_enable(); } } +void flush_tlb_kernel_range(unsigned long start, unsigned long end) +{ + __flush_tlb_kernel_range(NULL, start, end); +} + +void flush_tlb_kernel_range_deferrable(unsigned long start, unsigned long end) +{ + __flush_tlb_kernel_range(do_kernel_flush_defer_cond, start, end); +} + /* * This can be used from process context to figure out what the value of * CR3 is without needing to do a (slow) __read_cr3(). diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 1d13d71687d73..10f99224e12e0 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -439,6 +439,11 @@ void vunmap_range_noflush(unsigned long start, unsigned long end) __vunmap_range_noflush(start, end); } +#ifndef flush_tlb_kernel_range_deferrable +#define flush_tlb_kernel_range_deferrable(start, end) \ + flush_tlb_kernel_range(start, end) +#endif + /** * vunmap_range - unmap kernel virtual addresses * @addr: start of the VM area to unmap @@ -452,7 +457,7 @@ void vunmap_range(unsigned long addr, unsigned long end) { flush_cache_vunmap(addr, end); vunmap_range_noflush(addr, end); - flush_tlb_kernel_range(addr, end); + flush_tlb_kernel_range_deferrable(addr, end); } static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, @@ -1747,7 +1752,7 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) list_last_entry(&local_purge_list, struct vmap_area, list)->va_end); - flush_tlb_kernel_range(start, end); + flush_tlb_kernel_range_deferrable(start, end); resched_threshold = lazy_max_pages() << 1; spin_lock(&free_vmap_area_lock); @@ -1849,7 +1854,7 @@ static void free_unmap_vmap_area(struct vmap_area *va) flush_cache_vunmap(va->va_start, va->va_end); vunmap_range_noflush(va->va_start, va->va_end); if (debug_pagealloc_enabled_static()) - flush_tlb_kernel_range(va->va_start, va->va_end); + flush_tlb_kernel_range_deferrable(va->va_start, va->va_end); free_vmap_area_noflush(va); } @@ -2207,7 +2212,7 @@ static void vb_free(unsigned long addr, unsigned long size) vunmap_range_noflush(addr, addr + size); if (debug_pagealloc_enabled_static()) - flush_tlb_kernel_range(addr, addr + size); + flush_tlb_kernel_range_deferrable(addr, addr + size); spin_lock(&vb->lock); @@ -2260,7 +2265,7 @@ static void _vm_unmap_aliases(unsigned long start, unsigned long end, int flush) mutex_lock(&vmap_purge_lock); purge_fragmented_blocks_allcpus(); if (!__purge_vmap_area_lazy(start, end) && flush) - flush_tlb_kernel_range(start, end); + flush_tlb_kernel_range_deferrable(start, end); mutex_unlock(&vmap_purge_lock); }