From patchwork Wed Oct 18 06:03:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuyi Zhou X-Patchwork-Id: 154672 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4588744vqb; Tue, 17 Oct 2023 23:04:32 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFYX6DSiyQX3bDnRaorIH177+a5fi5AmQeEXkFCI04eZxypnL4PUtFVO/8igDH3pD0n1Yex X-Received: by 2002:a05:6359:8001:b0:166:db65:af9a with SMTP id rc1-20020a056359800100b00166db65af9amr3721703rwb.2.1697609072126; Tue, 17 Oct 2023 23:04:32 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697609072; cv=none; d=google.com; s=arc-20160816; b=ESiSHN9yiFtUkP6pBeMHZs+zBBi6jvB4pZ1Zi8OF32jpT6QEVlRVZFNSNYo9v2pGht Ztddxz1fsgjATutXNKAhnT/jhXJPFNHeBawVH2x4hAiTO7zCZmseMzJ2jOBWRtdt3WZp USNnvilEjWw+3AVBKwl6XtegLJGWeUWGHYsdGmI/wtithjWd6tdCc+daGNzwPs4s/7Bn ++GTkqTBpwpwgnYHXzOWuX1sV66zOks2r1v2x8VAL+yplLsXrdCxoRCgfoqsiUbOPnfG WD4y/DGplArgP+w6/O/PeeUA8fVtq/rvWoymwNkDLgmXDwueqnfTGs+XOYdRKDpU8SwW oKTw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=FNW2oYGtelVvgczyk2A13Ij/ocoUNYFTCyV6b+P6gh8=; fh=DR7g1EcWKOXTEoooUPBSJXUaklSrDEYzv6YDdhz1CwE=; b=IMXouEPgAzWg3M6utXOO1uJygh1zanG7+eF0GtsYDFRttp2bagdsEKw0+ieBHuE0sN RbWhalYyCFyeTVUKKtVOTtWgUt+7EfycZH0tQiO9f7vyP28/DIRjnExVQ21fQ5/Lkpl4 +Zl1uLfpTGi4sGqpPL5RGplGmE1gpSDZsWM+t5qbDCW9Y1CNShc+XEXtTOjjXehCSUbF kh2G1j+QBo7BI+LKtr5QFANEssPtfG7AzX0GNlgwhq8i1jSxeE7C1bBzWZDZVIjIHRZr OO+Uh9wjkXpToZ+emdTpWe4u4dVh367+6Tv6VoQsZXkby0PLWM01gzDoh4+TiCNoQDZ/ I5Yw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=bAuwMM04; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id u30-20020a056a00099e00b006bca1443f29si3506836pfg.297.2023.10.17.23.04.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 23:04:32 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=bAuwMM04; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 78A8780952D5; Tue, 17 Oct 2023 23:04:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344456AbjJRGEA (ORCPT + 23 others); Wed, 18 Oct 2023 02:04:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41256 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344419AbjJRGD6 (ORCPT ); Wed, 18 Oct 2023 02:03:58 -0400 Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF755F7 for ; Tue, 17 Oct 2023 23:03:34 -0700 (PDT) Received: by mail-pg1-x52e.google.com with SMTP id 41be03b00d2f7-5a9bf4fbd3fso3906115a12.1 for ; Tue, 17 Oct 2023 23:03:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1697609014; x=1698213814; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FNW2oYGtelVvgczyk2A13Ij/ocoUNYFTCyV6b+P6gh8=; b=bAuwMM04S+xXGF2mph5y8RlqZdBfuL+ajSBfC8QXGjoeoJLWhuDh9nA+BdUrXXdLWW rAcCpcs/Es1uO8VE3XLTgYaTkbQjVIEV9jUb7FbiWZw7IQFuWX4KPZshS/VpkfuxGY95 gG+7EPKiLPEgzcEEtBy0H+VoRbAHW+VPthYHyGF2s1f0FyHjGshSkGdg98/x2qOIMNX3 KzKzI0rP+BpU16oX2RhLPKA7HZ8oP3OY22w/9SvTGbbI+UBlJ4soHBifGHe1CQu6RrRg R3vJ9t8sdS0Zt4Wy4xat/oxRzcwuGr+5ANeSo+Tbq7RprU/9rGDHYL09ypIrKBbbatUI jiog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697609014; x=1698213814; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FNW2oYGtelVvgczyk2A13Ij/ocoUNYFTCyV6b+P6gh8=; b=BA3k0JmRTqUOp0ziil2uuPLvvmTj4Z1IZUEwlrPJ69PZCCHf16wxQsTtgTTzJtqFrp V5qy9cojHMickWNrwkITHliZD1C7yxHqCuztqU5/IzMgsw7SG9sXLLtnYa9GTfW8Z39x DqsNhl24phCvDqhs+onzvF0XSEmt1DIiRGeD9zkdkAboms1f3P1apzqJMDXIIrjuAIQM jqAzKTjI7EV6ZIGse3S/Le/jn/JmbJD5wfKpWevdGqHy6S+GJpagBpyT88rqK8prVrJd 9n3LjvZXHH62GZ8i+bpqMiURaDpMx2Qli1NP9Qj7QwtGvNHdKXrzIR2HfpiY5hY8js7U EAFw== X-Gm-Message-State: AOJu0Yx/csEb4W/7uRZkVOXTRxIbXW9vCybDwnmesCeMYWpDd6aBKwfj wQCh0yGhp3ONd8z68B6yEzD1UQ+LTpUEkT4msEc= X-Received: by 2002:a05:6a20:72ab:b0:15e:bb88:b76e with SMTP id o43-20020a056a2072ab00b0015ebb88b76emr4544752pzk.14.1697609014178; Tue, 17 Oct 2023 23:03:34 -0700 (PDT) Received: from n37-019-243.byted.org ([180.184.51.40]) by smtp.gmail.com with ESMTPSA id b14-20020a17090acc0e00b00276cb03a0e9sm505782pju.46.2023.10.17.23.03.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 23:03:33 -0700 (PDT) From: Chuyi Zhou To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, tj@kernel.org, linux-kernel@vger.kernel.org, Chuyi Zhou Subject: [PATCH bpf-next v6 1/8] cgroup: Prepare for using css_task_iter_*() in BPF Date: Wed, 18 Oct 2023 14:03:11 +0800 Message-Id: <20231018060318.105524-2-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20231018060318.105524-1-zhouchuyi@bytedance.com> References: <20231018060318.105524-1-zhouchuyi@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Tue, 17 Oct 2023 23:04:29 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780072130261077204 X-GMAIL-MSGID: 1780072130261077204 This patch makes some preparations for using css_task_iter_*() in BPF Program. 1. Flags CSS_TASK_ITER_* are #define-s and it's not easy for bpf prog to use them. Convert them to enum so bpf prog can take them from vmlinux.h. 2. In the next patch we will add css_task_iter_*() in common kfuncs which is not safe. Since css_task_iter_*() does spin_unlock_irq() which might screw up irq flags depending on the context where bpf prog is running. So we should use irqsave/irqrestore here and the switching is harmless. Suggested-by: Alexei Starovoitov Signed-off-by: Chuyi Zhou Acked-by: Tejun Heo --- include/linux/cgroup.h | 12 +++++------- kernel/cgroup/cgroup.c | 18 ++++++++++++------ 2 files changed, 17 insertions(+), 13 deletions(-) diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index b307013b9c6c..0ef0af66080e 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -40,13 +40,11 @@ struct kernel_clone_args; #define CGROUP_WEIGHT_DFL 100 #define CGROUP_WEIGHT_MAX 10000 -/* walk only threadgroup leaders */ -#define CSS_TASK_ITER_PROCS (1U << 0) -/* walk all threaded css_sets in the domain */ -#define CSS_TASK_ITER_THREADED (1U << 1) - -/* internal flags */ -#define CSS_TASK_ITER_SKIPPED (1U << 16) +enum { + CSS_TASK_ITER_PROCS = (1U << 0), /* walk only threadgroup leaders */ + CSS_TASK_ITER_THREADED = (1U << 1), /* walk all threaded css_sets in the domain */ + CSS_TASK_ITER_SKIPPED = (1U << 16), /* internal flags */ +}; /* a css_task_iter should be treated as an opaque object */ struct css_task_iter { diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 1fb7f562289d..b6d64f3b8888 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -4917,9 +4917,11 @@ static void css_task_iter_advance(struct css_task_iter *it) void css_task_iter_start(struct cgroup_subsys_state *css, unsigned int flags, struct css_task_iter *it) { + unsigned long irqflags; + memset(it, 0, sizeof(*it)); - spin_lock_irq(&css_set_lock); + spin_lock_irqsave(&css_set_lock, irqflags); it->ss = css->ss; it->flags = flags; @@ -4933,7 +4935,7 @@ void css_task_iter_start(struct cgroup_subsys_state *css, unsigned int flags, css_task_iter_advance(it); - spin_unlock_irq(&css_set_lock); + spin_unlock_irqrestore(&css_set_lock, irqflags); } /** @@ -4946,12 +4948,14 @@ void css_task_iter_start(struct cgroup_subsys_state *css, unsigned int flags, */ struct task_struct *css_task_iter_next(struct css_task_iter *it) { + unsigned long irqflags; + if (it->cur_task) { put_task_struct(it->cur_task); it->cur_task = NULL; } - spin_lock_irq(&css_set_lock); + spin_lock_irqsave(&css_set_lock, irqflags); /* @it may be half-advanced by skips, finish advancing */ if (it->flags & CSS_TASK_ITER_SKIPPED) @@ -4964,7 +4968,7 @@ struct task_struct *css_task_iter_next(struct css_task_iter *it) css_task_iter_advance(it); } - spin_unlock_irq(&css_set_lock); + spin_unlock_irqrestore(&css_set_lock, irqflags); return it->cur_task; } @@ -4977,11 +4981,13 @@ struct task_struct *css_task_iter_next(struct css_task_iter *it) */ void css_task_iter_end(struct css_task_iter *it) { + unsigned long irqflags; + if (it->cur_cset) { - spin_lock_irq(&css_set_lock); + spin_lock_irqsave(&css_set_lock, irqflags); list_del(&it->iters_node); put_css_set_locked(it->cur_cset); - spin_unlock_irq(&css_set_lock); + spin_unlock_irqrestore(&css_set_lock, irqflags); } if (it->cur_dcset) From patchwork Wed Oct 18 06:03:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuyi Zhou X-Patchwork-Id: 154670 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4588543vqb; Tue, 17 Oct 2023 23:04:04 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHl+t7NGinmwzkLXrQhLRrjJeH9rBhAujT+6b4vjbLhezytoZ7jZtdVq4pYaPfAtg7IvJ83 X-Received: by 2002:a05:6359:5e20:b0:166:d93e:339 with SMTP id pw32-20020a0563595e2000b00166d93e0339mr4217317rwb.0.1697609043591; Tue, 17 Oct 2023 23:04:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697609043; cv=none; d=google.com; s=arc-20160816; b=jE6ecM1dUsxAdC7sqh3l/hJwWc+kyIjzwG+D7PL13JnT1lEtKnhyuPbtHW383Ch43A Y0qu/To263CsrTbFDKAV+7tIz0z/O4e+mwXyQkmzdFV7zZWqytx52TRcSblIzFAsuDKd 93D/a9sO7fSPag6kxotD+v5V4COQyoudnypjIcYuxpf+GQURyn79RS4iLXlPWduPx4go yk5tBnE57JRk4t0igrGxaGp1x5AHCnddIdME1CzBBpqI+uiH/PRGXiB45cVTvWLdOYz7 9ZqZGQ27PJMtOnR84r1UEO1BsZF1o/olMXui+wm1HfLFeo5VsXmXcMPZzmCF5zTlM1pJ XzQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=BbIp3xjsMf+vYU1+9/Ac/+6x9YCK8BhSfh7PegmcPCk=; fh=DR7g1EcWKOXTEoooUPBSJXUaklSrDEYzv6YDdhz1CwE=; b=y0jzLUjG5wRNyJSeyv5yEPU+WCwwwzi1vrq4HFyOISdTm50TwzvrRtgW9L2MSHy4tO qQWM1vwELqssytCKqCaexj/TA/hf2WOCqn6kFq7YdMZIMzUnWaQA8TVfDdynvBhC3LV1 ZDkQfzA7ekYdA0L3L8B57Kg71uDpbdfie0YLxk1T5Fh+acAOvG0ozZ9bRZLnzM83oHIv Jf1jgv30mubkUO9FJPI6TDhnMaLtkmOGPp/V57w+sMhNJRYiZX75fM31uV8eYD3iH8Zv 5XTPjUaY1InZ5BdJYdNd15JqS/C8jZLjQb5K2aGMaCFsx1+yjMojVRuLs9kHzihXuc5c YQwQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=lNlW4MHS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from agentk.vger.email (agentk.vger.email. [2620:137:e000::3:2]) by mx.google.com with ESMTPS id x14-20020aa79ace000000b006bd8c5a3b52si3187408pfp.110.2023.10.17.23.04.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 23:04:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) client-ip=2620:137:e000::3:2; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=lNlW4MHS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:2 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by agentk.vger.email (Postfix) with ESMTP id A5B118024CCB; Tue, 17 Oct 2023 23:03:58 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at agentk.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229605AbjJRGDl (ORCPT + 23 others); Wed, 18 Oct 2023 02:03:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229570AbjJRGDj (ORCPT ); Wed, 18 Oct 2023 02:03:39 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7C03C6 for ; Tue, 17 Oct 2023 23:03:37 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id 41be03b00d2f7-5ae99bb5ccdso2404754a12.1 for ; Tue, 17 Oct 2023 23:03:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1697609017; x=1698213817; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BbIp3xjsMf+vYU1+9/Ac/+6x9YCK8BhSfh7PegmcPCk=; b=lNlW4MHStfBXSQ1ZYeV+XIS3QOZLzJkZOHDX+SGQImWJR7wjGSf+LEEKSXWalAzDgq pTspO64J2sy8MGGCQfunb4OU9Th+nqO2tKCiVZepLEuinRBm1hRJaRk+3jwVLGo5GCLQ o7ZhwvrYdBBzmIknzQxpxRAjd/EyvkzZwcohPREpsN73nDrgmzTkbAlJm2r0ENKyjmrR 0sBrxTdId0UKqNabUoYw4oOqYO5MXGAw9XPv15KLdJUjnWwX4NVJh93nb3Fq9ZTdxnpY P+YQHXbNtVJwCu367oMJCHfzwXVr8i21f1tKDYt5KrcOPMeSbSwAh/Kj68yg2oEfNZwh r5CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697609017; x=1698213817; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BbIp3xjsMf+vYU1+9/Ac/+6x9YCK8BhSfh7PegmcPCk=; b=vcTCp/wx2oka2hxHYyLMaqkUMBs7nUNzmzK4Oqjqn9eqUoeqs3SHYLtbvOyOab6x01 ESO6EWyWWb71kae48dIyIW/vce4ffYbHYxqK36SAQzAqqUl6vnaM+tVR7FAnXePsWczD nDqeFDOe4+O5TmrQ/WFE2QVMKFCtXPnreeqqr2AovhmRZBZfy7vPKhrEAXi/T2nt6RjO 2J8kdRnBf/fHaORyayP0wURrbR3JTsQjfwQSytS75dNJPyLofclxdhH7ckQVN5CJo/m2 xxTYNIwebFZ2WM5oBSYcNK536rzCJY1H0KMuGvu2XbwijnCIFTJwPov5FIW5oswRuuws BMwQ== X-Gm-Message-State: AOJu0YwhEmmdQb5tKD7HChbxEbc5CYXPUjtOQJv4N6iAZmZ4b2p3qtr6 5u8MHD2qlLOHzeoB+ylLmgkTBw== X-Received: by 2002:a05:6a21:999c:b0:17a:f2ed:e921 with SMTP id ve28-20020a056a21999c00b0017af2ede921mr4145757pzb.55.1697609017344; Tue, 17 Oct 2023 23:03:37 -0700 (PDT) Received: from n37-019-243.byted.org ([180.184.51.40]) by smtp.gmail.com with ESMTPSA id b14-20020a17090acc0e00b00276cb03a0e9sm505782pju.46.2023.10.17.23.03.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 23:03:37 -0700 (PDT) From: Chuyi Zhou To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, tj@kernel.org, linux-kernel@vger.kernel.org, Chuyi Zhou Subject: [PATCH bpf-next v6 2/8] bpf: Introduce css_task open-coded iterator kfuncs Date: Wed, 18 Oct 2023 14:03:12 +0800 Message-Id: <20231018060318.105524-3-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20231018060318.105524-1-zhouchuyi@bytedance.com> References: <20231018060318.105524-1-zhouchuyi@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on agentk.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (agentk.vger.email [0.0.0.0]); Tue, 17 Oct 2023 23:03:58 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780072100696327750 X-GMAIL-MSGID: 1780072100696327750 This patch adds kfuncs bpf_iter_css_task_{new,next,destroy} which allow creation and manipulation of struct bpf_iter_css_task in open-coded iterator style. These kfuncs actually wrapps css_task_iter_{start,next, end}. BPF programs can use these kfuncs through bpf_for_each macro for iteration of all tasks under a css. css_task_iter_*() would try to get the global spin-lock *css_set_lock*, so the bpf side has to be careful in where it allows to use this iter. Currently we only allow it in bpf_lsm and bpf iter-s. Signed-off-by: Chuyi Zhou Acked-by: Tejun Heo --- kernel/bpf/helpers.c | 3 + kernel/bpf/task_iter.c | 58 +++++++++++++++++++ kernel/bpf/verifier.c | 23 ++++++++ .../testing/selftests/bpf/bpf_experimental.h | 8 +++ 4 files changed, 92 insertions(+) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 61f51dee8448..c01441db9fd5 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2560,6 +2560,9 @@ BTF_ID_FLAGS(func, bpf_iter_num_destroy, KF_ITER_DESTROY) BTF_ID_FLAGS(func, bpf_iter_task_vma_new, KF_ITER_NEW | KF_RCU) BTF_ID_FLAGS(func, bpf_iter_task_vma_next, KF_ITER_NEXT | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY) +BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_css_task_destroy, KF_ITER_DESTROY) BTF_ID_FLAGS(func, bpf_dynptr_adjust) BTF_ID_FLAGS(func, bpf_dynptr_is_null) BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly) diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c index fef17628341f..e4126698cecf 100644 --- a/kernel/bpf/task_iter.c +++ b/kernel/bpf/task_iter.c @@ -894,6 +894,64 @@ __bpf_kfunc void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it) __diag_pop(); +struct bpf_iter_css_task { + __u64 __opaque[1]; +} __attribute__((aligned(8))); + +struct bpf_iter_css_task_kern { + struct css_task_iter *css_it; +} __attribute__((aligned(8))); + +__diag_push(); +__diag_ignore_all("-Wmissing-prototypes", + "Global functions as their definitions will be in vmlinux BTF"); + +__bpf_kfunc int bpf_iter_css_task_new(struct bpf_iter_css_task *it, + struct cgroup_subsys_state *css, unsigned int flags) +{ + struct bpf_iter_css_task_kern *kit = (void *)it; + + BUILD_BUG_ON(sizeof(struct bpf_iter_css_task_kern) != sizeof(struct bpf_iter_css_task)); + BUILD_BUG_ON(__alignof__(struct bpf_iter_css_task_kern) != + __alignof__(struct bpf_iter_css_task)); + kit->css_it = NULL; + switch (flags) { + case CSS_TASK_ITER_PROCS | CSS_TASK_ITER_THREADED: + case CSS_TASK_ITER_PROCS: + case 0: + break; + default: + return -EINVAL; + } + + kit->css_it = bpf_mem_alloc(&bpf_global_ma, sizeof(struct css_task_iter)); + if (!kit->css_it) + return -ENOMEM; + css_task_iter_start(css, flags, kit->css_it); + return 0; +} + +__bpf_kfunc struct task_struct *bpf_iter_css_task_next(struct bpf_iter_css_task *it) +{ + struct bpf_iter_css_task_kern *kit = (void *)it; + + if (!kit->css_it) + return NULL; + return css_task_iter_next(kit->css_it); +} + +__bpf_kfunc void bpf_iter_css_task_destroy(struct bpf_iter_css_task *it) +{ + struct bpf_iter_css_task_kern *kit = (void *)it; + + if (!kit->css_it) + return; + css_task_iter_end(kit->css_it); + bpf_mem_free(&bpf_global_ma, kit->css_it); +} + +__diag_pop(); + DEFINE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work); static void do_mmap_read_unlock(struct irq_work *entry) diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index bb58987e4844..974713185269 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -10472,6 +10472,7 @@ enum special_kfunc_type { KF_bpf_percpu_obj_new_impl, KF_bpf_percpu_obj_drop_impl, KF_bpf_throw, + KF_bpf_iter_css_task_new, }; BTF_SET_START(special_kfunc_set) @@ -10495,6 +10496,7 @@ BTF_ID(func, bpf_dynptr_clone) BTF_ID(func, bpf_percpu_obj_new_impl) BTF_ID(func, bpf_percpu_obj_drop_impl) BTF_ID(func, bpf_throw) +BTF_ID(func, bpf_iter_css_task_new) BTF_SET_END(special_kfunc_set) BTF_ID_LIST(special_kfunc_list) @@ -10520,6 +10522,7 @@ BTF_ID(func, bpf_dynptr_clone) BTF_ID(func, bpf_percpu_obj_new_impl) BTF_ID(func, bpf_percpu_obj_drop_impl) BTF_ID(func, bpf_throw) +BTF_ID(func, bpf_iter_css_task_new) static bool is_kfunc_ret_null(struct bpf_kfunc_call_arg_meta *meta) { @@ -11050,6 +11053,20 @@ static int process_kf_arg_ptr_to_rbtree_node(struct bpf_verifier_env *env, &meta->arg_rbtree_root.field); } +static bool check_css_task_iter_allowlist(struct bpf_verifier_env *env) +{ + enum bpf_prog_type prog_type = resolve_prog_type(env->prog); + + switch (prog_type) { + case BPF_PROG_TYPE_LSM: + return true; + case BPF_TRACE_ITER: + return env->prog->aux->sleepable; + default: + return false; + } +} + static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_arg_meta *meta, int insn_idx) { @@ -11300,6 +11317,12 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ break; } case KF_ARG_PTR_TO_ITER: + if (meta->func_id == special_kfunc_list[KF_bpf_iter_css_task_new]) { + if (!check_css_task_iter_allowlist(env)) { + verbose(env, "css_task_iter is only allowed in bpf_lsm and bpf iter-s\n"); + return -EINVAL; + } + } ret = process_iter_arg(env, regno, insn_idx, meta); if (ret < 0) return ret; diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 2c8cb3f61529..6792ed2b45d7 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -458,4 +458,12 @@ extern void bpf_throw(u64 cookie) __ksym; __bpf_assert_op(LHS, <=, END, value, false); \ }) +struct bpf_iter_css_task; +struct cgroup_subsys_state; +extern int bpf_iter_css_task_new(struct bpf_iter_css_task *it, + struct cgroup_subsys_state *css, unsigned int flags) __weak __ksym; +extern struct task_struct *bpf_iter_css_task_next(struct bpf_iter_css_task *it) __weak __ksym; +extern void bpf_iter_css_task_destroy(struct bpf_iter_css_task *it) __weak __ksym; + + #endif From patchwork Wed Oct 18 06:03:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuyi Zhou X-Patchwork-Id: 154671 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4588733vqb; Tue, 17 Oct 2023 23:04:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEORLiVwhlwIlLa6ckC2+6cMtWSwoS25op1ZNMq0goKZkmTdjdWFMcA219hdrjNZ1RHO4FO X-Received: by 2002:a05:6a21:617:b0:159:f5fb:bf74 with SMTP id ll23-20020a056a21061700b00159f5fbbf74mr3951602pzb.3.1697609069944; Tue, 17 Oct 2023 23:04:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697609069; cv=none; d=google.com; s=arc-20160816; b=cG4Vdw7qeEkN06csDxMheTjeff+ZRNo5812zApcbWf8wPX32Bh7EZFEHOeIdqpMoc5 5WZw/gAmwP0BwQRevShZTn6pXIPdYvyBqLSkAs6o0U7Q+BxoT4mMP9HTBmObGFGxJB1c h2dWfULLOaFcSwDkP2Z4a8cEIA+kGD861SxPhzbPiP0UTVoPTheEA/z9wwmzoBiSvut7 NG5pjvDE4eyI3xDY68YNs+n8jKX5Ij2UpLt+AT4AiorUlg39s8GAv4fJRbRG+RF2obiq dQqlNUIIlyNQ3LvZtV/XLRPXq/yqr77v9STusM3xlY3xPdZ/Vh/bRMo5YDq2mNDHUf3Z OKqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=6pA+uWgAid8oSLs2nLYNpXTwEjADAAqZra1pWMMUUeE=; fh=DR7g1EcWKOXTEoooUPBSJXUaklSrDEYzv6YDdhz1CwE=; b=wTL5MTMT/pbW0HiAC7wbHF8NX9O6Lx4OT5cTYZxu5Ibqz7bw74aBMrCdjadDtMzEqL XwFI5p1vj/babxSc3tr/DY2sSNWhQuvU4ZSdrJQsvM3LAwhTyLGjMlC48cg2TCsASmFl eE1NAPpqKO50efVLxn96AjY/OMI0Iskw9XZZ036+8N+Ter7zM7Wgpi6tOSh3b0QDAqJq 54FL6wq8FC9YnOmOWbL92VtL7JXb3jJ18Xz+qMqdYejF5TKmnVnfhEtqBfhcMoGV7MpL hgUnYUGlGpBMLBnr/SRbVt7Vv6lOVfBClGKnB0MdhnYg3rwwikDaGOP7J2kI5gEB22Ru BVfQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=YUZoUS8m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from groat.vger.email (groat.vger.email. [2620:137:e000::3:5]) by mx.google.com with ESMTPS id b13-20020a6567cd000000b0059c78eb3fe7si1468123pgs.838.2023.10.17.23.04.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 23:04:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) client-ip=2620:137:e000::3:5; Authentication-Results: mx.google.com; dkim=pass header.i=@bytedance.com header.s=google header.b=YUZoUS8m; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:5 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=bytedance.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by groat.vger.email (Postfix) with ESMTP id 2A4878038DB3; Tue, 17 Oct 2023 23:04:27 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at groat.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344429AbjJRGEH (ORCPT + 23 others); Wed, 18 Oct 2023 02:04:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344460AbjJRGEC (ORCPT ); Wed, 18 Oct 2023 02:04:02 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 834AAC6 for ; Tue, 17 Oct 2023 23:03:41 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id 98e67ed59e1d1-27d0acd0903so4116638a91.1 for ; Tue, 17 Oct 2023 23:03:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1697609021; x=1698213821; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6pA+uWgAid8oSLs2nLYNpXTwEjADAAqZra1pWMMUUeE=; b=YUZoUS8mKtF9eJp0E5jNZsZoUzKff6RJGp2trsjygUpRzQoecB84eC6OpgwkKq159u EPg0Q4cKZiZn5vtq0RZYe422+jQYoGmRayS506CQnsfqbaIb+oVehf0+9PRproDfY0SW CoHMQMUF8OvYEdHfyRxM+BAKY9QVc3aIOgzBXqrr6Z+ZLSYNEr9Dnpr6Ca2lgjG2vG1C +IlJILhZIjqr7dlgTi0MHDu+6AoW4+iQ98Fnou/GRa7cvmxFRnzY5m3yvJxGh8QrEcri CjVV2vVvCX2TbaJ3ilq4MCd766F58DFFISb2mE+UG2LEHzj+lgft6JraT1jIxldcLgGy MitA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697609021; x=1698213821; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6pA+uWgAid8oSLs2nLYNpXTwEjADAAqZra1pWMMUUeE=; b=LqqQ9eP0Zfkt2dMXCfC7DzHpxaM3PhQgLOEIlBGmwwNn9xLKL0oqUE1/UEe2nXtcZW NbR9zlIVmSwVoLsB/xIa8/28gJmYjp8ESLTYG0WZWhjpRFqd0CpsSbSEgSMSHhEWhuwI svsecGqfZtfI3ND5EHg/qgUpNuhJs5X7tY8tt085T5Vk0OozcGroJtN3hfc952mFDk1z AvC1oYFyqNQUBRL7y0YV9sN7TlfxAtBeTXT4l8pltPU51oqDX0jR0kwTpMTWMm8ssnas nfci3+felat+8e7UHfwzyGwcnPrUD4o+0NrTBbGpxAw/zq2VQWz3oOF0hRvT/8TIZpAz wcVg== X-Gm-Message-State: AOJu0YzfB15MAtE6F/T/bJkWdZT8La2FmdMKIyCsQCiezdsq3ykCtqqV z0SQ0W4b8eFbUOShJWeH0mu2rw== X-Received: by 2002:a17:90a:1141:b0:27d:61ff:3d3b with SMTP id d1-20020a17090a114100b0027d61ff3d3bmr4503004pje.38.1697609020936; Tue, 17 Oct 2023 23:03:40 -0700 (PDT) Received: from n37-019-243.byted.org ([180.184.51.40]) by smtp.gmail.com with ESMTPSA id b14-20020a17090acc0e00b00276cb03a0e9sm505782pju.46.2023.10.17.23.03.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Oct 2023 23:03:40 -0700 (PDT) From: Chuyi Zhou To: bpf@vger.kernel.org Cc: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, martin.lau@kernel.org, tj@kernel.org, linux-kernel@vger.kernel.org, Chuyi Zhou Subject: [PATCH bpf-next v6 3/8] bpf: Introduce task open coded iterator kfuncs Date: Wed, 18 Oct 2023 14:03:13 +0800 Message-Id: <20231018060318.105524-4-zhouchuyi@bytedance.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20231018060318.105524-1-zhouchuyi@bytedance.com> References: <20231018060318.105524-1-zhouchuyi@bytedance.com> MIME-Version: 1.0 X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on groat.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (groat.vger.email [0.0.0.0]); Tue, 17 Oct 2023 23:04:27 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780072128044579801 X-GMAIL-MSGID: 1780072128044579801 This patch adds kfuncs bpf_iter_task_{new,next,destroy} which allow creation and manipulation of struct bpf_iter_task in open-coded iterator style. BPF programs can use these kfuncs or through bpf_for_each macro to iterate all processes in the system. The API design keep consistent with SEC("iter/task"). bpf_iter_task_new() accepts a specific task and iterating type which allows: 1. iterating all process in the system (BPF_TASK_ITER_ALL_PROCS) 2. iterating all threads in the system (BPF_TASK_ITER_ALL_THREADS) 3. iterating all threads of a specific task (BPF_TASK_ITER_PROC_THREADS) Signed-off-by: Chuyi Zhou --- kernel/bpf/helpers.c | 3 + kernel/bpf/task_iter.c | 90 +++++++++++++++++++ .../testing/selftests/bpf/bpf_experimental.h | 5 ++ 3 files changed, 98 insertions(+) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index c01441db9fd5..c25941531265 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -2563,6 +2563,9 @@ BTF_ID_FLAGS(func, bpf_iter_task_vma_destroy, KF_ITER_DESTROY) BTF_ID_FLAGS(func, bpf_iter_css_task_new, KF_ITER_NEW | KF_TRUSTED_ARGS) BTF_ID_FLAGS(func, bpf_iter_css_task_next, KF_ITER_NEXT | KF_RET_NULL) BTF_ID_FLAGS(func, bpf_iter_css_task_destroy, KF_ITER_DESTROY) +BTF_ID_FLAGS(func, bpf_iter_task_new, KF_ITER_NEW | KF_TRUSTED_ARGS) +BTF_ID_FLAGS(func, bpf_iter_task_next, KF_ITER_NEXT | KF_RET_NULL) +BTF_ID_FLAGS(func, bpf_iter_task_destroy, KF_ITER_DESTROY) BTF_ID_FLAGS(func, bpf_dynptr_adjust) BTF_ID_FLAGS(func, bpf_dynptr_is_null) BTF_ID_FLAGS(func, bpf_dynptr_is_rdonly) diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c index e4126698cecf..faa1712c1df5 100644 --- a/kernel/bpf/task_iter.c +++ b/kernel/bpf/task_iter.c @@ -952,6 +952,96 @@ __bpf_kfunc void bpf_iter_css_task_destroy(struct bpf_iter_css_task *it) __diag_pop(); +struct bpf_iter_task { + __u64 __opaque[3]; +} __attribute__((aligned(8))); + +struct bpf_iter_task_kern { + struct task_struct *task; + struct task_struct *pos; + unsigned int flags; +} __attribute__((aligned(8))); + +enum { + /* all process in the system */ + BPF_TASK_ITER_ALL_PROCS, + /* all threads in the system */ + BPF_TASK_ITER_ALL_THREADS, + /* all threads of a specific process */ + BPF_TASK_ITER_PROC_THREADS +}; + +__diag_push(); +__diag_ignore_all("-Wmissing-prototypes", + "Global functions as their definitions will be in vmlinux BTF"); + +__bpf_kfunc int bpf_iter_task_new(struct bpf_iter_task *it, + struct task_struct *task, unsigned int flags) +{ + struct bpf_iter_task_kern *kit = (void *)it; + + BUILD_BUG_ON(sizeof(struct bpf_iter_task_kern) > sizeof(struct bpf_iter_task)); + BUILD_BUG_ON(__alignof__(struct bpf_iter_task_kern) != + __alignof__(struct bpf_iter_task)); + + kit->task = kit->pos = NULL; + switch (flags) { + case BPF_TASK_ITER_ALL_THREADS: + case BPF_TASK_ITER_ALL_PROCS: + case BPF_TASK_ITER_PROC_THREADS: + break; + default: + return -EINVAL; + } + + if (flags == BPF_TASK_ITER_PROC_THREADS) + kit->task = task; + else + kit->task = &init_task; + kit->pos = kit->task; + kit->flags = flags; + return 0; +} + +__bpf_kfunc struct task_struct *bpf_iter_task_next(struct bpf_iter_task *it) +{ + struct bpf_iter_task_kern *kit = (void *)it; + struct task_struct *pos; + unsigned int flags; + + flags = kit->flags; + pos = kit->pos; + + if (!pos) + return pos; + + if (flags == BPF_TASK_ITER_ALL_PROCS) + goto get_next_task; + + kit->pos = next_thread(kit->pos); + if (kit->pos == kit->task) { + if (flags == BPF_TASK_ITER_PROC_THREADS) { + kit->pos = NULL; + return pos; + } + } else + return pos; + +get_next_task: + kit->pos = next_task(kit->pos); + kit->task = kit->pos; + if (kit->pos == &init_task) + kit->pos = NULL; + + return pos; +} + +__bpf_kfunc void bpf_iter_task_destroy(struct bpf_iter_task *it) +{ +} + +__diag_pop(); + DEFINE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work); static void do_mmap_read_unlock(struct irq_work *entry) diff --git a/tools/testing/selftests/bpf/bpf_experimental.h b/tools/testing/selftests/bpf/bpf_experimental.h index 6792ed2b45d7..2f6c747aa874 100644 --- a/tools/testing/selftests/bpf/bpf_experimental.h +++ b/tools/testing/selftests/bpf/bpf_experimental.h @@ -465,5 +465,10 @@ extern int bpf_iter_css_task_new(struct bpf_iter_css_task *it, extern struct task_struct *bpf_iter_css_task_next(struct bpf_iter_css_task *it) __weak __ksym; extern void bpf_iter_css_task_destroy(struct bpf_iter_css_task *it) __weak __ksym; +struct bpf_iter_task; +extern int bpf_iter_task_new(struct bpf_iter_task *it, + struct task_struct *task, unsigned int flags) __weak __ksym; +extern struct task_struct *bpf_iter_task_next(struct bpf_iter_task *it) __weak __ksym; +extern void bpf_iter_task_destroy(struct bpf_iter_task *it) __weak __ksym; #endif