Message ID | 20230914172138.11977-5-james.morse@arm.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:172:b0:3f2:4152:657d with SMTP id h50csp512027vqi; Thu, 14 Sep 2023 10:34:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGn3q5nWRnUolLZgg1YbwMqLUwOvlXIrbEiZtLAgmmE/Uw6KUJPVwnt2twHpL/+Jil02OG8 X-Received: by 2002:a05:6a00:178f:b0:68e:2eed:5ab0 with SMTP id s15-20020a056a00178f00b0068e2eed5ab0mr7564008pfg.7.1694712877301; Thu, 14 Sep 2023 10:34:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1694712877; cv=none; d=google.com; s=arc-20160816; b=SquuqNOn1QelWxB60Z/k/hIxePX0wr3HxRAA9Qd9AleWb8DIQRSPPUNkejJG/AoSyN 2EEDz79ViPEwowaQ+o5WOAe26BXfAE7H1nXEz7wkmV4XV7UdxR8bHW6h9H3BZxZO9djM AckXqB85dxLbxu+yuNv7h2vZZUuYcvkSoYRh63VlK+vcsTvPkqzN4vYj4QyvO42dSq3m uFsT4BAyCiTXiwAxqi0RarDQYYqubm251XgfPfFk2Lf/matomidI9S0KhJ0yEdKwxmm5 rQq1np/vqrlf8Vj9CtL1HAwVPFBilMtFYSSDfuYo2Q0mUTdbv/cpf74wo3YMg0LTc7Gv ewYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=7Qfi7U9/nyKNa2E9HJI4AopuI/E5iLv+gcUeTYfR0kA=; fh=6Nl2MjDBXofbhMXyzAhf7L7loTtEBflFxRNey74qSFc=; b=b5lbavd7uYGFsFrTC99Kn5XyiINlVp37SEs5Y/c0hlAOhtyCr7RhaErK48oGbgTBx0 8/2reu7LnpE8vJ5tro5F3cGK+h7Vj2kvTuSeIbCtknC94we+/gbNxMIvOWMRGF8rmdz5 KgNf8MuTfaSZcNcgzFCBUVWL9l0V5CXI4YowgLZg6mYwAgR9mmqJctVkbLOH8bfsLU/M +ExAML/AxBvCpJ7CGOejrMUfGYd76/Rpi3/Snd7OMrOzBs6EMFOQCapyiskDNg5hy0On VJZfCM3TvkzAFn2apvCjC7xv9VLC9fqiFeuOy7kxeDdcAHe+EfG5LxGObjDuJUmAqxJC ZO5w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id y33-20020a056a00182100b0068ffe421443si2013430pfa.170.2023.09.14.10.34.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Sep 2023 10:34:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=arm.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 9BB9882114AA; Thu, 14 Sep 2023 10:22:29 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239269AbjINRWT (ORCPT <rfc822;pwkd43@gmail.com> + 33 others); Thu, 14 Sep 2023 13:22:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239182AbjINRWM (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Thu, 14 Sep 2023 13:22:12 -0400 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 65B031FF7 for <linux-kernel@vger.kernel.org>; Thu, 14 Sep 2023 10:22:08 -0700 (PDT) Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 582DB1476; Thu, 14 Sep 2023 10:22:45 -0700 (PDT) Received: from merodach.members.linode.com (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 5415B3F5A1; Thu, 14 Sep 2023 10:22:05 -0700 (PDT) From: James Morse <james.morse@arm.com> To: x86@kernel.org, linux-kernel@vger.kernel.org Cc: Fenghua Yu <fenghua.yu@intel.com>, Reinette Chatre <reinette.chatre@intel.com>, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>, H Peter Anvin <hpa@zytor.com>, Babu Moger <Babu.Moger@amd.com>, James Morse <james.morse@arm.com>, shameerali.kolothum.thodi@huawei.com, D Scott Phillips OS <scott@os.amperecomputing.com>, carl@os.amperecomputing.com, lcherian@marvell.com, bobo.shaobowang@huawei.com, tan.shaopeng@fujitsu.com, xingxin.hx@openanolis.org, baolin.wang@linux.alibaba.com, Jamie Iles <quic_jiles@quicinc.com>, Xin Hao <xhao@linux.alibaba.com>, peternewman@google.com, dfustini@baylibre.com, amitsinght@marvell.com Subject: [PATCH v6 04/24] x86/resctrl: Move rmid allocation out of mkdir_rdt_prepare() Date: Thu, 14 Sep 2023 17:21:18 +0000 Message-Id: <20230914172138.11977-5-james.morse@arm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20230914172138.11977-1-james.morse@arm.com> References: <20230914172138.11977-1-james.morse@arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 14 Sep 2023 10:22:29 -0700 (PDT) X-Spam-Status: No, score=-0.8 required=5.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1777035250015869043 X-GMAIL-MSGID: 1777035250015869043 |
Series |
x86/resctrl: monitored closid+rmid together, separate arch/fs locking
|
|
Commit Message
James Morse
Sept. 14, 2023, 5:21 p.m. UTC
RMID are allocated for each monitor or control group directory, because each of these needs its own RMID. For control groups, rdtgroup_mkdir_ctrl_mon() later goes on to allocate the CLOSID. MPAM's equivalent of RMID is not an independent number, so can't be allocated until the CLOSID is known. An RMID allocation for one CLOSID may fail, whereas another may succeed depending on how many monitor groups a control group has. The RMID allocation needs to move to be after the CLOSID has been allocated. Move the RMID allocation out of mkdir_rdt_prepare() to occur in its caller, after the mkdir_rdt_prepare() call. This allows the RMID allocator to know the CLOSID. Reviewed-by: Shaopeng Tan <tan.shaopeng@fujitsu.com> Tested-by: Shaopeng Tan <tan.shaopeng@fujitsu.com> Tested-By: Peter Newman <peternewman@google.com> Signed-off-by: James Morse <james.morse@arm.com> --- Changes since v2: * Moved kernfs_activate() later to preserve atomicity of files being visible Changes since v5: * Renamed out_id_free as out_closid_free. --- arch/x86/kernel/cpu/resctrl/rdtgroup.c | 35 +++++++++++++++++++------- 1 file changed, 26 insertions(+), 9 deletions(-)
Comments
Hi James, On 9/14/2023 10:21 AM, James Morse wrote: > RMID are allocated for each monitor or control group directory, because > each of these needs its own RMID. For control groups, > rdtgroup_mkdir_ctrl_mon() later goes on to allocate the CLOSID. > > MPAM's equivalent of RMID is not an independent number, so can't be > allocated until the CLOSID is known. An RMID allocation for one CLOSID > may fail, whereas another may succeed depending on how many monitor > groups a control group has. > > The RMID allocation needs to move to be after the CLOSID has been > allocated. > > Move the RMID allocation out of mkdir_rdt_prepare() to occur in its caller, > after the mkdir_rdt_prepare() call. This allows the RMID allocator to > know the CLOSID. > > Reviewed-by: Shaopeng Tan <tan.shaopeng@fujitsu.com> > Tested-by: Shaopeng Tan <tan.shaopeng@fujitsu.com> > Tested-By: Peter Newman <peternewman@google.com> > Signed-off-by: James Morse <james.morse@arm.com> > --- Reviewed-by: Reinette Chatre <reinette.chatre@intel.com> Reinette
Hi James, On 9/14/23 12:21, James Morse wrote: > RMID are allocated for each monitor or control group directory, because > each of these needs its own RMID. For control groups, > rdtgroup_mkdir_ctrl_mon() later goes on to allocate the CLOSID. > > MPAM's equivalent of RMID is not an independent number, so can't be > allocated until the CLOSID is known. An RMID allocation for one CLOSID > may fail, whereas another may succeed depending on how many monitor > groups a control group has. > > The RMID allocation needs to move to be after the CLOSID has been > allocated. > > Move the RMID allocation out of mkdir_rdt_prepare() to occur in its caller, > after the mkdir_rdt_prepare() call. This allows the RMID allocator to > know the CLOSID. > > Reviewed-by: Shaopeng Tan <tan.shaopeng@fujitsu.com> > Tested-by: Shaopeng Tan <tan.shaopeng@fujitsu.com> > Tested-By: Peter Newman <peternewman@google.com> > Signed-off-by: James Morse <james.morse@arm.com> > --- > Changes since v2: > * Moved kernfs_activate() later to preserve atomicity of files being visible > > Changes since v5: > * Renamed out_id_free as out_closid_free. > --- > arch/x86/kernel/cpu/resctrl/rdtgroup.c | 35 +++++++++++++++++++------- > 1 file changed, 26 insertions(+), 9 deletions(-) > > diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c > index 7a7369a323b5..d25cb8c9a20e 100644 > --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c > +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c > @@ -3189,6 +3189,12 @@ static int mkdir_rdt_prepare_rmid_alloc(struct rdtgroup *rdtgrp) > return 0; > } > > +static void mkdir_rdt_prepare_rmid_free(struct rdtgroup *rgrp) > +{ > + if (rdt_mon_capable) > + free_rmid(rgrp->mon.rmid); > +} > + > static int mkdir_rdt_prepare(struct kernfs_node *parent_kn, > const char *name, umode_t mode, > enum rdt_group_type rtype, struct rdtgroup **r) > @@ -3254,12 +3260,6 @@ static int mkdir_rdt_prepare(struct kernfs_node *parent_kn, > goto out_destroy; > } > > - ret = mkdir_rdt_prepare_rmid_alloc(rdtgrp); > - if (ret) > - goto out_destroy; > - > - kernfs_activate(kn); You should not remove "kernfs_activate(kn); from here (only the last line). kernfs_create_dir is called in this function. /* kernfs creates the directory for rdtgrp */ kn = kernfs_create_dir(parent_kn, name, mode, rdtgrp); There should be matching kernfs_activate. > - > /* > * The caller unlocks the parent_kn upon success. > */ > @@ -3278,7 +3278,6 @@ static int mkdir_rdt_prepare(struct kernfs_node *parent_kn, > static void mkdir_rdt_prepare_clean(struct rdtgroup *rgrp) > { > kernfs_remove(rgrp->kn); > - free_rmid(rgrp->mon.rmid); > rdtgroup_remove(rgrp); > } > > @@ -3300,12 +3299,21 @@ static int rdtgroup_mkdir_mon(struct kernfs_node *parent_kn, > prgrp = rdtgrp->mon.parent; > rdtgrp->closid = prgrp->closid; > > + ret = mkdir_rdt_prepare_rmid_alloc(rdtgrp); > + if (ret) { > + mkdir_rdt_prepare_clean(rdtgrp); > + goto out_unlock; > + } > + > + kernfs_activate(rdtgrp->kn); I dont see the need for this. There is kernfs_activate inside mkdir_rdt_prepare_rmid_alloc (mkdir_rdt_prepare_rmid_alloc ->mkdir_mondata_all) for all the files created. Also mkdir_rdt_prepare already has kernfs_activate for the files it created. > + > /* > * Add the rdtgrp to the list of rdtgrps the parent > * ctrl_mon group has to track. > */ > list_add_tail(&rdtgrp->mon.crdtgrp_list, &prgrp->mon.crdtgrp_list); > > +out_unlock: > rdtgroup_kn_unlock(parent_kn); > return ret; > } > @@ -3336,9 +3344,16 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn, > ret = 0; > > rdtgrp->closid = closid; > + > + ret = mkdir_rdt_prepare_rmid_alloc(rdtgrp); > + if (ret) > + goto out_closid_free; > + > + kernfs_activate(rdtgrp->kn); > + Same as above. > ret = rdtgroup_init_alloc(rdtgrp); > if (ret < 0) > - goto out_id_free; > + goto out_rmid_free; > > list_add(&rdtgrp->rdtgroup_list, &rdt_all_groups); > > @@ -3358,7 +3373,9 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn, > > out_del_list: > list_del(&rdtgrp->rdtgroup_list); > -out_id_free: > +out_rmid_free: > + mkdir_rdt_prepare_rmid_free(rdtgrp); > +out_closid_free: > closid_free(closid); > out_common_fail: > mkdir_rdt_prepare_clean(rdtgrp);
Hi Babu, On 04/10/2023 19:01, Moger, Babu wrote: > On 9/14/23 12:21, James Morse wrote: >> RMID are allocated for each monitor or control group directory, because >> each of these needs its own RMID. For control groups, >> rdtgroup_mkdir_ctrl_mon() later goes on to allocate the CLOSID. >> >> MPAM's equivalent of RMID is not an independent number, so can't be >> allocated until the CLOSID is known. An RMID allocation for one CLOSID >> may fail, whereas another may succeed depending on how many monitor >> groups a control group has. >> >> The RMID allocation needs to move to be after the CLOSID has been >> allocated. >> >> Move the RMID allocation out of mkdir_rdt_prepare() to occur in its caller, >> after the mkdir_rdt_prepare() call. This allows the RMID allocator to >> know the CLOSID. >> diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c >> index 7a7369a323b5..d25cb8c9a20e 100644 >> --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c >> +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c >> @@ -3189,6 +3189,12 @@ static int mkdir_rdt_prepare_rmid_alloc(struct rdtgroup *rdtgrp) >> return 0; >> } >> >> +static void mkdir_rdt_prepare_rmid_free(struct rdtgroup *rgrp) >> +{ >> + if (rdt_mon_capable) >> + free_rmid(rgrp->mon.rmid); >> +} >> + >> static int mkdir_rdt_prepare(struct kernfs_node *parent_kn, >> const char *name, umode_t mode, >> enum rdt_group_type rtype, struct rdtgroup **r) >> @@ -3254,12 +3260,6 @@ static int mkdir_rdt_prepare(struct kernfs_node *parent_kn, >> goto out_destroy; >> } >> >> - ret = mkdir_rdt_prepare_rmid_alloc(rdtgrp); >> - if (ret) >> - goto out_destroy; >> - >> - kernfs_activate(kn); > > You should not remove "kernfs_activate(kn); from here (only the last line). > > kernfs_create_dir is called in this function. > > /* kernfs creates the directory for rdtgrp */ > kn = kernfs_create_dir(parent_kn, name, mode, rdtgrp); > > > There should be matching kernfs_activate. I think your point is kernfs_activate() should have been called by the time mkdir_rdt_prepare() returns because it creates other directories. I don't think this matters because kernfs_activate() is a tree operation. Sure, the control/monitor group directory isn't visible once mkdir_rdt_prepare() returns, but by the time either of its two callers return, changes to the directory tree have been activated. Moving these lines is the to ensure user-space doesn't see the control/monitor group as existing without the mon_data directory that is created by mkdir_rdt_prepare_rmid_alloc(). >> - >> /* >> * The caller unlocks the parent_kn upon success. >> */ >> @@ -3278,7 +3278,6 @@ static int mkdir_rdt_prepare(struct kernfs_node *parent_kn, >> static void mkdir_rdt_prepare_clean(struct rdtgroup *rgrp) >> { >> kernfs_remove(rgrp->kn); >> - free_rmid(rgrp->mon.rmid); >> rdtgroup_remove(rgrp); >> } >> >> @@ -3300,12 +3299,21 @@ static int rdtgroup_mkdir_mon(struct kernfs_node *parent_kn, >> prgrp = rdtgrp->mon.parent; >> rdtgrp->closid = prgrp->closid; >> >> + ret = mkdir_rdt_prepare_rmid_alloc(rdtgrp); >> + if (ret) { >> + mkdir_rdt_prepare_clean(rdtgrp); >> + goto out_unlock; >> + } >> + >> + kernfs_activate(rdtgrp->kn); > > I dont see the need for this. There is kernfs_activate inside > mkdir_rdt_prepare_rmid_alloc (mkdir_rdt_prepare_rmid_alloc > ->mkdir_mondata_all) for all the files created. > Also mkdir_rdt_prepare already has kernfs_activate for the files it created. It does, and this makes the mon_data directory visible in the parent control/monitor group - but that control/monitor group isn't visible until this kernfs_activate(rdtgrp->kn) makes it visible. The scope of these tree operations is different. Looking at this again, there is an existing problem with the mon_groups directory not being visible until after the control/monitor group is visible, worse is that if the mon_group directory creation fails, the control/monitor group is removed. Chances are no-one is depending on this. I do think ultimately these kernfs_activate() calls should be moved to the end of the syscall helpers that change the directory structure. This would stop things being briefly visible. Thanks! James
diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c index 7a7369a323b5..d25cb8c9a20e 100644 --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c @@ -3189,6 +3189,12 @@ static int mkdir_rdt_prepare_rmid_alloc(struct rdtgroup *rdtgrp) return 0; } +static void mkdir_rdt_prepare_rmid_free(struct rdtgroup *rgrp) +{ + if (rdt_mon_capable) + free_rmid(rgrp->mon.rmid); +} + static int mkdir_rdt_prepare(struct kernfs_node *parent_kn, const char *name, umode_t mode, enum rdt_group_type rtype, struct rdtgroup **r) @@ -3254,12 +3260,6 @@ static int mkdir_rdt_prepare(struct kernfs_node *parent_kn, goto out_destroy; } - ret = mkdir_rdt_prepare_rmid_alloc(rdtgrp); - if (ret) - goto out_destroy; - - kernfs_activate(kn); - /* * The caller unlocks the parent_kn upon success. */ @@ -3278,7 +3278,6 @@ static int mkdir_rdt_prepare(struct kernfs_node *parent_kn, static void mkdir_rdt_prepare_clean(struct rdtgroup *rgrp) { kernfs_remove(rgrp->kn); - free_rmid(rgrp->mon.rmid); rdtgroup_remove(rgrp); } @@ -3300,12 +3299,21 @@ static int rdtgroup_mkdir_mon(struct kernfs_node *parent_kn, prgrp = rdtgrp->mon.parent; rdtgrp->closid = prgrp->closid; + ret = mkdir_rdt_prepare_rmid_alloc(rdtgrp); + if (ret) { + mkdir_rdt_prepare_clean(rdtgrp); + goto out_unlock; + } + + kernfs_activate(rdtgrp->kn); + /* * Add the rdtgrp to the list of rdtgrps the parent * ctrl_mon group has to track. */ list_add_tail(&rdtgrp->mon.crdtgrp_list, &prgrp->mon.crdtgrp_list); +out_unlock: rdtgroup_kn_unlock(parent_kn); return ret; } @@ -3336,9 +3344,16 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn, ret = 0; rdtgrp->closid = closid; + + ret = mkdir_rdt_prepare_rmid_alloc(rdtgrp); + if (ret) + goto out_closid_free; + + kernfs_activate(rdtgrp->kn); + ret = rdtgroup_init_alloc(rdtgrp); if (ret < 0) - goto out_id_free; + goto out_rmid_free; list_add(&rdtgrp->rdtgroup_list, &rdt_all_groups); @@ -3358,7 +3373,9 @@ static int rdtgroup_mkdir_ctrl_mon(struct kernfs_node *parent_kn, out_del_list: list_del(&rdtgrp->rdtgroup_list); -out_id_free: +out_rmid_free: + mkdir_rdt_prepare_rmid_free(rdtgrp); +out_closid_free: closid_free(closid); out_common_fail: mkdir_rdt_prepare_clean(rdtgrp);