From patchwork Wed Nov 2 02:02:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Leonardo Bras Soares Passos X-Patchwork-Id: 13989 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp3337278wru; Tue, 1 Nov 2022 19:05:41 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6JVaK4yD1Xz8A3sjBbAw8wGMm10sDrri6kLXhoGsoNNTYNf1HDBBb9qXWgNH8EpNH9JDHG X-Received: by 2002:a17:902:8c92:b0:178:29d4:600f with SMTP id t18-20020a1709028c9200b0017829d4600fmr22556208plo.40.1667354740791; Tue, 01 Nov 2022 19:05:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667354740; cv=none; d=google.com; s=arc-20160816; b=UtEaAtkn2XdCBtbECyiEHph2D5d7vYbKZLSb0p34258uiZwPBtnAD1yrS1siS3z49F cKgpurV+HLNzwpVRYKXw6hoCrgr1RF/Hx51Df+I3cLl+zuIsAVTS+5D/vY6eQ5ZUNKdH TL3R54P8VyT1KTOAjNOoCiNPsu9B9lWtw3A96in7+rOx5xwMfdnd9xEtua0EF3Kn8YvW KbfScQ/NEcPWGqwF2zIRQfPfPE5S+mC2soe2zuqxvTuiDl0vPWYz5wx2utaYDWz8cew4 0rZpdpJEoRvdirqpjSAehOSaWm2myZhXQmTRGmk5G3xPY1P9P3bbUH3YhebX00oHch75 hc4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CMSAt3GthE2zWUBTJjLiVIVQICwCKfwnNgpg1Ix0oWM=; b=HHTgMX4Tx77HTEuYGfxXDa/lP+ft4GbH8UX3p72rpckULa/cqWqubqe3gOuEjGll3O 5gEpjB/YJbg7un0jdzQKTU3CAdWpXk5a3YvCxw2Y5iGj/U6OLhgxBw7N4HX/3ycINwgD 65SskEq6FTuGwAgKPUxWUTOuFrb6NXmWgJs8ch2kDy0NWoRnwgcRJlfSq/IDDG5N+IQT nAm/To8zYRBbIqS8XjajW0f8w6EA2Y4azjU5qoTswjQaoYKbA5RaW3h219tzpuONmmHa WuJ16nDW0I0rP1i+flJLpZVLVG5YM1dFW+iYcE9yQ8Dz0BvMoT1pWFoxvIoMPZrn4HzQ 30cQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZljSs01H; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id jd1-20020a170903260100b00186dcc35018si13320442plb.108.2022.11.01.19.05.28; Tue, 01 Nov 2022 19:05:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=ZljSs01H; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229927AbiKBCE4 (ORCPT + 99 others); Tue, 1 Nov 2022 22:04:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34566 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230133AbiKBCEp (ORCPT ); Tue, 1 Nov 2022 22:04:45 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C56B962F7 for ; Tue, 1 Nov 2022 19:03:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667354626; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CMSAt3GthE2zWUBTJjLiVIVQICwCKfwnNgpg1Ix0oWM=; b=ZljSs01HJWODvjWnTfjhaXLqetGUfY2Dbrl1RhsbQQnnejRqFWQA2C9tpUUWZ+hvgIpwiM HTwk/pBtyUkelu6N0Uh5jaPV+Xe2L0GeZps/1AynDYVU2dw4H4J6XjUGYkuO4l+A/tzDrE P13JMWMDRpTOeP9n+JmzLPV1lq6oI9s= Received: from mail-ot1-f72.google.com (mail-ot1-f72.google.com [209.85.210.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-60-PW_wU8tnPgigXsR4Ju4Szw-1; Tue, 01 Nov 2022 22:03:44 -0400 X-MC-Unique: PW_wU8tnPgigXsR4Ju4Szw-1 Received: by mail-ot1-f72.google.com with SMTP id ck9-20020a056830648900b0066c56ff7b33so3593178otb.20 for ; Tue, 01 Nov 2022 19:03:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CMSAt3GthE2zWUBTJjLiVIVQICwCKfwnNgpg1Ix0oWM=; b=SIFknT/q08sZI8/YjgsjL9aUL1OyazyzTRT7x8DiHfwRjFh+nYQcc0LDDkA8N3PXoQ fhW7kIH4gz2acZa7t3onlUb32JE5M3xCpj3xjl9khgi5l57zYPPV63nvXSjTezWszct5 lQ/5VSO0rOxQA3DY0+86pa5wEZv53nCmCk3OR0vtp9dGM1zO/GU7oN/OJ1EXAhyBWZ4P vZ0GLF/B4/oMGKtP4PMLjomIPsVM9+OcXZxFbRgftkf7IXGG7ouRxpLdIAVxCxMknBzu Yc35718IPv2ZrVtbnQCsiaZB5Qge8XhPiI1fk8Alt/Ull6K0fIfHk+ucJZoVzj6fInA4 pOqQ== X-Gm-Message-State: ACrzQf1kQsPU3dt+VsK4CyqvQwVOJHqGRwqi6xlVXoA9of8A2r1ky/Mg TpBVYRQN6I5acg0dIE/Iw3avDB2eqvwbh1FxgpF7JZEY8TzueHj6ya914JAH6pny9HXkEpiteh5 5Pm9fPCxuuLqxBgbcF5IyXcSb X-Received: by 2002:a05:6870:d250:b0:13b:3100:abcc with SMTP id h16-20020a056870d25000b0013b3100abccmr13089104oac.3.1667354624054; Tue, 01 Nov 2022 19:03:44 -0700 (PDT) X-Received: by 2002:a05:6870:d250:b0:13b:3100:abcc with SMTP id h16-20020a056870d25000b0013b3100abccmr13089090oac.3.1667354623748; Tue, 01 Nov 2022 19:03:43 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a802:1099:7cb2:3a49:6197:5307]) by smtp.gmail.com with ESMTPSA id h15-20020a9d6f8f000000b00665919f7823sm4526624otq.8.2022.11.01.19.03.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 19:03:43 -0700 (PDT) From: Leonardo Bras To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Frederic Weisbecker , Leonardo Bras , Phil Auld , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 1/3] =?utf-8?q?sched/isolation=3A_Add_housekeep=C3=ADng?= =?utf-8?q?=5Fany=5Fcpu=5Ffrom=28=29?= Date: Tue, 1 Nov 2022 23:02:41 -0300 Message-Id: <20221102020243.522358-2-leobras@redhat.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221102020243.522358-1-leobras@redhat.com> References: <20221102020243.522358-1-leobras@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-3.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748348164641155650?= X-GMAIL-MSGID: =?utf-8?q?1748348164641155650?= As of today, there is a function called housekeepíng_any_cpu() that returns a housekeeping cpu near the current one. This function is very useful to help delegate tasks to other cpus when the current one is isolated. It also comes with the benefit of looking for cpus in the same NUMA node as the current cpu, so any memory activity could be faster in NUMA systems. On the other hand, there is no function like that to find housekeeping cpus in the same NUMA node of another CPU. Change housekeepíng_any_cpu() into housekeepíng_any_cpu_from(), so it accepts a cpu_start parameter and can find cpus in the same NUMA node as any given CPU. Also, reimplements housekeepíng_any_cpu() as an inline function that calls housekeepíng_any_cpu_from() with cpu_start = current cpu. Signed-off-by: Leonardo Bras --- include/linux/sched/isolation.h | 11 ++++++++--- kernel/sched/isolation.c | 8 ++++---- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h index 8c15abd67aed9..95b65be44f19f 100644 --- a/include/linux/sched/isolation.h +++ b/include/linux/sched/isolation.h @@ -20,7 +20,7 @@ enum hk_type { #ifdef CONFIG_CPU_ISOLATION DECLARE_STATIC_KEY_FALSE(housekeeping_overridden); -extern int housekeeping_any_cpu(enum hk_type type); +extern int housekeeping_any_cpu_from(enum hk_type type, int cpu_start); extern const struct cpumask *housekeeping_cpumask(enum hk_type type); extern bool housekeeping_enabled(enum hk_type type); extern void housekeeping_affine(struct task_struct *t, enum hk_type type); @@ -29,9 +29,9 @@ extern void __init housekeeping_init(void); #else -static inline int housekeeping_any_cpu(enum hk_type type) +static inline int housekeeping_any_cpu_from(enum hk_type type, int cpu_start) { - return smp_processor_id(); + return cpu_start; } static inline const struct cpumask *housekeeping_cpumask(enum hk_type type) @@ -58,4 +58,9 @@ static inline bool housekeeping_cpu(int cpu, enum hk_type type) return true; } +static inline int housekeeping_any_cpu(enum hk_type type) +{ + return housekeeping_any_cpu_from(type, smp_processor_id()); +} + #endif /* _LINUX_SCHED_ISOLATION_H */ diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 373d42c707bc5..6ebeac11bb350 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -36,22 +36,22 @@ bool housekeeping_enabled(enum hk_type type) } EXPORT_SYMBOL_GPL(housekeeping_enabled); -int housekeeping_any_cpu(enum hk_type type) +int housekeeping_any_cpu_from(enum hk_type type, int cpu_start) { int cpu; if (static_branch_unlikely(&housekeeping_overridden)) { if (housekeeping.flags & BIT(type)) { - cpu = sched_numa_find_closest(housekeeping.cpumasks[type], smp_processor_id()); + cpu = sched_numa_find_closest(housekeeping.cpumasks[type], cpu_start); if (cpu < nr_cpu_ids) return cpu; return cpumask_any_and(housekeeping.cpumasks[type], cpu_online_mask); } } - return smp_processor_id(); + return cpu_start; } -EXPORT_SYMBOL_GPL(housekeeping_any_cpu); +EXPORT_SYMBOL_GPL(housekeeping_any_cpu_from); const struct cpumask *housekeeping_cpumask(enum hk_type type) { From patchwork Wed Nov 2 02:02:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras Soares Passos X-Patchwork-Id: 13991 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp3337646wru; Tue, 1 Nov 2022 19:06:29 -0700 (PDT) X-Google-Smtp-Source: AMsMyM6cZvkSGzQffOyasD0acBVZSTcFnum0SSVei2I6cRGpjfTe71yZFcM+eYla44UyubCqBTnk X-Received: by 2002:a17:90b:2741:b0:20a:ebc3:6513 with SMTP id qi1-20020a17090b274100b0020aebc36513mr23486456pjb.29.1667354789209; Tue, 01 Nov 2022 19:06:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667354789; cv=none; d=google.com; s=arc-20160816; b=E1cl2kfpUc2WD74//c1Zjsi9SZky1xZjZTpvLs8JoGUkkJG9SxQjQNrZ5hWPbyZTJr DjsW8Syv/dfp6D6RwTW6wZ91jdVt1dxfsb1iFcVyxC7zU/B+1PFQXsXz2PA/rDX6/H8u 3ZSBcHPGTh6VIz+5fqZWooFTjVyMNIQDx8r5dQiDHULKp6IJ8ququ8M1Mj5khqQiLr4o jdKOoqOps9R7w9qqiBEenKTdHcQp8JdVFRA6XpsoMtI+M80m4AcgsCM8B/OGC0I4YnlJ lMYntmCEBZprQGyZEJ5g52hMraAJfXTFbNB5pqCuAhJyGqfWoVi4C5d5mbjewP72MnhS B4zA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9awyJRy6zbIYnxU2Cyp8GDi5vivjY8I9wO3N/Lj5YCI=; b=uCyb1iaFG/UE1xmJEOwuvOhypr1BCaffe+N5Xcvn/VXI6z2nS/lfj9rHJIs0h6mTYK TIXSOJtsWPPoXj+yxo9D/Ra2eS9lUzK+NNtuPc49twpNKhNnjm9EBLd1H4NKWHZNMsol s8DNOf0Td+EueC8jmM/sRDfv5Z5DFtiPQUD0EGMoM1KwMsvrPDMSFZnJ6aEo4istTULD JH3Qq+/2CYsl9rBR6CSRiOXyiShrquvh0mjxjU0Abb8fl9sam3zxiyxeo0ADBAxePD5s qhnegiv2anYyD4s4WthGNwlXz/pN8FwwXM73Ypk0yrPI7IhTerBipgOw2HSnYDGIHks4 NaqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=FRgrnSxE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j68-20020a625547000000b00567719e34aesi12573859pfb.49.2022.11.01.19.06.14; Tue, 01 Nov 2022 19:06:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=FRgrnSxE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230202AbiKBCFF (ORCPT + 99 others); Tue, 1 Nov 2022 22:05:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230122AbiKBCEy (ORCPT ); Tue, 1 Nov 2022 22:04:54 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1FD6E6351 for ; Tue, 1 Nov 2022 19:03:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667354631; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9awyJRy6zbIYnxU2Cyp8GDi5vivjY8I9wO3N/Lj5YCI=; b=FRgrnSxEw6MrXo8g7bVS2LwGLrnJ/DyTJ6aDxu52eO3jWNDdp75fa+yu8ZL3P+m1w9CZpy dbE8JoyWEcsN+OeM6DQjn6rAMJT7f9obDeka0Zm4eHx4UzWMSXwXq7uMJeI14JgVSBhrXZ OUGRv4w3Ih4I9vbH7D0ob5E4t7BHnuY= Received: from mail-oa1-f71.google.com (mail-oa1-f71.google.com [209.85.160.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-548-y8dEA0X8PuaogSBUFbIpNw-1; Tue, 01 Nov 2022 22:03:49 -0400 X-MC-Unique: y8dEA0X8PuaogSBUFbIpNw-1 Received: by mail-oa1-f71.google.com with SMTP id 586e51a60fabf-13bf576ffa6so8095986fac.9 for ; Tue, 01 Nov 2022 19:03:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9awyJRy6zbIYnxU2Cyp8GDi5vivjY8I9wO3N/Lj5YCI=; b=FKliEiPpmxf2y01QRbKdhGdThKDeBZ7qlxKCH3x/YfncLKdulq2v+LDkaHlHpea5x1 /WTxGPPI4kgROHTX+YtWVjLoXjFU1P5qamc1muVLlqPp5loHtwJhmHeDxbxlMvL1ukqk 5lY1+9Aut9IuAVRUVPrGibA6dv4UM8b/IxmSPw/T1Kt/oRM62pT7MzNHwefrHpMJhrdE 7G4iTS+RRAsa0Zo3mh+GwW4FX31FlLozniEHEz22UsxseNT4xxOHq1YbPWawSTei2Fna nMAr8zKHE14s+dIzR5YSi4v2REgdX/BpxKgWBYseTtWqRqM7AEF6s9uuWQyZB4PttDS+ 2onQ== X-Gm-Message-State: ACrzQf2489St25aQCBKbbIkegZAqV4ff3a2PwDIrMDPpNZB3TRQR4hIu 40vPHe0FNjaJSEZFwBz7KI5KQ+lKPNrScMIf3fHGp23GyZWQvDrNVFIX1uIDxYnrK49U3zwPyCV cruh1tuWUqdYbyRFF8IOj+UQS X-Received: by 2002:a05:6871:58b:b0:13c:be46:a02 with SMTP id u11-20020a056871058b00b0013cbe460a02mr11636093oan.8.1667354629098; Tue, 01 Nov 2022 19:03:49 -0700 (PDT) X-Received: by 2002:a05:6871:58b:b0:13c:be46:a02 with SMTP id u11-20020a056871058b00b0013cbe460a02mr11636072oan.8.1667354628892; Tue, 01 Nov 2022 19:03:48 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a802:1099:7cb2:3a49:6197:5307]) by smtp.gmail.com with ESMTPSA id h15-20020a9d6f8f000000b00665919f7823sm4526624otq.8.2022.11.01.19.03.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 19:03:48 -0700 (PDT) From: Leonardo Bras To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Frederic Weisbecker , Leonardo Bras , Phil Auld , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 2/3] mm/memcontrol: Change stock_lock type from local_lock_t to spinlock_t Date: Tue, 1 Nov 2022 23:02:42 -0300 Message-Id: <20221102020243.522358-3-leobras@redhat.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221102020243.522358-1-leobras@redhat.com> References: <20221102020243.522358-1-leobras@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-3.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748348215131054909?= X-GMAIL-MSGID: =?utf-8?q?1748348215131054909?= In this context, since it's using per-cpu variables, changing from local_lock to spinlock should not deal much impact in performance and can allow operations such as stock draining to happen in remote cpus. Why performance would probably not get impacted: 1 - Since the lock is in the same cache line as the information that is used next, there is no extra memory access for caching the lock. 2 - Since it's a percpu struct, there should be no other cpu sharing this cacheline, so there is no need for cacheline invalidation, and writing to the lock should be as fast as the next struct members. 3 - Even the write in (2) could be pipelined and batched with following writes to the cacheline (such as nr_pages member), further decreasing the impact of this change. Suggested-by: Marcelo Tosatti Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2d8549ae1b300..add46da2e6df1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2167,7 +2167,7 @@ void unlock_page_memcg(struct page *page) } struct memcg_stock_pcp { - local_lock_t stock_lock; + spinlock_t stock_lock; /* Protects the percpu struct */ struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; @@ -2184,7 +2184,7 @@ struct memcg_stock_pcp { #define FLUSHING_CACHED_CHARGE 0 }; static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { - .stock_lock = INIT_LOCAL_LOCK(stock_lock), + .stock_lock = __SPIN_LOCK_UNLOCKED(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); @@ -2229,15 +2229,15 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (memcg == stock->cached && stock->nr_pages >= nr_pages) { stock->nr_pages -= nr_pages; ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); return ret; } @@ -2274,14 +2274,14 @@ static void drain_local_stock(struct work_struct *dummy) * drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + old = drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -2309,10 +2309,12 @@ static void __refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; + struct memcg_stock_pcp *stock; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); __refill_stock(memcg, nr_pages); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); } /* @@ -3157,8 +3159,8 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, unsigned long flags; int *bytes; - local_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); /* * Save vmstat data in stock and skip vmstat array update unless @@ -3210,7 +3212,7 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, if (nr) mod_objcg_mlstate(objcg, pgdat, idx, nr); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -3221,15 +3223,15 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) unsigned long flags; bool ret = false; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (objcg == stock->cached_objcg && stock->nr_bytes >= nr_bytes) { stock->nr_bytes -= nr_bytes; ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); return ret; } @@ -3319,9 +3321,9 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, unsigned long flags; unsigned int nr_pages = 0; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (stock->cached_objcg != objcg) { /* reset if necessary */ old = drain_obj_stock(stock); obj_cgroup_get(objcg); @@ -3337,7 +3339,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock->nr_bytes &= (PAGE_SIZE - 1); } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); From patchwork Wed Nov 2 02:02:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras Soares Passos X-Patchwork-Id: 13992 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:6687:0:0:0:0:0 with SMTP id l7csp3337662wru; Tue, 1 Nov 2022 19:06:32 -0700 (PDT) X-Google-Smtp-Source: AMsMyM4nIC6/N4GG72MhtpqP3EKIALH1KuRlHnuQglZqfZlAkM1QSWdKIn02tqY3yo6vzUrdR0jq X-Received: by 2002:a05:6a00:14cf:b0:56d:88a8:32cf with SMTP id w15-20020a056a0014cf00b0056d88a832cfmr11739227pfu.27.1667354791702; Tue, 01 Nov 2022 19:06:31 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1667354791; cv=none; d=google.com; s=arc-20160816; b=N3LwqMZGE8Pb6i4wUTjovcPQ5piJ3ZSdWTpqkQ40uwDsiyg3Fj+XtDsjMykZSFhkrT 8U9aYzt7U7olmam8vUNWYcWEU14YzPjPK30EdwybbTvXdTwPJGQCH+f0/uWzlRDNZjNi YvtSMbmL0fn0bCyMkPjTOmFO4i0+bIdr9cdOC1Ncu7JK0ImDe50VCOqSS0qBnsJdYIm9 r2ZOShRikq5tYWSQC7VR22un5y2sQg6yCiU10lCzhb4WT6vFk62qTQVyIZIEUwwqx6tG 4iSYkvqZq3wb5T2bx44HGnMKN3xNlKy8YQP9Xm/wa/ql3U3a/5Bg0IWvsiX6QAPj4TXN m1wQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=i2RYvVpIxnZB4mRhcfM8kpuAulQpsyxXS11J9gDFkxg=; b=FuPeLmVaFwec3kz68xJXS4MO64I4YRNr+BjncrOm+luipveSYus3cyEV0CYcnbryZq oXRyA58ZyU1z8ReKgRfjZsFMOaquvvIHdZzPtOBxv+qEr4Kznc53JOHW/pXAjxybiwNE 5f8OqOJe4c1aW5UuZi48pvER30WEAhZ7g1QInjr1wvAu7qzd1+YITVOabMradqeaB9sx g/IIPwH1EIH9QjxvpxpzYhk1enxsx0+SmIgvAz798d7PR1/GB2Fobebwf9HNNP4jAqR4 lvcPYDSBCHreWlg1R3ZCdzQFgdPiSU++d9jJA+qu+p0itjkBNjj3bgL5LnM77MwLSfYw HW9Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Zwzdmd6D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k3-20020aa79d03000000b0056b8f6cd527si13152175pfp.61.2022.11.01.19.06.18; Tue, 01 Nov 2022 19:06:31 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Zwzdmd6D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230280AbiKBCFJ (ORCPT + 99 others); Tue, 1 Nov 2022 22:05:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230185AbiKBCE5 (ORCPT ); Tue, 1 Nov 2022 22:04:57 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BABF863DB for ; Tue, 1 Nov 2022 19:03:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667354636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i2RYvVpIxnZB4mRhcfM8kpuAulQpsyxXS11J9gDFkxg=; b=Zwzdmd6DfrLyphZlYUUq0lM/DoE0GLwAiWkYmwOq4W2QG/2jVAwrI97VRX2M6GDGQ0QL2e smmMDj6UwOImmMZqm1vWfoekoH6A2gB8lguxmTKX+0/AFipCZ5oqHCPe9wtnTDoTBiyMed +avQ5VVJ/e1YFQXTqblIR0arPqNNiXk= Received: from mail-oo1-f71.google.com (mail-oo1-f71.google.com [209.85.161.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-246-kZgHHuwuMuK0uDK0MYwmmQ-1; Tue, 01 Nov 2022 22:03:55 -0400 X-MC-Unique: kZgHHuwuMuK0uDK0MYwmmQ-1 Received: by mail-oo1-f71.google.com with SMTP id g6-20020a4a6b06000000b0049d1e5cd0cfso59052ooc.4 for ; Tue, 01 Nov 2022 19:03:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i2RYvVpIxnZB4mRhcfM8kpuAulQpsyxXS11J9gDFkxg=; b=daQWhjFCY87bRZwHaEI7uE8crvpGuzT+af2x5ho1/LQscqPAXt0E7a+tvAO3gTk1bc RloW9fYfX9Csk7JzpU5wc4ROPR2lq0rTE3O46juYPjgLV+/5QlRdOdNKVYfaqQBMzyQR 7m9lWal2dPkxRiSZAxFyeuOSFTxcfsgm5r2NFMtow1WGZVXyo4eLgIMo+edLOQGGp1EP g6Ycsxxp1nrjYgB2ZiRoL4JU2gbvp+rOAYhWtDuqlbgBkAn+a/7evM20jEBJnDM+9SPa C8sYYMOH/kIs8FW3K8UHlGJv3sorwUS9bBq6rp/b4bnyhLrcCHSY0eJ9XPAqkX9bfZ3l NQWA== X-Gm-Message-State: ACrzQf0als6DnJLnEs67IKfUKCBR1/UdT7djIbiZ1r3Z/+LC/1ywNmGN qhrQq6FYyKwFzbq9id0w0Lbiar6WlYToKLaPyA1+OV6mI55CH6nUl6L0k0NcNORqJPtxqEA2CIZ 6tEqvb7cIGrwMTxUNSEO/qc4H X-Received: by 2002:a05:6870:4212:b0:13c:d544:8e2a with SMTP id u18-20020a056870421200b0013cd5448e2amr10233067oac.28.1667354634222; Tue, 01 Nov 2022 19:03:54 -0700 (PDT) X-Received: by 2002:a05:6870:4212:b0:13c:d544:8e2a with SMTP id u18-20020a056870421200b0013cd5448e2amr10233044oac.28.1667354633998; Tue, 01 Nov 2022 19:03:53 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a802:1099:7cb2:3a49:6197:5307]) by smtp.gmail.com with ESMTPSA id h15-20020a9d6f8f000000b00665919f7823sm4526624otq.8.2022.11.01.19.03.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 19:03:53 -0700 (PDT) From: Leonardo Bras To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Frederic Weisbecker , Leonardo Bras , Phil Auld , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 3/3] mm/memcontrol: Add drain_remote_stock(), avoid drain_stock on isolated cpus Date: Tue, 1 Nov 2022 23:02:43 -0300 Message-Id: <20221102020243.522358-4-leobras@redhat.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221102020243.522358-1-leobras@redhat.com> References: <20221102020243.522358-1-leobras@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-3.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1748348217812919327?= X-GMAIL-MSGID: =?utf-8?q?1748348217812919327?= When drain_all_stock() is called, some CPUs will be required to have their per-CPU caches drained. This currently happens by scheduling a call to drain_local_stock() to run in each affected CPU. This, as a consequence, may end up scheduling work to CPUs that are isolated, and therefore should have as little interruption as possible. In order to avoid this, make drain_all_stock() able to detect isolated CPUs and schedule draining the perCPU stock to happen in another non-isolated CPU. But since the current implementation only allows the drain to happen in local CPU, implement a function to drain stock on a remote CPU: drain_remote_stock(). Given both drain_local_stock() and drain_remote_stock() do almost the same work, implement a inline drain_stock_helper() that is called by both. Also, since drain_stock() will be able to run on a remote CPU, protect memcg_hotplug_cpu_dead() with stock_lock. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 47 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 9 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index add46da2e6df1..7ad6e4f4b79ef 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -2263,7 +2264,7 @@ static void drain_stock(struct memcg_stock_pcp *stock) stock->cached = NULL; } -static void drain_local_stock(struct work_struct *dummy) +static inline void drain_stock_helper(int cpu) { struct memcg_stock_pcp *stock; struct obj_cgroup *old = NULL; @@ -2271,10 +2272,9 @@ static void drain_local_stock(struct work_struct *dummy) /* * The only protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. - * drain_stock races is that we always operate on local CPU stock - * here with IRQ disabled + * drain_stock races is stock_lock, a percpu spinlock. */ - stock = this_cpu_ptr(&memcg_stock); + stock = per_cpu_ptr(&memcg_stock, cpu); spin_lock_irqsave(&stock->stock_lock, flags); old = drain_obj_stock(stock); @@ -2286,6 +2286,16 @@ static void drain_local_stock(struct work_struct *dummy) obj_cgroup_put(old); } +static void drain_remote_stock(struct work_struct *work) +{ + drain_stock_helper(atomic_long_read(&work->data)); +} + +static void drain_local_stock(struct work_struct *dummy) +{ + drain_stock_helper(smp_processor_id()); +} + /* * Cache charges(val) to local per_cpu area. * This will be consumed by consume_stock() function, later. @@ -2352,10 +2362,16 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) if (flush && !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) { - if (cpu == curcpu) + if (cpu == curcpu) { drain_local_stock(&stock->work); - else + } else if (housekeeping_cpu(cpu, HK_TYPE_WQ)) { schedule_work_on(cpu, &stock->work); + } else { + int hkcpu = housekeeping_any_cpu_from(HK_TYPE_WQ, cpu); + + atomic_long_set(&stock->work.data, cpu); + schedule_work_on(hkcpu, &stock->work); + } } } migrate_enable(); @@ -2367,7 +2383,9 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) struct memcg_stock_pcp *stock; stock = &per_cpu(memcg_stock, cpu); + spin_lock(&stock->stock_lock); drain_stock(stock); + spin_unlock(&stock->stock_lock); return 0; } @@ -7272,9 +7290,20 @@ static int __init mem_cgroup_init(void) cpuhp_setup_state_nocalls(CPUHP_MM_MEMCQ_DEAD, "mm/memctrl:dead", NULL, memcg_hotplug_cpu_dead); - for_each_possible_cpu(cpu) - INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, - drain_local_stock); + /* + * CPUs that are isolated should not spend cpu time for stock draining, + * so allow them to export this task to the nearest housekeeping enabled + * cpu available. + */ + for_each_possible_cpu(cpu) { + if (housekeeping_cpu(cpu, HK_TYPE_WQ)) { + INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, + drain_local_stock); + } else { + INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, + drain_remote_stock); + } + } for_each_node(node) { struct mem_cgroup_tree_per_node *rtpn;