From patchwork Wed Jan 25 07:34:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras Soares Passos X-Patchwork-Id: 48037 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp147170wrn; Tue, 24 Jan 2023 23:40:28 -0800 (PST) X-Google-Smtp-Source: AMrXdXt1Ei7jIqhAXYiuZH5jmtAMblS3OQFSD8xZgsd/cN2wLV2e5Jpx/uqvwHvESpkovugl4gCS X-Received: by 2002:a17:90a:6391:b0:228:f576:b016 with SMTP id f17-20020a17090a639100b00228f576b016mr33510684pjj.25.1674632428156; Tue, 24 Jan 2023 23:40:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674632428; cv=none; d=google.com; s=arc-20160816; b=VLlDdi1F7AqpltTOyesr1bjov5PlkTvYaEsImq4vINiovbkI7zETReVlAZ11Z4pX2j 9yUKtSH7C1NKrZ/VFBb60xCzDUoiORMthFVb3dKWlC8FZPdNzy2knAufOZr4MUrp6ZKe aSqF3swjjPkpzAyDm+CHR2s+9+IdFmIgg9yR7IzcJlmWIg1XqZ1gmPAHzlOyRVOsWpVw 1waXkI/kLjDqmMtEwHtsfTo4Qg/jM6fKtIta11p9WQ5pIIcuyjUhDWgt78rz7CEK1Sqi k2x6GlkQlb0dbC0zUlbWIbFPDjQFPMc2nbCjUStQF9Ea6u+iEqGP3UESrUxafw0b+DW6 EK7g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KVG76aWPU3FY34u+v28shR/7Qg472WeM1BX+arjDbFU=; b=vDA0yfZvwRqO6gxOtdNyh+IJZ67hiWvXJ1yhqnEJS1chFGqivPv43hHKEGEYFUe4+K jFMux8PGdoRFwhzns2Lrd/rwgxtH4dPOP2P3XRQOKIXTwSLMuCtS1t1ejYpU8OWYT9MY +rfEM49U5Xa+EK6Fd5EmYMovJ20e9ULAvGvUet0UvXOnvYmHZ9qXC3d1X1nUG9ffcBdn aQKLPUQFYhRvB1EA5XZIyQzrHRbPP/uKAKXIIfKfz4y0mCLFw3QPuh0wdc+Jqjgmgake mmkSWG513O2cDa9Mwcg2EOjQcWmPhWlr91NJM1k9vj1tjPF3BRojR/buGNf0q7QAVbWG HG8A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PFCf9y1a; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l70-20020a638849000000b0047875582864si4689013pgd.263.2023.01.24.23.40.16; Tue, 24 Jan 2023 23:40:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PFCf9y1a; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235085AbjAYHgo (ORCPT + 99 others); Wed, 25 Jan 2023 02:36:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35734 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235068AbjAYHgk (ORCPT ); Wed, 25 Jan 2023 02:36:40 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB23337F1E for ; Tue, 24 Jan 2023 23:35:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632152; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KVG76aWPU3FY34u+v28shR/7Qg472WeM1BX+arjDbFU=; b=PFCf9y1aILug4RuXuZ5VVuMIlfN+xTWii3VtkSter+AZ+IHrFtNKfLED+yybJr95iAOI+n GrMhDxUrRoVZrqgcv6kRf4+yLSNAuNniz2DLvBNJK5PUmo0ujYb0RsYMu2s35Qglrzitxu f2TRjQhFlfP6AoDIqpOz1lehtmC8YKo= Received: from mail-oi1-f199.google.com (mail-oi1-f199.google.com [209.85.167.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-217-LuheR7CmNoCPHCcc3-3L6g-1; Wed, 25 Jan 2023 02:35:50 -0500 X-MC-Unique: LuheR7CmNoCPHCcc3-3L6g-1 Received: by mail-oi1-f199.google.com with SMTP id j5-20020a056808118500b0036b1cfb4778so5496544oil.2 for ; Tue, 24 Jan 2023 23:35:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KVG76aWPU3FY34u+v28shR/7Qg472WeM1BX+arjDbFU=; b=W/JY/RbRuYTur30xeCwQUThgngLuFDVxA2xcX4abZXR4b2XlKLVxTuBt6gIRlXh9J0 NjJS092Y7i8Dos0EV2axAz1OsjronKI8nMNwXOUA1STiziWWhdFicLqE7zC9iz1mGd+R WqaXaPRYulAmFhZtuCSc+NYiVPzAb8QtDuRnW1a/q7zTIfVAngEda1BGTOo++irDIj2y 8gX7iAyLGzEQEttzCEYutn6eLgFSko0whPNJz52Kr5qrQ5iqtx11l9sqP8JbnTSnUFpL /oPmdRdBaJKCqfLN4qwQCh0CckO7bqVrIepThJIBSQ4vShTxDzwQQ7WXKKM/HIWPFB+A yw7g== X-Gm-Message-State: AFqh2krS9OZeJ5/itAUXxxL4beAIX5IXi7ppZGmOWXjVhiBcEJmmt6rn QtMefceixwOwJeTZvy7HLV9fwZL2GyPTJT8Dd7TBD/S+P4GK2Gi3XLWxBCY7fl4jsN+l5oD61dD OijwGDpoVmfxv6saxA7JDSZ4B X-Received: by 2002:a05:6870:8dc5:b0:15f:74b2:9168 with SMTP id lq5-20020a0568708dc500b0015f74b29168mr14415645oab.4.1674632149720; Tue, 24 Jan 2023 23:35:49 -0800 (PST) X-Received: by 2002:a05:6870:8dc5:b0:15f:74b2:9168 with SMTP id lq5-20020a0568708dc500b0015f74b29168mr14415637oab.4.1674632149469; Tue, 24 Jan 2023 23:35:49 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:35:48 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/5] mm/memcontrol: Align percpu memcg_stock to cache Date: Wed, 25 Jan 2023 04:34:58 -0300 Message-Id: <20230125073502.743446-2-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755979373309846568?= X-GMAIL-MSGID: =?utf-8?q?1755979373309846568?= When a struct smaller than a cacheline has an instance that is not aligned to cache block size, it could happen that data can be spread across two cachelines. For memcg_stock this could mean the need to fetch and get cache-exclusivity in 2 cachelines instead of 1 when we bounce the cacheline between local cpu functions and drain_all_stock(), which does remote read/write. This could also mean some false-sharing costs being paid due to the cacheline being shared between 2 unrelated structures. Avoid this issue by getting memcg_stock cacheline-aligned. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ab457f0394ab6..f8e86b88b3c7a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2188,7 +2188,8 @@ struct memcg_stock_pcp { unsigned long flags; #define FLUSHING_CACHED_CHARGE 0 }; -static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { + +static DEFINE_PER_CPU_SHARED_ALIGNED(struct memcg_stock_pcp, memcg_stock) = { .stock_lock = INIT_LOCAL_LOCK(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); From patchwork Wed Jan 25 07:34:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras Soares Passos X-Patchwork-Id: 48033 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp146804wrn; Tue, 24 Jan 2023 23:39:10 -0800 (PST) X-Google-Smtp-Source: AMrXdXtLYCvW7QhPJY+bkjcDIh7DK6Lzid4CnvxT5LVRH+fn9M+c+OLfavohprDEETPhnpij2gep X-Received: by 2002:a17:902:eb83:b0:194:9d31:f270 with SMTP id q3-20020a170902eb8300b001949d31f270mr34391390plg.43.1674632350000; Tue, 24 Jan 2023 23:39:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674632349; cv=none; d=google.com; s=arc-20160816; b=x+DwyrrNonoxbEZ2khNoms3UitD/7doN7UgK9Vf9QzjDGczgN82AfJkqlKRBYMpOyJ uu0YJi6zpAYpDTVWg8s9ehA7Qv4QOfeGsrZk01HQIDOjkrG79mak+xSDvikJ/2KihGBJ lF/AOLzWqO90zwEotXIug2sOlxurSTeDXetXje8I3Wdi7k8Eu6aooeFjshl2yYmrv9Mc ChQ57esF5ggwxEhU4TG4AvbmCrOAXR011OEdiH9ml1u4r/tCfsN5GRLcSikk43AiLNzz lJ5dKqbBZvQMosasQfQN+BpNP430wuvFt5uBON2VF3ZiqCSYUG9VuAXoXOCXV11t4LP8 UOlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1+Ul5sQzoaw3CKL+Rr3WW09PeQYvbUe0nkd8DHiRlc0=; b=LplT7sDR486qypHT7HpbaWk6H+uhOTqKlnSMAUx6IiSc/T4LsS768HXQa+Wj3vjVH1 jmJDFN1lJO1Sp401kSupzQ1IrqS2+vNl0WNlY4x6632spBtLH115pK2GBITMgXPrp4Dy sRAfdCdekUtIlVlFfJQDooxPDeKKjOuETNpPRI1gZHuyIzUZIkp9p05wfYnOb0SqvIBx uYoI9RbHdrX15m2X+qkbbu1UZCvVbVEitWJlYy6mjn37bYVPGculcIBM1avmYwwKHY7/ ntFCSVib1PFiQFVlq2tqftVhqA605xkqXgm9MFdOCs8famO1stRzSHcNlhAn17UrRQF7 OHaw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dekaFMgE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d16-20020a170902b71000b001926b6927fbsi4787585pls.10.2023.01.24.23.38.58; Tue, 24 Jan 2023 23:39:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=dekaFMgE; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235099AbjAYHgy (ORCPT + 99 others); Wed, 25 Jan 2023 02:36:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235086AbjAYHgp (ORCPT ); Wed, 25 Jan 2023 02:36:45 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A33D843457 for ; Tue, 24 Jan 2023 23:35:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632156; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1+Ul5sQzoaw3CKL+Rr3WW09PeQYvbUe0nkd8DHiRlc0=; b=dekaFMgEkNAX/BhHGbGph44K2HgSP8qDPl9IQHon3PZ4NJ0NNk9XlhxpUcfftryKp2Ej7Q DqZ/j7/wVZG7yOIuOIKWuryuA7cQvNR9GtQxtwMptvBE3KXLN4z/Vvbi8tXouihaYgcMLN +knAf4QuTh5iE7oq7bsUPxxEC1u0eck= Received: from mail-oa1-f71.google.com (mail-oa1-f71.google.com [209.85.160.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-304-gfHaajAoPI-POjm3Cq7Y9A-1; Wed, 25 Jan 2023 02:35:55 -0500 X-MC-Unique: gfHaajAoPI-POjm3Cq7Y9A-1 Received: by mail-oa1-f71.google.com with SMTP id 586e51a60fabf-15fe7396eb4so4588179fac.12 for ; Tue, 24 Jan 2023 23:35:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1+Ul5sQzoaw3CKL+Rr3WW09PeQYvbUe0nkd8DHiRlc0=; b=K2GyaO8RZiypo436qeWpLq6OAD809Nfgpa3OShPhz0OCAacG9DM/Uz2gqNJ9YH57rs nlVq9qirBZjl7uOvGGIyR2vIENjGpshevD6ge2h0xNyQ0PDamx2f9AMH5sAg6nXgUAp4 PX+maNW9wNaFK8HxGY30VzK/7fQfDIUSlroEyz2L83xcSfmri3U8B2K7KI0Oqqjnblxb 70cJD4I9VQoDkwB2WpxbUBwOF9yZnJjKr8GfZxc0i78RiKZ19bZrTZIEDiY4UCHBF1v5 2gXL6FB3FUJvQdgjIu8PFwzuIwVkP5MQGrq/A8ahrn8fmmTsvnuTHGRU7rvfYLMiwLVg pgQg== X-Gm-Message-State: AFqh2kri+IUsweoncuCeK8R+PPIU1dIIzVJI+carphH6Dq/PB9ZrViPz njpD7aEn4qnza1efyMRQm7VTC08FSS9NuW7Mf42GDVyMSGpUabTpt9n6JZCTB+NQqmlG6+oDg4/ 3r/UaMVb20BoKiiMr1Lwd8Jmu X-Received: by 2002:a05:6870:9124:b0:15f:c278:ef49 with SMTP id o36-20020a056870912400b0015fc278ef49mr9943008oae.6.1674632154323; Tue, 24 Jan 2023 23:35:54 -0800 (PST) X-Received: by 2002:a05:6870:9124:b0:15f:c278:ef49 with SMTP id o36-20020a056870912400b0015fc278ef49mr9943000oae.6.1674632154115; Tue, 24 Jan 2023 23:35:54 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:35:53 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/5] mm/memcontrol: Change stock_lock type from local_lock_t to spinlock_t Date: Wed, 25 Jan 2023 04:34:59 -0300 Message-Id: <20230125073502.743446-3-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755979290797825911?= X-GMAIL-MSGID: =?utf-8?q?1755979290797825911?= In this context, since it's using per-cpu variables, changing from local_lock to spinlock should not deal much impact in performance and can allow operations such as stock draining to happen in remote cpus. Why performance would probably not get impacted: 1 - Since the lock is in the same cache line as the information that is written next, there is no much extra memory access cost for using the lock. 2 - Since it's a percpu struct, there should be rare for other cpu to share this cacheline, so there should be rare to need cacheline invalidation, and writing to the lock should be cheap since there is always a write to next struct members. 3 - Even the write in (2) could be pipelined and batched with following writes to the cacheline (such as nr_pages member), further decreasing the impact of this change. Suggested-by: Marcelo Tosatti Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f8e86b88b3c7a..1d5c108413c83 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2172,7 +2172,7 @@ void unlock_page_memcg(struct page *page) } struct memcg_stock_pcp { - local_lock_t stock_lock; + spinlock_t stock_lock; /* Protects the percpu struct */ struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; @@ -2190,7 +2190,7 @@ struct memcg_stock_pcp { }; static DEFINE_PER_CPU_SHARED_ALIGNED(struct memcg_stock_pcp, memcg_stock) = { - .stock_lock = INIT_LOCAL_LOCK(stock_lock), + .stock_lock = __SPIN_LOCK_UNLOCKED(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); @@ -2235,15 +2235,15 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (memcg == stock->cached && stock->nr_pages >= nr_pages) { stock->nr_pages -= nr_pages; ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); return ret; } @@ -2280,14 +2280,14 @@ static void drain_local_stock(struct work_struct *dummy) * drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + old = drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -2315,10 +2315,12 @@ static void __refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; + struct memcg_stock_pcp *stock; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); __refill_stock(memcg, nr_pages); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); } /* @@ -3165,8 +3167,8 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, unsigned long flags; int *bytes; - local_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); /* * Save vmstat data in stock and skip vmstat array update unless @@ -3218,7 +3220,7 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, if (nr) mod_objcg_mlstate(objcg, pgdat, idx, nr); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -3229,15 +3231,15 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) unsigned long flags; bool ret = false; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (objcg == stock->cached_objcg && stock->nr_bytes >= nr_bytes) { stock->nr_bytes -= nr_bytes; ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); return ret; } @@ -3327,9 +3329,9 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, unsigned long flags; unsigned int nr_pages = 0; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (stock->cached_objcg != objcg) { /* reset if necessary */ old = drain_obj_stock(stock); obj_cgroup_get(objcg); @@ -3345,7 +3347,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock->nr_bytes &= (PAGE_SIZE - 1); } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); From patchwork Wed Jan 25 07:35:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras Soares Passos X-Patchwork-Id: 48035 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp146973wrn; Tue, 24 Jan 2023 23:39:48 -0800 (PST) X-Google-Smtp-Source: AMrXdXt/rDfyFqJ4HvAxy/mX0tG0GhWp7r0WqZCsuLw2AmhMScKj77jpjby/9m32slRTcU/m0Hwq X-Received: by 2002:a17:906:8d06:b0:7b5:911c:9b12 with SMTP id rv6-20020a1709068d0600b007b5911c9b12mr31129435ejc.1.1674632388157; Tue, 24 Jan 2023 23:39:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674632388; cv=none; d=google.com; s=arc-20160816; b=ZqILKppZ7Ak8ozI4/DSGmjzX5xByVCh/IL6AKDNOv8uq+sh3DGyhf/U5FGsfpYcDv/ m0I/FMZ6DSzcVTF2t/9OOxVOoXTzQ87+XON9frOWtpB+ihq9ZwDYgwjLTKvGwL7obpVP r2oF3LbxjbJcWNMlms1bVbkuyJiovJPI/lkZxcaiqxcr4yYUuzuCDK/GbjPHS6e/IfWw 5QXGkfcJMQNADu0AouuJttKb3vNotPAsAOaLfnGxc93IwFH0mvKt0J9U3w2hVC2wh6BV qcddSkx8QlTr+30lbMSQIKJo50NWL8AMu8hGddpK2EZgae25XRXhkMGZupPMMuYDoVMC 7MlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=i0PZh+yTsPG9Po4mo5Kh71H2anPuGxn31rIFI1zWHXw=; b=Y+yCtz4fF2Rc7LUhllWWWbC01A2z4QHDRZNSMbhuTIHd+Q33MEBwRKj0ZeS12SsDlD hSMFcOADyhdEAKEadA/FXjRpv/VsYZGkNbDW+A8O9YThDqGepfYV/pa6mIO/H7kPvW1E /IhlnZdffMyIrig79d4Ekfjg0Z2FAird/x+Z7i2WXzYPw5q+xiokSwFyeETGqlCMxIBr PLIEE18eXSMDKengy4V+FfN06hYMwKWaaWyeZysedbhBj2f3kEGu/5oOpd23Z9F93Z/y /vCW0OfCqicWiWhMEBUCbL0DrElf9Gx5nb0BPkNITM9VgenITXaZaH1O706tIydYv6C6 rzqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cEhyx4CV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w20-20020aa7d294000000b0049eebc96d4fsi5578033edq.372.2023.01.24.23.39.23; Tue, 24 Jan 2023 23:39:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=cEhyx4CV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235182AbjAYHhC (ORCPT + 99 others); Wed, 25 Jan 2023 02:37:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36072 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235109AbjAYHgs (ORCPT ); Wed, 25 Jan 2023 02:36:48 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E9CC4A223 for ; Tue, 24 Jan 2023 23:36:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632161; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i0PZh+yTsPG9Po4mo5Kh71H2anPuGxn31rIFI1zWHXw=; b=cEhyx4CVXxxP7oH0boIAzIj4YO3rZjYnryslJLa+Rx9MeLb9JMXiId3qKVrxhU4KnBgREv 7K3uhEBsjOWnFhkMrwB0T5Lhjdwfn4Bq4OY54//l4L/BTbTEMWU+i4sFlixT6KhRDdUA9L NmsUkmfMGPPcqhFYN34dGmk6OHu5h/g= Received: from mail-ot1-f70.google.com (mail-ot1-f70.google.com [209.85.210.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-649-xazf8DXcNbiW-sKrngbvQA-1; Wed, 25 Jan 2023 02:36:00 -0500 X-MC-Unique: xazf8DXcNbiW-sKrngbvQA-1 Received: by mail-ot1-f70.google.com with SMTP id a4-20020a9d5c84000000b00686ab49309fso6873980oti.19 for ; Tue, 24 Jan 2023 23:35:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i0PZh+yTsPG9Po4mo5Kh71H2anPuGxn31rIFI1zWHXw=; b=LkOC7uqvuFL1d18YOqAx5RIBeaAYZch6w8/I2Wi7HJzn5h7D75Z2MIsLtmltPDt3Tw sFAi+5KacNi1t0/QNhgyn05UzrOPDe7fhjJD2A+9xeN1OYmk/Q169TJxpqcikogvVF6Y pS77/LPwlBmnsE5N937A1BuCfhivxsmarlQ+4nVzM71FVFHroXDMj8eLgfvYqFldqTQp unk+HgVywnVF/0ma99Hy33qM6TdrL4pJodx5JkzDWOO6n+c8Yzhkyj2vikAv9W63MvIb xKIZINdMRULD7p1wMevCKGBS33FgH+H5VbDuy11GjdyJaj8E5UvJISj0PjGe82GtxTfo uwXg== X-Gm-Message-State: AO0yUKVyf52DnTA3N3glWLo1FQoCu8UqC2RzeKO8X5YMg6k4tubTPp5p tzAwLch86F60BYUdyU5dWV2/zDfbKdXA7VCRtRmT+QhGOwBwiG/waS6QpCJR17J767XIi0owEC3 25UgVaeAZKKlF2YJHgljgKdnb X-Received: by 2002:a05:6870:a788:b0:163:2d2d:beb with SMTP id x8-20020a056870a78800b001632d2d0bebmr1040179oao.26.1674632159218; Tue, 24 Jan 2023 23:35:59 -0800 (PST) X-Received: by 2002:a05:6870:a788:b0:163:2d2d:beb with SMTP id x8-20020a056870a78800b001632d2d0bebmr1040175oao.26.1674632158988; Tue, 24 Jan 2023 23:35:58 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:35:58 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/5] mm/memcontrol: Reorder memcg_stock_pcp members to avoid holes Date: Wed, 25 Jan 2023 04:35:00 -0300 Message-Id: <20230125073502.743446-4-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755979331227666490?= X-GMAIL-MSGID: =?utf-8?q?1755979331227666490?= In 64-bit architectures, the current layout of memcg_stock_pcp should look like this: struct memcg_stock_pcp { spinlock_t stock_lock; /* 0 4 */ /* 4 bytes hole */ struct mem_cgroup * cached; /* 8 8 */ unsigned int nr_pages; /* 16 4 */ /* 4 bytes hole */ [...] }; This happens because pointers will have 8 bytes (64-bit) and ints will have 4 bytes. Those 2 holes are avoided if we reorder nr_pages and cached, effectivelly putting nr_pages in the first hole, and saving 8 bytes. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1d5c108413c83..373fa78c4d881 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2173,8 +2173,8 @@ void unlock_page_memcg(struct page *page) struct memcg_stock_pcp { spinlock_t stock_lock; /* Protects the percpu struct */ - struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; + struct mem_cgroup *cached; /* this never be root cgroup */ #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cached_objcg; From patchwork Wed Jan 25 07:35:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras Soares Passos X-Patchwork-Id: 48034 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp146942wrn; Tue, 24 Jan 2023 23:39:40 -0800 (PST) X-Google-Smtp-Source: AMrXdXtuyK/i+1kqK7OkWspC45n7l1EbY7vKP7zNxTQB0DxCCcPeJ31Lj9JAmc27EYQX529XTWVh X-Received: by 2002:a17:907:2b01:b0:877:6e09:468d with SMTP id gc1-20020a1709072b0100b008776e09468dmr20360257ejc.7.1674632380285; Tue, 24 Jan 2023 23:39:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674632380; cv=none; d=google.com; s=arc-20160816; b=aqz5vJZgjBBGFTYejBdtIi7wZZuVG0PvZpSVcYweP6BVoCyWGKB4srGi8wCNrKJ3Vq JRRb4iLIRLLwAKJgd/lOaAXbX/Q3kGZXg2904HQBfhILCos2TxH6dhqwdjRDaO/vOUMk rb1zfGDFe/1G6Cvjy7rxvvGkKDYLFL2N97GSLgNZOZolZZ/U2liImta/GcuU883Hw6pJ l/vvXeRhS9SfN9qLrGjHYTbKE8XY2mKWLVo1Ldu6LGimgARQF7cXJyYYdBmVjIqqvdGh nJ//EHrESkMYdJNx7eeKMem2mGF/P5PK1qlr+wZXQFC46M8pTd0Df5mcW0TFYe50mrXi g3mQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=cta4F9u/56j6UuYJx1HDD/ltBRbZgzhdAn9u+4PdIyQ=; b=iiPNFPvGIQSDRCcL09u8Igyh1BMcVFCnGssDEQFUQ807TieP9P2ZnWCIjwbiLFPbUU IZxz3yEQU1B9rjUz422RzrGdKi57U0anfBLarShsvn7mjYuaeQuzdZWT/s1Jfv8eQswZ qGHacUHpVwmC2VG/NCALzTsXJcd9YQLCEpVVdIp6RWLTcUjD6DTLUTIm860IUPjh9F4K 6kO6qdesSfmqDzPWCz2bZViCNtCM2Vi0hk838ylhY03HJGsWSIJb84dsdMLKhQ8CPxU5 vFHmk/3wzPIf7GiP3dk1JjnuCTOKe1HvTqN7p0RJq3Gorv9JHICnm5CgptSIh1wY9Wve 6wRg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PX8YfOcU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fm20-20020a1709072ad400b0084b9b15e1cfsi5106546ejc.41.2023.01.24.23.39.16; Tue, 24 Jan 2023 23:39:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=PX8YfOcU; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235090AbjAYHg7 (ORCPT + 99 others); Wed, 25 Jan 2023 02:36:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235097AbjAYHgq (ORCPT ); Wed, 25 Jan 2023 02:36:46 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 023BA3E602 for ; Tue, 24 Jan 2023 23:36:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cta4F9u/56j6UuYJx1HDD/ltBRbZgzhdAn9u+4PdIyQ=; b=PX8YfOcUQ9+4SqkymxPco8kbg9Q5hvtS/sv54cDrH/YEloZa3Wf9aLDfRKHh0LQ5t0htcf TML3pLGjtZkvua6rFoZUytr6W3EmSl3Yh9U+W+5dfAC8IgkQZBAdYZljKi3Hru0WQkwK61 JCQwKwmbUNomzoIEoLRW+syeqiDDZ98= Received: from mail-oo1-f70.google.com (mail-oo1-f70.google.com [209.85.161.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-619-7c0AlrY2Pte6H5VOdvYvNg-1; Wed, 25 Jan 2023 02:36:05 -0500 X-MC-Unique: 7c0AlrY2Pte6H5VOdvYvNg-1 Received: by mail-oo1-f70.google.com with SMTP id w2-20020a4aca02000000b0050d6685961bso1101249ooq.19 for ; Tue, 24 Jan 2023 23:36:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cta4F9u/56j6UuYJx1HDD/ltBRbZgzhdAn9u+4PdIyQ=; b=tN48J5ehtAOjYetksweifoUeBnPsUmMEAlQGs8oX2zcM0MLN02oc7nP88p073rLkmk zIv1OWiHDOTeqDpfRRXfY1OnP/iruH7thrvoCvhDGCqQD72HabMq5xO/IVL6QNP2FxaL bOUVlAKE+lpQwFWRonY6ywHOl8bInb591+F6TvuQ7fBb0jymlu3+sdRFR9hzk/REi6X5 VAyBJE09688N2kb4OqtJXThPT1ET/h+Ar9PnadFTlhMK1kC7IaxpHZJptVbVsNoAVznv rIP1MDVJq5qkdbTFbVl4URIA7We994kCS+L/LjG9FoplvNGln84SftIdKQ9izxfWQhb1 r5mQ== X-Gm-Message-State: AFqh2kobcyfiFQd5YYbw6eL5N7SAyenZm+1KPRV0Le8uCmUUge2Snk1z LN2C3W30L5gNDH5uUfSqu14/H10R0p/HNdptxHeQ5BgZ52LfQVv+j1MUjBCi417Ufw4rOxSimus fRzRdByXJWird/4LtFnxT8ALA X-Received: by 2002:a4a:d307:0:b0:4f5:2a03:b22a with SMTP id g7-20020a4ad307000000b004f52a03b22amr14131796oos.4.1674632164303; Tue, 24 Jan 2023 23:36:04 -0800 (PST) X-Received: by 2002:a4a:d307:0:b0:4f5:2a03:b22a with SMTP id g7-20020a4ad307000000b004f52a03b22amr14131784oos.4.1674632164048; Tue, 24 Jan 2023 23:36:04 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:36:03 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/5] mm/memcontrol: Perform all stock drain in current CPU Date: Wed, 25 Jan 2023 04:35:01 -0300 Message-Id: <20230125073502.743446-5-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755979322691063502?= X-GMAIL-MSGID: =?utf-8?q?1755979322691063502?= When drain_all_stock() is called, some CPUs will be required to have their per-CPU caches drained. This currently happens by scheduling a call to drain_local_stock() to run in each affected CPU. This, as a consequence, may end up scheduling work to CPUs that are isolated, and therefore should have as little interruption as possible. In order to avoid this, run all CPUs stock drain directly from the current CPU. This should be fine as long as drain_all_stock() runs fast enough so it don't often cause contention on consume_stock(), refill_stock(), mod_objcg_state(), consume_obj_stock() or refill_obj_stock(). Also, since drain_all_stock() will be able to run on a remote CPU, protect memcg_hotplug_cpu_dead() with stock_lock. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 373fa78c4d881..5b7f7c2e0232f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2184,7 +2184,6 @@ struct memcg_stock_pcp { int nr_slab_unreclaimable_b; #endif - struct work_struct work; unsigned long flags; #define FLUSHING_CACHED_CHARGE 0 }; @@ -2269,18 +2268,15 @@ static void drain_stock(struct memcg_stock_pcp *stock) stock->cached = NULL; } -static void drain_local_stock(struct work_struct *dummy) +static void drain_stock_from(struct memcg_stock_pcp *stock) { - struct memcg_stock_pcp *stock; struct obj_cgroup *old = NULL; unsigned long flags; /* - * The only protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. - * drain_stock races is that we always operate on local CPU stock - * here with IRQ disabled + * The protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. + * drain_stock races is stock_lock, a percpu spinlock. */ - stock = this_cpu_ptr(&memcg_stock); spin_lock_irqsave(&stock->stock_lock, flags); old = drain_obj_stock(stock); @@ -2329,7 +2325,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) */ static void drain_all_stock(struct mem_cgroup *root_memcg) { - int cpu, curcpu; + int cpu; /* If someone's already draining, avoid adding running more workers. */ if (!mutex_trylock(&percpu_charge_mutex)) @@ -2341,7 +2337,6 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) * per-cpu data. CPU up doesn't touch memcg_stock at all. */ migrate_disable(); - curcpu = smp_processor_id(); for_each_online_cpu(cpu) { struct memcg_stock_pcp *stock = &per_cpu(memcg_stock, cpu); struct mem_cgroup *memcg; @@ -2357,12 +2352,8 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) rcu_read_unlock(); if (flush && - !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) { - if (cpu == curcpu) - drain_local_stock(&stock->work); - else - schedule_work_on(cpu, &stock->work); - } + !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) + drain_stock_from(stock); } migrate_enable(); mutex_unlock(&percpu_charge_mutex); @@ -2373,7 +2364,9 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) struct memcg_stock_pcp *stock; stock = &per_cpu(memcg_stock, cpu); + spin_lock(&stock->stock_lock); drain_stock(stock); + spin_unlock(&stock->stock_lock); return 0; } @@ -7328,7 +7321,7 @@ __setup("cgroup.memory=", cgroup_memory); */ static int __init mem_cgroup_init(void) { - int cpu, node; + int node; /* * Currently s32 type (can refer to struct batched_lruvec_stat) is @@ -7341,10 +7334,6 @@ static int __init mem_cgroup_init(void) cpuhp_setup_state_nocalls(CPUHP_MM_MEMCQ_DEAD, "mm/memctrl:dead", NULL, memcg_hotplug_cpu_dead); - for_each_possible_cpu(cpu) - INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, - drain_local_stock); - for_each_node(node) { struct mem_cgroup_tree_per_node *rtpn; From patchwork Wed Jan 25 07:35:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras Soares Passos X-Patchwork-Id: 48036 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp147051wrn; Tue, 24 Jan 2023 23:40:03 -0800 (PST) X-Google-Smtp-Source: AMrXdXsqKWY39R07SwNah9pOSG+/I5gCPsQHdliAJuubPp7si/bki9K85nAgNKInnyFmhOwFtZBa X-Received: by 2002:a05:6a20:9f90:b0:b6:597f:66a1 with SMTP id mm16-20020a056a209f9000b000b6597f66a1mr33467258pzb.60.1674632403371; Tue, 24 Jan 2023 23:40:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674632403; cv=none; d=google.com; s=arc-20160816; b=hUNG1kqb0zwpLgtLhNXMnJxYwFuY1QmQrRseuFuHQotNNGV53NhTDE/iKMkW2J5ZJO +waOVrdxXQtn7Sz5IKj1xhcVhooaSruKd7/EMu3dR7sZdPCiVst9/ikrh0pCogDcwb/6 YFpf5wtPfal0JZXCQInudvRdiYDoatf+NrrYLacmXXSVXaY6APw3EHFpUMsuL+EiMA7M WNZFm0lmYVOsnNBkB1oYXt4kVAzYCPzJqddDJabKC+ZcY6ek3N5JFkf5z2evH3loCx8Y a0lAVygQNYW023Tie39yd/T2YU4idSxc7HgsIaazkwL+KSY5NsY//BiFDPw3LmiB5uNS TPgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NWp8AY96vdXmiMbopROrFgs2cU/VADiUd7iIJJXBCxY=; b=fmDfbzRsYb4CqyeJY1U9tIbDaf3CuZu8uZ9S6R4wgNMKkPfNP5CvnvZXZc/M9jZ5TL iekyhWaKFif7XHhNsC3Bz+9Trbb4mtXbA8+hx++eBrqdv+nJaAx324iQBnZzouNE3Mb2 RvR0Woqyzyuz56oX7iGFRQpF8ZabITHmiVEnfEHUJZtKWxGQ7rO8qR6hC/JsZN6M+yKT IbIrqSO/JkJ86dz+8tunTKsdoMPpXu8Ps11rSBL7i3rVIqvYbUxxl73hCMp1cf2Mnu/D s9+HRpC+OKxr69Zpo/I5rPyfQqOA/GOIvNYlLZ+pqdjEIEz+a7CDci/avF5VJThVuvLe aunA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="el/47FiS"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k9-20020a63ab49000000b004788e156154si4533866pgp.360.2023.01.24.23.39.51; Tue, 24 Jan 2023 23:40:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="el/47FiS"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235172AbjAYHhS (ORCPT + 99 others); Wed, 25 Jan 2023 02:37:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235129AbjAYHhC (ORCPT ); Wed, 25 Jan 2023 02:37:02 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A87737F1F for ; Tue, 24 Jan 2023 23:36:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632171; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NWp8AY96vdXmiMbopROrFgs2cU/VADiUd7iIJJXBCxY=; b=el/47FiS49Sk8mreZ2OWz2iX8VjGh8taqZbHwZv74qFL3bVKOKHuGUe1yFvUTuI0sjc6m5 llrVLhXki9/43TklZkbLmHtvBtMoQwdOMp6NRXwtztEQzAPcDlrBLP9En/LYAUcaV9QHtN FG92PPC+IlBX9V4mlINT0wPRgM/jh1U= Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-67-s94wKnlcOyi377eKMrntwg-1; Wed, 25 Jan 2023 02:36:10 -0500 X-MC-Unique: s94wKnlcOyi377eKMrntwg-1 Received: by mail-oi1-f198.google.com with SMTP id u9-20020a544389000000b00363be5d9f42so5551447oiv.15 for ; Tue, 24 Jan 2023 23:36:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NWp8AY96vdXmiMbopROrFgs2cU/VADiUd7iIJJXBCxY=; b=KHuHECozvbsms4X3V9pEXeDQ/1z0byeTw18eZsFxL08f5rbD6E/81vVHCBJ10/xfJ6 eZ0cyWffoOx1q2nfms8vxwngp3e/dc7kOzRY8XDZKgWFfZFEaSye0OFyIfM+NVFWZEdm erd4IKTopqKVmS9v/aRMqhBBF1wFV4VRoocMu63xGxi1zqj+hilmrCtNTDwNgeHWU23M qqNm8S0LvcCvuGwfiryq5jUQLgzMQkY+W/8MkATI4tPH2BJUoYR5zkE3SH+ALtC8Uea/ +dvb8rcfnEacfaxyh9EX+75PzuP16hCNxW0yxlB3Z4FKPjj/y8SZXfPNOB/wNxobk1wm j1ZQ== X-Gm-Message-State: AFqh2kpki+vNd7tIhh4oHEdLkHXZYWoEWKDJmTeYxtbPtHWWMK6B2n4w KK6ma08AlWyfcoYD9QfyNUPlkGa3jCWiLxviUKAu7y1tQ2pmH5WNKM5h2qumKHLm2eS8eczJHR3 iwPdkYxaP2u2pcQUEtV/7vWNk X-Received: by 2002:a05:6870:e0ce:b0:15f:3bb9:7b3d with SMTP id a14-20020a056870e0ce00b0015f3bb97b3dmr17462992oab.28.1674632169346; Tue, 24 Jan 2023 23:36:09 -0800 (PST) X-Received: by 2002:a05:6870:e0ce:b0:15f:3bb9:7b3d with SMTP id a14-20020a056870e0ce00b0015f3bb97b3dmr17462978oab.28.1674632169128; Tue, 24 Jan 2023 23:36:09 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.36.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:36:08 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 5/5] mm/memcontrol: Remove flags from memcg_stock_pcp Date: Wed, 25 Jan 2023 04:35:02 -0300 Message-Id: <20230125073502.743446-6-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755979346677472397?= X-GMAIL-MSGID: =?utf-8?q?1755979346677472397?= The flags member of struct memcg_stock_pcp has only one used bit: FLUSHING_CACHED_CHARGE Both struct member and flag were created to avoid scheduling multiple instances of kworkers running drain_local_stock() for a single cpu. How could this scenario happen before: - drain_all_stock() gets called, get ownership of percpu_charge_mutex, schedules a drain_local_stock() on cpu X, and drops ownership of percpu_charge_mutex. - Another thread calls drain_all_stock(), get ownership of percpu_charge_mutex, schedules a drain_local_stock() on cpu X, ... Since the stock draining is now performed by the thread running drain_all_stock(), and happens before letting go of the percpu_charge_mutex, there is no chance of another drain happening between test_and_set_bit() and clear_bit(), so flags is now useless. Remove the flags member of memcg_stock_pcp, its usages and the FLUSHING_CACHED_CHARGE define. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5b7f7c2e0232f..60712f69595e4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2183,9 +2183,6 @@ struct memcg_stock_pcp { int nr_slab_reclaimable_b; int nr_slab_unreclaimable_b; #endif - - unsigned long flags; -#define FLUSHING_CACHED_CHARGE 0 }; static DEFINE_PER_CPU_SHARED_ALIGNED(struct memcg_stock_pcp, memcg_stock) = { @@ -2281,7 +2278,6 @@ static void drain_stock_from(struct memcg_stock_pcp *stock) old = drain_obj_stock(stock); drain_stock(stock); - clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) @@ -2351,8 +2347,7 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) flush = true; rcu_read_unlock(); - if (flush && - !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) + if (flush) drain_stock_from(stock); } migrate_enable();