From patchwork Mon Nov 20 02:35:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhiguo Jiang X-Patchwork-Id: 166920 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:9910:0:b0:403:3b70:6f57 with SMTP id i16csp1943098vqn; Sun, 19 Nov 2023 18:36:36 -0800 (PST) X-Google-Smtp-Source: AGHT+IGL9GEvmlP5X/5bjBrwGHiZXCfN4tx/KY5a82v/TMMsl3nVOftERq/lLSWdo4Gef9mNZKdC X-Received: by 2002:a05:6808:3999:b0:3b2:e5f2:5a59 with SMTP id gq25-20020a056808399900b003b2e5f25a59mr1110246oib.35.1700447796157; Sun, 19 Nov 2023 18:36:36 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1700447796; cv=pass; d=google.com; s=arc-20160816; b=UQyS6nXdeUvSAoIMnWqLfUGJXQvS2hA7gGJGikFyktA8qUn6tWZKE6hsawdhEnMO3k TxL3QVEI9tuinS0jfhzozqwtRPgCtMqYiUQ08c7kcXq1CpDeLKdKPxKWieShHqs/KRW8 VjIUl9+ZplsSmgs9lM9gM7VnGXCdUf9sbiE1K9qFm9PGtr12gos1YhMKQbUEZFZBYD7q LSdfSC7kNQxJQa/l2Lc6MfUlsQ+LTgq8CCVivLbDtPxTOA5FmnSjmNVzHkwO2iJgG0os QmDwDe/GQ1jeubj25M56JSHR5AFy7eCaDKrK6p2yUgm8cwOxOCNvnC4ESmmF6uDCHC8k /3Gg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :message-id:date:subject:cc:to:from:dkim-signature; bh=xBeZTAyGspQwRPIMNiuVt1kp+s/UGwHOXaMl87chj/U=; fh=eO6uY58qo6Irpb51LjbHpHfACnP+e1Mre/LkeZaqQxU=; b=zFe4MniIbXf4HFXKZ3gbWfVV3Zky4FXf36r7gJ3ZCHKdKZI9HMCwSQk+r9f+qCKFHn hUigE2Lgp476tIDMqieN48MZrIgue4ToCRrDiCsVArpavhlvUkawdCXw1HkdqzlV6pIb A2p2j+BzM6qMTms3J8KWPLurBqBtj5PfgWtnHV3srrJkI5qFfPqbvCfph9pPwPFevLEZ Cn8jFYb59AESkecx7edbWhcs0+SAtaa/pkaBI+jfQghFSFKan9DQ+La6/iDqYPpmnSXi t9/iQUjv2Yz/fX8ubOem0F032gcMIYEQcfs164jp90TjZTleviap2fsfHZqsdW7izNFM ybtA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@vivo.com header.s=selector2 header.b=QyiZTQym; arc=pass (i=1 spf=pass spfdomain=vivo.com dkim=pass dkdomain=vivo.com dmarc=pass fromdomain=vivo.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=vivo.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id z26-20020a63191a000000b005a9f776c57dsi7278879pgl.644.2023.11.19.18.36.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 19 Nov 2023 18:36:36 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@vivo.com header.s=selector2 header.b=QyiZTQym; arc=pass (i=1 spf=pass spfdomain=vivo.com dkim=pass dkdomain=vivo.com dmarc=pass fromdomain=vivo.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=vivo.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 4B10680B231B; Sun, 19 Nov 2023 18:36:35 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231740AbjKTCg3 (ORCPT + 28 others); Sun, 19 Nov 2023 21:36:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230339AbjKTCg2 (ORCPT ); Sun, 19 Nov 2023 21:36:28 -0500 Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01on2128.outbound.protection.outlook.com [40.107.117.128]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9264136 for ; Sun, 19 Nov 2023 18:36:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XDvwIN9CiU6JTmeGqfgbE4EYd/OMZbbfS809mvgCfuPc8cQnkVrbFSGIn1lnhDsMFBjb46GWLGjRLDtlLISFEWamRgB+Ch68zh/8LMqQbHyy7ac4p4IlD8yqJ9IoOaPKmCoSDC8jSakMD9SVTJ2gnLkGiss6rQiFklw++37lxVUM8CZf9/XiuGlc7qN4cyCSHXQqzWFLhUOyJqHQBDw51PGjSyXPXNUTNtkhjv3sP7AKh1qS/u03bRyGnZuEbvetpWeQ1jZ6pwYc5JHv8Fw1FnhP2EzOOUEyqSRo7fdR1dhppCdFLPaJtVsnqnocXrSx36Q8jBXJ4STGy+8gDQNGkg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xBeZTAyGspQwRPIMNiuVt1kp+s/UGwHOXaMl87chj/U=; b=k+DTft1EJlAsqLfUDP+aTIXU/3AAD2n/oEqVQ54dbd6T7TWFfkkTPpjKUH0d7eLQctdxay7CwT5wcfY3ura9Jez/VBZFHhpWKA+LGaC1KKtTNslYWiWM77vmPrVf+431fTeL6JlFsJwB0VTKuMpCL8J+XAYrJvD5tfxTd+s4ozybhVEZnoxx42pm656i4tPsG5jhR23VVyGCTSdQ+LCMrt5fV08Sp3LHEoFMy/Vql7j7XF7TE9BQiyoAtAXs1QyLDVJVZvMjgdE5S+BZWRt9SsK4G/bJALUutGRiQwjNcaoVvdG0ohYjGZXCTTcnGo/JVBOymRgObNPuVJb3G7TJuw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vivo.com; dmarc=pass action=none header.from=vivo.com; dkim=pass header.d=vivo.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vivo.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xBeZTAyGspQwRPIMNiuVt1kp+s/UGwHOXaMl87chj/U=; b=QyiZTQymHzekfNYfZxg3qbrywd2gTMkik/NrhLW8AnTi2FcLjeSsGtsXlB8/FTbxvcn0hyAWD0bJJ7qEGXJKZMd7nYBtVxtM1TXet8bhPR334iu+UBmT3PgsUWq1IprI8vJJ7uB2nztXOC4mryMXGj/DIxUcyCZIjpstAFuiNF9NeKQTBG8bdXawq6xd0Nkg2CdfwobN645ksfDa2ajVn9KbFlOVP/MntJrx1aiYMqdwhPa+psjLtQiKDHP7RAZnpueFGQ7Bo4MwPg3Dg7osNMeOLSqiTZGjPd3HBBzTGaylK2N0i7rOOSvUihIb+12eDDYDh4L7zg0Tk09pQ2H6fQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=vivo.com; Received: from JH0PR06MB6849.apcprd06.prod.outlook.com (2603:1096:990:47::12) by JH0PR06MB6679.apcprd06.prod.outlook.com (2603:1096:990:3d::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7002.26; Mon, 20 Nov 2023 02:36:15 +0000 Received: from JH0PR06MB6849.apcprd06.prod.outlook.com ([fe80::bdde:31ee:f13c:79fb]) by JH0PR06MB6849.apcprd06.prod.outlook.com ([fe80::bdde:31ee:f13c:79fb%4]) with mapi id 15.20.7002.027; Mon, 20 Nov 2023 02:36:15 +0000 From: Zhiguo Jiang To: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Johannes Weiner , Matthew Wilcox , opensource.kernel@vivo.com, Zhiguo Jiang Subject: [PATCH] mm: ALLOC_HIGHATOMIC flag allocation issue Date: Mon, 20 Nov 2023 10:35:36 +0800 Message-ID: <20231120023537.1785-1-justinjiang@vivo.com> X-Mailer: git-send-email 2.41.0.windows.3 X-ClientProxiedBy: SG2PR06CA0221.apcprd06.prod.outlook.com (2603:1096:4:68::29) To JH0PR06MB6849.apcprd06.prod.outlook.com (2603:1096:990:47::12) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: JH0PR06MB6849:EE_|JH0PR06MB6679:EE_ X-MS-Office365-Filtering-Correlation-Id: 0d6e01ab-624e-47a6-f838-08dbe97177b9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: DlfYmKiPQbZ/mGFN6ow7HrSerKg1ON2pOsStCNaJ52VcNs6MpcxHiuWwOGc7N5YKKcrT5wZ4lqkH9NLDt6KJnO2s9tVQfFClqItLTWWGHiVB7aEdEetF9Y7xm9nX/AohvuHRg1NSBhFSJaaYgIrtT7pFhGqXX3+kRuZFygZPnZ+5o2WDM0yFrClF5st4kcTFgXA7JO7KQJdOhqJZKFTUhy4TU1tEKGPUlifKKTu+w0IBDgF7hOKSdbRP/klTAA7PcLWJ/Kkhk+D1bZKOvupmENLBp9LYcRRq402AuWvJuU3arAZp2rSosxhX2ktAm6Sr0JxvxhE1W7WQjAF2yOaB29E69tZ642dmsHn2XIQ3Amijr9T0OUOWnymwNv2PqqvHIJRWLfs4W6yGC4b0spBSDZcekfx80HUaUZAgPC3UZKuJMI1DkcDBxk47BG8vRMLWO7083r0BphPEjS5wInsx7cwNi7Kt3vUFiMoBQQDeus5hA/JB30/FYsGy2TuGI/VEMyI5mfDGUzL+mmrKcxx6Y5nZKFa0t22wV3cCJS18uxBkudt10Vq0ZbFNKDzAO7aFFFVMr+Xcv1oh+xrqbuhJC97bJoqhiTp6oS2cWRtqgy2Rwkz3mX1oM7vmah6F/85z X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:JH0PR06MB6849.apcprd06.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(136003)(366004)(346002)(376002)(39850400004)(230922051799003)(451199024)(186009)(64100799003)(1800799012)(1076003)(107886003)(2616005)(52116002)(6506007)(41300700001)(6512007)(38350700005)(83380400001)(36756003)(26005)(4326008)(8676002)(8936002)(5660300002)(478600001)(86362001)(6486002)(6666004)(2906002)(38100700002)(66476007)(66556008)(66946007)(316002)(54906003);DIR:OUT;SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: vXvb4Zrne1IJQOK9cU7nHKI+I4nxRjNSEk/RU+7JyZn3TfVRJHhoXAe//TwDQUHtuDF1UsRgMek4RDxpdM9lOFBPSIS3Co6iGKpEVYDTyuKqeQQjFUClC31KP/yw+BIOFwdF9VLp9gpfgdS21Qth40GEArW7+3ppo7kAOPNLjIoZSDil7PacuuKOA4X+D6svX2EEsAgdHwqQrvVrQgstuqafzDwHF0DOzJyTtjvSefzQxPdhr562tH+LoO9My5vHdsphv7FiMAbr/DlXJROkJoKzjSoHeew5mT3s4FdUAN1usXh4aqjvE6X9wKD4jEzzTFPCfyUHFQZIqF7w5wi+4cvAEF4Rqp+Xt8L2nLHYVQvtX2U1LE5x/3BZCzPJhEbtqUsQQFq4GYgzOjpc88EIV+K+xfNniR+dR9DzoKnEubDUrZvn6gJAF3ufxlypvKOJa3Wr2epNddPREVWuXANRraVAT9c4iHlgJ1rEzHo4FYDC0U4Cn8qVnyrC5TCTIUPuldnn5nXHzyX0kJRQCLvb+9pKTGNj1F/NJr3D12Eun+ZageAbwYfWtQBqJk1hwOTaV/N6VXIIctt1babL/PpChsyuhaf4mwQlGu7h5TLg0R6v57lenbFX23VBGRdS5LreeMANkL5hbAT3W8UW3zKWBlZHWfkfF7SqEBEyUDLo1cQHc+nuEGV0cuvqVZxFRLCEphnTRxHmtefkHlWIdy2flpFrU/Y5Q+O2k5y4myewNmypcv+lcR0xMbQmyqELqsmTgIqMC3ssnkOE7VkeLm2k37zY9u3S1VtkcmkC9J2JW9vvQusbkGMHTWXDZqgAuY+/Su3gvPE9/SJn4BOo3+lxPYJuHoNECewlJCvafnSZDonm+z+A2EDXO4dzpHXNUcZdZkL3SqBhEadN5MEsZ1MIyUN0cJeSPefEuNEoh06rEZEvhtshHzdF7H40JF06uPoIpVymp3Izwmzaty+kRh2SaZW5t/0DeqsUcpx1P/jETzvhUlK7FWU2zfpMvT8n+iU04T0RbH9/Olp99pOiFHeNQEL6xau7/OVZyIlNOfboDcI8pbXYE5S11LFxU45jUBAtlX44/vTnPgkBZQ69maENLZwNQvquhpuZ+okdMY0MZsndg3g0B/eLBw9FFrEzHmauyEbjBOEHwv2AMxo2YPRrjaUU5g1OKnMH1CIR9uiuRIH9x61KuoS0iBcpmHs0heoL7bXVNGxkautvv/88K4rr3lyW7DVxw+pPNiZ1+PRQPWDqmAaO5AhBjHqv8wAZVq6O4gntRnr4F/5z9UmWtSKUOTTynwQuY/KXY45sWuJvtzL9qQ5mGrr4LaarwlB9MFAb12JEAIAle1fMLKOnOsiqUKQLVgqOeafw6wG8DCdm7N5LFXZJawBCQYeq81YypDBfZSFZ7oC0JZ3WXdGsm5NFQCvDsaAFvBcuEq4M3JjT7kI1NLf4qHU5sKbjCH+tbyOt/GVqb1h4Xfxvc/FxEgja0IKKQr3Umal729eD08rgVJv/Ymbj9tJ+Dr+YAtLIX68aLLoEMEP7t6HTVVjua+Yko+foO5PWZaYp37ig92YeUKZ7u2DS+V62aKxoeSnqWHJ5 X-OriginatorOrg: vivo.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0d6e01ab-624e-47a6-f838-08dbe97177b9 X-MS-Exchange-CrossTenant-AuthSource: JH0PR06MB6849.apcprd06.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Nov 2023 02:36:15.3714 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 923e42dc-48d5-4cbe-b582-1a797a6412ed X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: IbiRN8ccgMeh+OqjfihootNs2MYFu1KRCaeFeBShT1QP4LalJd5aGilekP2Mbq6AcbmqK45AfAtV9FaRxRzxRg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: JH0PR06MB6679 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_BLOCKED, RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Sun, 19 Nov 2023 18:36:35 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783048747948745902 X-GMAIL-MSGID: 1783048747948745902 In case that alloc_flags contains ALLOC_HIGHATOMIC and alloc order is order1/2/3/10 in rmqueue(), if pages are alloced successfully from pcplist, a free pageblock will be also moved from the alloced migratetype freelist to MIGRATE_HIGHATOMIC freelist, rather than alloc from MIGRATE_HIGHATOMIC freelist firstly, so this will result in an increasing number of pages on the MIGRATE_HIGHATOMIC freelist, pages in other migratetype freelist are reduced and more likely to allocation failure. Currently the sequence of ALLOC_HIGHATOMIC allocation is: pcplist --> rmqueue_bulk() --> rmqueue_buddy() MIGRATE_HIGHATOMIC --> rmqueue_buddy() allocation migratetype. Due to the fact that requesting pages from the pcplist is faster than buddy, the sequence of modifying the ALLOC_HIGHATOMIC allocation is: pcplist --> rmqueue_buddy() MIGRATE_HIGHATOMIC --> rmqueue_buddy() allocation migratetype. This patch can solve the failure problem of allocating other types of pages due to excessive MIGRATE_HIGHATOMIC freelist reservations. In comparative testing, cat /proc/pagetypeinfo and the HighAtomic freelist size is: Test without this patch: Node 0, zone Normal, type HighAtomic 2369 771 138 15 0 0 0 0 0 0 0 Test with this patch: Node 0, zone Normal, type HighAtomic 206 82 4 2 1 0 0 0 0 0 0 Signed-off-by: Zhiguo Jiang --- mm/page_alloc.c | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) mode change 100644 => 100755 mm/page_alloc.c diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7a8dac0c1c74..49890d00cc3c --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2850,11 +2850,20 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order, int batch = nr_pcp_alloc(pcp, zone, order); int alloced; + /* + * If pcplist is empty and alloc_flags is with ALLOC_HIGHATOMIC, + * it should alloc from buddy highatomic migrate freelist firstly + * to ensure quick and successful allocation. + */ + if (alloc_flags & ALLOC_HIGHATOMIC) + goto out; + alloced = rmqueue_bulk(zone, order, batch, list, migratetype, alloc_flags); pcp->count += alloced << order; +out: if (unlikely(list_empty(list))) return NULL; } @@ -2918,7 +2927,7 @@ static inline struct page *rmqueue(struct zone *preferred_zone, struct zone *zone, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags, - int migratetype) + int migratetype, bool *highatomc_allocation) { struct page *page; @@ -2938,6 +2947,23 @@ struct page *rmqueue(struct zone *preferred_zone, page = rmqueue_buddy(preferred_zone, zone, order, alloc_flags, migratetype); + /* + * The high-order atomic allocation pageblock reserved conditions: + * + * If the high-order atomic allocation page is alloced from pcplist, + * the highatomic pageblock does not need to be reserved, which can + * void to migrate an increasing number of pages into buddy + * MIGRATE_HIGHATOMIC freelist and lead to an increasing risk of + * allocation failure on other buddy migrate freelists. + * + * If the high-order atomic allocation page is alloced from buddy + * highatomic migrate freelist, regardless of whether the allocation + * is successful or not, the highatomic pageblock can try to be + * reserved. + */ + if (unlikely(alloc_flags & ALLOC_HIGHATOMIC)) + *highatomc_allocation = true; + out: /* Separate test+clear to avoid unnecessary atomics */ if ((alloc_flags & ALLOC_KSWAPD) && @@ -3208,6 +3234,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, struct pglist_data *last_pgdat = NULL; bool last_pgdat_dirty_ok = false; bool no_fallback; + bool highatomc_allocation = false; retry: /* @@ -3339,7 +3366,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, try_this_zone: page = rmqueue(ac->preferred_zoneref->zone, zone, order, - gfp_mask, alloc_flags, ac->migratetype); + gfp_mask, alloc_flags, ac->migratetype, &highatomc_allocation); if (page) { prep_new_page(page, order, gfp_mask, alloc_flags); @@ -3347,7 +3374,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, * If this is a high-order atomic allocation then check * if the pageblock should be reserved for the future */ - if (unlikely(alloc_flags & ALLOC_HIGHATOMIC)) + if (unlikely(highatomc_allocation)) reserve_highatomic_pageblock(page, zone); return page;