From patchwork Fri Jan 13 11:12:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 43189 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp213220wrt; Fri, 13 Jan 2023 03:23:00 -0800 (PST) X-Google-Smtp-Source: AMrXdXvYGSZ33qBGJjJvREO+84LzDzQJdUsi2tSLhoiDRpCKfB8/3OZHFrMnqMUK8dNGNc119L8Z X-Received: by 2002:a05:6402:538c:b0:483:21d8:87af with SMTP id ew12-20020a056402538c00b0048321d887afmr60213005edb.24.1673608980743; Fri, 13 Jan 2023 03:23:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673608980; cv=none; d=google.com; s=arc-20160816; b=sggNo+rf106kKGfIBVUQ/Qe6VeYoWkDEMRZEcWOe2hoB9zmVpHo3QrsgZzebFTlH2C XZCDp0LlaM8QGdLSZfewagXLUR9kPNhhfJmZ8s6fE7+A6wkBECx335rTS0O+GWtYH3IW assdWI4rdTWaNz04gxQ73TAWBbdxeeyZCihZDzHGMKJSJtSxNxlvulqQ9Tqt470wa3NF Mm8+GDbBR1ZvPbkEPh0StsRmI3/qwU19OChE/g/1ts+Nmnlf6cxRX7FpvpAZJ2DtPQwS vSyGY6YCi2InA19rUdiDkV4KD+XKxDtLbD6DT+ZHaxBsEitJcfSQ9g99gkp6Y35hZwVV hHkQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=LnNHzsxLBAqkc6G9HCgyzTdpj5TzheSV3RrT+CCgXUE=; b=IFv6JVcdNxDMVhxA8QMvZVCJg71xbv6AZbBkebZEDQIdXl2/wrCo2rju3p2x4AmCVH i9hXCsug0pgZXgj2mvqOIfM57Ao8Bmpa4CZrp9nJtDwSaZk70urNbZP1ZKxOk7IIHo4f c/fus9hXXQcjx5jebGhGZCIWZ0Hju9Xz1O6nBYIQtnE0jYEuNxpbmT9WHVZJ1V5qXjQf CmWWq8CznGzto6Wr+YrHfmfJrmKcTf6KPoLYa8ERapegSNT6+CufqtaJVBcKr4gzykPY dss/0MYr9ewR89KD+UqGjlkxoE5PjZJ8e/brTjEzv9M8nYi5B3oevAxNGqqCg3YwCsjW uD8Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xh10-20020a170906da8a00b0084d37abf43csi18691291ejb.919.2023.01.13.03.22.34; Fri, 13 Jan 2023 03:23:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232755AbjAMLSX (ORCPT + 99 others); Fri, 13 Jan 2023 06:18:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36908 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241343AbjAMLRU (ORCPT ); Fri, 13 Jan 2023 06:17:20 -0500 Received: from outbound-smtp29.blacknight.com (outbound-smtp29.blacknight.com [81.17.249.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68D42659F for ; Fri, 13 Jan 2023 03:12:39 -0800 (PST) Received: from mail.blacknight.com (pemlinmail02.blacknight.ie [81.17.254.11]) by outbound-smtp29.blacknight.com (Postfix) with ESMTPS id 6BE4C18E031 for ; Fri, 13 Jan 2023 11:12:38 +0000 (GMT) Received: (qmail 8795 invoked from network); 13 Jan 2023 11:12:38 -0000 Received: from unknown (HELO morpheus.112glenside.lan) (mgorman@techsingularity.net@[84.203.198.246]) by 81.17.254.9 with ESMTPA; 13 Jan 2023 11:12:38 -0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , NeilBrown , Thierry Reding , Matthew Wilcox , Vlastimil Babka , Linux-MM , LKML , Mel Gorman Subject: [PATCH 1/6] mm/page_alloc: Rename ALLOC_HIGH to ALLOC_MIN_RESERVE Date: Fri, 13 Jan 2023 11:12:12 +0000 Message-Id: <20230113111217.14134-2-mgorman@techsingularity.net> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230113111217.14134-1-mgorman@techsingularity.net> References: <20230113111217.14134-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754906210668353148?= X-GMAIL-MSGID: =?utf-8?q?1754906210668353148?= __GFP_HIGH aliases to ALLOC_HIGH but the name does not really hint what it means. As ALLOC_HIGH is internal to the allocator, rename it to ALLOC_MIN_RESERVE to document that the min reserves can be depleted. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka Acked-by: Michal Hocko --- mm/internal.h | 4 +++- mm/page_alloc.c | 8 ++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index bcf75a8b032d..403e4386626d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -736,7 +736,9 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #endif #define ALLOC_HARDER 0x10 /* try to alloc harder */ -#define ALLOC_HIGH 0x20 /* __GFP_HIGH set */ +#define ALLOC_MIN_RESERVE 0x20 /* __GFP_HIGH set. Allow access to 50% + * of the min watermark. + */ #define ALLOC_CPUSET 0x40 /* check for correct cpuset */ #define ALLOC_CMA 0x80 /* allow allocations from CMA areas */ #ifdef CONFIG_ZONE_DMA32 diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0745aedebb37..244c1e675dc8 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3976,7 +3976,7 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, /* free_pages may go negative - that's OK */ free_pages -= __zone_watermark_unusable_free(z, order, alloc_flags); - if (alloc_flags & ALLOC_HIGH) + if (alloc_flags & ALLOC_MIN_RESERVE) min -= min / 2; if (unlikely(alloc_harder)) { @@ -4818,18 +4818,18 @@ gfp_to_alloc_flags(gfp_t gfp_mask) unsigned int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET; /* - * __GFP_HIGH is assumed to be the same as ALLOC_HIGH + * __GFP_HIGH is assumed to be the same as ALLOC_MIN_RESERVE * and __GFP_KSWAPD_RECLAIM is assumed to be the same as ALLOC_KSWAPD * to save two branches. */ - BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_HIGH); + BUILD_BUG_ON(__GFP_HIGH != (__force gfp_t) ALLOC_MIN_RESERVE); BUILD_BUG_ON(__GFP_KSWAPD_RECLAIM != (__force gfp_t) ALLOC_KSWAPD); /* * The caller may dip into page reserves a bit more if the caller * cannot run direct reclaim, or if the caller has realtime scheduling * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will - * set both ALLOC_HARDER (__GFP_ATOMIC) and ALLOC_HIGH (__GFP_HIGH). + * set both ALLOC_HARDER (__GFP_ATOMIC) and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |= (__force int) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); From patchwork Fri Jan 13 11:12:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 43190 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp213414wrt; Fri, 13 Jan 2023 03:23:30 -0800 (PST) X-Google-Smtp-Source: AMrXdXt3jQiXl0qk9WSFG7NSVVi6OobG18otqsA0MFuyY7cVzgZX4CKGXHqp2mDNrZOCVzO0+J/Q X-Received: by 2002:a05:6402:5245:b0:499:bcd7:a968 with SMTP id t5-20020a056402524500b00499bcd7a968mr17387620edd.22.1673609009986; Fri, 13 Jan 2023 03:23:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673609009; cv=none; d=google.com; s=arc-20160816; b=HXsRVxxWsduDZ930qocPDg1/FBGVxjHhUYEZvazKe59NARdhTe5lh74K2RuJjIRvFi g8S/K4+sXqDabsUK3a1GcjnXVK9TrLYAx0Eaky+CF9VaypH30WVrJkSgkmiSFZdqVLyg LLQAn7dwJ+QprlovAGMJI2T0dTMegIzIGOHcZac98a2PUdNuoDch3aEbu9ZslD/WawZj JMadwBrFy8oe39AV/Ry8L9oy02R1Rep5k1O2A7C5TT3la4tKSEtqgv0vWrukth49H6tc gm+7Y4QpW2kiEqa5ehzIGbRVhn7BmJDIESc2G8L9MLaeCV3ycFuYO2ft0sgcBUJhayyX xdwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=FiHLbgT+/V2AjHkJD/pCH3t4rdxztdy//5LcZVO0gHM=; b=h9dMRoBPmC78spzedI9Ae+UaBJbfX6m5dg8ry2hWXq4VjYFkJPKBeOhMuffVI+KQUc idIZhJyV6wdNpIdlH4aJvLzN97/BuoXn5TLMYBNCmDDq+QwuLQ1SZrT/IYosr0o4vF3l HajJgc4th8hY7jkTmht3HYizI34T6SXI8s82kZvLfheT+mpYqSBM13J9DsHQbNR9Mvpy IRn4j6ZbAuWVV3+Km1GzVDXko+P2IUW/QyfRFed4vQlwjUYkiQrtQbh3xoAVKR+Ahjcf CSs9v22bsSbh8hpwrg4DF3fBCia15cfSs21f4Bplzm9A1V9NB2o5V/LMrgyeGmlhEmC8 hMlQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p34-20020a056402502200b00488c68b1998si19843872eda.103.2023.01.13.03.23.02; Fri, 13 Jan 2023 03:23:29 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239679AbjAMLTA (ORCPT + 99 others); Fri, 13 Jan 2023 06:19:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241326AbjAMLR5 (ORCPT ); Fri, 13 Jan 2023 06:17:57 -0500 Received: from outbound-smtp26.blacknight.com (outbound-smtp26.blacknight.com [81.17.249.194]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 063CC7D9D5 for ; Fri, 13 Jan 2023 03:12:49 -0800 (PST) Received: from mail.blacknight.com (pemlinmail02.blacknight.ie [81.17.254.11]) by outbound-smtp26.blacknight.com (Postfix) with ESMTPS id 8A53ECAE82 for ; Fri, 13 Jan 2023 11:12:48 +0000 (GMT) Received: (qmail 9303 invoked from network); 13 Jan 2023 11:12:48 -0000 Received: from unknown (HELO morpheus.112glenside.lan) (mgorman@techsingularity.net@[84.203.198.246]) by 81.17.254.9 with ESMTPA; 13 Jan 2023 11:12:48 -0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , NeilBrown , Thierry Reding , Matthew Wilcox , Vlastimil Babka , Linux-MM , LKML , Mel Gorman Subject: [PATCH 2/6] mm/page_alloc: Treat RT tasks similar to __GFP_HIGH Date: Fri, 13 Jan 2023 11:12:13 +0000 Message-Id: <20230113111217.14134-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230113111217.14134-1-mgorman@techsingularity.net> References: <20230113111217.14134-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754906241271732139?= X-GMAIL-MSGID: =?utf-8?q?1754906241271732139?= RT tasks are allowed to dip below the min reserve but ALLOC_HARDER is typically combined with ALLOC_MIN_RESERVE so RT tasks are a little unusual. While there is some justification for allowing RT tasks access to memory reserves, there is a strong chance that a RT task that is also under memory pressure is at risk of missing deadlines anyway. Relax how much reserves an RT task can access by treating it the same as __GFP_HIGH allocations. Note that in a future kernel release that the RT special casing will be removed. Hard realtime tasks should be locking down resources in advance and ensuring enough memory is available. Even a soft-realtime task like audio or video live decoding which cannot jitter should be allocating both memory and any disk space required up-front before the recording starts instead of relying on reserves. At best, reserve access will only delay the problem by a very short interval. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka Acked-by: Michal Hocko --- mm/page_alloc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 244c1e675dc8..0040b4e00913 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4847,7 +4847,7 @@ gfp_to_alloc_flags(gfp_t gfp_mask) */ alloc_flags &= ~ALLOC_CPUSET; } else if (unlikely(rt_task(current)) && in_task()) - alloc_flags |= ALLOC_HARDER; + alloc_flags |= ALLOC_MIN_RESERVE; alloc_flags = gfp_to_alloc_flags_cma(gfp_mask, alloc_flags); From patchwork Fri Jan 13 11:12:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 43192 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp215661wrt; Fri, 13 Jan 2023 03:30:05 -0800 (PST) X-Google-Smtp-Source: AMrXdXtJgQIVRkH0lRv4V1Y2ocPyvDU6Di2Aer3obrayavUOxuqyOLRMqEgUCUEQOPLL9hq4nSFq X-Received: by 2002:a05:6a20:d013:b0:af:ae14:9ecb with SMTP id hu19-20020a056a20d01300b000afae149ecbmr78148056pzb.17.1673609405077; Fri, 13 Jan 2023 03:30:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673609405; cv=none; d=google.com; s=arc-20160816; b=Hbfnt2WtbQOXNbDVfprMIWWRPkS4MKs7ABjYsDvI4LQhjciKu+uvinZAc9ljh50nXU jclaaS0FYJfQ2cAxrINfUKbsvB/ALObd2LjxPiQcuYuZVcaDZllbiqio0oMH5UrpAMXc GRV6LYZoPK38srMcpWV8HWk8XHieKOHSEDOSSnnvQZaFOuK897DFIGJjpqp4S74MaCHp 3Zz4K8YyYMA3wqGlNnWWmcUy69PrPRhc44tI0CPvDNCHJU/Kb4I43NtLFcNQWTiPYzsY eIGqfRNQXTX0HS/I8i1lSKb3+SyyDhXCgw6k7Bw6FHDVypfqSZLASw7zHqQXuehQpF0/ mPVw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=4dtuJMuLjAPooRkMDKhq25jF6dppLdwvssJeQCbIWts=; b=H1sl749CsoP2fytnkV0TxO9K4p0WSmt/CIZmW2bebHoNyUN+77DXGjUMXdVSfVpVL9 TzKpZTPyT5bHGY6M8RvFdUvA71OHxnxxeGi2FkbHSI+e3mQ7C9zJskokv1cMY3I5hh4Y QOh31obQwu82eC5E1B1J42ZTKiCgVMDdm1HcoaP5D2WpiTDfHqm9GhhmPu0J+vxA+0U1 RyK6xv4sR600GAReUoU7fNCC5iqC8my2Ay5BBmE/W3jZQeQfDo7nBRp0LficUx3TPpvu HfEI6kfvOXe63T43OB8C1U3xKfWhI1vUh4ew83I9/B2vbeeRy/Vz2Vuz/ypQXBGVv9Gd 8ZXQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m14-20020a63fd4e000000b0046f583c8c4asi20525529pgj.315.2023.01.13.03.29.50; Fri, 13 Jan 2023 03:30:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240753AbjAMLTK (ORCPT + 99 others); Fri, 13 Jan 2023 06:19:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241245AbjAMLSJ (ORCPT ); Fri, 13 Jan 2023 06:18:09 -0500 Received: from outbound-smtp06.blacknight.com (outbound-smtp06.blacknight.com [81.17.249.39]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1021213D00 for ; Fri, 13 Jan 2023 03:12:59 -0800 (PST) Received: from mail.blacknight.com (pemlinmail02.blacknight.ie [81.17.254.11]) by outbound-smtp06.blacknight.com (Postfix) with ESMTPS id 99AA3C2BBB for ; Fri, 13 Jan 2023 11:12:58 +0000 (GMT) Received: (qmail 9751 invoked from network); 13 Jan 2023 11:12:58 -0000 Received: from unknown (HELO morpheus.112glenside.lan) (mgorman@techsingularity.net@[84.203.198.246]) by 81.17.254.9 with ESMTPA; 13 Jan 2023 11:12:58 -0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , NeilBrown , Thierry Reding , Matthew Wilcox , Vlastimil Babka , Linux-MM , LKML , Mel Gorman Subject: [PATCH 3/6] mm/page_alloc: Explicitly record high-order atomic allocations in alloc_flags Date: Fri, 13 Jan 2023 11:12:14 +0000 Message-Id: <20230113111217.14134-4-mgorman@techsingularity.net> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230113111217.14134-1-mgorman@techsingularity.net> References: <20230113111217.14134-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754906655591122023?= X-GMAIL-MSGID: =?utf-8?q?1754906655591122023?= A high-order ALLOC_HARDER allocation is assumed to be atomic. While that is accurate, it changes later in the series. In preparation, explicitly record high-order atomic allocations in gfp_to_alloc_flags(). Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka Acked-by: Michal Hocko --- mm/internal.h | 1 + mm/page_alloc.c | 29 +++++++++++++++++++++++------ 2 files changed, 24 insertions(+), 6 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 403e4386626d..178484d9fd94 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -746,6 +746,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #else #define ALLOC_NOFRAGMENT 0x0 #endif +#define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ enum ttu_flags; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0040b4e00913..0ef4f3236a5a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3706,10 +3706,20 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, * reserved for high-order atomic allocation, so order-0 * request should skip it. */ - if (order > 0 && alloc_flags & ALLOC_HARDER) + if (alloc_flags & ALLOC_HIGHATOMIC) page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); if (!page) { page = __rmqueue(zone, order, migratetype, alloc_flags); + + /* + * If the allocation fails, allow OOM handling access + * to HIGHATOMIC reserves as failing now is worse than + * failing a high-order atomic allocation in the + * future. + */ + if (!page && (alloc_flags & ALLOC_OOM)) + page = __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC); + if (!page) { spin_unlock_irqrestore(&zone->lock, flags); return NULL; @@ -4023,8 +4033,10 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, return true; } #endif - if (alloc_harder && !free_area_empty(area, MIGRATE_HIGHATOMIC)) + if ((alloc_flags & (ALLOC_HIGHATOMIC|ALLOC_OOM)) && + !free_area_empty(area, MIGRATE_HIGHATOMIC)) { return true; + } } return false; } @@ -4286,7 +4298,7 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, * If this is a high-order atomic allocation then check * if the pageblock should be reserved for the future */ - if (unlikely(order && (alloc_flags & ALLOC_HARDER))) + if (unlikely(alloc_flags & ALLOC_HIGHATOMIC)) reserve_highatomic_pageblock(page, zone, order); return page; @@ -4813,7 +4825,7 @@ static void wake_all_kswapds(unsigned int order, gfp_t gfp_mask, } static inline unsigned int -gfp_to_alloc_flags(gfp_t gfp_mask) +gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) { unsigned int alloc_flags = ALLOC_WMARK_MIN | ALLOC_CPUSET; @@ -4839,8 +4851,13 @@ gfp_to_alloc_flags(gfp_t gfp_mask) * Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule. */ - if (!(gfp_mask & __GFP_NOMEMALLOC)) + if (!(gfp_mask & __GFP_NOMEMALLOC)) { alloc_flags |= ALLOC_HARDER; + + if (order > 0) + alloc_flags |= ALLOC_HIGHATOMIC; + } + /* * Ignore cpuset mems for GFP_ATOMIC rather than fail, see the * comment for __cpuset_node_allowed(). @@ -5048,7 +5065,7 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, * kswapd needs to be woken up, and to avoid the cost of setting up * alloc_flags precisely. So we do that now. */ - alloc_flags = gfp_to_alloc_flags(gfp_mask); + alloc_flags = gfp_to_alloc_flags(gfp_mask, order); /* * We need to recalculate the starting point for the zonelist iterator From patchwork Fri Jan 13 11:12:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 43195 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp217267wrt; Fri, 13 Jan 2023 03:33:57 -0800 (PST) X-Google-Smtp-Source: AMrXdXs+jFpJSCK5f+Bx5K0SQQXytrFqyvlZ3Z/JlQ1mmdVJaJLPoz4vd9bulROavfD5qSh1YDY+ X-Received: by 2002:a17:906:6dd4:b0:836:e6f7:8138 with SMTP id j20-20020a1709066dd400b00836e6f78138mr82167605ejt.13.1673609637502; Fri, 13 Jan 2023 03:33:57 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673609637; cv=none; d=google.com; s=arc-20160816; b=TsNkCdTRQ7s+9EoX02P/5xu7GPX2wWnFv9ZLac7j1ay0nO7TTKP4EkHC51Z/VXcoun RUOPGxIFwaqE65+J1ve7z89jw9jT1xANCgyhrO0Oxg7SK6vNG0mIdItlMigT+c6x6IGC 8mpDqGlOf0aNFiHAcQ3muxjGbSh/z8ahXnKNQjHkaHbc8ynQaXy4kf4oXnFiEFrRdMGG L3dVHgl4u1O29eNo1w/ehVp3YMmDtFUjzI5jPjt3bIiJ4Lh9L8JVEJLVB9UtkI5Cn7fl 5grlP/0ul9hat2+1ZhUqde0T8OVsDKawUgcKGIQtuUVos7Fi2PeCTFyZjt5gqBprmo8j srdg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=vABlLWU4vOV7PbsFCMQ7RZft5Uue6QSR/R93b7+7cFQ=; b=AlWUfhVVmNrO96ppvSzYOwGYytX/bmXjYl88d/fLoIdNtDHAccxS3ep2KX2VIguUiq MgWNEHdDypEWZai9LFlpFbW96vRC+mSUARdr+DZs/JBlRpDcFX251yhCGPKaFnxRTt4J ugnP7+gKLRpWg4rknMC2TYTBQuLHJfv8KmdxQzooW7yrlO/qPlhDE4ea9V34ZSDObPaM a/fJBfDmTowlavO8TCbFeee/4gBiR3NnqQror5n7XLi3rjZiZ6jlr2oynjlvUL9dMs+u ZbQ6G+HuC+xliJR7qAMYJVWKLmXgeD4CPh5s0nDdmafkenhnTt/lmdPi+Ws6Ddjqxdmn WRiw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qk32-20020a1709077fa000b007ada2ec1a28si22111291ejc.165.2023.01.13.03.33.33; Fri, 13 Jan 2023 03:33:57 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241061AbjAMLTX (ORCPT + 99 others); Fri, 13 Jan 2023 06:19:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241489AbjAMLSW (ORCPT ); Fri, 13 Jan 2023 06:18:22 -0500 Received: from outbound-smtp27.blacknight.com (outbound-smtp27.blacknight.com [81.17.249.195]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71AC9175BF for ; Fri, 13 Jan 2023 03:13:10 -0800 (PST) Received: from mail.blacknight.com (pemlinmail02.blacknight.ie [81.17.254.11]) by outbound-smtp27.blacknight.com (Postfix) with ESMTPS id BEE6CCAE83 for ; Fri, 13 Jan 2023 11:13:08 +0000 (GMT) Received: (qmail 10332 invoked from network); 13 Jan 2023 11:13:08 -0000 Received: from unknown (HELO morpheus.112glenside.lan) (mgorman@techsingularity.net@[84.203.198.246]) by 81.17.254.9 with ESMTPA; 13 Jan 2023 11:13:08 -0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , NeilBrown , Thierry Reding , Matthew Wilcox , Vlastimil Babka , Linux-MM , LKML , Mel Gorman Subject: [PATCH 4/6] mm/page_alloc: Explicitly define what alloc flags deplete min reserves Date: Fri, 13 Jan 2023 11:12:15 +0000 Message-Id: <20230113111217.14134-5-mgorman@techsingularity.net> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230113111217.14134-1-mgorman@techsingularity.net> References: <20230113111217.14134-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754906899425307951?= X-GMAIL-MSGID: =?utf-8?q?1754906899425307951?= As there are more ALLOC_ flags that affect reserves, define what flags affect reserves and clarify the effect of each flag. Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka Acked-by: Michal Hocko --- mm/internal.h | 3 +++ mm/page_alloc.c | 34 ++++++++++++++++++++++------------ 2 files changed, 25 insertions(+), 12 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 178484d9fd94..8706d46863df 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -749,6 +749,9 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_HIGHATOMIC 0x200 /* Allows access to MIGRATE_HIGHATOMIC */ #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ +/* Flags that allow allocations below the min watermark. */ +#define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) + enum ttu_flags; struct tlbflush_unmap_batch; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0ef4f3236a5a..6f41b84a97ac 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3949,15 +3949,14 @@ ALLOW_ERROR_INJECTION(should_fail_alloc_page, TRUE); static inline long __zone_watermark_unusable_free(struct zone *z, unsigned int order, unsigned int alloc_flags) { - const bool alloc_harder = (alloc_flags & (ALLOC_HARDER|ALLOC_OOM)); long unusable_free = (1 << order) - 1; /* - * If the caller does not have rights to ALLOC_HARDER then subtract - * the high-atomic reserves. This will over-estimate the size of the - * atomic reserve but it avoids a search. + * If the caller does not have rights to reserves below the min + * watermark then subtract the high-atomic reserves. This will + * over-estimate the size of the atomic reserve but it avoids a search. */ - if (likely(!alloc_harder)) + if (likely(!(alloc_flags & ALLOC_RESERVES))) unusable_free += z->nr_reserved_highatomic; #ifdef CONFIG_CMA @@ -3981,25 +3980,36 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, { long min = mark; int o; - const bool alloc_harder = (alloc_flags & (ALLOC_HARDER|ALLOC_OOM)); /* free_pages may go negative - that's OK */ free_pages -= __zone_watermark_unusable_free(z, order, alloc_flags); - if (alloc_flags & ALLOC_MIN_RESERVE) - min -= min / 2; + if (unlikely(alloc_flags & ALLOC_RESERVES)) { + /* + * __GFP_HIGH allows access to 50% of the min reserve as well + * as OOM. + */ + if (alloc_flags & ALLOC_MIN_RESERVE) + min -= min / 2; - if (unlikely(alloc_harder)) { /* - * OOM victims can try even harder than normal ALLOC_HARDER + * Non-blocking allocations can access some of the reserve + * with more access if also __GFP_HIGH. The reasoning is that + * a non-blocking caller may incur a more severe penalty + * if it cannot get memory quickly, particularly if it's + * also __GFP_HIGH. + */ + if (alloc_flags & ALLOC_HARDER) + min -= min / 4; + + /* + * OOM victims can try even harder than the normal reserve * users on the grounds that it's definitely going to be in * the exit path shortly and free memory. Any allocation it * makes during the free path will be small and short-lived. */ if (alloc_flags & ALLOC_OOM) min -= min / 2; - else - min -= min / 4; } /* From patchwork Fri Jan 13 11:12:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 43197 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp217456wrt; Fri, 13 Jan 2023 03:34:24 -0800 (PST) X-Google-Smtp-Source: AMrXdXv3Fslxd1D/r6LPkysHOHQC/h8xYyJXSq/Z62E5SRByrqbqTGBZ53mh3vXnRveEIyg9u3Zx X-Received: by 2002:a17:906:6d14:b0:7c1:765:9cfc with SMTP id m20-20020a1709066d1400b007c107659cfcmr2738085ejr.34.1673609664696; Fri, 13 Jan 2023 03:34:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673609664; cv=none; d=google.com; s=arc-20160816; b=J3ci8pF847ZnMDWX3nDEu9xREzLY1k8MnPyOHrtIOw5kJkr4kzg5DkmVWKiz04yX6Y Y+KhwqqEAQ4zcHTgopQ3Ggy/tvAPRsnT2tEaEjY511anzrjO4sVxjxppWA+4YN86CUQW x6um7HPOd0cibBeWXa0fPx6VwiOqXR+8L1ovE4y8t8OebAfSh4m1rkHY61yeDJ8sUuEo MprvY9BBT4KqEvtfm12akVQaRkFPbsWPnTgAodgqiL1Kj9CcfAgRZ/oyteehhIEA770H L8K9n5snFYNuseiuE7qxZj6UDapmdjtSPWdPCLcKfcLDk7xupUEZ0d8XdEogfPZxaYob U9BQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=nmoznsuK0w9j+EF/lpoydTt9vDWWrFHmoqctmIZ8VnY=; b=jmsi8YIjJX0510tlIhyPGmxxfmuNKIjy7MElAojLGmPpWcmB1aWXiM+7S6ZykRMzWp UrXXXr/a4eSAsLXaGhwRmLoMo/ZOr4UCglU49mlN20Bez5dbosK4cl+mqjZ1MKO89yTC ESbXv2Xhvz9WF0+7RBLMdTEPHs5YTz7PVo0FSk8wn/bwia2ryC/Ln0W8wMBE8hBq7lE+ uvmI2YglABp31WjwW8SZy5W/DBNZBhYs+QNHOz43BPzTWHGcAVZLToqNUHZf/fLMStZg fNMk/Kf2soD+rOxEdhMjwv0coF2okDIYgUJDCj5l/vylyMKUiGNn0/f3csGq5n1Ci1g0 F5zg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c10-20020aa7df0a000000b00497779c45dfsi8228562edy.538.2023.01.13.03.33.59; Fri, 13 Jan 2023 03:34:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241215AbjAMLTy (ORCPT + 99 others); Fri, 13 Jan 2023 06:19:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241585AbjAMLSj (ORCPT ); Fri, 13 Jan 2023 06:18:39 -0500 Received: from outbound-smtp55.blacknight.com (outbound-smtp55.blacknight.com [46.22.136.239]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B991543DA2 for ; Fri, 13 Jan 2023 03:13:20 -0800 (PST) Received: from mail.blacknight.com (pemlinmail02.blacknight.ie [81.17.254.11]) by outbound-smtp55.blacknight.com (Postfix) with ESMTPS id EB420FAEB9 for ; Fri, 13 Jan 2023 11:13:18 +0000 (GMT) Received: (qmail 10897 invoked from network); 13 Jan 2023 11:13:18 -0000 Received: from unknown (HELO morpheus.112glenside.lan) (mgorman@techsingularity.net@[84.203.198.246]) by 81.17.254.9 with ESMTPA; 13 Jan 2023 11:13:18 -0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , NeilBrown , Thierry Reding , Matthew Wilcox , Vlastimil Babka , Linux-MM , LKML , Mel Gorman Subject: [PATCH 5/6] mm/page_alloc: Explicitly define how __GFP_HIGH non-blocking allocations accesses reserves Date: Fri, 13 Jan 2023 11:12:16 +0000 Message-Id: <20230113111217.14134-6-mgorman@techsingularity.net> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230113111217.14134-1-mgorman@techsingularity.net> References: <20230113111217.14134-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754906927783233869?= X-GMAIL-MSGID: =?utf-8?q?1754906927783233869?= GFP_ATOMIC allocations get flagged ALLOC_HARDER which is a vague description. In preparation for the removal of GFP_ATOMIC redefine __GFP_ATOMIC to simply mean non-blocking and renaming ALLOC_HARDER to ALLOC_NON_BLOCK accordingly. __GFP_HIGH is required for access to reserves but non-blocking is granted more access. For example, GFP_NOWAIT is non-blocking but has no special access to reserves. A __GFP_NOFAIL blocking allocation is granted access similar to __GFP_HIGH if the only alternative is an OOM kill. Signed-off-by: Mel Gorman Acked-by: Michal Hocko Acked-by: Vlastimil Babka --- mm/internal.h | 7 +++++-- mm/page_alloc.c | 44 ++++++++++++++++++++++++-------------------- 2 files changed, 29 insertions(+), 22 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 8706d46863df..23a37588073a 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -735,7 +735,10 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_OOM ALLOC_NO_WATERMARKS #endif -#define ALLOC_HARDER 0x10 /* try to alloc harder */ +#define ALLOC_NON_BLOCK 0x10 /* Caller cannot block. Allow access + * to 25% of the min watermark or + * 62.5% if __GFP_HIGH is set. + */ #define ALLOC_MIN_RESERVE 0x20 /* __GFP_HIGH set. Allow access to 50% * of the min watermark. */ @@ -750,7 +753,7 @@ unsigned int reclaim_clean_pages_from_list(struct zone *zone, #define ALLOC_KSWAPD 0x800 /* allow waking of kswapd, __GFP_KSWAPD_RECLAIM set */ /* Flags that allow allocations below the min watermark. */ -#define ALLOC_RESERVES (ALLOC_HARDER|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) +#define ALLOC_RESERVES (ALLOC_NON_BLOCK|ALLOC_MIN_RESERVE|ALLOC_HIGHATOMIC|ALLOC_OOM) enum ttu_flags; struct tlbflush_unmap_batch; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 6f41b84a97ac..b9ae0ba0a2ab 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3989,18 +3989,19 @@ bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, * __GFP_HIGH allows access to 50% of the min reserve as well * as OOM. */ - if (alloc_flags & ALLOC_MIN_RESERVE) + if (alloc_flags & ALLOC_MIN_RESERVE) { min -= min / 2; - /* - * Non-blocking allocations can access some of the reserve - * with more access if also __GFP_HIGH. The reasoning is that - * a non-blocking caller may incur a more severe penalty - * if it cannot get memory quickly, particularly if it's - * also __GFP_HIGH. - */ - if (alloc_flags & ALLOC_HARDER) - min -= min / 4; + /* + * Non-blocking allocations (e.g. GFP_ATOMIC) can + * access more reserves than just __GFP_HIGH. Other + * non-blocking allocations requests such as GFP_NOWAIT + * or (GFP_KERNEL & ~__GFP_DIRECT_RECLAIM) do not get + * access to the min reserve. + */ + if (alloc_flags & ALLOC_NON_BLOCK) + min -= min / 4; + } /* * OOM victims can try even harder than the normal reserve @@ -4851,28 +4852,30 @@ gfp_to_alloc_flags(gfp_t gfp_mask, unsigned int order) * The caller may dip into page reserves a bit more if the caller * cannot run direct reclaim, or if the caller has realtime scheduling * policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will - * set both ALLOC_HARDER (__GFP_ATOMIC) and ALLOC_MIN_RESERVE(__GFP_HIGH). + * set both ALLOC_NON_BLOCK and ALLOC_MIN_RESERVE(__GFP_HIGH). */ alloc_flags |= (__force int) (gfp_mask & (__GFP_HIGH | __GFP_KSWAPD_RECLAIM)); - if (gfp_mask & __GFP_ATOMIC) { + if (!(gfp_mask & __GFP_DIRECT_RECLAIM)) { /* * Not worth trying to allocate harder for __GFP_NOMEMALLOC even * if it can't schedule. */ if (!(gfp_mask & __GFP_NOMEMALLOC)) { - alloc_flags |= ALLOC_HARDER; + alloc_flags |= ALLOC_NON_BLOCK; if (order > 0) alloc_flags |= ALLOC_HIGHATOMIC; } /* - * Ignore cpuset mems for GFP_ATOMIC rather than fail, see the - * comment for __cpuset_node_allowed(). + * Ignore cpuset mems for non-blocking __GFP_HIGH (probably + * GFP_ATOMIC) rather than fail, see the comment for + * __cpuset_node_allowed(). */ - alloc_flags &= ~ALLOC_CPUSET; + if (alloc_flags & ALLOC_MIN_RESERVE) + alloc_flags &= ~ALLOC_CPUSET; } else if (unlikely(rt_task(current)) && in_task()) alloc_flags |= ALLOC_MIN_RESERVE; @@ -5303,12 +5306,13 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, WARN_ON_ONCE_GFP(costly_order, gfp_mask); /* - * Help non-failing allocations by giving them access to memory - * reserves but do not use ALLOC_NO_WATERMARKS because this + * Help non-failing allocations by giving some access to memory + * reserves normally used for high priority non-blocking + * allocations but do not use ALLOC_NO_WATERMARKS because this * could deplete whole memory reserves which would just make - * the situation worse + * the situation worse. */ - page = __alloc_pages_cpuset_fallback(gfp_mask, order, ALLOC_HARDER, ac); + page = __alloc_pages_cpuset_fallback(gfp_mask, order, ALLOC_MIN_RESERVE, ac); if (page) goto got_pg; From patchwork Fri Jan 13 11:12:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 43193 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4e01:0:0:0:0:0 with SMTP id p1csp215862wrt; Fri, 13 Jan 2023 03:30:32 -0800 (PST) X-Google-Smtp-Source: AMrXdXsynkEBXHpq1a2LtkpXAnDQuGuf0nI/ZwMOGhoenwobTrkvqvlYD4KCKiw/HtVy7t94dZbS X-Received: by 2002:a17:902:ef8e:b0:193:2bed:3325 with SMTP id iz14-20020a170902ef8e00b001932bed3325mr10805407plb.15.1673609432136; Fri, 13 Jan 2023 03:30:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1673609432; cv=none; d=google.com; s=arc-20160816; b=gF2E+wzpPx9+0NO7pUHVVTiuOetMDAuq729Z6iuwbszqP/QbhHQ9USdC8G1dIXCGw9 QtTHow8JVSCspTUWqvuc3y6BIW6bDtUg/z5d/Dqt8T02blN/mT5uH8aD1/NgtvriUs23 443Fs9zwG0y5hSHBO3B7Q5Kflidk1RP9wYIJGNc6zfLN98HWGlLeJLHL+p9oQ26RZLVp ToE/cSoW9n/ijr9g52fR4M5I7617MPU6+GX81Sz9nNwISSRtgCyezJ2eU+kgJLVsgeQe 72vezazE8UmvcPmh5VKM7IeV62IFzp/ulYpSO1j55fLyBxTyRpOq+DmjTw51z2U88xNX 7BGw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=Tm+wGdOtKgrascPjaaR4v1bCmFAQxbOESbe2IeSexTk=; b=NC0Em9ikf8mNjo2TODm25WHrIKLZolFPtF5ta3s2b+xfAG4azqXGaYVcKtNTOSqM1j TAPDEfP9SBziCxBfqdnFpD2bS7e2HBDmVEueHLAYw5a+ALf9X2cigmWfQxj3HAFUZlCj xUYdHa+u8w7PtUbXhlZ4tfXYAEWvJic7ekKrcNv5qcfbu9JH1A44QltBjVFI95O6IyNC 2L74OGtxjdK0OLLFCY36N9hSw76FjO+T07ss2VUQYWwVe1aQzw+P1pFYy/pneogjHaC2 RYzoZqhaTz4GcDXQX/N47iKFqF6h2qzTtbOK9tZhi1RkqisJPjjCNyeOFRpHmEdiZN1j AvDA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l6-20020a170903120600b0017f7faef235si22621791plh.148.2023.01.13.03.30.19; Fri, 13 Jan 2023 03:30:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231154AbjAMLUb (ORCPT + 99 others); Fri, 13 Jan 2023 06:20:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232723AbjAMLS5 (ORCPT ); Fri, 13 Jan 2023 06:18:57 -0500 Received: from outbound-smtp05.blacknight.com (outbound-smtp05.blacknight.com [81.17.249.38]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7FF2482B2 for ; Fri, 13 Jan 2023 03:13:30 -0800 (PST) Received: from mail.blacknight.com (pemlinmail02.blacknight.ie [81.17.254.11]) by outbound-smtp05.blacknight.com (Postfix) with ESMTPS id 143F3CCCE4 for ; Fri, 13 Jan 2023 11:13:29 +0000 (GMT) Received: (qmail 11467 invoked from network); 13 Jan 2023 11:13:28 -0000 Received: from unknown (HELO morpheus.112glenside.lan) (mgorman@techsingularity.net@[84.203.198.246]) by 81.17.254.9 with ESMTPA; 13 Jan 2023 11:13:28 -0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , NeilBrown , Thierry Reding , Matthew Wilcox , Vlastimil Babka , Linux-MM , LKML , Mel Gorman Subject: [PATCH 6/6] mm: discard __GFP_ATOMIC Date: Fri, 13 Jan 2023 11:12:17 +0000 Message-Id: <20230113111217.14134-7-mgorman@techsingularity.net> X-Mailer: git-send-email 2.35.3 In-Reply-To: <20230113111217.14134-1-mgorman@techsingularity.net> References: <20230113111217.14134-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1754906684082017035?= X-GMAIL-MSGID: =?utf-8?q?1754906684082017035?= From: NeilBrown __GFP_ATOMIC serves little purpose. Its main effect is to set ALLOC_HARDER which adds a few little boosts to increase the chance of an allocation succeeding, one of which is to lower the water-mark at which it will succeed. It is *always* paired with __GFP_HIGH which sets ALLOC_HIGH which also adjusts this watermark. It is probable that other users of __GFP_HIGH should benefit from the other little bonuses that __GFP_ATOMIC gets. __GFP_ATOMIC also gives a warning if used with __GFP_DIRECT_RECLAIM. There is little point to this. We already get a might_sleep() warning if __GFP_DIRECT_RECLAIM is set. __GFP_ATOMIC allows the "watermark_boost" to be side-stepped. It is probable that testing ALLOC_HARDER is a better fit here. __GFP_ATOMIC is used by tegra-smmu.c to check if the allocation might sleep. This should test __GFP_DIRECT_RECLAIM instead. This patch: - removes __GFP_ATOMIC - allows __GFP_HIGH allocations to ignore watermark boosting as well as GFP_ATOMIC requests. - makes other adjustments as suggested by the above. The net result is not change to GFP_ATOMIC allocations. Other allocations that use __GFP_HIGH will benefit from a few different extra privileges. This affects: xen, dm, md, ntfs3 the vermillion frame buffer hibernation ksm swap all of which likely produce more benefit than cost if these selected allocation are more likely to succeed quickly. [mgorman: Minor adjustments to rework on top of a series] Link: https://lkml.kernel.org/r/163712397076.13692.4727608274002939094@noble.neil.brown.name Signed-off-by: NeilBrown Signed-off-by: Mel Gorman Acked-by: Vlastimil Babka Acked-by: Michal Hocko --- Documentation/mm/balance.rst | 2 +- drivers/iommu/tegra-smmu.c | 4 ++-- include/linux/gfp_types.h | 12 ++++-------- include/trace/events/mmflags.h | 1 - lib/test_printf.c | 8 ++++---- mm/internal.h | 2 +- mm/page_alloc.c | 13 +++---------- tools/perf/builtin-kmem.c | 1 - 8 files changed, 15 insertions(+), 28 deletions(-) diff --git a/Documentation/mm/balance.rst b/Documentation/mm/balance.rst index 6a1fadf3e173..e38e9d83c1c7 100644 --- a/Documentation/mm/balance.rst +++ b/Documentation/mm/balance.rst @@ -6,7 +6,7 @@ Memory Balancing Started Jan 2000 by Kanoj Sarcar -Memory balancing is needed for !__GFP_ATOMIC and !__GFP_KSWAPD_RECLAIM as +Memory balancing is needed for !__GFP_HIGH and !__GFP_KSWAPD_RECLAIM as well as for non __GFP_IO allocations. The first reason why a caller may avoid reclaim is that the caller can not diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c index 5b1af40221ec..af8d0e685260 100644 --- a/drivers/iommu/tegra-smmu.c +++ b/drivers/iommu/tegra-smmu.c @@ -671,12 +671,12 @@ static struct page *as_get_pde_page(struct tegra_smmu_as *as, * allocate page in a sleeping context if GFP flags permit. Hence * spinlock needs to be unlocked and re-locked after allocation. */ - if (!(gfp & __GFP_ATOMIC)) + if (gfpflags_allow_blocking(gfp)) spin_unlock_irqrestore(&as->lock, *flags); page = alloc_page(gfp | __GFP_DMA | __GFP_ZERO); - if (!(gfp & __GFP_ATOMIC)) + if (gfpflags_allow_blocking(gfp)) spin_lock_irqsave(&as->lock, *flags); /* diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index d88c46ca82e1..5088637fe5c2 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -31,7 +31,7 @@ typedef unsigned int __bitwise gfp_t; #define ___GFP_IO 0x40u #define ___GFP_FS 0x80u #define ___GFP_ZERO 0x100u -#define ___GFP_ATOMIC 0x200u +/* 0x200u unused */ #define ___GFP_DIRECT_RECLAIM 0x400u #define ___GFP_KSWAPD_RECLAIM 0x800u #define ___GFP_WRITE 0x1000u @@ -116,11 +116,8 @@ typedef unsigned int __bitwise gfp_t; * * %__GFP_HIGH indicates that the caller is high-priority and that granting * the request is necessary before the system can make forward progress. - * For example, creating an IO context to clean pages. - * - * %__GFP_ATOMIC indicates that the caller cannot reclaim or sleep and is - * high priority. Users are typically interrupt handlers. This may be - * used in conjunction with %__GFP_HIGH + * For example creating an IO context to clean pages and requests + * from atomic context. * * %__GFP_MEMALLOC allows access to all memory. This should only be used when * the caller guarantees the allocation will allow more memory to be freed @@ -135,7 +132,6 @@ typedef unsigned int __bitwise gfp_t; * %__GFP_NOMEMALLOC is used to explicitly forbid access to emergency reserves. * This takes precedence over the %__GFP_MEMALLOC flag if both are set. */ -#define __GFP_ATOMIC ((__force gfp_t)___GFP_ATOMIC) #define __GFP_HIGH ((__force gfp_t)___GFP_HIGH) #define __GFP_MEMALLOC ((__force gfp_t)___GFP_MEMALLOC) #define __GFP_NOMEMALLOC ((__force gfp_t)___GFP_NOMEMALLOC) @@ -329,7 +325,7 @@ typedef unsigned int __bitwise gfp_t; * version does not attempt reclaim/compaction at all and is by default used * in page fault path, while the non-light is used by khugepaged. */ -#define GFP_ATOMIC (__GFP_HIGH|__GFP_ATOMIC|__GFP_KSWAPD_RECLAIM) +#define GFP_ATOMIC (__GFP_HIGH|__GFP_KSWAPD_RECLAIM) #define GFP_KERNEL (__GFP_RECLAIM | __GFP_IO | __GFP_FS) #define GFP_KERNEL_ACCOUNT (GFP_KERNEL | __GFP_ACCOUNT) #define GFP_NOWAIT (__GFP_KSWAPD_RECLAIM) diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 412b5a46374c..9db52bc4ce19 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -31,7 +31,6 @@ gfpflag_string(__GFP_HIGHMEM), \ gfpflag_string(GFP_DMA32), \ gfpflag_string(__GFP_HIGH), \ - gfpflag_string(__GFP_ATOMIC), \ gfpflag_string(__GFP_IO), \ gfpflag_string(__GFP_FS), \ gfpflag_string(__GFP_NOWARN), \ diff --git a/lib/test_printf.c b/lib/test_printf.c index d34dc636b81c..46b4e6c414a3 100644 --- a/lib/test_printf.c +++ b/lib/test_printf.c @@ -674,17 +674,17 @@ flags(void) gfp = GFP_ATOMIC|__GFP_DMA; test("GFP_ATOMIC|GFP_DMA", "%pGg", &gfp); - gfp = __GFP_ATOMIC; - test("__GFP_ATOMIC", "%pGg", &gfp); + gfp = __GFP_HIGH; + test("__GFP_HIGH", "%pGg", &gfp); /* Any flags not translated by the table should remain numeric */ gfp = ~__GFP_BITS_MASK; snprintf(cmp_buffer, BUF_SIZE, "%#lx", (unsigned long) gfp); test(cmp_buffer, "%pGg", &gfp); - snprintf(cmp_buffer, BUF_SIZE, "__GFP_ATOMIC|%#lx", + snprintf(cmp_buffer, BUF_SIZE, "__GFP_HIGH|%#lx", (unsigned long) gfp); - gfp |= __GFP_ATOMIC; + gfp |= __GFP_HIGH; test(cmp_buffer, "%pGg", &gfp); kfree(cmp_buffer); diff --git a/mm/internal.h b/mm/internal.h index 23a37588073a..71b1111427f3 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -24,7 +24,7 @@ struct folio_batch; #define GFP_RECLAIM_MASK (__GFP_RECLAIM|__GFP_HIGH|__GFP_IO|__GFP_FS|\ __GFP_NOWARN|__GFP_RETRY_MAYFAIL|__GFP_NOFAIL|\ __GFP_NORETRY|__GFP_MEMALLOC|__GFP_NOMEMALLOC|\ - __GFP_ATOMIC|__GFP_NOLOCKDEP) + __GFP_NOLOCKDEP) /* The GFP flags allowed during early boot */ #define GFP_BOOT_MASK (__GFP_BITS_MASK & ~(__GFP_RECLAIM|__GFP_IO|__GFP_FS)) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b9ae0ba0a2ab..78ffebc4798b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4087,13 +4087,14 @@ static inline bool zone_watermark_fast(struct zone *z, unsigned int order, if (__zone_watermark_ok(z, order, mark, highest_zoneidx, alloc_flags, free_pages)) return true; + /* - * Ignore watermark boosting for GFP_ATOMIC order-0 allocations + * Ignore watermark boosting for __GFP_HIGH order-0 allocations * when checking the min watermark. The min watermark is the * point where boosting is ignored so that kswapd is woken up * when below the low watermark. */ - if (unlikely(!order && (gfp_mask & __GFP_ATOMIC) && z->watermark_boost + if (unlikely(!order && (alloc_flags & ALLOC_MIN_RESERVE) && z->watermark_boost && ((alloc_flags & ALLOC_WMARK_MASK) == WMARK_MIN))) { mark = z->_watermark[WMARK_MIN]; return __zone_watermark_ok(z, order, mark, highest_zoneidx, @@ -5058,14 +5059,6 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, unsigned int zonelist_iter_cookie; int reserve_flags; - /* - * We also sanity check to catch abuse of atomic reserves being used by - * callers that are not in atomic context. - */ - if (WARN_ON_ONCE((gfp_mask & (__GFP_ATOMIC|__GFP_DIRECT_RECLAIM)) == - (__GFP_ATOMIC|__GFP_DIRECT_RECLAIM))) - gfp_mask &= ~__GFP_ATOMIC; - restart: compaction_retries = 0; no_progress_loops = 0; diff --git a/tools/perf/builtin-kmem.c b/tools/perf/builtin-kmem.c index e20656c431a4..173d407dce92 100644 --- a/tools/perf/builtin-kmem.c +++ b/tools/perf/builtin-kmem.c @@ -641,7 +641,6 @@ static const struct { { "__GFP_HIGHMEM", "HM" }, { "GFP_DMA32", "D32" }, { "__GFP_HIGH", "H" }, - { "__GFP_ATOMIC", "_A" }, { "__GFP_IO", "I" }, { "__GFP_FS", "F" }, { "__GFP_NOWARN", "NWR" },