From patchwork Wed Feb 1 08:02:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 51216 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp145105wrn; Wed, 1 Feb 2023 00:08:39 -0800 (PST) X-Google-Smtp-Source: AK7set9CxB38ZyKKegjUm1rrmCY/1xY3uB7c65d8uP0oC4l9dSTIz2bixmYZYUMsq8ot3NzNE3zy X-Received: by 2002:a17:906:184a:b0:886:50d:be8d with SMTP id w10-20020a170906184a00b00886050dbe8dmr1496843eje.13.1675238919808; Wed, 01 Feb 2023 00:08:39 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675238919; cv=pass; d=google.com; s=arc-20160816; b=mOGtrAc17zyvyisSZcDf2/7UDIJFjEvKGfeFE9OGvliCfXJQG0l6HE0E5WSo7+CCuC eWxe6Z8vFKXcwMwFrBbPhUm40AGSH48th1EAD0nFkTNKbYUfobCNAFJRstYDNDFQBD7h RkW5PD/JIX/3AVNclV+CZYaDx2/8Z4N5h1gRM2ezyuHN8Do/fGY8K3JmuDdbZlx/lmp2 HoET6nontn9KAU7i5+GND2l2jQgYGJ9TQDI9bH2LjKH+/rHWF2S2haHLHNP1UHZbZ0rT lw135rh7AUcTGR8PM6Ca+gvJfHk+PZQHw/aX6l0+wcQuOppghpEm4AjbGH8Lr1+erQBL 3CeQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AhcuMGmyMR6wFmrQd9SM/5hyh6AFl2dn/jX8xfA+fTk=; b=QO19uHFY1ZK7aQb7WkFn7cpRMJX3nWLn+z76Iw7dCf+OkAg0Ft6sEp8VSElko7XkN9 AJKp2cZ15TuM6Axge3GBivjFar2C2DKRgKGKyRkp8A1Np/u0TAFDjsww12v51w4y9jxX VQ+WcjQ60khhnfJETP2N1vuotZDmoYwtOniWbqHXTB/t3XEtdk224c+0AeKSrm3bxlIL 9v7S46xR+N3hRCu0eU7MTMPiShG8E1a07Kh3BNDs3KbwzXjoQ3dfRgvI3ULBWKU7S6bp 7bAUR2v7WJSwVORuoIvdAf87IP4QCyM/vFUpoAZkt01fDC/2ge1L7vJCBDAbgeyF848c tCww== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=RxcmXlie; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id fl15-20020a1709072a8f00b00877df2d5137si18655452ejc.475.2023.02.01.00.08.16; Wed, 01 Feb 2023 00:08:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=RxcmXlie; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231857AbjBAIDl (ORCPT + 99 others); Wed, 1 Feb 2023 03:03:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52676 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230437AbjBAIDg (ORCPT ); Wed, 1 Feb 2023 03:03:36 -0500 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2044.outbound.protection.outlook.com [40.107.243.44]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61C4A2BF3A for ; Wed, 1 Feb 2023 00:03:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Mvsd5PIG9G1vJY0uDpjV3o1hPM7aBxusFhRgoWan/BGg8YWM8Ho4fhUvoz470GV/sZ23w1BumzF1K0jYVFYteHuyEpa61GM2pReYrjpNXpcGm3ua0pYjtgfKW90wvYek6rt1eqWhkg+7CvLTC8HXSObkyvkXy3GbCU0tgjJTXufCOmJhllazG6PAjgB62t34dko59qxLJQlBaYWwvw4XZyvFcGchSfAJSYAzsZ8Fhqmv3aARpdvCON2rHvFZEymhtBO6h3Nol7NZI1ug4xCtWPrQF38ncNkLukx1Sbj8barPrvB1g/I5MIr3qe21pQ/VckFauVy9cJ4ClhhgNg+Q4g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AhcuMGmyMR6wFmrQd9SM/5hyh6AFl2dn/jX8xfA+fTk=; b=etD+lxv6XdLA/T2Myd7/YFDB4Z1HxELSMpa+/0tLmP2jADZDSYoYqrHC9NwySwXNlr67+gaqCyAO7rOY+OuALoIb8CbdSdi8bebXEnIq/7j5yio6KLMqiYkyBb5unqlc6Zah4yTPTZwpHmJo0HOvSmzPdkVI1E7Hy6oCAYwFz0Fg9W+vna+19KxVzRx5N1xK2BiNdH/4dM+fGCm40d2rIvTAAXR8j1uSqe6T8YaDJzUOpN1bmGg1ISwHGGALZtfSh3a3CH5l6mK/KkXPL/8asck/ieQFJ1Q57OGbkNePVivRN4yDSQ64cpRI0/fnyiwb1l8DbPRhBQmGIIjEwKXBtA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AhcuMGmyMR6wFmrQd9SM/5hyh6AFl2dn/jX8xfA+fTk=; b=RxcmXlieyT4aNlBaMdoa68K87VGD0TBpMjYRJVq2wmpnq7veMTzPq50p82cjoPt10eJ7t+ivKbziPkcANnrfBwUfkXJRpg6uXcFiyju7PAaJxYQvWkbNHxMqvZMSio+04WwOaRbPKHP+aZCbJ+J9x85Cyiua6he5W47yaOAetkY= Received: from BN9P223CA0019.NAMP223.PROD.OUTLOOK.COM (2603:10b6:408:10b::24) by CH3PR12MB8511.namprd12.prod.outlook.com (2603:10b6:610:15c::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.38; Wed, 1 Feb 2023 08:03:26 +0000 Received: from BN8NAM11FT082.eop-nam11.prod.protection.outlook.com (2603:10b6:408:10b:cafe::36) by BN9P223CA0019.outlook.office365.com (2603:10b6:408:10b::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.38 via Frontend Transport; Wed, 1 Feb 2023 08:03:25 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT082.mail.protection.outlook.com (10.13.176.94) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6064.22 via Frontend Transport; Wed, 1 Feb 2023 08:03:25 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Wed, 1 Feb 2023 02:03:22 -0600 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Bharata B Rao , Disha Talreja , Mel Gorman , Raghavendra K T Subject: [PATCH V2 1/3] sched/numa: Apply the scan delay to every vma instead of tasks Date: Wed, 1 Feb 2023 13:32:20 +0530 Message-ID: <1aebc55030925998a3df3cafb79c5cd28b199ea8.1675159422.git.raghavendra.kt@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT082:EE_|CH3PR12MB8511:EE_ X-MS-Office365-Filtering-Correlation-Id: 510611f5-71e3-49c2-9d40-08db042acbef X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /SRV4htTNbr7JEQTc+CKomAgA6e1uQtQT7+e74B3FoFmk66xOXpU5iqvOZ4kljvy6KwcH/DsdEX0hUnJlt/DgtUq1ZGjWh90uqDpL/BzgD5wjwX1jkwqGfBCa7pf1C30QN7RwAUEytf04cnHYb59cPTS4Revv6OOY1OD8Pb/4dV6oinvQ3TnYWcGC37eIoyHV2FpmYnRd0MaJgFgA3gRNX2GE1h84X0D0IggUOGZSCvMgI92UZVdc+X+D7L0un7r6Y1ggh8NRJ1ZYCJa7xuBT/MeXYvTPCysXLDyCD+EZH0cKZomBK+hT9h68Uyge2d1pPcKtj4jPJhPQfCfM7XqFfgQ9uKA25zH04fScUXebJla/L1F5gbD+BreG1CCburGOKUvHXj0D8p6LrEwHWzu9B01I1D9kJf8TE9EMLjetS1vroXTi9zgxhbhz+OB5WUqawekepNfg0/16czmXpmFmZn5m6TuxIuCSfAHILG32p2VpkRbsVIfRDFhYiVzfi9qaoyfw+NuA6BhRX9Eft6I2D3fAQSN8D54Av3mYkXLuaMK2jPlZ6/9kdVovXq6u49q+p4h8P8XZZPeM8P3BVSn3J1OZhZ82uACNFqMCQdsCKD5C/acJfcahgwprlvkFYxlJpxDzewXUk1rIOJ5OF9deeozAuIcUTKaybDBq928Zps7kdNudPiNLFS3PhTa/34JAIP3NgfCjNqB8SE/ErhPKO1HrJdMijTaUH09qFQ8LTQjkdjUp6o0hhWA/cxZfzmg X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(346002)(39860400002)(451199018)(46966006)(36840700001)(40470700004)(54906003)(316002)(8936002)(110136005)(4326008)(70586007)(8676002)(41300700001)(70206006)(356005)(40480700001)(82740400003)(36756003)(36860700001)(81166007)(40460700003)(186003)(26005)(2906002)(82310400005)(7696005)(47076005)(478600001)(426003)(83380400001)(2616005)(5660300002)(16526019)(336012)(2101003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2023 08:03:25.9204 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 510611f5-71e3-49c2-9d40-08db042acbef X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT082.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB8511 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756615325786931088?= X-GMAIL-MSGID: =?utf-8?q?1756615325786931088?= From: Mel Gorman Avoid scanning new or very short-lived VMAs. (Raghavendra: Add initialization in vm_area_dup()) Signed-off-by: Mel Gorman Signed-off-by: Raghavendra K T --- include/linux/mm.h | 9 +++++++++ include/linux/mm_types.h | 7 +++++++ kernel/fork.c | 2 ++ kernel/sched/fair.c | 17 +++++++++++++++++ 4 files changed, 35 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 974ccca609d2..74d9df1d8982 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -611,6 +611,14 @@ struct vm_operations_struct { unsigned long addr); }; +#ifdef CONFIG_NUMA_BALANCING +#define vma_numab_init(vma) do { (vma)->numab = NULL; } while (0) +#define vma_numab_free(vma) do { kfree((vma)->numab); } while (0) +#else +static inline void vma_numab_init(struct vm_area_struct *vma) {} +static inline void vma_numab_free(struct vm_area_struct *vma) {} +#endif /* CONFIG_NUMA_BALANCING */ + static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) { static const struct vm_operations_struct dummy_vm_ops = {}; @@ -619,6 +627,7 @@ static inline void vma_init(struct vm_area_struct *vma, struct mm_struct *mm) vma->vm_mm = mm; vma->vm_ops = &dummy_vm_ops; INIT_LIST_HEAD(&vma->anon_vma_chain); + vma_numab_init(vma); } static inline void vma_set_anonymous(struct vm_area_struct *vma) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 500e536796ca..e84f95a77321 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -435,6 +435,10 @@ struct anon_vma_name { char name[]; }; +struct vma_numab { + unsigned long next_scan; +}; + /* * This struct describes a virtual memory area. There is one of these * per VM-area/task. A VM area is any part of the process virtual memory @@ -504,6 +508,9 @@ struct vm_area_struct { #endif #ifdef CONFIG_NUMA struct mempolicy *vm_policy; /* NUMA policy for the VMA */ +#endif +#ifdef CONFIG_NUMA_BALANCING + struct vma_numab *numab; /* NUMA Balancing state */ #endif struct vm_userfaultfd_ctx vm_userfaultfd_ctx; } __randomize_layout; diff --git a/kernel/fork.c b/kernel/fork.c index 08969f5aa38d..ac6f0477cf6e 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -474,6 +474,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) */ *new = data_race(*orig); INIT_LIST_HEAD(&new->anon_vma_chain); + vma_numab_init(new); dup_anon_vma_name(orig, new); } return new; @@ -481,6 +482,7 @@ struct vm_area_struct *vm_area_dup(struct vm_area_struct *orig) void vm_area_free(struct vm_area_struct *vma) { + vma_numab_free(vma); free_anon_vma_name(vma); kmem_cache_free(vm_area_cachep, vma); } diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index e4a0b8bd941c..060b241ce3c5 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -3015,6 +3015,23 @@ static void task_numa_work(struct callback_head *work) if (!vma_is_accessible(vma)) continue; + /* Initialise new per-VMA NUMAB state. */ + if (!vma->numab) { + vma->numab = kzalloc(sizeof(struct vma_numab), GFP_KERNEL); + if (!vma->numab) + continue; + + vma->numab->next_scan = now + + msecs_to_jiffies(sysctl_numa_balancing_scan_delay); + } + + /* + * After the first scan is complete, delay the balancing scan + * for new VMAs. + */ + if (mm->numa_scan_seq && time_before(jiffies, vma->numab->next_scan)) + continue; + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE); From patchwork Wed Feb 1 08:02:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 51215 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp144817wrn; Wed, 1 Feb 2023 00:08:00 -0800 (PST) X-Google-Smtp-Source: AK7set9wOq+zp/KUSiKJLDvK+j+DEfaoUB1DF5aoEjD2sHXa4jHFKp5yPB3vW8JFahCkTNbnqtXq X-Received: by 2002:a05:6a20:d49a:b0:bf:e6c:758b with SMTP id im26-20020a056a20d49a00b000bf0e6c758bmr1803808pzb.4.1675238880096; Wed, 01 Feb 2023 00:08:00 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675238880; cv=pass; d=google.com; s=arc-20160816; b=xhCPn2N7yTNNdTFTHrtlwthbgHHC/b5eFVYgf790vS+5TAcs0YVlWA+rZBy9RVLRCM AuaWEy1/Jijm+aDzV99bLMyhz44CDKC1jIE+0DSdgwx+fLNPw4wHpl4t9Y1d9a94QBep OHRUv8WZrlQgtjhf99Xu0grNLtWxWIOUtjm7ayjv2wlecS65v/5eBIdX8xuIBOEEPUfE ZeroEQ0If8yA+H7jnznOxv1GvvKhIrdwDDzc61Dpre486t08h5WrniIxEusDKanGwtoP P2cHi+LfeQwQrpmCXQZmcdtBECneM7k2eNulcyjDNOAQ6vhSBUk3FjlpsET0N08Ldj7L zuiA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=1R9KAiSIKM0EdbYofwwJVHMq44auVW63YveLhtMSQ3A=; b=K80Q8MToAhvN9EvB2ptzyoy3Q1r9iMHBSlAba0sfV8/Y7xouJEnnyaQ8DdEy9JP1oL yi671Ze2cz8ljaDL+Le4lcwUsCE6vGFP9GDPxV1MlIDcdbmKRcFMOnd4LjBjmOHjjU0T YKhbwjcs/C4nySyLU0gwjDCH2JrQoi/JocQeODWb9+rJfHWc//88UA5jSYWjTZaHfBnZ 81uOd73WYxta0TuMyyapumztST7IGsOECbt+/uNU6Y2Wusadx7mzt+MWS1u94ADy8LhU yVi9QCN8vN/sZp7lD3EkDRg62r8pVRbW1/iYpSPrIVitvsFIc09MGpQRV2L1t0uvKIX+ gfgQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=TtZGqDs1; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id l190-20020a6388c7000000b004a58ff2e054si18249878pgd.834.2023.02.01.00.07.48; Wed, 01 Feb 2023 00:08:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=TtZGqDs1; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232000AbjBAIDn (ORCPT + 99 others); Wed, 1 Feb 2023 03:03:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231985AbjBAIDh (ORCPT ); Wed, 1 Feb 2023 03:03:37 -0500 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D63D53D912 for ; Wed, 1 Feb 2023 00:03:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EaisipyW1DNecXe/OL8nAK6OSSIwpNOx0jfBHm7SEMEY9I3la64UaqB+2679vkJ1I/X6Wm4OAnFV4eja279oqg6F71z4lcTfsmmmmAw3OP50+z9u3Fx9U1mB1nopLbhKC2NAvit5ul0zHffO5q7uycBk2GQUq4Ph0TysYd8rt50jnLj5PMcdr7GF3GNXWKQs4/Q9Os5gXdSg5tBP2HHLmgYwNHkrwoX6wmpEV0mMNb/oXiOPnvhIDlXM56IoQReG2fHo3N5XANltCgrSVzBuGVLv1w/XcK5uyWEP3KxVdSzU/gHz9F5CpTxesA0BVm1Q2udIiEMrAmFePDglZVppUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1R9KAiSIKM0EdbYofwwJVHMq44auVW63YveLhtMSQ3A=; b=fEpiMe2C2Xd93Q4f0+OwDRVDogTu5jIwL8ROm7N68EYy1wuKsgiNSzWyAVwG4AWQp6bYv/XauLL9YnHuug3R+ZkPhur3htRYUTNLWzOC6Cz9Jr6sFJUfkM3+Gx1vIRE918iBZABglicdB3jBWluxcCzWIPJZYObwj9smbMFztqPC/BCFxHi9SPxw/SBthQNYrAp3yxdCIaVEzHdDgRKA+NBeiCwvwO2mBKhlXY6ILIrGTOmKgBtQcgg/mt7DOe1pwlnaXcKyUPktVndFjtrEKkYvU7n66ZTbfX4tFg/eT6XV363RWzYKra89OPZ4LbQzr80+/AkQhfwEiq/DRl4YwA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1R9KAiSIKM0EdbYofwwJVHMq44auVW63YveLhtMSQ3A=; b=TtZGqDs1IjOIhDWRY8F4GtSlIuDzjxWQq9V96wz5C88B+zMctmpv9PpPg0Y8kZxdeJ/t5aflF15zs/cJo8QLyhAwvlgbepVLWBQr6Tfewa5bC6/XppcyiEuHaerWVi72GxA2nDnMbtXCB9ghkg67RNeudzwJ8Vt55Br0d2YLs8s= Received: from BN1PR12CA0029.namprd12.prod.outlook.com (2603:10b6:408:e1::34) by CH3PR12MB7689.namprd12.prod.outlook.com (2603:10b6:610:14d::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.36; Wed, 1 Feb 2023 08:03:29 +0000 Received: from BN8NAM11FT096.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e1:cafe::ee) by BN1PR12CA0029.outlook.office365.com (2603:10b6:408:e1::34) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.24 via Frontend Transport; Wed, 1 Feb 2023 08:03:29 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT096.mail.protection.outlook.com (10.13.177.195) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6064.24 via Frontend Transport; Wed, 1 Feb 2023 08:03:29 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Wed, 1 Feb 2023 02:03:25 -0600 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Bharata B Rao , Disha Talreja , Raghavendra K T Subject: [PATCH V2 2/3] sched/numa: Enhance vma scanning logic Date: Wed, 1 Feb 2023 13:32:21 +0530 Message-ID: <5f0872657ddb164aa047a2231f8dc1086fe6adf6.1675159422.git.raghavendra.kt@amd.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT096:EE_|CH3PR12MB7689:EE_ X-MS-Office365-Filtering-Correlation-Id: 84e0c807-f189-4def-00f7-08db042ace20 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1x5Eufv38AZdDH+X32cME7v+8pJxC7VkM+4RaTgfaFTEGV+5Hmh59KNIC6uH7TaaloDWI+XpkEslufPnVS9HVJouLjrm5i2diGoO6gSmRky022Yl94Q+pZkUaCAnbJmll0nAfxlT66qcNYrivpR8bZbYYDd772Kdip0gEbLbtczngrLLYs+T+XRt5Uhvf7dbPcobVbx2iZzBosMCNEO7PqEyTATMotHLYcFyG/nNUmVXdtOGBS8QaSusXbdW0iDR8RURSV2edkvkYpYLNjKgmYhQtR1HoAdlk7QZ785P3txKCOzkUO/kOAXdffwc9u3/t05JpV7EuljVjtu/wyxXdAqlnccRwaX0++BIJ36ZeQQFj45frhMSpkOBXTRWjItTknNrnoKbQ7r8RarJvcigu3Ot/WVd5Ar4zt+g1goWhfvF/FvjVC1JlImGsbeUK/5raAGeqKgeytcjbnynmX6ht6wMF89EvWiY0ZUM73Ry1/pwyinvN11Vib8/b4dFUBkb1mjl+HV2vmPYlSxOZSun4yhEIjorMw9VPcEMzfTJ3kYXJ/3bbgAENyi5rfZGtgfHeCuDVuLrhI0YSKJfwc/4z3YRQnz+XF1GgzJuYFFoRBOmmVTpxD5foh2HpG67YyucvrF3kwA0NxilSEkeoqntFoASK9jmvCO1RnFbj/VwSUSrhAUS5nbO4+oqe2sQ0iK/SNYTAOfGslx6GxG/PDMl36ER4BX9hON07cRL/FZCjKOuKQluvoRoJfAsIE5ex0Ux X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230025)(4636009)(396003)(346002)(376002)(136003)(39860400002)(451199018)(40470700004)(36840700001)(46966006)(8676002)(81166007)(36860700001)(356005)(82740400003)(70206006)(316002)(36756003)(82310400005)(8936002)(5660300002)(4326008)(110136005)(41300700001)(70586007)(2906002)(40460700003)(2616005)(336012)(40480700001)(83380400001)(426003)(7696005)(478600001)(47076005)(26005)(186003)(16526019)(6666004)(54906003)(2101003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2023 08:03:29.5929 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 84e0c807-f189-4def-00f7-08db042ace20 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT096.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7689 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756615283708766066?= X-GMAIL-MSGID: =?utf-8?q?1756615283708766066?= During the Numa scanning make sure only relevant vmas of the tasks are scanned. Before: All the tasks of a process participate in scanning the vma even if they do not access vma in it's lifespan. Now: Except cases of first few unconditional scans, if a process do not touch vma (exluding false positive cases of PID collisions) tasks no longer scan all vma. Logic used: 1) 6 bits of PID used to mark active bit in vma numab status during fault to remember PIDs accessing vma. (Thanks Mel) 2) Subsequently in scan path, vma scanning is skipped if current PID had not accessed vma. 3) First two times we do allow unconditional scan to preserve earlier behaviour of scanning. Acknowledgement to Bharata B Rao for initial patch to store pid information. Suggested-by: Mel Gorman Signed-off-by: Raghavendra K T --- include/linux/mm.h | 14 ++++++++++++++ include/linux/mm_types.h | 1 + kernel/sched/fair.c | 15 +++++++++++++++ mm/huge_memory.c | 1 + mm/memory.c | 1 + 5 files changed, 32 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 74d9df1d8982..489422942482 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1381,6 +1381,16 @@ static inline int xchg_page_access_time(struct page *page, int time) last_time = page_cpupid_xchg_last(page, time >> PAGE_ACCESS_TIME_BUCKETS); return last_time << PAGE_ACCESS_TIME_BUCKETS; } + +static inline void vma_set_active_pid_bit(struct vm_area_struct *vma) +{ + unsigned int active_pid_bit; + + if (vma->numab) { + active_pid_bit = current->pid % BITS_PER_LONG; + vma->numab->accessing_pids |= 1UL << active_pid_bit; + } +} #else /* !CONFIG_NUMA_BALANCING */ static inline int page_cpupid_xchg_last(struct page *page, int cpupid) { @@ -1430,6 +1440,10 @@ static inline bool cpupid_match_pid(struct task_struct *task, int cpupid) { return false; } + +static inline void vma_set_active_pid_bit(struct vm_area_struct *vma) +{ +} #endif /* CONFIG_NUMA_BALANCING */ #if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index e84f95a77321..980a6a4308b6 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -437,6 +437,7 @@ struct anon_vma_name { struct vma_numab { unsigned long next_scan; + unsigned long accessing_pids; }; /* diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 060b241ce3c5..3505ae57c07c 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2916,6 +2916,18 @@ static void reset_ptenuma_scan(struct task_struct *p) p->mm->numa_scan_offset = 0; } +static bool vma_is_accessed(struct vm_area_struct *vma) +{ + unsigned int active_pid_bit; + + if (READ_ONCE(current->mm->numa_scan_seq) < 2) + return true; + + active_pid_bit = current->pid % BITS_PER_LONG; + + return vma->numab->accessing_pids & (1UL << active_pid_bit); +} + /* * The expensive part of numa migration is done from task_work context. * Triggered from task_tick_numa(). @@ -3032,6 +3044,9 @@ static void task_numa_work(struct callback_head *work) if (mm->numa_scan_seq && time_before(jiffies, vma->numab->next_scan)) continue; + if (!vma_is_accessed(vma)) + continue; + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 811d19b5c4f6..d908aa95f3c3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1485,6 +1485,7 @@ vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf) bool was_writable = pmd_savedwrite(oldpmd); int flags = 0; + vma_set_active_pid_bit(vma); vmf->ptl = pmd_lock(vma->vm_mm, vmf->pmd); if (unlikely(!pmd_same(oldpmd, *vmf->pmd))) { spin_unlock(vmf->ptl); diff --git a/mm/memory.c b/mm/memory.c index 8c8420934d60..2ec3045cb8b3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4718,6 +4718,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) bool was_writable = pte_savedwrite(vmf->orig_pte); int flags = 0; + vma_set_active_pid_bit(vma); /* * The "pte" at this point cannot be used safely without * validation through pte_unmap_same(). It's of NUMA type but From patchwork Wed Feb 1 08:02:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 51219 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp145562wrn; Wed, 1 Feb 2023 00:10:01 -0800 (PST) X-Google-Smtp-Source: AK7set880IMcxmCI1NdNKtPfxRNnak5zjD0nZmYu7iYTyZCXHCLGOBfCcqhhroR1uPbG0pI/QIun X-Received: by 2002:a17:902:e5c8:b0:196:5425:9eea with SMTP id u8-20020a170902e5c800b0019654259eeamr2358646plf.41.1675239001234; Wed, 01 Feb 2023 00:10:01 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675239001; cv=pass; d=google.com; s=arc-20160816; b=CUe0SEZ2K6pch3jc2uoagLdzhNZT18wFLjsK5ZpSkxpFJj/dw07gPRdpvUf2LT2CzA hkUs0z9bY4Fmrm0SWNdiI3YkdeXn3hKuDBBQUBTOQ3zWN/BNQr9bm4JXfJJrZ6egpfJs ecIyv/swseaEUcuAq8rY3YnJo7x/G7LX4sKrBauJoZ1fjOW887YJ637QXdTV+ih63vbX c+tyyq0iU7MQNHy/r5ofhOqrszTscPQYovq2aIK75UNBqmv59/9I7oajXntgkOgE/xrS Mkx+WoFkSkcHUH/y1L+7c4TlIf1PvOazO/VzosohAw8OVkZQEE0nkTy/tDHEae49p07q 0h4g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=zBy421iEXPoI1DU2317/qdpzKhPgWFnxaUG/6sW3lQg=; b=KsrQk7BwHRSvLk/ceSGhlahPzYUy588X5NSVcFjeEI8MLvi/YyksusEt/1gwHTG6HI tCBcjZqYITHqWZsE6wZIHHfWg9it9OqQdRhDmKO865mfHWvFzdOp48DR6NLPvjK3CSCd er5fT13OWKfs1MUMgFnWlpois3a4Rcq19RlOuvVAb6o48FAZj7pnLymjWjEWGUXnssjd rYqk5jT8bwdWx/eHIL0L8+/gys0ktX1NW9L4+KmQszF1UHX/c+/zWD5QepChzfyQQEY8 NDDExLBhGEPxXJjFFkfdMAFAYr3cylGhDXR9qUxeBwJQHQX5yNhrNSgMA2LNOnHHto7/ gW5g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=qirlsF8c; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q17-20020a17090311d100b00174c5fdc8d2si21304780plh.307.2023.02.01.00.09.48; Wed, 01 Feb 2023 00:10:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=qirlsF8c; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232017AbjBAIEB (ORCPT + 99 others); Wed, 1 Feb 2023 03:04:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232030AbjBAIDs (ORCPT ); Wed, 1 Feb 2023 03:03:48 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2076.outbound.protection.outlook.com [40.107.223.76]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A6225D122 for ; Wed, 1 Feb 2023 00:03:43 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QutHDQm4lZeAIrwxYF/HoNq/20PqkqrHp+UedaZawYjNZfMVLUGMahVGSeIwQkpgGs0mpXNZRqCh/WlcqU3UdfJePrtjOJgb+AV+NDyq19nYXlU0v35hQNjDpfzcreIRrU3h5yVjz1UObAz+jds8X7Zpt7df4TZv2hBVgC4A7yJI0dHMjnXm9HBJ0mjaeS0UXY0gy4Ib5mVbYrGNocplsUBu3kHnsvGZKtb+D+GhVT/knKyeRkSYCdd/VqIUlzOaJU3pV7vpd4IlNpn8y06YtSo95Qj2eNvsDJ4IHIA7Gt11isAdUW9FamhAZaOrSpWnTbrRF4EwoasZUbzdqF8M2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=zBy421iEXPoI1DU2317/qdpzKhPgWFnxaUG/6sW3lQg=; b=EVxB//Q5DkKMXQ+ICFbhLggcbgDfTtefYMDneP9KRL360VFdla2R3Uf4Vm8sVVx8gebO+PoTLQN8RmW1ETaKAV7qbCI7uSXivdGubmjpUdnbt96PkBQat+aiAHegartWGH/bpEV1jf8U6ILc3IUOa7deaCx/nJtUjgQ9/cWxDlxj+tY7btoPH/nx+aMh+/6lqB8nWA4C7K/2mtMWoK9NVvfzdA2lqoasdHiJbeitutXFPZ5oNNSDVlAaljxKwA5bamCQIPbfCe2md+mtKOUwwJa70ROfMf7GhNwDQHRxQKG5vpiyar1oXHIker9wiBIXX3LvwfAfXSEKwjwnMXiNgw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zBy421iEXPoI1DU2317/qdpzKhPgWFnxaUG/6sW3lQg=; b=qirlsF8c/7kHm5TSItSbvRRqT6avHT5eRJREx0qNRCRuqsknuODdeUwxfE79Kx/tRwUrv6CpkRz2EHutgBB80NaBe6hUzYBAUMPyOXZ84sceCFCz2NiDP6yOsu7dZMKC6VE6qbSPxRHVuhKakwG4KRIwJsUtBZStMk+pMd4g7hE= Received: from BN1PR14CA0014.namprd14.prod.outlook.com (2603:10b6:408:e3::19) by MN0PR12MB6272.namprd12.prod.outlook.com (2603:10b6:208:3c0::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.38; Wed, 1 Feb 2023 08:03:41 +0000 Received: from BN8NAM11FT086.eop-nam11.prod.protection.outlook.com (2603:10b6:408:e3:cafe::fd) by BN1PR14CA0014.outlook.office365.com (2603:10b6:408:e3::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6043.38 via Frontend Transport; Wed, 1 Feb 2023 08:03:41 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by BN8NAM11FT086.mail.protection.outlook.com (10.13.176.220) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6064.22 via Frontend Transport; Wed, 1 Feb 2023 08:03:41 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Wed, 1 Feb 2023 02:03:29 -0600 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Bharata B Rao , Disha Talreja , Raghavendra K T Subject: [PATCH V2 3/3] sched/numa: Reset the accessing PID information periodically Date: Wed, 1 Feb 2023 13:32:22 +0530 Message-ID: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BN8NAM11FT086:EE_|MN0PR12MB6272:EE_ X-MS-Office365-Filtering-Correlation-Id: 753a8f99-9e26-4f55-bbcb-08db042ad528 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oLTh+cidC2YJSLzJPpFtDSWZ3aa4tbq+BHJD3eBC6XHLkMqyP9hHdLxDGiybeu5SHHs/kpch6WSBZhoXfUFK5xYGVaPOfwpkY/94OX9RiWssyqd9f9gb+TwEmz2ZLq58Ef2blCJxcQN3jO5l3XXwNQgyGike9bC8tmDCDzIdf8ScSMObuW/nq69xTH0h0PInmlUjH3SwHLXDaquVLALw23NPkhKM2hoVrmpNpQm5pfEUprNrM2+ddxBzwU7wcELSL/pzWU4MP6TxkQ+wXtyP+9Ed3papyn+MhAH6RpVvbjfa+dfdaBa/TSu5UBUbi9pDa987wufya1V50YG7e91qHLbq/snFoqj0m6hf7YRUnGqhQ4WoHlaDjMcP/4ycYQtOCCyJnI5tMLHiTH6P1t0sA8rGYqRreldX6xuMWlVaZ9BRtENHtqTO6/gfkZg0P+I9NIq7Jbxu/ziJnU3DlJJXbpnF8HH8hTlWlWfMuv3tEnIz2DOodjXTQyNtvH5HTTxjGUPmbnF9aLHwQP2IuWRZLTtd4BWX5+FqM+BL15qADgK1iv/iYaJCCDfleHHx6586LSrf/2s7NfR5hThK/bX2e8AlB3ii+6CkqQtrb4WrzqkeNEhRuOGDuF01hw4XksVZUvAbaGtFC1eqjZijvxWVZsoLsUrFz8b1y2VQ0U5/DCEidpIovQekiK+STLU+kpSuX1r7+tfitM1G1MG0Q8A+2LYBij/w1RoOuDUE3ikwaJJCjc5pWqLnWxJ6Rw15ws5Q X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230025)(4636009)(346002)(39860400002)(376002)(396003)(136003)(451199018)(46966006)(40470700004)(36840700001)(356005)(81166007)(82740400003)(40480700001)(82310400005)(36756003)(40460700003)(336012)(110136005)(426003)(7696005)(316002)(54906003)(47076005)(2616005)(6666004)(478600001)(26005)(16526019)(186003)(41300700001)(36860700001)(70206006)(70586007)(5660300002)(4326008)(83380400001)(2906002)(8676002)(8936002)(36900700001)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Feb 2023 08:03:41.3942 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 753a8f99-9e26-4f55-bbcb-08db042ad528 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT086.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN0PR12MB6272 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756615410809384224?= X-GMAIL-MSGID: =?utf-8?q?1756615410809384224?= This helps to ensure, only recently accessed PIDs scan the VMAs. Current implementation: Reset accessing PIDs every (4 * sysctl_numa_balancing_scan_delay) interval after initial scan delay period expires. The reset logic is implemented in scan path Suggested-by: Mel Gorman Signed-off-by: Raghavendra K T --- Some of the potential ideas for clearing the accessing PIDs 1) Flag to indicate phase in life cycle of vma and tie with timestamp (reuse next_scan or so) VMA life cycle t1 t2 t3 t4 t5 t6 |<- DS ->|<- US ->|<- CS ->|<- US ->|<- CS ->| flags used to indicate whether we are in DS/CS/US phase DS (delay scan): Initial phase where scan is avoided for new VMA US (unconditional scan): Brief period where scanning is allowed irrespective of task faulting the VMA CS (conditional scan) : Longer conditiona scanning phase where task scanning is allowed only for VMA of interest 2) Maintain duplicate list of accessing PIDs to keep track of history of access. and switch/reset. use OR operation during iteration Two lists of PIDs maintained. At regular interval old list is reset and we make current list as old list At any point of time tracking of PIDs accessing VMA is determined by ORing list1 and list2 accessing_pids_list1 <- current list accessing_pids_list2 <- old list 3) Maintain per vma numa_seq also Currently numa_seq (how many times we are scanning entire set of VMAs) is maintained at mm level. Having numa_seq (almost like how many times the current VMA considered for scanning) per VMA may be helpful in some context (for e.g., whether we need to allow VMA scanning unconditionally for a newly created VMA). 4) Reset accessing PIDs at regular intervals (current implementation) t1 t2 t3 t4 t5 t6 |<- DS ->|<- CS ->|<- CS ->|<- CS ->|<- CS ->| The current implementation resets accessing PIDs every 4*scan_delay intervals after initial scan delay time expires. The reset logic is implemented in scan path include/linux/mm_types.h | 1 + kernel/sched/fair.c | 17 +++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 980a6a4308b6..08a007744ea1 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -437,6 +437,7 @@ struct anon_vma_name { struct vma_numab { unsigned long next_scan; + unsigned long next_pid_reset; unsigned long accessing_pids; }; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3505ae57c07c..14db6d8a5090 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2928,6 +2928,8 @@ static bool vma_is_accessed(struct vm_area_struct *vma) return vma->numab->accessing_pids & (1UL << active_pid_bit); } +#define VMA_PID_RESET_PERIOD (4 * sysctl_numa_balancing_scan_delay) + /* * The expensive part of numa migration is done from task_work context. * Triggered from task_tick_numa(). @@ -3035,6 +3037,10 @@ static void task_numa_work(struct callback_head *work) vma->numab->next_scan = now + msecs_to_jiffies(sysctl_numa_balancing_scan_delay); + + /* Reset happens after 4 times scan delay of scan start */ + vma->numab->next_pid_reset = vma->numab->next_scan + + msecs_to_jiffies(VMA_PID_RESET_PERIOD); } /* @@ -3047,6 +3053,17 @@ static void task_numa_work(struct callback_head *work) if (!vma_is_accessed(vma)) continue; + /* + * RESET accessing PIDs regularly for old VMAs. Resetting after checking + * vma for recent access to avoid clearing PID info before access.. + */ + if (mm->numa_scan_seq && + time_after(jiffies, vma->numab->next_pid_reset)) { + vma->numab->next_pid_reset = vma->numab->next_pid_reset + + msecs_to_jiffies(VMA_PID_RESET_PERIOD); + vma->numab->accessing_pids = 0; + } + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE);