From patchwork Wed May 3 02:05:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 89543 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1031685vqo; Tue, 2 May 2023 19:35:24 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5HqpphnWEXj6MfD2TJmvYk94QKfeXxjlvEFWb3xOjie0FTT2JzbEiRVF5/+wR2skfcM3JM X-Received: by 2002:a17:902:ec8a:b0:1a9:9ace:3e74 with SMTP id x10-20020a170902ec8a00b001a99ace3e74mr642622plg.65.1683081324049; Tue, 02 May 2023 19:35:24 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1683081324; cv=pass; d=google.com; s=arc-20160816; b=CTVIJk7egQhtYyab8AztVjzL1moDZrxXNgshPnmoKeB1x+yZw2ctAchO1zDqYq6eSw ogMw5U7LSDFneTq1LO7I8NT3hicmCnFIMtV7uKZVH9KB7xazug43L/3sU8RkWOjRZxW8 8ACY/PHxEGNqyBd+ksNRjiP67TGl23hSeh7JVeUq09/Mm7nRrQMv88G5pXqGNA+zY1JC gNBdGirQm15KEsKVDM2TK12kPZ0pCA3MaFFTBUAoWF1YDQ2jSwtM66pRyaqqx70FdFUQ 12Mu/Hfulk/BDnRXJuS+0zzmKfoiB3c78tvzERaW3L1duXQjrLIiYFX+a5jK0cwlOa1i oYcg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=XRFZzU6RWuHttsW+R6U0rWTSI9QV998SO9eAEjJvhs0=; b=Cg1GcVtRY9Cq+5Qe1kAvSvPBH62TNPqGJNp75rpie3QkEkgblQlTlB0K8ouqNw8Ee+ 0tTgqho8/YYAqxJnbinWI0oBYdGZU8MD9LawkPPs3T9PifbUPqyMP6MFEsu8/tNPGkEe zEVU8xHp47pbjNCpLBzZroF4lhk4tLkiABa8AeBGE+Bz+rBew32uEr0HKdfn9d00dvyu 6OCGmveYBQJoV9gM+o4qJ9yT7RhgxpckTx+y0WeDalFr9kwQjR1RPI/lreyx0kdAf7LR QynTzGR+xfXgzMd4iHrbRjso/Hna9wX+hZwazspgo3w70xoMwuHUpaj2soRfdcK1X7R0 NKoA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=1EQOKnSJ; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g12-20020a170902c38c00b001a6555fb904si1880003plg.187.2023.05.02.19.35.09; Tue, 02 May 2023 19:35:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=1EQOKnSJ; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229562AbjECCGp (ORCPT + 99 others); Tue, 2 May 2023 22:06:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38814 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229482AbjECCGn (ORCPT ); Tue, 2 May 2023 22:06:43 -0400 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2059.outbound.protection.outlook.com [40.107.93.59]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F22119BE for ; Tue, 2 May 2023 19:06:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HD8HeAy/i88DWSkG6wfyuBbUrB2m3xS5Sti6djynH3AjcdEkZRtjWeYCmFssOwOaacygPxLkAw5rLwd8czC0647UfG+F5/m8YsClKotoq6sqdWOcgNyvNfQ0r/Q/Vid9fi2n5stC4V4AAfR969l/UZZboQn7QHaEQF0EHGLYofUXZAnIEPUdIi3Ez49kqfAzsRzcabpvLtw55yG44USozsCDteiOb8Z/VQZAFoJTnIwmnvqM0co0jqHo/4droTZrD65NB6dEhT4ZHzHdyfATjtdBYSwSzWR8YPOdcf9YbXLYBa305Kl+HjPT0TUnQKDTUaLhiISuV7t/3AV8EPQ1UQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XRFZzU6RWuHttsW+R6U0rWTSI9QV998SO9eAEjJvhs0=; b=kTQj9uT6vf+Z6YdHxnG5iJSzERfWQgDekahSBh0OjHNws7aIl9s1dejIFs7lj57ZrqjHDr8147GN2cUdAavAHMO6k4M79ZnGZvfRz2I716z/X2x4AcqktL/907+K06lI9kgC4bb6VgAOoPTo4c20Jks9AgZzlqLtO5LbU+6ET25e1HZzouRzh7jHvqv9FjwVZnTQNchCKqbFaAiX9uROvq5/6TQtwoLI8vczysZXwDMy26UEj+D6ACaMjqZgjLndbiccf0OgzEZzN5jMyzz6eBDZjTKWhhvinQw0ZGKXlc+nEN/4DK5AFSJpavhgdJ6mG1F6LMsA6bvftEoH0r8a2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XRFZzU6RWuHttsW+R6U0rWTSI9QV998SO9eAEjJvhs0=; b=1EQOKnSJ8TQhWcOkcnUGphbfkYzMDGvn6/htBYVdutzbRlQXRRUwYexucICLXcb1rofi+5khDgicK27OQ7mRMkLPcax/ecfjUiVmpRNaznwIL5C4iYkeumdEBPkANpnQde+ei0KiGkcA4CU2Z5IVR0lZQ4o+ijEmwKiWmJrkTeM= Received: from DS7PR03CA0021.namprd03.prod.outlook.com (2603:10b6:5:3b8::26) by LV2PR12MB6014.namprd12.prod.outlook.com (2603:10b6:408:170::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.24; Wed, 3 May 2023 02:06:37 +0000 Received: from DM6NAM11FT100.eop-nam11.prod.protection.outlook.com (2603:10b6:5:3b8:cafe::a7) by DS7PR03CA0021.outlook.office365.com (2603:10b6:5:3b8::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.31 via Frontend Transport; Wed, 3 May 2023 02:06:37 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT100.mail.protection.outlook.com (10.13.172.247) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6363.21 via Frontend Transport; Wed, 3 May 2023 02:06:36 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Tue, 2 May 2023 21:06:32 -0500 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Juri Lelli , Vincent Guittot , Bharata B Rao , Raghavendra K T Subject: [RFC PATCH V1 1/2] sched/numa: Introduce per vma scan counter Date: Wed, 3 May 2023 07:35:48 +0530 Message-ID: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT100:EE_|LV2PR12MB6014:EE_ X-MS-Office365-Filtering-Correlation-Id: ca225477-f148-458f-8270-08db4b7b06b9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GaWWE2a8aH4wq1brpUM3aZy651lHgYS9qioOGFgkFKO8ONZpuS0qZxu/wAYlMQnqScm4Ka7JQvmZfKgkQC73cdyo0BCfsTEB9wwpK2SzqxAsSr7vrG8RoY+HR2zd5eKhDoYtnUnObDh8wb1i8pi+skofl4VDVSTGA+pVQF61jxdSUttbocsOPx1xz9PWiQHfLPvozNVS4LzUvFC0wJraK9YhgUdlPJQ7yKpj2rHM8divMvNvbczp3MSRoXZ8owe95r4R/5BYKjXNYyh1/GH750hNDDtUyint+c9CPwaV7YwF8dpVE9urbCTuQb4ZxR3SAQVL3EOfHwYcmYhoDK5Qr94nWoowh2fhwKr1uNBWN2GqsWjJvNtcqnhQhUPALTI/+BgVlWysfztLJ8Y2362PzzyXRqXdQLCOX61h6TXez/7H8Lhmc5WlUS0Xdwe6oFKHykbKIuRFEoeLX/QBryCB1QXCiib0rXurfjzm8seNn8cHUJcK+L1Y2UOu/camWbNG7mB4ZYcrYPqsRpEPgaxQnwfJEMQvNrcB35Y/UpAnqoc5UqOtHPh7PcUeeGhcSGt+7nx/DTHypbAGA6xH/0eioAsBVcTodjdpow4X5uKp8xRmguMmGZ7aFuXXt7FE3dGcYUiesS5QIr6waglcRQun4YN+zBspfQ83Sd5dNi2v9+EDnleAoQcoiY9OMCSR2L0zZtTIFJ5gNBsPS/RMfi8lRbFjN+QWzE4zR71iTiJK6X9lQOf1qQEuGmds2O5+tuaB X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230028)(4636009)(376002)(39860400002)(346002)(136003)(396003)(451199021)(36840700001)(40470700004)(46966006)(81166007)(82740400003)(356005)(7696005)(26005)(6666004)(82310400005)(336012)(83380400001)(47076005)(426003)(70586007)(36860700001)(16526019)(186003)(2616005)(8936002)(36756003)(2906002)(40460700003)(5660300002)(8676002)(478600001)(110136005)(70206006)(54906003)(316002)(7416002)(41300700001)(4326008)(40480700001)(36900700001)(2101003);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 May 2023 02:06:36.8157 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: ca225477-f148-458f-8270-08db4b7b06b9 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT100.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV2PR12MB6014 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764838681974822710?= X-GMAIL-MSGID: =?utf-8?q?1764838681974822710?= With the recent numa scan enhancements, only the tasks which had previously accessed vma are allowed to scan. While this has improved significant system time overhead, there are corner cases, which genuinely needs some relaxation for e.g., concern raised by PeterZ where unfairness amongst the theread belonging to disjoint set of VMSs can potentially amplify the side effects of vma regions belonging to some of the tasks being left unscanned. To address this, allow scanning for first few times with a per vma counter. Signed-off-by: Raghavendra K T --- include/linux/mm_types.h | 1 + kernel/sched/fair.c | 30 +++++++++++++++++++++++++++--- 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 3fc9e680f174..f66e6b4e0620 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -479,6 +479,7 @@ struct vma_numab_state { unsigned long next_scan; unsigned long next_pid_reset; unsigned long access_pids[2]; + unsigned int scan_counter; }; /* diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index a29ca11bead2..3c50dc3893eb 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2928,19 +2928,38 @@ static void reset_ptenuma_scan(struct task_struct *p) p->mm->numa_scan_offset = 0; } +/* Scan 1GB or 4 * scan_size */ +#define VMA_DISJOINT_SET_ACCESS_THRESH 4U + static bool vma_is_accessed(struct vm_area_struct *vma) { unsigned long pids; + unsigned int windows; + unsigned int scan_size = READ_ONCE(sysctl_numa_balancing_scan_size); + + if (scan_size < MAX_SCAN_WINDOW) + windows = MAX_SCAN_WINDOW / scan_size; + + /* Allow only half of the windows for disjoint set cases */ + windows /= 2; + + windows = max(VMA_DISJOINT_SET_ACCESS_THRESH, windows); + /* - * Allow unconditional access first two times, so that all the (pages) - * of VMAs get prot_none fault introduced irrespective of accesses. + * Make sure to allow scanning of disjoint vma set for the first + * few times. + * OR At mm level allow unconditional access first two times, so that + * all the (pages) of VMAs get prot_none fault introduced irrespective + * of accesses. * This is also done to avoid any side effect of task scanning * amplifying the unfairness of disjoint set of VMAs' access. */ - if (READ_ONCE(current->mm->numa_scan_seq) < 2) + if (READ_ONCE(vma->numab_state->scan_counter) < windows || + READ_ONCE(current->mm->numa_scan_seq) < 2) return true; pids = vma->numab_state->access_pids[0] | vma->numab_state->access_pids[1]; + return test_bit(hash_32(current->pid, ilog2(BITS_PER_LONG)), &pids); } @@ -3058,6 +3077,8 @@ static void task_numa_work(struct callback_head *work) /* Reset happens after 4 times scan delay of scan start */ vma->numab_state->next_pid_reset = vma->numab_state->next_scan + msecs_to_jiffies(VMA_PID_RESET_PERIOD); + + WRITE_ONCE(vma->numab_state->scan_counter, 0); } /* @@ -3084,6 +3105,9 @@ static void task_numa_work(struct callback_head *work) vma->numab_state->access_pids[1] = 0; } + WRITE_ONCE(vma->numab_state->scan_counter, + READ_ONCE(vma->numab_state->scan_counter) + 1); + do { start = max(start, vma->vm_start); end = ALIGN(start + (pages << PAGE_SHIFT), HPAGE_SIZE); From patchwork Wed May 3 02:05:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Raghavendra K T X-Patchwork-Id: 89540 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1026944vqo; Tue, 2 May 2023 19:20:56 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4L5gzjqiRn2yiqPJWEvTPYVMBN7lEE7XkWr8YVWjtsr/HbyRByDQNDcq63oXxrdNfU++/Q X-Received: by 2002:a17:902:830c:b0:1a7:a541:742a with SMTP id bd12-20020a170902830c00b001a7a541742amr518393plb.28.1683080456429; Tue, 02 May 2023 19:20:56 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1683080456; cv=pass; d=google.com; s=arc-20160816; b=IvKYckXqbp62uUF8Lno7kXAqz1M4EBipFImVBHsknN2DqqjhRCCh60ivjsgrf5eCiY rlCAiKIU3ZpK7M0czKuX9aITMZS7GoWoPmGt0YqAGdRSeOsnh7K91vHMoJBKQD1QKTU4 V4E7tvhlpgHTU7Gs3yq3XtPJZMQQkSG/bigFftksHkzQnpZqYi1OADrDayCop3636PlP yzwZGmkbqmXTr6gXOYawFu+I9Au2Jc9LF1unksFHSyjadHuCT/lmEj0RcX0hBcjyn+bi npsAsnKMd86xR/e8NglXnFhaORxZH0wIhl365zvwhUCSX0fKJH8kGPz0OzwwnbDrIxNs EusA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/NSBlCz8xCMXbjCQo62cKyUj6pVFVaXwQU03xdm2Tgw=; b=TeimlwB0QemP3wGls7275isOgwUCezeMonQ52v0Bda4LbqVHARUCDcmYYIujT/hURf ueBGgl0gAXADm3QSU5K7oW7uiXacXONl4BnWochz0lfkZg9MYnjuYc+7E0j1C7hiFZf5 MnSOY/y9RbWg51bgQBb0ICrlw0+aJ44UmazSaazC3n9oozpYR+p9JnHqerA0nNdPmYhQ +wV2ZHequhuSdgM4Hk/J3L6CVCLDfk/AnVnhp7T6oU8aT0CIKwXWMl5y7l/X8m96EMHZ kYjPic0T6jNWpuCxC7Tr74H65TfYAWE2RcqP6P7QQa2O50qva9oWgO2PcejV+UFCZ/EX Tzyg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b="Zy/YQ+9l"; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q7-20020a17090311c700b001a526bc2b84si33034015plh.620.2023.05.02.19.20.41; Tue, 02 May 2023 19:20:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b="Zy/YQ+9l"; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229582AbjECCGt (ORCPT + 99 others); Tue, 2 May 2023 22:06:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229569AbjECCGr (ORCPT ); Tue, 2 May 2023 22:06:47 -0400 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2048.outbound.protection.outlook.com [40.107.223.48]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00D4D2700 for ; Tue, 2 May 2023 19:06:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Z1KuHjiTQrqEax+O7Oi6Qf+ny/e8rxUr02tfWKch0+rm2tLgxd40QJVG3Xo1lbjwh25IzXw3GI6MORyDJz1+zgnSLONiPKfle1gEj0iwLjhk/Ot88VpY7aSwPylDcx89xkpW3xh62+Osu0QS7PCqfYVjQKxXb/DOAkUg/Uo0RD6LHfO5i7o8S10JcchcAtS7GZYSBDLx2cOcBmTOAJcFNiNwRdrGjtWUyNJiVPmQEBEESAzAciPSQk1GypjXYC2vfABIoB1huus9UJ3aFc+m3EVAZhulcC8Oss2XfZ9Y4Nk8sg2yTD0LdSB7++KhoNu9y14DJpVHbOcYPEADSujypg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/NSBlCz8xCMXbjCQo62cKyUj6pVFVaXwQU03xdm2Tgw=; b=a+3pb+bhcBSbo6PS9G1eqM2U/+Gg+1phV3cZv/0GaoMOJXUs8c3WM9vV8xUb+wWM0QYkMDS0MUDA9FBVBJzgDUtSHwB7I9I/arrOpjyfkoqZ4L2BMUqWAFX/DMtC79ASPcXhBDRNi9ESPbEt1WAnaU4w9sV3RAjtx3QF6C20Kwa4USVJ69cCeRvqx2zWW7vgZLdGY3Y1ZS1OGpKjswaVp1JKhpiiw9YkQz2EfJTKqSEcypqfSXFtdkCAHj9CMwXn9ZcMmi2IMGylIXR3SnxWRAI32IJmoMjXRQGk6VgdIVP5P9U1/flyHLK7spyApyilugeuA6Lkz5TmhaxITQiBOg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=vger.kernel.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/NSBlCz8xCMXbjCQo62cKyUj6pVFVaXwQU03xdm2Tgw=; b=Zy/YQ+9l7CXzGRtV9waTkX3XOILR3EOBYLbbyVNz2WFmOqAcleJErve6IS6VEmFtEEPaMHX87BAlGMOhRPCoPTyjES9BXVBTbwsT8UWo5bqGmeosndfkP0dIRNh9WSVV70XapfpS5ir6ZR4PbHUSHpMu7xLRjzAclXIESadsGkc= Received: from DS7PR03CA0321.namprd03.prod.outlook.com (2603:10b6:8:2b::6) by PH7PR12MB7212.namprd12.prod.outlook.com (2603:10b6:510:207::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6340.27; Wed, 3 May 2023 02:06:41 +0000 Received: from DM6NAM11FT009.eop-nam11.prod.protection.outlook.com (2603:10b6:8:2b:cafe::e5) by DS7PR03CA0321.outlook.office365.com (2603:10b6:8:2b::6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6363.22 via Frontend Transport; Wed, 3 May 2023 02:06:40 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT009.mail.protection.outlook.com (10.13.173.20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6363.20 via Frontend Transport; Wed, 3 May 2023 02:06:40 +0000 Received: from BLR-L-RKODSARA.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Tue, 2 May 2023 21:06:36 -0500 From: Raghavendra K T To: , CC: Ingo Molnar , Peter Zijlstra , "Mel Gorman" , Andrew Morton , "David Hildenbrand" , , Juri Lelli , Vincent Guittot , Bharata B Rao , Raghavendra K T Subject: [RFC PATCH V1 2/2] sched/numa: Introduce per vma numa_scan_seq Date: Wed, 3 May 2023 07:35:49 +0530 Message-ID: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT009:EE_|PH7PR12MB7212:EE_ X-MS-Office365-Filtering-Correlation-Id: 99af254d-92aa-494e-3ffa-08db4b7b091a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9hpY/bkoeOJl1NcUKrxEdLPh9lQKx9iqVb2oNVuzDoywOQIZHA+rCjlXhKg3SMeLC3lCKbVze8kmykuUzXAYXwQSZqHQ9Fo5bIsC+JUnWCOfc1XWPhp7MHiVJ8lZmKM1yas2RxUXOjsTAZL4FciCQnmpWmmbnkDrTCPh7WAczWat/kRpZ0ovsNaOQF/TVTQ5hC+nuntpT2Y30q0KMLBt+TCJiIsGAvZCokns9qxtMw8CDqw/iSxrIcWU5ScnWsXh+i0tckJCmTxR2SKjTxKN2fxFS9qdsUHjHl09yQQHifvGtEkGgXWMCRxePvnP08ZWBdUhXUKfQEP1m8IWyK7C/YDsdb+sLoY1WU9zOCOvSbaT8NLrM6uAoqnfEGH8HS2jDGDP6oacBO84cckiCwTmq4YQ3S61e5vIcJ2Uwq2kYo2y1wxeACwOz93yYhgra6CENGidFmjc1HvSsjkGeU6/sOgKuk+9kqwdEY/S0fwaCjaYmXHtSI5YvulSi4voGFFr0C09eNiWSSOCf1s0ectK4L2EE1W9yIfrm6gp2ix3Q5LmBESlDN6rBbEEHz4qVL73Gz/ccI7e72E4Ynnnyrh6mHjYAh4ceAKeOKmS8Nz/9Um5NuEWcs47Z+hFMk8XdXtox3XnJKiZ58c3ujGzQ5wWU8y9OD9S2i3aUSP3lvrf28/Gd7TfjCWAz7W2VfWDiVEO43lcBnHb2XlCBXR6RGvnoESFs38VTOlOE2nGonN8K2jkEJ9jZIKv+E/LFdX+OlLY X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230028)(4636009)(136003)(39860400002)(376002)(396003)(346002)(451199021)(36840700001)(46966006)(40470700004)(4326008)(82310400005)(26005)(40460700003)(40480700001)(316002)(7416002)(8936002)(8676002)(16526019)(5660300002)(478600001)(36860700001)(186003)(70206006)(54906003)(70586007)(336012)(47076005)(426003)(83380400001)(41300700001)(110136005)(2616005)(356005)(36756003)(81166007)(2906002)(7696005)(6666004)(82740400003)(2101003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 May 2023 02:06:40.8051 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 99af254d-92aa-494e-3ffa-08db4b7b091a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT009.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR12MB7212 X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1764837772653571552?= X-GMAIL-MSGID: =?utf-8?q?1764837772653571552?= Per vma scan counter was introduced to aid disjoint set vma scanning in corner cases. But that counter needs reset regularly. Reset is achieved after full round of mm scanning by per vma numa_scan_sequence that follows mm->numa_scan_seq. Result: With this patch series we recover mmtest's numa01_THREAD_ALLOC performance as below Base 11-apr-next w/numascan w/o numascan numascan+patch real 1m33.579s 1m2.042s 1m11.738s user 280m46.032s 213m38.647s 231m40.226s sys 0m18.061s 6m54.963s 4m43.174s In summary: it adds back some system overhaed of scanning disjoint vma scanning, But still we are at huge advantage w.r.t base kernel Signed-off-by: Raghavendra K T --- include/linux/mm_types.h | 1 + kernel/sched/fair.c | 18 ++++++++++++++++-- 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index f66e6b4e0620..9c0fc83118da 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -479,6 +479,7 @@ struct vma_numab_state { unsigned long next_scan; unsigned long next_pid_reset; unsigned long access_pids[2]; + unsigned int vma_scan_seq; unsigned int scan_counter; }; diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c index 3c50dc3893eb..dc011a2a31ac 100644 --- a/kernel/sched/fair.c +++ b/kernel/sched/fair.c @@ -2935,6 +2935,7 @@ static bool vma_is_accessed(struct vm_area_struct *vma) { unsigned long pids; unsigned int windows; + unsigned int mm_seq, vma_seq; unsigned int scan_size = READ_ONCE(sysctl_numa_balancing_scan_size); if (scan_size < MAX_SCAN_WINDOW) @@ -2945,6 +2946,18 @@ static bool vma_is_accessed(struct vm_area_struct *vma) windows = max(VMA_DISJOINT_SET_ACCESS_THRESH, windows); + mm_seq = READ_ONCE(current->mm->numa_scan_seq); + vma_seq = READ_ONCE(vma->numab_state->vma_scan_seq); + + if (vma_seq != mm_seq) { + /* + * One more round of whole mm scan was done. Reset the vma scan_counter + * and sync per vma numa_scan_seq. + */ + WRITE_ONCE(vma->numab_state->vma_scan_seq, + READ_ONCE(current->mm->numa_scan_seq)); + WRITE_ONCE(vma->numab_state->scan_counter, 0); + } /* * Make sure to allow scanning of disjoint vma set for the first * few times. @@ -2954,8 +2967,7 @@ static bool vma_is_accessed(struct vm_area_struct *vma) * This is also done to avoid any side effect of task scanning * amplifying the unfairness of disjoint set of VMAs' access. */ - if (READ_ONCE(vma->numab_state->scan_counter) < windows || - READ_ONCE(current->mm->numa_scan_seq) < 2) + if (READ_ONCE(vma->numab_state->scan_counter) < windows || mm_seq < 2) return true; pids = vma->numab_state->access_pids[0] | vma->numab_state->access_pids[1]; @@ -3078,6 +3090,8 @@ static void task_numa_work(struct callback_head *work) vma->numab_state->next_pid_reset = vma->numab_state->next_scan + msecs_to_jiffies(VMA_PID_RESET_PERIOD); + WRITE_ONCE(vma->numab_state->vma_scan_seq, + READ_ONCE(current->mm->numa_scan_seq)); WRITE_ONCE(vma->numab_state->scan_counter, 0); }