From patchwork Fri Oct 6 03:20:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 149092 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp61512vqo; Thu, 5 Oct 2023 20:22:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFE4un4hunzbazOI5Ttm3O5odIPJytwA/W6X5MCxl3P3fgrxPrlMg1E/GeOj6jyZ2HSWoHA X-Received: by 2002:a05:6a20:a11f:b0:15d:d250:d240 with SMTP id q31-20020a056a20a11f00b0015dd250d240mr7936244pzk.50.1696562545150; Thu, 05 Oct 2023 20:22:25 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696562545; cv=pass; d=google.com; s=arc-20160816; b=jXOk+C88vnVYHQfkdjYB39/luNieaBbkiOLJICMXGO8un42TAptkBseqfEmjE8iOdN i0AR8BYsteRRN+RwJIXCqESGo4eYfDVodjdHNcbCnE29RJ3M3aLWG8JHCtnkWNnO3TPh jBuVfVYyK5Nj5qJQDAUCfEH9wwYdNBdyLgANt+kthoL8KvoWYP5qBDRPeRfaW7g0cVOj 9PJ6OmfkmjaeKkDIDcLxWIiMFTB76H27nplhsopXKeOJjiCBA2fX9reBSFR11kcDmEhz h1SU2z025PYPNwzxRDrIpQnHto1bDEoP096nXGrqF974tQSc+grudpOTS4JCqE+cLRZY sjVA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=yWCzkeiwrw7E7zmdYoM/+u9CKp7c2Frl8b8zefaOlJQ=; fh=1vpEKG5MNy2vyykgaUfqR+bhVuR6RQQ9ADpwA0vfD/0=; b=DWxKuW5HbD6ywK/7AdBnAX9lFcWHvcCtLyhHCZCkZFyjC3fDHTPWw05Qe0M1ea3mS0 l+F0uUk/I2Os3lrZuQxrcVhPKbFsrRWdISWoXte79d9lqB40Qw+WEXd26bZpiqIiq7x2 cZG7eM0ZSG8sWZsqd45FiU9WFz5WMu+SzqGrdXNJM5mvpORPbp67aZGf3nGyj0j59E/g OIhf+iqEK4Le75o7Cnr/NPd/e4JpT1LRPvcUVx/51ydoGzcFOjKnM8FWYlGpF+d3+Oui GkjHYrRb6s//3/YhiqvHU32mCflDEdW3dR0CH7sKYMbt2d0dJ9/xBqjPZu0oApLVuvT5 Wy0Q== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=ErFHN6DH; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ohwveF5C; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id cq24-20020a056a00331800b0068fba70d25dsi586258pfb.33.2023.10.05.20.22.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 20:22:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=ErFHN6DH; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=ohwveF5C; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 25EE2806BC12; Thu, 5 Oct 2023 20:22:21 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230083AbjJFDV5 (ORCPT + 18 others); Thu, 5 Oct 2023 23:21:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229920AbjJFDVa (ORCPT ); Thu, 5 Oct 2023 23:21:30 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7AB0DE for ; Thu, 5 Oct 2023 20:21:25 -0700 (PDT) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3962NikX027817; Fri, 6 Oct 2023 03:20:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=yWCzkeiwrw7E7zmdYoM/+u9CKp7c2Frl8b8zefaOlJQ=; b=ErFHN6DH0CjI6WYkViznXYwspj5E/1iK1/JG7CMCJSucvkNz9wK9UyJEFMrCY+4fiuuL Ibbe/UTZUeX7E+569HmhlarZVdIQpOkEMQO6QCl58L34lVNSMQZ1hB+yYJCq3pXWvpe9 oXca5mWOTqCSmxWsRUY2CStXf+BlM6gedwGJ3GpA0vx8qyD2LKCus48kweTiLAjAfvsP 51ISE6oamM9U1JMP9SuIYVqYKUJRLug82hAqIBv4hgcsSdX98XgPyotk8O9zPkRCjDaG +YF6WmX8mnfG7A6ksC5FKchS5Zr8924XtVi1abMWa9BQb4lBErcEp0edAYlgn2LkctbJ vA== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tea3ejyap-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:26 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 39632iwP002972; Fri, 6 Oct 2023 03:20:25 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2169.outbound.protection.outlook.com [104.47.56.169]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3tea49ypp5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:25 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PbkGBnMM+Mi7zIqYiY7Xgvtuv4FDZi7ODEOURs8bnLQVqxR0RJ1zSdS52vfVLWVdvY9HPvi9lEHMzN/woUsbjmvtlTJ6Gy4NxUT9BTqQ0rxr3QJwIWpQ33OU8qT5V6dlo1wmXK9tQwlvhCpexVAszjOoYqbay9ORoLKRows0kqzWjrJeKLMG6rONe5YYO/ISxsJp9pBdnp3J9hiwaAG1T2u9BsaBdQ2qlwcX6GiWBgvO8o8ilACf5/H55nhFE8fity8lOfsg/oOMOMvDLaTRtv18m5516gbxq4x3O4mA1nkCx0d1Xn8GyBOEtMbTUklgAh85XQoMGVfVmTHkvT6Zgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yWCzkeiwrw7E7zmdYoM/+u9CKp7c2Frl8b8zefaOlJQ=; b=HUPBR9wqfB1vK2Th9vsFP8bh0/lZrn4qoEikGjIFp+JQrplDQGdtKk9CklR4U+xSAV8xnVBtjzEFQZ8fiBLZRGQYLBzMAYU3De2ar+zOJahZoLwhSz1xgYoTu8o497wj9GfbF/rBsduAdPO9K6c97YNwt6Z7B2byyw/CVxskeM8TtWRIjaBfxZODWQWNWgEw/TwIJmFFKM+3+cKo+48jyQeydcdhb3igIFzCnzRU/JKqsrlOKzZVAtM5MipXKd2ludOaEoMebUbq2JUdvNFXOlp0+Aqn1WaDnkPOHJ/b3XSFg1Ds6HIKfKxOnjZz/UovkBHPOeeYqPa5abuNMZhd1w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yWCzkeiwrw7E7zmdYoM/+u9CKp7c2Frl8b8zefaOlJQ=; b=ohwveF5CBGp9RVww7j17AGxt9fv4bbeH7xY8TTTaK89Opd+SezQXyFkMtDS215cUvD/6v5sLTooflYvhnBSBLc1Ad47qoe6H+tiR4tH3cPWlOw045CQGfzxftbquH/W0cvZhtRksdb5z4V8+9rTb4+0XO5j3OVpdi4Ac++LRqWk= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SA1PR10MB6615.namprd10.prod.outlook.com (2603:10b6:806:2b8::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.38; Fri, 6 Oct 2023 03:20:23 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2%6]) with mapi id 15.20.6838.033; Fri, 6 Oct 2023 03:20:23 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Konrad Dybcio , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz , James Houghton Subject: [PATCH v7 1/8] hugetlb: optimize update_and_free_pages_bulk to avoid lock cycles Date: Thu, 5 Oct 2023 20:20:03 -0700 Message-ID: <20231006032012.296473-2-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231006032012.296473-1-mike.kravetz@oracle.com> References: <20231006032012.296473-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0244.namprd04.prod.outlook.com (2603:10b6:303:88::9) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|SA1PR10MB6615:EE_ X-MS-Office365-Filtering-Correlation-Id: 808d10d8-a751-44ac-a47a-08dbc61b2d44 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LU+MksQPgRH3m8lI3fLDSEeT5cpAQ+EFJD4GTQJSVnwr9I9d4R0qVytU9SvZ/AMM7HPESkJdhY6WsLqviHR2/XMK5aHxncrohde0iGSCY7PlQbHftfhStsUENBa7mAXPUiFyx+T1y7b5xF4RHb7gyZFDh9RDXkbi8ZG6DbbXQ6jw1TPrQD4tkKRO3bjVrwyD1cvyXftB3zWU/aWZCyULtMuP/jbXR9H4R84FL4Y151ihmk+UOFEyU41/QKuqExgYWOGx8EtLFS6muM62wDCeiZ0GtydoD2/P1yTWkXDb5bnuV1CPRfrf06G399PWoujfziX+lcVX4nsGvq1uRYip/kZLKS9+2QpUAsrdqd1g0uG7LdsLPepuCPpGUX9MEb6IwwE31TIlg8z0hZDa4Zk5TwKs7qCUl2bpXmZm/tKFbPn+NdOvgSLFok36s6pU6wkIFbNl7nRPlsmJdYRsqgjbuNPck47Snptb8sJcvPzkY7ua7S/XgSlz/6jT08rpVfHF4orqIDIyqAPcDib9puEWVh0wihMQPZBZdqutJ+nijpd5r8avEApYoik1SnhKU/+j X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(366004)(376002)(39860400002)(346002)(136003)(230922051799003)(64100799003)(186009)(451199024)(1800799009)(316002)(54906003)(66946007)(66476007)(66556008)(41300700001)(1076003)(2616005)(86362001)(6512007)(6506007)(26005)(6666004)(36756003)(6486002)(478600001)(38100700002)(83380400001)(44832011)(15650500001)(7416002)(2906002)(4326008)(8936002)(8676002)(5660300002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: gbX9EgG9E4uTEYIUU4irzj3U4mcv18aknhvQRsTG1OwPH6wK47J1UPj53mvMm/w8AHqPY4TMbS7nPgpIVFBST9nclezWgXJiXr1ni4J8VGbgcSSOKQOnQJNYBRguidTc0qlHpCYuM/64Vb/wIWubnPCBNNqTZMyA6KB2qsQ+y2g6Rxjfb9PG2sN0kbwbuMZAhADInsJrZqXTswz1SYIQ/tbegY7Ker/X8ptvrf8kAxnJYpIn05vYURLbK5UalsPB5i+cUb4D9fm9Safw3I0eYyX9sL1ucHT5kAlTYglAKfPAqpn9jpXUmOT9AlptrpZxU2FQOKKZlEB/SHDhBjqmBPorEORwf/4kofjJcoQi/yzvK1RYnLNanF8Z7R/6Bd77I9TJT4ikCvJ19/jE/sCvdj5dM6q6fdoc+GWoDbE+bnDJ1XnSGJouWkaYZ54PT3rSj3bHYMrQeNgpPnvgqU1d5JUY3EpjUAMet5YkPYbcVYAqvWdQCiVgxAt420V9uQpqukQhiJO28y/xRv+dZCxxX71bHkWlu6mF7pow6t39bGnwAKztKtwHbUNid+7LB3g5sbeDuZxX1qW0TdQAYf2+Pfevg+IW1dvKEtqj0UPVk7dXtv3JbLQ6d76e6yNb948QmQBXV650pA4MPFsIjgdagu3BlVbIDn7ZOrbUQG+QkQAb3YTtawLtGdyVEPjKQ2RCy/gc8VTCpeaQZVy3WZwjMSdWwr3tQeqU9LHNkGhuvZuGznwk0ruK1rxf2kGuAhVba50WV7iC+1PAt/4CcelG7vJkTtrFKKxuHPhb8Ti2BWpfILvHfw7jupm0aVrjq96MehQUnkM/zFhrLP++GiTHIWBnqv4LZvnXmENcDcvq289A3WjuC4OAHhZD/ieKSU086WqwprYMQl6XNIfnYQMAiL/jSbe1cqtF9rpara+Crwf5+hJOqQefYnmmBFKxglnYvt+++H4CIvUF2G4j2n9lo4hYz5n7FGdMVsJyej8R9Jby57XkAzUhbalpK0vGZlEaMbMWvz37tMWjwjVexQBsjF6o4E4mZZ4dwMt9BYrFkYjz7kJwEtTYgUrMt58uxhJ1pL7nfCkmgV1y8m4WYRIFu7M303Ga2hRF7UTngDfOeC1mG7R0162S/JjK83xWGjQxWZVE8F1KeAFDf1izvOAJX1owk8OevfR2npEWxKkNjFA+Vi5EZSQ7OhspgHLU4KFNRIFmZLVD9I4+ug5NjeC9I25AeUj/wem8zc9QZ0VHdxaAT0O8Ou3XTDePbIRXI+jkBVGm5fAIE/yUj2YztLdzTY4IEvqC6Yn9CnkQP4z7RYKbhWXHPFT6QWHgy8kyvYLJaEGVd+M0IJ81ufDevWTa34tZjfPDHTerQ4XPhbkJ5yc7762RjqeOKE0JcpsYKYuUCgJqEd/a76OONDpyQgLbY2c3CroTabn/lqo4VMLFeMbARrJOl6vUuhLH7C5AVImVG6xGFjE1eP4WMfJBeGClI20CuF2XFHerbww+EkS7r7l9vslI1rDQZC25XBVwBgSkZ+fs6nT6zZm6zREx1t8QB/tS9QOSUVf1QBtjvUdzClJl6vvUonbuUBa/SFmPVsa7 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: +fI7QeKj1IWlcrDz6kQRckh8/gjLlKdMesLWUF87GXYZv86rP3EE8Gs4nW75wwyu59bRQIwVvvvPGJanUPgSmfqEhupm2o3KaKNcJgTcjv4UKfYW659aOOk5anT69Jg2hc1RQTogvuzAfjDzlFCm3ZFUEzQ209TRhiGCJw38SfFxeqzNg5/t7Lxddbj5v1eeA/hA2BMya5YZ9fOvCihkE5m/awjqJ4EAVvws+hMrvCmhyASurL9D9Pc1E+g0tXosbBwlf+GYs2afsPsnrNpbPLxM7B7AxKJSk+uf7HdxmyVY/ACzIvA31YZ6gfODue/hapN1IB5Ab+TGr8ik6ySFuaNWud/1cBtlXoQieArwTEoKI0skKmMzEfLr0aJXTljHz12LNnlo5nUS2+TPn14l5C/mWkQb8QacQb1QRLxSDRxj/ihyabpCAm9cX1iGOsHybOV1FO31MZp1DzI7cy9bSpzEDfrNcixNsvy3MxJKpXMQH4mU7sF49cAqTdQjsvXovaAehgs85aVWN+K6fC9rMJ6h0W49ai1K29BNSlK3lZdCT2HBGlzSWR9ND1m5rXdJwHkR1ULE8JK8COYoBJ5cg5RKfza7eCOjL5JGZfEn/Lym34pd9H5X0FtNdMQ6k6hguT+BUOd9wcyILFdrBGeRPxAd7SH5soVwhrJvDM06w9h64T059EbmdV5n7QKPxu1AvH6i1nvsShTODBhUlzzmW7Ix4Fx3WMFemjpoNC56oQlSziFFunlcmx9/Tura33HjuuPr/uwS3V059au7XRsgSboSjyHD6EhSjqoPI/lXVlJEU63j7vkjA6Xvz87qPm/FXV/HBS9yxM4N10YhGACeERX2etBQfjLQBwC4rBpNWCe0yBgaSq/uKdFwjAx1vB7p3mzjrFgPQ2Ej8xtW74vHuBiArbiuXG7XawsI7r2C6ZxM/h6YEJCewWiyU92LqzDmMFJUli4U7tLLMKwyKpJqkOwKmjFP09qTWQO2xqpe6ttd3r+9GP5gzq7Ckobc1r+SYAF7kxRwb/R0RyC/oNyYjpvo01zzJACdgygcUY5ViaR+hWb3BPdzHGVDJ0WhIjARXWeXyjH6cuMOPfLk9vqJqPcbJ7frZHcpfTpMJmLEaKuHBhWDeO+epevNrAWlqBSa9Udb5rL+A/t1c/sz7WD6fw== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 808d10d8-a751-44ac-a47a-08dbc61b2d44 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2023 03:20:23.0321 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: n4Xi6UUWRCzUVqX1FUSsq2KaKY1S3DqeIy7x7kBgHg65Rgli3b/mU6HsuPgK9sgTWNkhZhE0MWEW2G5zrUW5tw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6615 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-05_18,2023-10-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310060024 X-Proofpoint-ORIG-GUID: NrqZk-iaAy5Mme0Hx-MT1mRc6TmxSkx- X-Proofpoint-GUID: NrqZk-iaAy5Mme0Hx-MT1mRc6TmxSkx- X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 05 Oct 2023 20:22:21 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778974767237964617 X-GMAIL-MSGID: 1778974767237964617 update_and_free_pages_bulk is designed to free a list of hugetlb pages back to their associated lower level allocators. This may require allocating vmemmmap pages associated with each hugetlb page. The hugetlb page destructor must be changed before pages are freed to lower level allocators. However, the destructor must be changed under the hugetlb lock. This means there is potentially one lock cycle per page. Minimize the number of lock cycles in update_and_free_pages_bulk by: 1) allocating necessary vmemmap for all hugetlb pages on the list 2) take hugetlb lock and clear destructor for all pages on the list 3) free all pages on list back to low level allocators Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song Acked-by: James Houghton --- mm/hugetlb.c | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c1171801a3ba..d7679d37d072 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1830,7 +1830,46 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) { struct folio *folio, *t_folio; + bool clear_dtor = false; + /* + * First allocate required vmemmmap (if necessary) for all folios on + * list. If vmemmap can not be allocated, we can not free folio to + * lower level allocator, so add back as hugetlb surplus page. + * add_hugetlb_folio() removes the page from THIS list. + * Use clear_dtor to note if vmemmap was successfully allocated for + * ANY page on the list. + */ + list_for_each_entry_safe(folio, t_folio, list, lru) { + if (folio_test_hugetlb_vmemmap_optimized(folio)) { + if (hugetlb_vmemmap_restore(h, &folio->page)) { + spin_lock_irq(&hugetlb_lock); + add_hugetlb_folio(h, folio, true); + spin_unlock_irq(&hugetlb_lock); + } else + clear_dtor = true; + } + } + + /* + * If vmemmmap allocation was performed on any folio above, take lock + * to clear destructor of all folios on list. This avoids the need to + * lock/unlock for each individual folio. + * The assumption is vmemmap allocation was performed on all or none + * of the folios on the list. This is true expect in VERY rare cases. + */ + if (clear_dtor) { + spin_lock_irq(&hugetlb_lock); + list_for_each_entry(folio, list, lru) + __clear_hugetlb_destructor(h, folio); + spin_unlock_irq(&hugetlb_lock); + } + + /* + * Free folios back to low level allocators. vmemmap and destructors + * were taken care of above, so update_and_free_hugetlb_folio will + * not need to take hugetlb lock. + */ list_for_each_entry_safe(folio, t_folio, list, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); From patchwork Fri Oct 6 03:20:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 149089 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp61347vqo; Thu, 5 Oct 2023 20:21:55 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH76yQK6dwH8VHzKajxkEuNQX1jP8ZVciBHTLmTXsKSZiYs1yzC1fmfXwyXoD3ZtMc3QG3N X-Received: by 2002:a05:6a21:7892:b0:15e:96d3:a31b with SMTP id bf18-20020a056a21789200b0015e96d3a31bmr8294786pzc.39.1696562515081; Thu, 05 Oct 2023 20:21:55 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696562515; cv=pass; d=google.com; s=arc-20160816; b=fCMp4JKlWSQsJL3L9fiKvZ/2fCHcKC3cWY/kRG2+V9NAzS/PN3V6GIEt1fUOUI2t4U qfF/MzFGUnnYxcNARKFnyHhyzQ1nVbzbMadrZrJeleNwEXR+6S0p85IPaPiNbIJ9teFE F4OQN9pm2kH5hnLexrFeL9aHw3nlJcpXZdb/CE5j+f4hUUUEt7UOIj6lPXzlov6c14R0 /s15RwO3QbsaMmUx53tmXBwyawGNgk08PytyKEZlrzz/Ldmsi+mgZ3iUz/Tll0q05+je eOz5bnLJw3Q4noc+IqSzJ5Lo6ax+m4xk9LSh8lBaczr/4M/DNOyUdIikw1KII3L3Mwht wX0w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=xoACsKEuLfdGzaswTJlVElsJTMkB6WtgGWU8LDfGKEQ=; fh=d5TJywUWx1H4J8STZuTxM2xyaY/BETP1UBu+8XkSxoA=; b=fcweJe4VVcZ0mSVQv6sTQCZxFT+ndeQPldaJoEF4TsIE+YcTUsnPTEEG36hYxIzRDg 2pvd6egFskexoHIXGMH9h4111q37mWHuu/Zrj8t6M7FZVkN4rIWbrzx3TBBM/BpLkxoO L4bwZbgQneYJ8Mwv/qtTtety6gt+074foiv/wkvn4uwQgeuu+uL3Lz0kMVyTME+n+CAZ l5/aoGJ9fs27jjav8lkz4NpY8gWH3Tl1Sncdv1/0NWkkwKhU5DzZwVMyiTdBTyYXBswE S0dtwoABNQtbcWNhAZR25/X5KTAmZn2k2RwAXBEVqfT6Ip9Nyi/o8zuGEHBFsS61PgYl cIfw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=YF1CbUhN; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=RiMHOq2J; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id e14-20020a17090301ce00b001c73e886f68si3035183plh.316.2023.10.05.20.21.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 20:21:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=YF1CbUhN; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=RiMHOq2J; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id B8C4C8343505; Thu, 5 Oct 2023 20:21:53 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229973AbjJFDVh (ORCPT + 18 others); Thu, 5 Oct 2023 23:21:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229870AbjJFDVY (ORCPT ); Thu, 5 Oct 2023 23:21:24 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49F1AEB for ; Thu, 5 Oct 2023 20:21:20 -0700 (PDT) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3962OHKq004811; Fri, 6 Oct 2023 03:20:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=xoACsKEuLfdGzaswTJlVElsJTMkB6WtgGWU8LDfGKEQ=; b=YF1CbUhNRlcYBGRQCe2pCeqQnaFkJyUqW5NB5lroLJFmULY/Sk+p2vAdCTATvMu0RARA OJFsmLVWX7OT3S188RbuZbbE/ER1903O1/wyUHRriotGNnbycP46c/r7ufkhtMPCEc8w pzSpQ6yj8w0H6Oi7NNcRDcK2pXfH7RIuxM/RChEmwzJCMLjqvVGPsL4wV0e4/aBWG0Do W8U46qyHAI1UTGKOOJkVxYOPMrU62VkXBni0zHnK+twrpFv11DaLNKx3kruhVK3Mi1t8 zCjcAFdbHfpUWakMU5nFICp4k+DPmzFLcyq1MqmEwDx/nj/47860n9Fs/QQ858bJhQ+b RQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3teb9uk15a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:29 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3960X2DP000467; Fri, 6 Oct 2023 03:20:28 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2171.outbound.protection.outlook.com [104.47.55.171]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3tea49xvyv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:28 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BObRPbY1NENFSsKHphpwfHGrQ/umd1I67fYh/eV4p5BPLDIyng26MLmGxjiuSSoLMViPccSbgK6QrlrmZmKmi1QL7Yr6616IsNdd9JaGSHfeRYecUt3rFI01CJI2E8CV6qF/CgADu6UF9l6P4vFf79zCdwTUY/NOkRWN+i3hn2bFcOUb5DYwmZoxvFjIxpp3l1hECt++NdGZhyMD8V3dHLN5OJg32kYjpzNyQ/ZUZ3oC9au56QEtltYL5EU8LobjNup+EXQ1Cl+EPBwjyHIvB4epTOZz0UV9EJHjQfXMlwo3U6c6e9xnu8O4fi3njIDeJVOrc8yzrZSPHauCAsqYtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xoACsKEuLfdGzaswTJlVElsJTMkB6WtgGWU8LDfGKEQ=; b=Vv8xCcH1QN79jYJ4U+QDyIC7SZibx9V5x/BUnIbdcbPVuOqsEtDbOeL6bs3LJCok69JNhTBlrdUsp3BTE4JJwXDkS7tLTiR0sk5ZSf0tpV+DLH540x7R4UhSC6hZ/80UHhdmi+7u6KFL3RP3B7KFhEdDK1i9foWIeepvbpP6tDO9PNA+tcR9iC6JDE0cEncBuCzMrSntSHVv839moIEgy9u98s7MASgLkisHxC763LU0TSxXftLLQTQof5KjAErpv8JcP/JISXUHu/5IGC6t1IiIQ5UsZOAr653YWVjCwSM4QZ+U1NcNH7OUA+8OXo+dsfYCFeToskUyG6ND9AkBwA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xoACsKEuLfdGzaswTJlVElsJTMkB6WtgGWU8LDfGKEQ=; b=RiMHOq2Jj8umYsIfxgWusmKcR1/PPKhmEPBOcpOs8n3+O9HvLsa3/qy0UzTbX8OOVNWQYOcdx7vSAZQ71h3s7nWPzS2TuKny4LDnOsQWsPONdO7TRvW3XvbndIVJdV0ntmxfE+iL9G0bsqlKsjXmJXXMnljIh+IwvNbvh2mBrY8= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SA1PR10MB6615.namprd10.prod.outlook.com (2603:10b6:806:2b8::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.38; Fri, 6 Oct 2023 03:20:26 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2%6]) with mapi id 15.20.6838.033; Fri, 6 Oct 2023 03:20:26 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Konrad Dybcio , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v7 2/8] hugetlb: restructure pool allocations Date: Thu, 5 Oct 2023 20:20:04 -0700 Message-ID: <20231006032012.296473-3-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231006032012.296473-1-mike.kravetz@oracle.com> References: <20231006032012.296473-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0098.namprd04.prod.outlook.com (2603:10b6:303:83::13) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|SA1PR10MB6615:EE_ X-MS-Office365-Filtering-Correlation-Id: 3b8ea91f-2405-4158-c1f4-08dbc61b2f61 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1SjVoW38GIdNkUWRiIh7mwCynK06x3jn4QMzXerEpf7dHgkbyyyWdL+CXUWB7yu8mGz17aCLBfIcEdqHn6gw7WPLkQnJ9qDpu4zrt5D1/aeyiLbHcC1sgyUyMSUTsH75FbKs3yP+O2bL8nuFPC7pLwc9kjH3xi1zoVqJBNQNW/vM1gzvDFABHEsY91IukpVEhV/ma2mAQBGuUBLaPqyeg4UA1YGLEki7DMqn90XjnIUkNaEMlWv0As8WIfapBfN3cIkud//Zb0jSFlOMw6WctkBUijQ2nfUkYt9AtBZo50TGqMCC+n8wC1s33dEUTCAKgyHUHqxf4masN9qF1wLcYuepVDI/1TO/itWG/E42Dz6wMXBHIKCqV4bOnjSKS5+0o8ESOUKjET+vVl3+6Pp5rd+SzECIdFbwFzkDmoiILISx+hL7AoArnQdOKzq4ZelBwrvYeB18Gv6NI+nHyp8kwIxKd+w1MDWcNlOw1NfjU+oL4J3TAP2XHQ7hDiAzRw3wu/tBgdsHpURNkvygs17rziKjQ2rQp5YcGOEyK+HOD6MaZPmiJMu3LQ7x2j3BYcCP X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(366004)(376002)(39860400002)(346002)(136003)(230922051799003)(64100799003)(186009)(451199024)(1800799009)(316002)(54906003)(66946007)(66476007)(66556008)(41300700001)(1076003)(2616005)(86362001)(6512007)(107886003)(6506007)(26005)(6666004)(36756003)(6486002)(478600001)(38100700002)(83380400001)(44832011)(7416002)(30864003)(2906002)(4326008)(8936002)(8676002)(5660300002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: BJ5RM90mkTVE9gA9QXRR4aoXZ0bETMWZwsm2TMVCbu4mBbTWv05Pwg3Cy54vUM35xwlL9HWVPiAh2+mCLcFAp4c0f4cc9vLfvlODaPMEP7mjQjx0hQ8dJe5lc1MaoedWp5vVUWls5dOzrClFuFKk+sUmvlhie3Fj7qyspe1v3Pa4SxRfmatxgO1cztAKsvQlrXEa0jcMm9C1ftW5xKIw9k3TAmxIgwrYFRmye4J4hTHj8LV2Ty8x+XkjLIrdDexj8Jhm6JO5S+YO5NmGtvuD/Scbc1uSALAN6HOIeXBauzs+eIGpjv50SP1xrs+4xVYpVdk89CQwv80XFUFTUPAJ/J1Lyoi1pV7HoSWbWk12r/ibjjXm8JsyCiSq86GZwv+f4Pb5XlPH+R7Qg6+y5dJQjYshVZs9/yhFYxskwOl7JT+6ntDAlgirLfh7IaVpstOqlNgUe1XBgkOnnFe2v1J5olnW3uvMswEf09ATRv0/JrR43lFHtDwxPYx/aUJV41VGsvSjxzH+zMd2oxCsWQ/oZqVnLSUJXzp7a0rDXDbBEM+WkGEHHmSEnX2xCRlWOJzR1+Ifvnkp80UCEl/kCjRre2ax6mo312ePTkwzw7KkhxwSHi36YoGp6xBuI00m6PWk8ehQIfl2sbhncluUK2g3DsnrNoU1WBtz4+Aud9nvdtA7O9pTKZo3dlUpobMTD2tbGjxQf5P8/4mSII82YEGwRscR0yAJVD+mWHrQ5mqptGXyrhMYZvbxojiFjT9sxW/sLTR7T+mfIoNaUGchJV+T2UmOJvfOv5uXeE/ulgquxin3eIpI9pqN4t+xEAtZm37kqPFcSfCVHBzRJ3QW/uwg2/ZlhTvkekz0k5Y1IIQZFrZjai7pzlzcUtCU5iiRT5FVonJGeQG1Fop+/Jw/WhMWzpEB1+CP04aPTZGOPYu5nIX3LUlfMzXZg7HBScV79mSWUZHwFCvnYFXKlhPkEdilIvOTD2vU0MYfpW2f+qw4Og0AvhpukDOA1QIyCH5Pp3DhDDNrhm75v9JdfVLhH2azlxOURQhrPBykUxu1OO9xd8ThYG73o2SNjztNDcV7jPKmFXrOUGNf7FKAcU/xdcIht+g5lMCDZ1UKfo5qrtbi9JeSa+iI4wLU7mmrS0xgfaa1F6ccal0Cu0MuwljrOTLKfwL/dFvKaPYBh7gGTdF2SPU4wG63tNFGbE0BaU4UV2YxZMPCGR7ej5tIx4qgf44Qc6zt+Bcqy5OyVZ4s40lgx1kolY2JfFzAyvqYjXVwDgPkyV7nRA2XJy0kq5acTb4MZ/fkWJiEuNn65O+r3ufvkgX0s/PXme7hvLE3iLNT08Sedps6ZWsjD/3Wt6ZVKvd4+cBMN7jZtiZOZZ5zwO87HStpPVLzfIKVzyYwJ9grcXbBb3PM4UEUvl+/0TOfo1/FjTTu4C0/i4bAmppGFeus3PGr2sGoC+Sv502PGqBnt7fpfgHwimJje896Kd7QOjgQdYDkCF4w5DKrgTcLq6U7NzvhwECRyGYfJf9cjKadMNjD2glIZPDGJATDPhXhtgr3/nUNaZEIOO2FbgwCeJhBZ9B2yDMyVMn44kvyVDY/NCG0 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: xVb4yXmg1Eew1PjHBVPoCCMohd9f7EmbYJaOJAWmjkYOIbO0Au2hdc1YZpELo8ocFiMynv+JJmVkJC4lXtNnLYejFC4BhHejUzlbV3cJ0W7W00lyEQ4/NuOC+gDKR6+rFgvmLrJYAh64qP5JrOiAXNX3VJEeDjdJpbBfAzTAwOI0WAWhIh/72djgdTxFSnyj0/Gs6LJNA9QIqnFQ8+uEJhe1lLXizCyQ0DXW3iVNRSd0hAIY2Vqu2vgyqPmTQGdPdjVBkLmdVKPb+FLUK2+sy4RPyWZcgfYe7Rv+T8K7SwNf3zVe3cMTSaZZPPETXQb8IjKiac70b4Sn0JCORQ8iI+AeQbG+Bf7M7KjvClml7t+rbl92quI5JoHXy82jotKuRZLbw5Val3Fr7FXcpU4w8WjcC+4GdTrd5tSAUNkexodQjf6dp2Y+Xw7dDtZrJgTUdjUujzDVe/QxDyqgKCgRm6rqHH3n4RyTnxPtawzbsz9sSiTI9j/0lmK/KObflfx2S76tPLFsqPIJZIiFXtHES4/LFY8L2YQE+1ExKPdoKUph1Yam3sFebfX7KltxAIpUM3hLZKwmwLoZfWSYzkcu4QyREwoYB8LG4DUHZLTvUl4OvYgmHBhMYFoOINE3MlETkf7CXMhluXeXn1s+CK49+e1DwGd/8WvMuQ0IfSWDfKe8sXg4yZfP6k+zmME0hn+7mLdGCsn58kuxT2/CL1ceTZ0v27D9nHm2v1ZGJAGldSx5TUI9713DxTtX3eReXVoUuf5RifumD1HKS0QyIRTWSG6u4HNlDV8HMNpLxp7LTQKkIMXannbwFzLHW4bBN5eM3/42K1owL9zD70ali3T5GEtzyLudo2uqQHb8B77L72ePVftCihCapH1J0XbvKIqEbc4uFLaZ6e/jHKlB12wQJ5S9z0pdeZX4uGiq7+WDC2gggNbIfxR1XLU/Z9t+b4z/d40JiYUydAMS2CRTZPt/rUMKl18Qhm7QmMPy5n0BWvwx3TZqDd0Aajc2XE/BHFG6Ah1QjHnZAWkf2hgzgEW6NgnxXUEQF413RIpqQA00zBVsr4oF/SP7mt/7+P5I7xm4TiFM1PAxbRRBbQyzoB8XEyjX9EyHrAwL4I6ftt/NHMmzGpa6wDzBuUmfBisuShpj/FccwKMOZeLd8KK3u36klg== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3b8ea91f-2405-4158-c1f4-08dbc61b2f61 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2023 03:20:26.5812 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kSjA2nueTEoNuKK/IIP9UDMUMMHCSusV9AaNnZkJemfnJDQ8J4gcsqOKaMk++C1giHTT2yGbgBH5ETrmzfrnfQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6615 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-05_18,2023-10-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 adultscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310060024 X-Proofpoint-ORIG-GUID: YnXg2MWcJqtoZt3H-8LbkffOaDQ4f8Og X-Proofpoint-GUID: YnXg2MWcJqtoZt3H-8LbkffOaDQ4f8Og X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 05 Oct 2023 20:21:53 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778974736151196660 X-GMAIL-MSGID: 1778974736151196660 Allocation of a hugetlb page for the hugetlb pool is done by the routine alloc_pool_huge_page. This routine will allocate contiguous pages from a low level allocator, prep the pages for usage as a hugetlb page and then add the resulting hugetlb page to the pool. In the 'prep' stage, optional vmemmap optimization is done. For performance reasons we want to perform vmemmap optimization on multiple hugetlb pages at once. To do this, restructure the hugetlb pool allocation code such that vmemmap optimization can be isolated and later batched. The code to allocate hugetlb pages from bootmem was also modified to allow batching. No functional changes, only code restructure. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb.c | 179 ++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 140 insertions(+), 39 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d7679d37d072..4ccb54824daa 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1964,16 +1964,21 @@ static void __prep_account_new_huge_page(struct hstate *h, int nid) h->nr_huge_pages_node[nid]++; } -static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) +static void init_new_hugetlb_folio(struct hstate *h, struct folio *folio) { folio_set_hugetlb(folio); - hugetlb_vmemmap_optimize(h, &folio->page); INIT_LIST_HEAD(&folio->lru); hugetlb_set_folio_subpool(folio, NULL); set_hugetlb_cgroup(folio, NULL); set_hugetlb_cgroup_rsvd(folio, NULL); } +static void __prep_new_hugetlb_folio(struct hstate *h, struct folio *folio) +{ + init_new_hugetlb_folio(h, folio); + hugetlb_vmemmap_optimize(h, &folio->page); +} + static void prep_new_hugetlb_folio(struct hstate *h, struct folio *folio, int nid) { __prep_new_hugetlb_folio(h, folio); @@ -2170,16 +2175,9 @@ static struct folio *alloc_buddy_hugetlb_folio(struct hstate *h, return page_folio(page); } -/* - * Common helper to allocate a fresh hugetlb page. All specific allocators - * should use this function to get new hugetlb pages - * - * Note that returned page is 'frozen': ref count of head page and all tail - * pages is zero. - */ -static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, - gfp_t gfp_mask, int nid, nodemask_t *nmask, - nodemask_t *node_alloc_noretry) +static struct folio *__alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; bool retry = false; @@ -2192,6 +2190,7 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, nid, nmask, node_alloc_noretry); if (!folio) return NULL; + if (hstate_is_gigantic(h)) { if (!prep_compound_gigantic_folio(folio, huge_page_order(h))) { /* @@ -2206,32 +2205,80 @@ static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, return NULL; } } - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); return folio; } +static struct folio *only_alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) +{ + struct folio *folio; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (folio) + init_new_hugetlb_folio(h, folio); + return folio; +} + /* - * Allocates a fresh page to the hugetlb allocator pool in the node interleaved - * manner. + * Common helper to allocate a fresh hugetlb page. All specific allocators + * should use this function to get new hugetlb pages + * + * Note that returned page is 'frozen': ref count of head page and all tail + * pages is zero. */ -static int alloc_pool_huge_page(struct hstate *h, nodemask_t *nodes_allowed, - nodemask_t *node_alloc_noretry) +static struct folio *alloc_fresh_hugetlb_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nmask, + nodemask_t *node_alloc_noretry) { struct folio *folio; - int nr_nodes, node; + + folio = __alloc_fresh_hugetlb_folio(h, gfp_mask, nid, nmask, + node_alloc_noretry); + if (!folio) + return NULL; + + prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + return folio; +} + +static void prep_and_add_allocated_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + /* Add all new pool pages to free lists in one lock cycle */ + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { + __prep_account_new_huge_page(h, folio_nid(folio)); + enqueue_hugetlb_folio(h, folio); + } + spin_unlock_irq(&hugetlb_lock); +} + +/* + * Allocates a fresh hugetlb page in a node interleaved manner. The page + * will later be added to the appropriate hugetlb pool. + */ +static struct folio *alloc_pool_huge_folio(struct hstate *h, + nodemask_t *nodes_allowed, + nodemask_t *node_alloc_noretry) +{ gfp_t gfp_mask = htlb_alloc_mask(h) | __GFP_THISNODE; + int nr_nodes, node; for_each_node_mask_to_alloc(h, nr_nodes, node, nodes_allowed) { - folio = alloc_fresh_hugetlb_folio(h, gfp_mask, node, + struct folio *folio; + + folio = only_alloc_fresh_hugetlb_folio(h, gfp_mask, node, nodes_allowed, node_alloc_noretry); - if (folio) { - free_huge_folio(folio); /* free it into the hugepage allocator */ - return 1; - } + if (folio) + return folio; } - return 0; + return NULL; } /* @@ -3246,25 +3293,35 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, */ static void __init gather_bootmem_prealloc(void) { + LIST_HEAD(folio_list); struct huge_bootmem_page *m; + struct hstate *h, *prev_h = NULL; list_for_each_entry(m, &huge_boot_pages, list) { struct page *page = virt_to_page(m); struct folio *folio = (void *)page; - struct hstate *h = m->hstate; + + h = m->hstate; + /* + * It is possible to have multiple huge page sizes (hstates) + * in this list. If so, process each size separately. + */ + if (h != prev_h && prev_h != NULL) + prep_and_add_allocated_folios(prev_h, &folio_list); + prev_h = h; VM_BUG_ON(!hstate_is_gigantic(h)); WARN_ON(folio_ref_count(folio) != 1); hugetlb_folio_init_vmemmap(folio, h, HUGETLB_VMEMMAP_RESERVE_PAGES); - prep_new_hugetlb_folio(h, folio, folio_nid(folio)); + __prep_new_hugetlb_folio(h, folio); /* If HVO fails, initialize all tail struct pages */ if (!HPageVmemmapOptimized(&folio->page)) hugetlb_folio_init_tail_vmemmap(folio, HUGETLB_VMEMMAP_RESERVE_PAGES, pages_per_huge_page(h)); - free_huge_folio(folio); /* add to the hugepage allocator */ + list_add(&folio->lru, &folio_list); /* * We need to restore the 'stolen' pages to totalram_pages @@ -3274,6 +3331,8 @@ static void __init gather_bootmem_prealloc(void) adjust_managed_page_count(page, pages_per_huge_page(h)); cond_resched(); } + + prep_and_add_allocated_folios(h, &folio_list); } static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) @@ -3307,9 +3366,22 @@ static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) h->max_huge_pages_node[nid] = i; } +/* + * NOTE: this routine is called in different contexts for gigantic and + * non-gigantic pages. + * - For gigantic pages, this is called early in the boot process and + * pages are allocated from memblock allocated or something similar. + * Gigantic pages are actually added to pools later with the routine + * gather_bootmem_prealloc. + * - For non-gigantic pages, this is called later in the boot process after + * all of mm is up and functional. Pages are allocated from buddy and + * then added to hugetlb pools. + */ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long i; + struct folio *folio; + LIST_HEAD(folio_list); nodemask_t *node_alloc_noretry; bool node_specific_alloc = false; @@ -3351,14 +3423,25 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) for (i = 0; i < h->max_huge_pages; ++i) { if (hstate_is_gigantic(h)) { + /* + * gigantic pages not added to list as they are not + * added to pools now. + */ if (!alloc_bootmem_huge_page(h, NUMA_NO_NODE)) break; - } else if (!alloc_pool_huge_page(h, - &node_states[N_MEMORY], - node_alloc_noretry)) - break; + } else { + folio = alloc_pool_huge_folio(h, &node_states[N_MEMORY], + node_alloc_noretry); + if (!folio) + break; + list_add(&folio->lru, &folio_list); + } cond_resched(); } + + /* list will be empty if hstate_is_gigantic */ + prep_and_add_allocated_folios(h, &folio_list); + if (i < h->max_huge_pages) { char buf[32]; @@ -3492,7 +3575,9 @@ static int adjust_pool_surplus(struct hstate *h, nodemask_t *nodes_allowed, static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, nodemask_t *nodes_allowed) { - unsigned long min_count, ret; + unsigned long min_count; + unsigned long allocated; + struct folio *folio; LIST_HEAD(page_list); NODEMASK_ALLOC(nodemask_t, node_alloc_noretry, GFP_KERNEL); @@ -3569,7 +3654,8 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, break; } - while (count > persistent_huge_pages(h)) { + allocated = 0; + while (count > (persistent_huge_pages(h) + allocated)) { /* * If this allocation races such that we no longer need the * page, free_huge_folio will handle it by freeing the page @@ -3580,15 +3666,32 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, /* yield cpu to avoid soft lockup */ cond_resched(); - ret = alloc_pool_huge_page(h, nodes_allowed, + folio = alloc_pool_huge_folio(h, nodes_allowed, node_alloc_noretry); - spin_lock_irq(&hugetlb_lock); - if (!ret) + if (!folio) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + list_add(&folio->lru, &page_list); + allocated++; /* Bail for signals. Probably ctrl-c from user */ - if (signal_pending(current)) + if (signal_pending(current)) { + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); goto out; + } + + spin_lock_irq(&hugetlb_lock); + } + + /* Add allocated pages to the pool */ + if (!list_empty(&page_list)) { + spin_unlock_irq(&hugetlb_lock); + prep_and_add_allocated_folios(h, &page_list); + spin_lock_irq(&hugetlb_lock); } /* @@ -3614,8 +3717,6 @@ static int set_max_huge_pages(struct hstate *h, unsigned long count, int nid, * Collect pages to be removed on list without dropping lock */ while (min_count < persistent_huge_pages(h)) { - struct folio *folio; - folio = remove_pool_hugetlb_folio(h, nodes_allowed, 0); if (!folio) break; From patchwork Fri Oct 6 03:20:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 149093 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp61513vqo; Thu, 5 Oct 2023 20:22:25 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGK88cRYLub5QbLxHdqdlCCidDNHFyDO4lciB65D0hrLSYBPUSN7F3bcZIoNlGQeg941BBX X-Received: by 2002:a05:6808:210f:b0:3ae:5743:533a with SMTP id r15-20020a056808210f00b003ae5743533amr9016604oiw.47.1696562545345; Thu, 05 Oct 2023 20:22:25 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696562545; cv=pass; d=google.com; s=arc-20160816; b=PF1E2HUueo410hOedJzasfwTMr22lTUULgYOTlsPjZicpf5ZezOhK5EzY23CKHuLJU fmqbOMnsp6Z0kHbgXcoucLqzGCAfOEDeXLkQFDo7M/pK7zL4H2gZJ96T5D1p33C65+eZ fmF4VndMDUe3KVVjiJ2eBvu1KiNAQR8LybxwZX+cxW1J0Lv6c0k+zkBL9xpSaFMr7nhb 7Y5j/Ww6FHIOPg7lWu8S/M3mFZvrNDTQ4J7P/RTLCUAVSfzYe4I1RISV4Wg/NrviGMmI zQrWeeTyG2FqJMZPr2TRPeNfLJFGJYH2T4eVDEz2Q/saQe54gBnmt5NgjN9JuM3EfYJN 7jBA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=FiLgl5+dGvdhCQpbuKeyP18InceJB4CZHOC4Sn8wCck=; fh=d5TJywUWx1H4J8STZuTxM2xyaY/BETP1UBu+8XkSxoA=; b=sBaRZJ4U34jetbrifXxd8AK0PdUXoFcoX5bsR1lKlZYerMst3R+DuE7q68M+mSL/7C bfk7W/8MqRmsY80SNeoqaTC5/CHUKxypgPBKKyS83p/13jbsd/P/4WEa/Y5i5zog3SWU CdHu4lrmbVYbYJGL+/ow0o0lO+4Fw5/AT5zLbwJSFydw7qzv9VJ954XbcKkOWcAmaRp8 j0xSYTZhS1aLvnx+h3tkTLOynM2bfAR8GTWpexkXkuACfLHRdwpyLtBJ7o6DMSwUS1IC Ke4A7cWyrTiQcOIJpctCRf+v4tKOmf/Wnmxm3OH8w/1aIpU/SycQDqK9XhykOw7V7N12 x/zg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=TdlnLAfl; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=E30BDezB; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from morse.vger.email (morse.vger.email. [23.128.96.31]) by mx.google.com with ESMTPS id a70-20020a639049000000b00585a16c1077si2696621pge.367.2023.10.05.20.22.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 20:22:25 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) client-ip=23.128.96.31; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=TdlnLAfl; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=E30BDezB; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.31 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id E2054806D82E; Thu, 5 Oct 2023 20:22:22 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230109AbjJFDWC (ORCPT + 18 others); Thu, 5 Oct 2023 23:22:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229931AbjJFDVa (ORCPT ); Thu, 5 Oct 2023 23:21:30 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09212E4 for ; Thu, 5 Oct 2023 20:21:25 -0700 (PDT) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3962O7cs004502; Fri, 6 Oct 2023 03:20:34 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=FiLgl5+dGvdhCQpbuKeyP18InceJB4CZHOC4Sn8wCck=; b=TdlnLAflYg8Ww39hJKCavLByw8jIZFZ5pzPpXfOB81aUORbhk2TIXUwPpt0QlevR6j3U +NUgDo0/R/kCpfMoaEvh4ZVhIssjH125Pfo4yaFdtBEUJVNUzm5M9clqdwHC0X2IjNKQ EwvFZUN1v2KwilFHik9yXiCe/FrrBWegQgdT8VY1i5jDSSylx39EzabtbYzAN9rbt/jK nGhMPBYDlnXOmd3YPeOiHbAB9fDmPpJmJvq8HZIIUy4RcHllS3JfYeghR9hZDQyh3HvQ 58w3JyLtqHhCBT7irCdCRXpSqeYhJVJrO7pnopuRttjgSA6J66Y6qDVHtP/mPYTxzzcq fA== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3teb9uk15c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:33 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3963Isvi008714; Fri, 6 Oct 2023 03:20:32 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2174.outbound.protection.outlook.com [104.47.55.174]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3tea4ahawt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:32 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AZQCOZQpUhHHjDDkHVRv2pYljcsenKpzvi9+l7P35DgJJNyoaCN30RuIf7NpmgJWg/WKOyP9nIbK6GmPJwaGmBjIw7mFQUXyFaTLOb5iwo6wEF8i23GCZYjKFPh2M9ANk6SoF/X3MfnCCYfhtH+6BdDpL2vl5+MIcxJgve1b95wgd2Fd4swYs3k8+j0k1yFq7zSdb2fqBA02/NkB4q/e6me2O/DCJUpa5DQclT5/7DPCp9JLk72BbFkRhstJVNY1blfSq56Le6/XG1L9wkEIAc4io44QA2nZHxDUxN3Z4XNh2ZU6f6hmjJVDqpIhS6FMbW+eN7IRTw6aEN5k1G34/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=FiLgl5+dGvdhCQpbuKeyP18InceJB4CZHOC4Sn8wCck=; b=Z15LA5QrLuD+d1XU1OxM7CKWJ3t5sMcCa2Axii4U0ZOzhaAHBuN5ho7l5jSVeSKIgMbD8rVeCny6fmlj032d2n2wwj7f/4GKhPBmqDWTdHxBhqP8Z/INdDWuXKFM7QxalMnPh6hBxetpuuml3Ivh9XHzI9wMQ4xRhKzbkLCH47HOdxZCIF5FnUO9AvHXNDyc8aXknkyQAo1ux1zRLkOqtSL8+zXCXbiPC1sH3D2gtuoTGiGZVUGAUgXCifiRAGa/veunaATPj+M9ATimAzzwDBjjviAARhVZiKPYp8TWjDJvnO9gFtyhCmNim0vn4UI4Lroa3IH37MumnYkIxfgwJA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FiLgl5+dGvdhCQpbuKeyP18InceJB4CZHOC4Sn8wCck=; b=E30BDezBkl0iroS/3DvHkHaJ7Dwc3tTbg27YZcercRDHA3dVEwjFR0XYjtULOcMbNiliqGR9YzTTiXYIIdRXhqWJfSbSM/IabgjFl5T7x13itQeA7QAStAFiFel6+f298UY9nkd4C00fiElZcvTAWiZLLVc2wH9AyrCwcI7+7S8= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SA1PR10MB6615.namprd10.prod.outlook.com (2603:10b6:806:2b8::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.38; Fri, 6 Oct 2023 03:20:30 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2%6]) with mapi id 15.20.6838.033; Fri, 6 Oct 2023 03:20:30 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Konrad Dybcio , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v7 3/8] hugetlb: perform vmemmap optimization on a list of pages Date: Thu, 5 Oct 2023 20:20:05 -0700 Message-ID: <20231006032012.296473-4-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231006032012.296473-1-mike.kravetz@oracle.com> References: <20231006032012.296473-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0057.namprd04.prod.outlook.com (2603:10b6:303:6a::32) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|SA1PR10MB6615:EE_ X-MS-Office365-Filtering-Correlation-Id: 2c44c973-ba7d-4d91-687c-08dbc61b3188 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: V8Qf8pT81GSXXkYF2F5YwBrYEY0of5pFJuzAIGys5eCoIv/WCQx0umZ20nFahqL9hXTN0/L1SseS0kZOqaIDhML93THcFEJg2/i91sOUO02IAbnkxflEtarve8x4kzMe47bzcqzW12gcv0rY+Sq3DtgjAzZaXpric+r7xBZtnLZrA0l9hf40IT6MqvG3tutNiQjmB/0gITjhgrlb6OPaTwpI++PfWARuQBAAEC/XvcfEazwBmMJOnDHNxGoYUPrJCM7FwhWWjpKO0FnJWnzFsqqD7HRKOKB8Tb0c9Ezyc/g1Ljv+Yb6fZ6Ce9xfCqsLoidvXC4mT5FG02dDy32hX4rfjDMW/2SwDjP6rKL8Gpn5d70iMGZ/6gQKsfFGYY4udC3SD8Me56BjX15G0aEDh4fmKKEPq/GHm9nsoxleIF2w/WGGsODy3fhKNPQEEcKenRTe1Yhq4okBHlpXSZUjhX+HHul3dPcOwbRNFfeqHrgrqFct31ZDRVapd0BbkSINOfgYv68bhzblLgDZogxdqUxOlvcDHLOkKIVc8YY9IT4J3WGNQTDuQjlAsTlfkTu6O X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(366004)(376002)(39860400002)(346002)(136003)(230922051799003)(64100799003)(186009)(451199024)(1800799009)(316002)(54906003)(66946007)(66476007)(66556008)(41300700001)(1076003)(2616005)(86362001)(6512007)(107886003)(6506007)(26005)(6666004)(36756003)(6486002)(478600001)(38100700002)(83380400001)(44832011)(7416002)(2906002)(4326008)(8936002)(8676002)(5660300002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: FRTkTxz3gm90NtwPK7Nsq7Unn5wzfugGjK4veGra0BSBZ7zxXiATvg8YbT065YDGShCOL9J+86qRuSS62FdMQP3qKQGyCxUbUL9sfRxrkgYQ1qPp3uTnE9slxLEVRmtLtmgdhN8qpX5SvmTNqlotXiBmpLlkU9hQf/BxSi2ubHoDdYOszQ4RzM+hWbL4qUg/uPRW7W9b0hYu4SG+hb+QOV2cE5Cu65DH8ZaRIIjoLw4tN/KzG8juY5yqedUjG1HXKKR48wuc0/88ANzKOyjFPgf2ItrxKixjCe7HKyYyFm1gm8IzJxmalOHqE0+lSWmsJRc1qsApuX2jfP33JUerfKegUp0QSXIAhdD+n69wE6PtopNXdRuJUocBCHAp92G9xkR1uKWach3H6Cu0bqjvFI4oYrhE03TCmdCSZKJFewe7VDRhXY0pmNh5EzNhaXY5RRX32P05kwROwza4r+nLYjyNXoXaxZl2H7Wrp3LE9FsL8/WDMcXfx4GhlbcujaNDF/r4ODhvD6KDlZkD1BRkHeDfPbxW4LZf5QCT2JQMvXDWl8ZMUyqr4OnZ21g22ygV8b1Zqm+sGfOgoyadPYYSmwwkJIUXwekjednMWXHj+uG2Tx+dwRTG1rCyQ1mNLWqkdqAJiGCrapzRMHXiQInq9um3UNrV0UrktJVUZhMvf2R1JFlFPrPz7ymWZ312dGNmNYqpYRW8uwQKZxwhW/Z+kBf7WA5ts8B5UwXFyy03ypMpozd1iMZTU2M1wKwwrKb/FbSL36e6HxozBCcMuzVCNt+3BC8UEocRvK0hNymTdrwalTDfN/ToOjRMkLsIc+kJCX8oCKdu07EzJD925860m72BcVPmqMITrvnv3bP4m2Dggqjb5zWSvK9fzh/Zt9/1/OU64+fBvrQ7Kem1oZlLijOplhPbV0FL73HWoSgAFEQyOfKQfYzeyPWZzO8DrBmxZf4f4MTTz8tNJJWGFuj6bgupfQ+WgE7nLqTYyrhA4/c813A6tzLDGokBoqfF08nW9xTOjjsBf8nKlc3CtYNSa6c+mbxySjf5Cf6fJSwzdXQbbMNI/6wpbcBwnu0gtzYfpJBn9vC27KcItyyAEhWZUNXYnH4dlHCu0wvOmFY2+XFZ08y00QiNc+32WG0gWcSNhIMAQiaz2dd+h2Tbw6eKVuKVVanDlOB3uPnwkCU3o/f11GGVLppih/xzqSsv/+XbJO/J/CoJNPneTzqRferChbTptpMzcv4YOf9AK4XrNL9Q6xySZVgv95WhBZhUd+8hkAF1/Ulv+SgH2FbpcT/8BCwxFqLZAoPQ/zdpxzxNadCAXvp/TAahNqS64kW2/PMs/w8OpsZ26ixw09VM1N/ewYEB16maP3+xK4EX/SHxxpemRRT5jT1+dZqIhIPZMQAEt9HAo7fuzyP8EsfI6mZcJZEP3a9eoQcMmyGovmlyB2e5ACDb6uoxb/DqYl58os9wzA/wJaUI43o/FwGJScWG2CVSI/bbYuPrh/VorxVzfblcwQLQeR0dRQBhk9pYwdfikcnKIK+YnmJaVgTZRsxsU1Gaw3wcCGwdbrT+jc/T8n2ptNK2t3ddIVLG7z5j3s19 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: QHXCsKytGmI/APSq/c++lAA2f2V7cnUUEM1GzUgEFlKjvD0rxjT17crxRXZifMc15hFiV6t4msRckYMvkRvLkzlb03IBXXSyjp2sKjBlkLi62uAQdKldfBvjIypI4AlhMPred/jFbugJ6ih2qUAyOHMh1+Vyd5tjQZp9ngUTax2S+uqcPL331Mwlvwr/kt631zuqGdxE5NPatfV5YDlV0oyIJan8e+jjhF5J3Jx+wp+f1bPo2yMpoNfUZnIDD7QeagH5pKcrTe6bDEbuTPCkBARLcweJlEngGXIkqpKSb0ezhjl7JKN+B2xsDo5MijwHqfH5ICIFqmqKFUvM/oRiLcFiF74VWInGCkaBsQXJU6foBCGtEdLNDH8CnDhYg5y4zo6lQdMMVKahkyTHOty0FZI071ZH3USramFsHPB44kjSA85hnA2ROy2Vn834A//hUm3dY2xSWQDEMd/tEfPmCCVeVhxAmnF2l6ecAMj78ShEfmP+6W5R7arIXVNy6q+xLs008ygdNh9842d5rPoTO+E1TA5E2/WwAJS2JZlluL36124RrRjlCH1M94hSrmcQ6D+ABm0xMxtiwr3SXU6/sjeH9nWuXU3u1V9KvoF6dbRxNYaAbHq88ldC7i9w6mbGRaHQnzmxmvNcfgKlo+UZWfTyFCMMm1fmgubGqfRNwBoIK5a5YGNSok5nkQiWE9yzmM48Qawd0evtwgd102dUjF3VbQMp4nVGnwCOwglzsEVKCoqts+cCw1pprKih6yZlcaF9pvbTBpcRk4Leokh0ACrO8jS8dhxwxLznO5NQKrouZLlQhi5oKxss1iOi1r2vYAikcQ98b36w9P7Ul5acVomjEpKmuI7uvK6r1F0/uOZr7bbaN3Icy888ax7g4mU+329W9m0WGMkZwY/xLQUFVUs1ilDnUzX/QMen4UHxh+evvSa8EP/FV9Y3BxwjDvQGO2f9BuK9jfDbhl/oRf+9Q5mDQSzZkH9mzYHuO+OeS4aBGNvO67rY4KYIBwCk3YEgc44e3P4D2dSPmIDkkNJCiOrf/GDeYfbnfp0gCiWVHNyHkGCjPsl2EGlzhDiFAC0Afvh5IKXq397M1jTrEkSXuqlhtT18bW6rJMN0YQjc2gY3RC/g4FYSirDJc5OfsUjoEp8rlaaxwkof5i8u2GozYA== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2c44c973-ba7d-4d91-687c-08dbc61b3188 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2023 03:20:30.1724 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MMUbCMxxC+h7Un2yLot5dzYblEotCo47rUDWcQf2blekbtkPa/q4Imzg5W6km39qbyRTvUR/i/UcSGDHkK5r9Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6615 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-05_18,2023-10-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 spamscore=0 phishscore=0 bulkscore=0 mlxlogscore=999 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310060024 X-Proofpoint-ORIG-GUID: nE8fvQnH6Hbh3Ti7U5BawZ3ypMOM8UWy X-Proofpoint-GUID: nE8fvQnH6Hbh3Ti7U5BawZ3ypMOM8UWy X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 05 Oct 2023 20:22:22 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778974767806473116 X-GMAIL-MSGID: 1778974767806473116 When adding hugetlb pages to the pool, we first create a list of the allocated pages before adding to the pool. Pass this list of pages to a new routine hugetlb_vmemmap_optimize_folios() for vmemmap optimization. Due to significant differences in vmemmmap initialization for bootmem allocated hugetlb pages, a new routine prep_and_add_bootmem_folios is created. We also modify the routine vmemmap_should_optimize() to check for pages that are already optimized. There are code paths that might request vmemmap optimization twice and we want to make sure this is not attempted. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb.c | 42 ++++++++++++++++++++++++++++++++++-------- mm/hugetlb_vmemmap.c | 11 +++++++++++ mm/hugetlb_vmemmap.h | 5 +++++ 3 files changed, 50 insertions(+), 8 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 4ccb54824daa..2df9435afa48 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -2249,6 +2249,9 @@ static void prep_and_add_allocated_folios(struct hstate *h, { struct folio *folio, *tmp_f; + /* Send list for bulk vmemmap optimization processing */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + /* Add all new pool pages to free lists in one lock cycle */ spin_lock_irq(&hugetlb_lock); list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { @@ -3287,6 +3290,34 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, prep_compound_head((struct page *)folio, huge_page_order(h)); } +static void __init prep_and_add_bootmem_folios(struct hstate *h, + struct list_head *folio_list) +{ + struct folio *folio, *tmp_f; + + /* Send list for bulk vmemmap optimization processing */ + hugetlb_vmemmap_optimize_folios(h, folio_list); + + /* Add all new pool pages to free lists in one lock cycle */ + spin_lock_irq(&hugetlb_lock); + list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { + if (!folio_test_hugetlb_vmemmap_optimized(folio)) { + /* + * If HVO fails, initialize all tail struct pages + * We do not worry about potential long lock hold + * time as this is early in boot and there should + * be no contention. + */ + hugetlb_folio_init_tail_vmemmap(folio, + HUGETLB_VMEMMAP_RESERVE_PAGES, + pages_per_huge_page(h)); + } + __prep_account_new_huge_page(h, folio_nid(folio)); + enqueue_hugetlb_folio(h, folio); + } + spin_unlock_irq(&hugetlb_lock); +} + /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_ORDER) pages. @@ -3307,7 +3338,7 @@ static void __init gather_bootmem_prealloc(void) * in this list. If so, process each size separately. */ if (h != prev_h && prev_h != NULL) - prep_and_add_allocated_folios(prev_h, &folio_list); + prep_and_add_bootmem_folios(prev_h, &folio_list); prev_h = h; VM_BUG_ON(!hstate_is_gigantic(h)); @@ -3315,12 +3346,7 @@ static void __init gather_bootmem_prealloc(void) hugetlb_folio_init_vmemmap(folio, h, HUGETLB_VMEMMAP_RESERVE_PAGES); - __prep_new_hugetlb_folio(h, folio); - /* If HVO fails, initialize all tail struct pages */ - if (!HPageVmemmapOptimized(&folio->page)) - hugetlb_folio_init_tail_vmemmap(folio, - HUGETLB_VMEMMAP_RESERVE_PAGES, - pages_per_huge_page(h)); + init_new_hugetlb_folio(h, folio); list_add(&folio->lru, &folio_list); /* @@ -3332,7 +3358,7 @@ static void __init gather_bootmem_prealloc(void) cond_resched(); } - prep_and_add_allocated_folios(h, &folio_list); + prep_and_add_bootmem_folios(h, &folio_list); } static void __init hugetlb_hstate_alloc_pages_onenode(struct hstate *h, int nid) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 76682d1d79a7..4558b814ffab 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -483,6 +483,9 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { + if (HPageVmemmapOptimized((struct page *)head)) + return false; + if (!READ_ONCE(vmemmap_optimize_enabled)) return false; @@ -572,6 +575,14 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) SetHPageVmemmapOptimized(head); } +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ + struct folio *folio; + + list_for_each_entry(folio, folio_list, lru) + hugetlb_vmemmap_optimize(h, &folio->page); +} + static struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 4573899855d7..c512e388dbb4 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -20,6 +20,7 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -48,6 +49,10 @@ static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page { } +static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ +} + static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { return 0; From patchwork Fri Oct 6 03:20:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 149090 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp61382vqo; Thu, 5 Oct 2023 20:22:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFOkcAMekT3PW6ey86XUk8Z87g/UJ67Rd3aMaxNk2bMgbTEG8iQqbsqyaMEV5kZlYMTEIWB X-Received: by 2002:a17:90a:c293:b0:268:18e:9dfa with SMTP id f19-20020a17090ac29300b00268018e9dfamr7552334pjt.5.1696562522686; Thu, 05 Oct 2023 20:22:02 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696562522; cv=pass; d=google.com; s=arc-20160816; b=AVpr0jV8UX3RQ3yWTJGVSCwCrNofmqwdIjm3LRjCtbXFLfegU/uyZFEr8siHlUk+1M 2kPYsDDskpVbuF7EGE5aPeVUvm8dr1Sx9OdVq0MefzB0pOBYzLwet0I96ijdgjEpKGwk yx5uuXiScsDt+ZgK9UNbMsrWVT86hUo3iVtfPMo+gVOco8QmBh18z+udW55Q/meeA2Ej h0v1kiV5xJnFc2UKGKSZ1SZ3EdpfLosBWsJYVcjt7V23LMTFyTvwjOiPozl0RDDRYNE0 GC8EUkbS2BJegg9Uvdj7DzkDZucpL2Z6fVtfWsfZxH7whD5Js0IEGi3eHyW7Kg8M7c9Z N6Sg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=x+PUOq+oletQu3/ZNtEomgJpvs6MncZM8flj0pLsxs4=; fh=d5TJywUWx1H4J8STZuTxM2xyaY/BETP1UBu+8XkSxoA=; b=p8bjag9MosmsB/mhvm+RH8Acl/JfhYquS8Z6OW+kqWknKQVmltcbgwq0TPf5jd8JOg Naz1F58YowsY3Hp8nrzgnb5AEwg9k/hZFOYFVrGQES8oCXpHkvXeFzkJ90hsdhO6sKAn g0L/KkYZgjqEBcKv3S1VaMj0175mpH6uKhB4rbw8wvoPcILdfXFTcbq4XvcKGgSexAmd FwW3NKZG97EMdkpmu2WeK2j9slxmH2iR86CH25p4V/GHYSUsV4UKmE3S5ujCVk0Apk99 jcxhSiNlju4pjCjpEyIwlIi7kG6Ff3xIbCBK72nJkixZnZW1f6E8jX8qrT51Rro26vWF hkUQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=z+LNyort; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=KV2cBI43; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id x8-20020a17090aca0800b00268278c51e3si4997542pjt.49.2023.10.05.20.22.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 20:22:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=z+LNyort; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=KV2cBI43; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 08E8A8343502; Thu, 5 Oct 2023 20:22:02 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230001AbjJFDVr (ORCPT + 18 others); Thu, 5 Oct 2023 23:21:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38522 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229670AbjJFDVY (ORCPT ); Thu, 5 Oct 2023 23:21:24 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71D74D6 for ; Thu, 5 Oct 2023 20:21:19 -0700 (PDT) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3962NikZ027817; Fri, 6 Oct 2023 03:20:37 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=x+PUOq+oletQu3/ZNtEomgJpvs6MncZM8flj0pLsxs4=; b=z+LNyortT5Vsjyqti+Sqbvo/8AnfV4e/iGi7vqyfTX7XN/C+YXSnk//de282gkYKj0Dx FH6vB9cBuOFHgPZaClLnKB/gu7UkM+h9oWvDLp23l+QsucNymu2jdMJDscCZFMWA9CoZ RREJnsSRjVMDmIRk/sP6QkRYl38k/8rCDYDG8u0fgm9O+LCXtjQW2TA0W8vmORMrK177 srVMmTtNyEaVVcylSsO/0KyOjdDq35hrpIucWXfInd1bEK5ENq+abPpaIzzepXp8LtgD iqF20Zw9R8cVTHBf6k/37XV4aW8sesfXc+OriY8U+/gveiQ9Q3jarcvNJHogXTQBGlsQ 1w== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tea3ejyav-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:37 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3960oAAH002983; Fri, 6 Oct 2023 03:20:36 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2176.outbound.protection.outlook.com [104.47.55.176]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3tea49ypv9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:36 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Qq+li12vyGovG5vuLGaf+Y05pldHB29LMR9RVYLvn7dfeNeCn9ZERPdpbK53mbUeYiHgtL4UlWEQoVZkzYjyzwRfAighkHK3QTr1DAf4Yw1fMU3c+oy3qeXmV3NboeoOmhLeYvcCiW65NhFUs2d+iQftWD15uo91K1CD3CCFsZUjea9HMddo7O5G22VufKvCEpfizocgscMpEh8ymmr0Plx6pvEHLV5dMYMRgR4Jl3BlDdJTF0ArZkawA7znwcfRHka6peeWIm0kJAkB5E35UP2MvHntH1JVVITtx4ddfEnpymHmUqIYZwUDsq817fvlq1oAmxdT8y+Ec/AQ6DvSEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=x+PUOq+oletQu3/ZNtEomgJpvs6MncZM8flj0pLsxs4=; b=Ooh2EAVkJqDjUlD8WWOUmKsdjF0Bq59BDGWide4Jb4Ac9oJxdU1P4zQIPkGSPKby0SkIMO8c03Xj9K8h+WW7ua1VJQxqzd5osaZgMusoNqOwT2zwdKIBeR04eI1MQRB8cNLRH7z8Mi5+P/XcZlNCv7lOBzvFEusSaexfFkcKyOtakRizDUI7HkfXMcE8uJy0fSTKNI0C7aPP7iVGVS+/1iw3mTLCTunelNQ+RB4AAcBD6JT6rdPSCoPF+xsI9QUXL7CJlULGVR5oiuwUy+K7ZdPJZMcKpyF6wktYGPmvmW6E4dlSRsH4DZNASyoPt9wylZlff8hXGEor3jvBQlukqw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=x+PUOq+oletQu3/ZNtEomgJpvs6MncZM8flj0pLsxs4=; b=KV2cBI43WNNBVcO7PYngeH4b2RP/7AGsYz17LMeF46JmgyJTs6IQLz5qUleez0rEJwe+hNq8Nc+GIA/nhbLGWTwKehHrTtgkFvG8Q+S754pdx6RPvq+v7CdzRo6EQRol2JBQz7BqZB/eEu83AnSjtEd/j/musNct6nwlnyublmg= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by SA1PR10MB6615.namprd10.prod.outlook.com (2603:10b6:806:2b8::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.38; Fri, 6 Oct 2023 03:20:32 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2%6]) with mapi id 15.20.6838.033; Fri, 6 Oct 2023 03:20:32 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Konrad Dybcio , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v7 4/8] hugetlb: perform vmemmap restoration on a list of pages Date: Thu, 5 Oct 2023 20:20:06 -0700 Message-ID: <20231006032012.296473-5-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231006032012.296473-1-mike.kravetz@oracle.com> References: <20231006032012.296473-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0182.namprd04.prod.outlook.com (2603:10b6:303:86::7) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|SA1PR10MB6615:EE_ X-MS-Office365-Filtering-Correlation-Id: 7572bb18-5e93-46e8-c077-08dbc61b32d3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ceRUW4T+bwz0tbI7XfIqZ+LrFjpUovO1slddtjc11+TaTNqoadqhZR+ljegEFX5CkCVovA5ybp5lItZmpf1tFH94sv2zgl2op4JvazroVQSeOY/ykz4Bil8EfNiuT8nFcmlfWXqBRwpMuY31mdtu3P0AcLrSVNf12eCfdXO2MUUcQRtCENwF35NBnJqic9iPz/Ynz4otXbj1K3mSx5xhFWrXbblDxttGuuzWzbg8kiGC9tjBonk09Y0FJUKODYUtwHiuS+59WcyjUMGRSuolOsYka2yBgpTcaFz+vyTCTuNZlRUJFimjE+exI2sXZSB92lW10vfJFmBRpSFgh6HPejOWCxfn1fV8sFdwlpYmDFtNy174Qm9FW8ERfR7lRvI7GrRypPP4jK/Rhhgilsy83Cj7A6/iYBQRcx/Z9YDtGQRf8ye+J1ayZhkbBlQPg2UmokkKWS4vd9A4iq7ksXdOOIa8rzHA9lyRhIo+dG/SUeQzUszXg27pkyrE+Hkc5fC8c3y4BeVXls1wdL+i6qYxwNafLNKIWHc3S66nJu/EI/SWZYTt5SiUQQGrh301/hiZ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(396003)(366004)(376002)(39860400002)(346002)(136003)(230922051799003)(64100799003)(186009)(451199024)(1800799009)(316002)(54906003)(66946007)(66476007)(66556008)(41300700001)(1076003)(2616005)(86362001)(6512007)(107886003)(6506007)(26005)(6666004)(36756003)(6486002)(478600001)(38100700002)(83380400001)(44832011)(7416002)(2906002)(4326008)(8936002)(8676002)(5660300002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: giXkcGpXxL6oo5pUcwaoyCHKuB2qJxh3yo9T9fUgWkyREb3xRJ41arJzYp/IApWhc2I/g+gDEdsbODlvdwpICVZllWeFpZaBpNwxLpMR+DKpyvMiILtHb0Zdnl+G1p2bgRNjQr5WfcC6ObvBHadXjP0U16SjorbTvr1E0/8tIP0Nvm0rM+ZkDvN7nLoQ+XCHd3Rs1vvTUns16NAppYcoKBuT6nfvWvvKEGzKahxNb+b8Wgo5C0vjlKvNNs/CFimCHxtsaMZiOM45fc70MtYMLMzzNiP8hfoJXsOUdFoE9IpQ3qUVI2KG3HEBe0k4BN4JLaLyZwfS3OmHH791wVPJ6OLn5YHu+UztDmsbjD12Qk6E5ZBiTdYHpvio0+pxX1F3WWhEpwUCOJBI3k0VQW2BZOFu8FMWHKxqpvc0rwZnwenGQXCQipNXnUJvfZs9tvYAItTEGFqdhfwBe184duC83YYD5bXqqMbvbbY40WjXllaZli/PtPhkchnA8GdGORoFAMpRg+9nKpSaBofp4S26KER75NVcip8yBl6N2zZVesxf9NEgdFcgmtwfA4KiknEwKlliLy94qVivOZG/kmv1bewoHMEJhPCjJRFvAXLTtYtT1ZiyseIEgFZwCoO7ebOw7ksR2TS+X0lD7V1t7qg9XCP77XMmAGCJT1mgrfSnvqGWv8jAz3F9euSu9CNWt/ZBuv5HXJ2C+MtXjm3cNU3tx8n3BxxQh3/tMYok8pdr8H/806cfh0wiFtUl0gcDJNILj9LKNYF+MMiLlmTI9CpkoBhTFKfBO3Q+FV4iOKvldBJo7sdaaHqWelYA0+nkyG4VQj95oVVKwCm5xOGtqxqOMlE+EMyzNc4gk4cJkOPmu1KqPKcpvuUjZGk+2MVYWqcbzeyIwnaB4nVLvGeuARnjD4dUg2hwloVaLiSvWlTHvdfEPxftE03iQJL5KjF2GXwenZLStorWfZv/XrCtIJUy349PMYP2is3r26qbJf2j/2Q6cms5ENa+zwQiGOU/rF31q7btp6UePbtS22z+IFFHHGtDZTJ+pofuRkBQdHlR+0VuCFK4CI/OqK/5T0QKSNNdb1EfJNmDOD/o9sDJg9P1+VOooKTgZ4kbMQLUugMix79B3g4A74h1bbAeTRzWegd6+j5HgBfG1GiO8kPE9BVxlxCV7V5DffB3zG9HzzLDZpoVt55KdtIqNDubdS0HUfDg22afT5qNfg7egUBaJa1UYs7a4o65LsFt4+Opy0FtVGzCiPAFNGAO05gCD9cXGQG+vYlWa8IOjIOGBIcgQvSFQQHFoxAtFKHCX4O7CeUJYCCvnrYxrED0q4bsKpMBGD3EZF8+Pje4MpB/ETq2siZnYaXXqJVhpDIbYpGzss1G9P+FM5ueKzr0ZqxQgSfWP8x8z2u65cKrya2Vr5eRumie4RUwB9wWb7SgETdhyccPAnoxuWlFHWcvlqcCJe0WC9EfGPOriovOaVts4kpbDlm9nbZBrRU0pg29grlXFaz+i/NTWI0DpLwbooHPu6qliFiDbmd+TDH2w715VtnGV+7gqUCde/hcEDSdvzxgiVdjkdCOykcqDmFHsQemN4m/Gm0b X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: d+n2S5K5Xu5DsI1jzCMLFG8ZAliW0bEiQLOQuzTJhPQLRgo5wlPGeZ41CO0zctFYofkb1AtdJ2nsfs+29Ua5Dob/J9/qBFK1qs/d3E+/OYw3MMr2M/v6sBoNZY1WWgJ/Ke57W8os376o3m1NdRNA11SmJWDpQDx03f1vxIt1+D9CO4yTx/ubLwcTAuDDq8sIEXhPiAb6G/RW1f/TMyphGQi40Iqm3QAZiXnNP6B/k5qYgRsXP4Ti8gVXXjRiNuaaMVIFO+HDAH1Zin6Ztd0+1KpdGAhPMKMzsIrccoojD/Da1scS020NmMY/3F1MQqI1IVOyGNEmaUGgyx7i2fXYUQpOMqdJegsh2L5NDGhwISH9kPV8vdsZP/ec/gPJ6Fa7F79R0JxLpGJvR6/tS3FCA+RhF83YULwIFI2n36dGrjmIG0jDV9wdM7znGt6l1ZUmGrDMl0pB/1SqBGPtAjbf8kVZWgsB0lwcAUaRUHzpIE9r3To5NuvOOdoYbuae3Ghr2vL0rUYoUb8DlAB3PU/0gCZaFLQd042+vasNue0eg/W902nIxgzXdL/TUx7gw4Z1rUnvcozgVcI7HmuwU61fIr0+MXXRtE2UnBj3OmvVz0K6b+OmGS3cTaObEYopmDMYLC1LNUbGTl80NFDk+DdiH8K6viTRmUZmaJ1UBP5VMTPwgyoNSMuZIdCqcMjT8XzdXwfPtAUWuYm8WLrOO1jeo4OcuLPFeuyTAa4xEFK23sRo1tfxFHmm0l5BLIx2o25Brzbm5/QowQNKkMyYTcTdINVDeAdtd9UrFttGNmBSPUta54ayCKAzKYdrjW9vszaJatUHnCnaG6HwcsiLXpV5a/nFzbnuKnsqenSTt99+NhQZHW6wMk7j9YiDXggIZYJ7gi+OQ5BscBY2ii5TUgvN2JKN+aJku2v9FujWdI+lHOHCdg/sZs5O+KCg7y5GCmay1QLn2u/cFQgfgh2lX8/S/hnvnHY7u0PvX8Q57XebMs60ObV4nypunxXHtaYnsTtCKq/IfUCcwmnUrPEqTPNe5yZSHdz5ff4c0awRxtetHVWH2SQDlm8HCV7GMqn/F1DHk+i0QE5Rjfef8Ph5PUkRahtJESN6+HwX5QwXQRgjyS1IPIl/7j+SiZ0rYII+yRPmbMHGjXPmWPhjkRb5D7VmIw== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7572bb18-5e93-46e8-c077-08dbc61b32d3 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2023 03:20:32.3249 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mYfzFeJB0FlB0j2sC7qjreOfy7RF7hlwlUgasZOYLryp/DaBFV7xEqFUQkuBs9ucwkDh+X+IvFXqh1EF8e53Gg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6615 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-05_18,2023-10-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310060024 X-Proofpoint-ORIG-GUID: VwFMwQSqVAyeeRen7QJyrmNdGfttsAxt X-Proofpoint-GUID: VwFMwQSqVAyeeRen7QJyrmNdGfttsAxt X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW, RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Thu, 05 Oct 2023 20:22:02 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778974744034579083 X-GMAIL-MSGID: 1778974744034579083 The routine update_and_free_pages_bulk already performs vmemmap restoration on the list of hugetlb pages in a separate step. In preparation for more functionality to be added in this step, create a new routine hugetlb_vmemmap_restore_folios() that will restore vmemmap for a list of folios. This new routine must provide sufficient feedback about errors and actual restoration performed so that update_and_free_pages_bulk can perform optimally. Special care must be taken when encountering an error from hugetlb_vmemmap_restore_folios. We want to continue making as much forward progress as possible. A new routine bulk_vmemmap_restore_error handles this specific situation. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb.c | 99 +++++++++++++++++++++++++++++++------------- mm/hugetlb_vmemmap.c | 38 +++++++++++++++++ mm/hugetlb_vmemmap.h | 11 +++++ 3 files changed, 120 insertions(+), 28 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 2df9435afa48..2c491effba39 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1827,50 +1827,93 @@ static void update_and_free_hugetlb_folio(struct hstate *h, struct folio *folio, schedule_work(&free_hpage_work); } -static void update_and_free_pages_bulk(struct hstate *h, struct list_head *list) +static void bulk_vmemmap_restore_error(struct hstate *h, + struct list_head *folio_list, + struct list_head *non_hvo_folios) { struct folio *folio, *t_folio; - bool clear_dtor = false; - /* - * First allocate required vmemmmap (if necessary) for all folios on - * list. If vmemmap can not be allocated, we can not free folio to - * lower level allocator, so add back as hugetlb surplus page. - * add_hugetlb_folio() removes the page from THIS list. - * Use clear_dtor to note if vmemmap was successfully allocated for - * ANY page on the list. - */ - list_for_each_entry_safe(folio, t_folio, list, lru) { - if (folio_test_hugetlb_vmemmap_optimized(folio)) { + if (!list_empty(non_hvo_folios)) { + /* + * Free any restored hugetlb pages so that restore of the + * entire list can be retried. + * The idea is that in the common case of ENOMEM errors freeing + * hugetlb pages with vmemmap we will free up memory so that we + * can allocate vmemmap for more hugetlb pages. + */ + list_for_each_entry_safe(folio, t_folio, non_hvo_folios, lru) { + list_del(&folio->lru); + spin_lock_irq(&hugetlb_lock); + __clear_hugetlb_destructor(h, folio); + spin_unlock_irq(&hugetlb_lock); + update_and_free_hugetlb_folio(h, folio, false); + cond_resched(); + } + } else { + /* + * In the case where there are no folios which can be + * immediately freed, we loop through the list trying to restore + * vmemmap individually in the hope that someone elsewhere may + * have done something to cause success (such as freeing some + * memory). If unable to restore a hugetlb page, the hugetlb + * page is made a surplus page and removed from the list. + * If are able to restore vmemmap and free one hugetlb page, we + * quit processing the list to retry the bulk operation. + */ + list_for_each_entry_safe(folio, t_folio, folio_list, lru) if (hugetlb_vmemmap_restore(h, &folio->page)) { + list_del(&folio->lru); spin_lock_irq(&hugetlb_lock); add_hugetlb_folio(h, folio, true); spin_unlock_irq(&hugetlb_lock); - } else - clear_dtor = true; - } + } else { + list_del(&folio->lru); + spin_lock_irq(&hugetlb_lock); + __clear_hugetlb_destructor(h, folio); + spin_unlock_irq(&hugetlb_lock); + update_and_free_hugetlb_folio(h, folio, false); + cond_resched(); + break; + } } +} + +static void update_and_free_pages_bulk(struct hstate *h, + struct list_head *folio_list) +{ + long ret; + struct folio *folio, *t_folio; + LIST_HEAD(non_hvo_folios); /* - * If vmemmmap allocation was performed on any folio above, take lock - * to clear destructor of all folios on list. This avoids the need to - * lock/unlock for each individual folio. - * The assumption is vmemmap allocation was performed on all or none - * of the folios on the list. This is true expect in VERY rare cases. + * First allocate required vmemmmap (if necessary) for all folios. + * Carefully handle errors and free up any available hugetlb pages + * in an effort to make forward progress. */ - if (clear_dtor) { +retry: + ret = hugetlb_vmemmap_restore_folios(h, folio_list, &non_hvo_folios); + if (ret < 0) { + bulk_vmemmap_restore_error(h, folio_list, &non_hvo_folios); + goto retry; + } + + /* + * At this point, list should be empty, ret should be >= 0 and there + * should only be pages on the non_hvo_folios list. + * Do note that the non_hvo_folios list could be empty. + * Without HVO enabled, ret will be 0 and there is no need to call + * __clear_hugetlb_destructor as this was done previously. + */ + VM_WARN_ON(!list_empty(folio_list)); + VM_WARN_ON(ret < 0); + if (!list_empty(&non_hvo_folios) && ret) { spin_lock_irq(&hugetlb_lock); - list_for_each_entry(folio, list, lru) + list_for_each_entry(folio, &non_hvo_folios, lru) __clear_hugetlb_destructor(h, folio); spin_unlock_irq(&hugetlb_lock); } - /* - * Free folios back to low level allocators. vmemmap and destructors - * were taken care of above, so update_and_free_hugetlb_folio will - * not need to take hugetlb lock. - */ - list_for_each_entry_safe(folio, t_folio, list, lru) { + list_for_each_entry_safe(folio, t_folio, &non_hvo_folios, lru) { update_and_free_hugetlb_folio(h, folio, false); cond_resched(); } diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4558b814ffab..77f44b81ff01 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -480,6 +480,44 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/** + * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list. + * @h: hstate. + * @folio_list: list of folios. + * @non_hvo_folios: Output list of folios for which vmemmap exists. + * + * Return: number of folios for which vmemmap was restored, or an error code + * if an error was encountered restoring vmemmap for a folio. + * Folios that have vmemmap are moved to the non_hvo_folios + * list. Processing of entries stops when the first error is + * encountered. The folio that experienced the error and all + * non-processed folios will remain on folio_list. + */ +long hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, + struct list_head *non_hvo_folios) +{ + struct folio *folio, *t_folio; + long restored = 0; + long ret = 0; + + list_for_each_entry_safe(folio, t_folio, folio_list, lru) { + if (folio_test_hugetlb_vmemmap_optimized(folio)) { + ret = hugetlb_vmemmap_restore(h, &folio->page); + if (ret) + break; + restored++; + } + + /* Add non-optimized folios to output list */ + list_move(&folio->lru, non_hvo_folios); + } + + if (!ret) + ret = restored; + return ret; +} + /* Return true iff a HugeTLB whose vmemmap should and can be optimized. */ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *head) { diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index c512e388dbb4..a0dcf49f46ba 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -19,6 +19,9 @@ #ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head); +long hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, + struct list_head *non_hvo_folios); void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); @@ -45,6 +48,14 @@ static inline int hugetlb_vmemmap_restore(const struct hstate *h, struct page *h return 0; } +static long hugetlb_vmemmap_restore_folios(const struct hstate *h, + struct list_head *folio_list, + struct list_head *non_hvo_folios) +{ + list_splice_init(folio_list, non_hvo_folios); + return 0; +} + static inline void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { } From patchwork Fri Oct 6 03:20:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 149096 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp61975vqo; Thu, 5 Oct 2023 20:24:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGAik6471Rmb7NfGDs90H+zBV05LqTT+RSbKz863Mug5t/8/pD0XDodI/uqrsuGP3yO9RQ9 X-Received: by 2002:a17:90b:4a48:b0:274:2523:fc7f with SMTP id lb8-20020a17090b4a4800b002742523fc7fmr6870529pjb.47.1696562643054; Thu, 05 Oct 2023 20:24:03 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696562643; cv=pass; d=google.com; s=arc-20160816; b=Kb765MyQwOpy6mNE2XWjIm8G2iD7BpIHeBpTuXJTfgsWoY1ahd3b9HCY/vg2uellIu SAaq9wizLclFGN+TIE1xhlVe2RZk3AqqXkyJOrz6tZ2cZEoJBPwJsMX6/sZ3dqsbBAc4 Aj8Hcrd7/cKilV/g80QsYwJ6CLQb650TINvFpB304VggbjAjWde1HSjjktsmz8x1Nz1L C9HE1K6k3kI4D/Bcq4H1WQ3BXXrELpkAmwsia1K5ub7D57vJtQ9gCJdxTqtA9vn3wF7L xwEPMimofPBILb1csZZj7VBCIKAtkxnvn2j1ZWb6Gh3ZbwY7DRH0D3pVRRc4y8gArocQ D0Aw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=o8COXuu6/TtJ5IDPv6UZqVnZDWI+bSmwncS3etnX9L8=; fh=d5TJywUWx1H4J8STZuTxM2xyaY/BETP1UBu+8XkSxoA=; b=R20+BHioFYBPHOrqwS7/wG4QoOpHN80euQ7WrhVTmW0bSIfKBwgw/Pu1LdcXlxHO9t evx3VNOXa8mI46zk73IwyJaJeAVBXX3CE/dezQSI47s4eJBQloSJkaXJHPsAKtZ3q6Jf M4tEtBwaXUTH2O/Nrly8cYOMFf8M8Q1w863i+1TPjyAiNQeExQtLvSvrZUhpOvaflwqx qpOj8Sk9FDZ0mwIF0f7AlDirJ3CBoV2lfEDt2sogptgtXQoynuExqS+F9WRsJ+sLrizs lCfeET/cHNQNdZHoan44rILxqNpGJ9/r2QbvpKs0z8VqrRe+X+DKTRk5Q27WLMTRW8xm IWMw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b="OEk0D+/S"; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=hqdXZDn6; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id hk8-20020a17090b224800b00274c6997c25si3209814pjb.110.2023.10.05.20.24.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 20:24:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b="OEk0D+/S"; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=hqdXZDn6; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id A25CF8053672; Thu, 5 Oct 2023 20:22:12 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230019AbjJFDVn (ORCPT + 18 others); Thu, 5 Oct 2023 23:21:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229879AbjJFDVY (ORCPT ); Thu, 5 Oct 2023 23:21:24 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D89B5ED for ; Thu, 5 Oct 2023 20:21:21 -0700 (PDT) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3962O544027984; Fri, 6 Oct 2023 03:20:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=o8COXuu6/TtJ5IDPv6UZqVnZDWI+bSmwncS3etnX9L8=; b=OEk0D+/STrWX9EUMhMW2zvjYPmAQg8LOFsTTTh596Nb3LrprpEkveGX5uoIGk3ScorIc na+r+H19hlZUVzZx4SV7miFEZhi3S9zzkTp7QrWnk3py2Zm19Ds/ayU6enMSrcwTkFKH lkpySEktYiA/JST5f5v4lypSMs1cjUL8fV3dcsxshdCUYSJ+kLchk3dU98QGDhxTA74R Ny/dmrv2q3fjXNCNp62SNSdKLYn+7EYAKMUYBWDXArrJhGl5+P9CH52zySYw0PWLWz7m y9hpnQ1RTmRvJvXx14l3+TJj5qtOR9teXAJjV30XXMOk3Vb8jY8dGTWHm8gd3zmpnf/L zQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tea3ejyaw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:40 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3960oAAI002983; Fri, 6 Oct 2023 03:20:38 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam02lp2042.outbound.protection.outlook.com [104.47.56.42]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3tea49ypvt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:38 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aKuQ9KTc9TZOnvgn38k7hiYiB/H7nsXpWldNju2fGtyl079qgXnCUNEk/d7nf/3jgAgZ8EzkHIj56EptInA1uEpdGF3J630WjYyNgZyts5llkzWkXHgmiiGhC9WjHOFhhJFzhPUcppmB0BPg1/JQ44vweeFvpyK8s6W5Jp7o5WNY4q7xpluZFAWOTa0gMfMb0F9XzkmebfygRXbfBoxQ9MJlTMiejs4meHsetmlWkO0Iqv/+OIhCQXwoiCPWxEzBF18q/CcixFULQXBcpQIxLre4bt0iJj4AJtE8q/Aam/yXRgNGQQk4huaRwm3C+p3uKxCIuMR98YBTfYgAAYTIYA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=o8COXuu6/TtJ5IDPv6UZqVnZDWI+bSmwncS3etnX9L8=; b=OArL9szItFuqafH+CnCfaJgcd1zA1dmfmATuAHDyOTRlbEIAFaeMvJFlJzBc0IeqDBGKYt4cGpWsjdFpJsBbRR1nPDZiweCh5Re0z/CzeRyuZy9vz7AAM/vxpNemSozpZZWcCJGLIK1WHtNSQbJ0cbB95iMW7PdEx8cmm9nIor5ZrBDXNrjtxnIKT39K4vxnplWUnASZQk1AGDl+IluOHepcgkRX6XeWg+clRwnAyRWoBvln81CgQ8EupDO6+iwPVePeWFjDDSgxEOvrUYlAal68chRuWGEFydCJzmUQXdY4VrO+aIRwo0ZXxs/b2vOytDg1Ugt6GosaktWDTks9eA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o8COXuu6/TtJ5IDPv6UZqVnZDWI+bSmwncS3etnX9L8=; b=hqdXZDn62+w89Ktzhq/d/oPlNu4Cg0WOYAtVFVksm1eBEPin9KPnuY/w9xIHUL1Jd9GJbJki7BXUwgdTXvFHRRC/gi9N9KfVc4nS8NRve6x2MbLoMDznqWYIcoHkjoE4N2SvS673/l+jD9wLHxAupjCEaXBgzd6XGSmAFUDJGJA= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS7PR10MB5391.namprd10.prod.outlook.com (2603:10b6:5:3ac::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.38; Fri, 6 Oct 2023 03:20:36 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2%6]) with mapi id 15.20.6838.033; Fri, 6 Oct 2023 03:20:36 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Konrad Dybcio , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v7 5/8] hugetlb: batch freeing of vmemmap pages Date: Thu, 5 Oct 2023 20:20:07 -0700 Message-ID: <20231006032012.296473-6-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231006032012.296473-1-mike.kravetz@oracle.com> References: <20231006032012.296473-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0213.namprd03.prod.outlook.com (2603:10b6:303:b9::8) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|DS7PR10MB5391:EE_ X-MS-Office365-Filtering-Correlation-Id: 44630959-3a0a-4db7-c92b-08dbc61b34fc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9bpA4f6UR5D9j2bqtkunDVibRvmFBe3ABM6pzFIOsahecWryycRDHoC1c3QCDerpxbZmCkor3qUJSxNQzKCaD6WuyfjXIG0CnxkAzT/MriKA5sLGxyqMuTD7om2VEFMAWEpyTw2OUq+GJlXLePPSmc+npy4qJbtMMrsIGHMAATHt0Z7iT6GVwcudOLQKSnHE5kmG2h9XamK8GIZl+jsQ/qb6jOdYdBgaYFHDuz68ut/78eqSstcgu7jhAPHZ7lZNMlc2D94jjPiIbHVOeS28ZbUwKU+ln3gSoSeC2mTg2avyMRcoG23K5O3FKKMV1+wUHhKIlCxMAFrCJPqkUrKUvnyLGkygEKjO+9NnZDMz6Qz+u+yMJmXriCmcsWS7cHAZygNrGNqF1Njm9wXB4J9vYbyF9Kf90G3akPRG0h0S/btWMOrIkanljj8aAqOYfDkZkmHk0o8ObqXm+N38dGi/FlMW7uZhME1wZ8wdSiiJ8tGeRJ09dJGzFYYFlvB9XP+n94+BcuoU//l1FjPCzdi4ZRllu1oZCnQCnMBWMp535gX5TkmmBNXMAqbr/ReNNKoF X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(366004)(376002)(136003)(346002)(230922051799003)(64100799003)(451199024)(186009)(1800799009)(6506007)(6666004)(6512007)(6486002)(478600001)(38100700002)(86362001)(2906002)(7416002)(316002)(41300700001)(2616005)(83380400001)(1076003)(107886003)(26005)(36756003)(44832011)(66946007)(66556008)(66476007)(5660300002)(54906003)(4326008)(8936002)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: tiygdZgaRfsSFuT7EBEHjXhyIog5em+8pNMlh5Psfc56nmsAMTjTgKYKASXfXmPlJnzlq1jquiOLCUn8XSZF0ZaoekA1nAuxAPM1+L3T4YQ6ahsI9fnplO6V9CbH4mc7kK67vy1fUn/nLFQJ0UPKPbAE+xcmclLWSpj/Wv55O3jneS58EL34rIDB74MvQ6gpUhRpyhwmGeCRhF+UQMTZBflQO7DfGn/GsBxjVSl0DA9PqYhwVVX1o9CQR2uZ3v+cP7NOjtmpc/zduW4V3QXOfrkrQPpiT48n77gKU8f56qc4jm24GYVk/lOw6Y0+i73UL+gAAzYDHq+Z1NzOyDTInYc4wqOGKzjaO1B3VmRsnAiC4lt2KlnKfdp3zJ/MI0D9qVCqrxksw5ik7HJIR78ORJytnUWQZFqdYwJqMOZAmbezOLv7nEbcHFSSh8f6FVSLMAUKdAm5Z0cyaelCFlo889MQsP5dpdMTwzaJkXHGHpf+wY5lu+CKUJP4SZ/Umg3qWqCgRLobo1gpLI2/TD3nh5VcAspXW0oQgp+wqr1xp0x0U69hqpj0AVgtSwni3vpknJZAMDPqLp3niVlpXAC0DVoldw2wAdgcrKIamIEqzRvxR4OSGgukOSqt38exRt272ZW/XjUiH1zKdWmyrYKMAFkMnoHGLW5R5AooBKliR4ed52Ynf420KxNk+qy0cN2UzeyAMfaj9tGsMqCJ1eDk4syiqd0aKTDxMYORv9wDS/iKVnGO28h7Bq2lfANf8dqAajbW6+MIZEEklPELe1WQRRXgMANWZ8hlH2CtknTQXNl4IswaIKJd4SgBL8xv4K9a3bihs/vuXEc5CNBtUx0WWiYiB2c+8sVTQNHXhCFCi5X05zJ7afvC5MCBwYWZ8RChSSoiWeDJEfnlpGhz/cXh+pE/IPW9TJO8jWUCZyE9M8+Vqj/8oeLIktO7JV0DDfWMaMeMSSWvWnvTW6rVPn97i9zmbpO3YhH9FEPyZlN1I6bh+cePyUZISC3G9eJmbB9o9mlpX8lhEU77txAGzC0lHc+13oHQzD7BIZsgxSbRZMoHYEglLbAELg4hIIAE12QyJpfXDN1lXnkw+nPr4RDCVGrVTh/6MYUl9IW5e9RLlb7/lOSvyH1XF8nZuxGMeueJkELRmvkupeDifSAPkl6hCCvF9doSzGPcleELZ9UynmuiOyBx+jfmBp2GQDeWuo9BnQFAjLUjd26sIfeTjbUR8yIXGVkwpQBqmVycagYHFnNV6rtgKMrwfhhVcwEq7mLX/XdPJiPH+ufK6b2CxVYspAHoxDdAwkGTAJSgDnLAfZaHjCjYp1o/6MQ0qDM0pCN2xFUNGbl97gIwW5PCSRAxMZCEgZhSuUNe3NjfP8Y8sqAzM9i9esi05fzridf2lMwBidPBvU+MLU/0lmBpvLjYgoSEpJd5TTcB5h8DRhAib09SSvdNtQ+KFgjPS+DZ2fRvze/TrFl/2OPVb7RH2uVPCCR7REculbTJQc5tDBdqtc6qOc8EhklGi1lEA+VUVUtuABFZlO4bA8EGLcnF6TEgS7z7GK07uF6D0ibtk/JquiWoBQjFaUgKTPWnIJygNHn+HnGPZlVTT8NsgInMln+mvg== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Mi/TM4QQOpAH61zW4tLN1/md73USVSUVh6Ds5kuEEBwKBXWbt4Hne62y7pvLuT8BiHwQbAAa4RtTx0PZsv/5eh+huFoBy2iQiDhYqWiMGm3aKa5gzGgIAC92bUhxADHq8uunAv0TCXAF0+kJlOvzEk65e9f5m3mttux81twj62wJ7efY/xTwAFmro7Dd3+OadoTFAehl/yD+/bnub4FpxUA/H9N4V+4Cv4PwCxCN633GiueVTKOMZOV/2z6xz/K7BLoYBLeDBvjdwr06EFasDoKX+Gw+0iPCWWgEHpeszYXI6BKWrIWfLMHuzHxV3fOo7cjEqIePb0tgn7FHt4MrSd3Xymqsg8dOXWk08R+ed84bPQdKe3Mvu2jvzG+dpPpEzfGljYvqzYRvJXnGJgBdzkYP9F4iR5dqWAKQ639HGVAFMguCIcKWwldhrkTL8g1XfsEcLwMWtgHpUTF5/8x8xVExhh7/GJKEC0ZQfB+XirLEeRVx+NUEaHUxko/Ahmpd5G0Kdy9l4tVtybgx1oWq0zbOYAQvFRRq2HQMaIom1LT/aoJUTO8o5yuDW0i/tpfMRW3lSZsBhd1IzBfkPDNSL7BW9t4RJ/B25vRpURv/DNXYLlmEGak4LFEZ7ULk08EmjViOijYdm0vygxFCpjpEdfNsDnHJKDOiw17ZQZoBcOnpxxP83vvqGGXWy0IDTgKWUOo4xqyaeurMUN2uUFcJzkAHnvhaC5CQNM1VkdLxaHPlevu+gnvxzi18ii8cB3qi7MAld42frafxugYNnQriETrjc1ObV94TjywRVck6wdUR6pWXd6XE71DlzB7sa4eIFiWNm6Bsn7ooiUyJ18p0OoN7TIanuBHhk2cwu7HQc6YVwE/MXEpWxjuPLfFFXNuf4weAQ252mHzA6HRCahyaiCH5DCPJre3iGuuQCvjjLR7qFv9fnB5LqzASyipkgPo2K3SNAuLhVkW9LB4SfIaTUmLIEpwUSkocozuHahYoVNs2HkaFIzbKR54DXzB7vVP0sinDe/LIF6NkXu9S/lIYIqqQxepK8ahTqhp5e66OUHJxxgHY5LcF4FpMh4qjA4lHPl9ovdoSw+FrF9dF/TxnmNR4InNpG1MIEq3X0Y0bB9Ek68F7VjZrqcA2EaM95pmDHWwjeCduQXaYweu75Cquyw== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 44630959-3a0a-4db7-c92b-08dbc61b34fc X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2023 03:20:35.9889 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Fl/GEpIwq/Js5ji+zrzq2fDW4R/D49PWwqBVgCwh1W821ZWc9uenMfDtIw9yCv/KWUPfjD4dsZwRH1/z76gpLA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5391 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-05_18,2023-10-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310060024 X-Proofpoint-ORIG-GUID: t2hYD3M9Pk0TB8r8dPy7YM-f1gXPPTdz X-Proofpoint-GUID: t2hYD3M9Pk0TB8r8dPy7YM-f1gXPPTdz X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Thu, 05 Oct 2023 20:22:13 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778974870344566418 X-GMAIL-MSGID: 1778974870344566418 Now that batching of hugetlb vmemmap optimization processing is possible, batch the freeing of vmemmap pages. When freeing vmemmap pages for a hugetlb page, we add them to a list that is freed after the entire batch has been processed. This enhances the ability to return contiguous ranges of memory to the low level allocators. Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 82 ++++++++++++++++++++++++++++++-------------- 1 file changed, 56 insertions(+), 26 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 77f44b81ff01..4ac521e596db 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -251,7 +251,7 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr, } entry = mk_pte(walk->reuse_page, pgprot); - list_add_tail(&page->lru, walk->vmemmap_pages); + list_add(&page->lru, walk->vmemmap_pages); set_pte_at(&init_mm, addr, pte, entry); } @@ -306,18 +306,20 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers + * responsibility to free pages. * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, + struct list_head *vmemmap_pages) { int ret; - LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, + .vmemmap_pages = vmemmap_pages, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -334,7 +336,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, if (walk.reuse_page) { copy_page(page_to_virt(walk.reuse_page), (void *)walk.reuse_addr); - list_add(&walk.reuse_page->lru, &vmemmap_pages); + list_add(&walk.reuse_page->lru, vmemmap_pages); } /* @@ -365,15 +367,13 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, walk = (struct vmemmap_remap_walk) { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, - .vmemmap_pages = &vmemmap_pages, + .vmemmap_pages = vmemmap_pages, }; vmemmap_remap_range(reuse, end, &walk); } mmap_read_unlock(&init_mm); - free_vmemmap_page_list(&vmemmap_pages); - return ret; } @@ -389,7 +389,7 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, page = alloc_pages_node(nid, gfp_mask, 0); if (!page) goto out; - list_add_tail(&page->lru, list); + list_add(&page->lru, list); } return 0; @@ -577,24 +577,17 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h return true; } -/** - * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be optimized. - * - * This function only tries to optimize @head's vmemmap pages and does not - * guarantee that the optimization will succeed after it returns. The caller - * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages - * have been optimized. - */ -void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +static int __hugetlb_vmemmap_optimize(const struct hstate *h, + struct page *head, + struct list_head *vmemmap_pages) { + int ret = 0; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; unsigned long vmemmap_reuse; VM_WARN_ON_ONCE(!PageHuge(head)); if (!vmemmap_should_optimize(h, head)) - return; + return ret; static_branch_inc(&hugetlb_optimize_vmemmap_key); @@ -604,21 +597,58 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) /* * Remap the vmemmap virtual address range [@vmemmap_start, @vmemmap_end) - * to the page which @vmemmap_reuse is mapped to, then free the pages - * which the range [@vmemmap_start, @vmemmap_end] is mapped to. + * to the page which @vmemmap_reuse is mapped to. Add pages previously + * mapping the range to vmemmap_pages list so that they can be freed by + * the caller. */ - if (vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse)) + ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages); + if (ret) static_branch_dec(&hugetlb_optimize_vmemmap_key); else SetHPageVmemmapOptimized(head); + + return ret; +} + +/** + * hugetlb_vmemmap_optimize - optimize @head page's vmemmap pages. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be optimized. + * + * This function only tries to optimize @head's vmemmap pages and does not + * guarantee that the optimization will succeed after it returns. The caller + * can use HPageVmemmapOptimized(@head) to detect if @head's vmemmap pages + * have been optimized. + */ +void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) +{ + LIST_HEAD(vmemmap_pages); + + __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages); + free_vmemmap_page_list(&vmemmap_pages); } void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; + LIST_HEAD(vmemmap_pages); + + list_for_each_entry(folio, folio_list, lru) { + int ret = __hugetlb_vmemmap_optimize(h, &folio->page, + &vmemmap_pages); + + /* + * Pages to be freed may have been accumulated. If we + * encounter an ENOMEM, free what we have and try again. + */ + if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { + free_vmemmap_page_list(&vmemmap_pages); + INIT_LIST_HEAD(&vmemmap_pages); + __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); + } + } - list_for_each_entry(folio, folio_list, lru) - hugetlb_vmemmap_optimize(h, &folio->page); + free_vmemmap_page_list(&vmemmap_pages); } static struct ctl_table hugetlb_vmemmap_sysctls[] = { From patchwork Fri Oct 6 03:20:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 149091 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp61461vqo; Thu, 5 Oct 2023 20:22:17 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFW9muWpLG4UrWzqmQalSgyIx0UTD0ZcKPoEXIbnlhGQzs/2HXF4WQfoOrJ1uvg5afgtRW2 X-Received: by 2002:a05:6a00:1781:b0:690:2e46:aca3 with SMTP id s1-20020a056a00178100b006902e46aca3mr8591928pfg.25.1696562537478; Thu, 05 Oct 2023 20:22:17 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696562537; cv=pass; d=google.com; s=arc-20160816; b=CGSKjCYR9j9HkU5abFLyGfeu6RBjHxb/uesXij/t4QID+nD5Yd5gTO6o/s0iu+FjEc QpP0ili7ErcPVrH/KHaAoDyaW/bSOzXfYFIhxr7OhLBB+ob4n8YGfLZpvci+9ylYg+XF cOyj5hpQf8zBjorjwBwxddlBRCu5bnwDrofXuSoW4cT4sBWnKguz8DHI0+SK/aGugCNy o7vakmaSl2TVg8pqZa0gBXp2hA/BzZw3iqsELiTKnOTwNuiXBdLUQ/Q60riynB3BkZdy le6dvvazk37rR0O+pE+TbFTcBMjcg0lhi1NHlO7bB7co5cpsu6xRUpzPDcoHlDAI2Grr Zmog== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; fh=d5TJywUWx1H4J8STZuTxM2xyaY/BETP1UBu+8XkSxoA=; b=KXaI+v+yqxZYcflMHQo9k1oAht27y+PfbON7ajNV0nAc+ye5pO25X0VkgUiuDAuKG9 /uMvUGcEObH5tA+W1a07HPZnfa+YvDBe0MAdNeWs1THx6y9FP4qLt086xzpZBnOANHLD hiHIzBFr7vZ941PVZxR1GtYHXy+b9g3oGawM8ZePed05AamrNq944sb/xvpbwPIjnoL7 E9ZnDfj+W8WWcRaoZqZsBZHJMYS2nfQfR6V+cmALdhtFbxCnPs1AfcdZb8gPvs+iIidF 6dvd0MmSy/CnG+WcBqx70n1WAK2ILju2uEMdTRra0gWQBdrg9RFOhckOWBC4z53HPuE0 mX9g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=RIHQxFLL; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=IYLk0bXE; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from morse.vger.email (morse.vger.email. [2620:137:e000::3:1]) by mx.google.com with ESMTPS id h2-20020a056a00230200b0068fc49cc456si631202pfh.248.2023.10.05.20.22.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 20:22:17 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) client-ip=2620:137:e000::3:1; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=RIHQxFLL; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=IYLk0bXE; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:1 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by morse.vger.email (Postfix) with ESMTP id 5FA0C806D82E; Thu, 5 Oct 2023 20:22:12 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at morse.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229918AbjJFDVa (ORCPT + 18 others); Thu, 5 Oct 2023 23:21:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38540 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229861AbjJFDVY (ORCPT ); Thu, 5 Oct 2023 23:21:24 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BD3A2E9 for ; Thu, 5 Oct 2023 20:21:19 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3962Npww013412; Fri, 6 Oct 2023 03:20:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=RIHQxFLLHl5pKCFIPm193UARpvxBBQlCG4rARQolbUZliJePDcnn41cpNVML1tkCcBJP ClYs55qYfrWgCI+6S5FKvFbM75qXA37l2KtA1jmzpTIAEiR5zF6tzveG7UGwuH/G50za JVFu79DPDw0oP73OrhL2im7upLY5b67R7sY96DO5LNIiTivC5VFa0BV2CnYr1yO16vvz g0M6aQQH77OWT360jRzNNYz/e/a3lPE0z9tyRmN+NhYPduChQ4hsM8nmaHHuRZ285N8b 7F9ptB+rflHrE/nM125+wCGFPK4iUtywaa8bR7/X5w+8DnM86ppv/ZDGt9r3gjjBTuSA 1A== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tebqe31sm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:42 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3961NaBS033628; Fri, 6 Oct 2023 03:20:41 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam02lp2040.outbound.protection.outlook.com [104.47.56.40]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3tea4a7nqs-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:41 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=JF7QJqvT/N4XkALqP5VrmaUkNdFaOaqxNZmPNWNmyqN+wtWY4onWzRZ3XVFMF3mXHEn0rZlfZiJMW+TKxlO6U8YcH0Wkdh7kJdUAjSoHzBH9Bh5MJCAqGvTg7kLzDABsfLXeMz30csMbA3B2MWZbm28damZCuPczq8vzf7teUuqpiA9iI2vn1uKpxikJVH0zuKjHnfbh7as3s0XId6G692A6ha/vd46d4argvk3ZzlpHhCu4hJUJyLXlBhe5slBcnXBF07qUL4QBLDp/Sapb0E6WevR1PEQ0JyPqZ9iuQMY/r+jEFk/UpW1JrEvm0B7Scf/JKZNrpZzrAszeA5hhTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=KJztZ1iVQ/tChimxkPXTczkJDp+8zOICDDsrIdlPj/3/FS4F1qNLJrza+zVne3u0sNK01e2x96uav90XwF1GZTmQ1bwnhu7tR+Y1td/U1S3VPuYwNQNg7AbnnDTZfD0N/vhJCM8HbxAYduGnZmhz6OiLa0j5BA3VwOOkIE05pEV2PEpPxk8ewhht4BLuZUhA4bbi8tg8T36ZnJ6zWYHfJAXz2r1wqSLibT1Bt0inQih6+6hn3U5QFOEiCUwerzy3AswCFs1YAO1ulx9fQrnBrUeJF0sFZnAsh/O8BQid38ttaLREJfzkjQ+aPny5ZnP5To67o+PLdxzWxGp+x0a3eA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3vbONjsZvravfMZFWf/jFmc0JqQCVqxQwTo9QP2uYlc=; b=IYLk0bXEC8UvdUcEnJQU/juSVLoQjMhGaQ4qnrQ/k1laGmMQRvTH5zqrh1pjEt8F7HQf3LrgYeSTGQnk9WjfFnBjpiM/Pd5+isA3Lpq495ctrFW/o/8XVnNyF/DRtmrpUt1Ib3CM23WjqQ6n/ejX3rcYtCWNFiF1sAtXO+hghQA= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS7PR10MB5391.namprd10.prod.outlook.com (2603:10b6:5:3ac::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.38; Fri, 6 Oct 2023 03:20:39 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2%6]) with mapi id 15.20.6838.033; Fri, 6 Oct 2023 03:20:39 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Konrad Dybcio , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v7 6/8] hugetlb: batch PMD split for bulk vmemmap dedup Date: Thu, 5 Oct 2023 20:20:08 -0700 Message-ID: <20231006032012.296473-7-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231006032012.296473-1-mike.kravetz@oracle.com> References: <20231006032012.296473-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0210.namprd04.prod.outlook.com (2603:10b6:303:86::35) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|DS7PR10MB5391:EE_ X-MS-Office365-Filtering-Correlation-Id: fe40a5e3-3a64-463d-e971-08dbc61b36fa X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: b4Usg4BblSwYPNntvFmGcxx8MIDmg//lWw9CXWQ7S0hj6lDrni6y2Qq6fe91ka6eT1M314z/tjNumkHa264VaGiWHh98fbxpwuoXJhFoS2Pzeh5sN6vhfObZnANwS0FD4E1NRtw8KHQC9bLJEV4UFDhUxX2a+OXmJz05wbyKSJ6Cu2YqLpBAukUwb9ZJhLHuSyDRJtZMmPSm8WQ/tXHCtFeTd3T0lOJBJJBXz4qtm3tmLzy/w5/Nynns2ipy565Loo7dxYEv+E4tepfsYr6nf2DZftnZ+s3Xj9+2n4qggWZZ5Cxu8RQnPtiUtzZzIDGmLrSIUEJBEgLkb9vxQ1pV8O/xtmYcua9PLD0qSNR9VKENloW+v42BzMin9g+SXuFM9z4Ee0+vyCAGXLGUAmbsocaj3tkOFa/Q1/3CHpbzqi6CBcnqWU2ibQQkEhyiv3dnvB4yY8Rf32lVvqiI/TLM7diP8bMJsq5fjV8zAynAoJ6iXapu4g9rgLg6xAvujLUkIGV97pcIU0pEkrbfHzvkm0YflQraPgrfn5XDjycKdKCSWKL9CPMm4Ttiy9b7dfi/ X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(366004)(376002)(136003)(346002)(230922051799003)(64100799003)(451199024)(186009)(1800799009)(6506007)(6666004)(6512007)(6486002)(478600001)(38100700002)(86362001)(2906002)(7416002)(316002)(41300700001)(2616005)(83380400001)(1076003)(107886003)(26005)(36756003)(44832011)(66946007)(66556008)(66476007)(5660300002)(54906003)(4326008)(8936002)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: rypyXgOsNl+2WmRYsTx3N/O/2jQQBxON5VOUTmr+1rHG2LcFtKbsGjHmilkcibcfVYjAsmtTCNf2F9Pl9bnkia8an5aKOnoxYTg/Dvo0RcJaU9L9A8Ny+ZIeFYMToafgb0d+0hglVJ9EUoWfKvWwKjdZV8iX1bG3njh1sRyhLZAeBn9sMaagyIP4FuYwjV1ZTRbxWZTqMVtDx4YkXYgkmL0pK634JnLx0r4thMOhwiUXmInlq6PEC79ACzuTUuW0Vyr7h/2/3bk/1LnbsTSdEQ9ifXZ4tBoHYx1Yet/rcWQOJWoZgl2JB2wndt2RqEH7+bfRKh4ADHUfYwdsiaH1T4vZF1+8JE6KDuMglwjbhX0rCnZyqFkjPmxAYGiTfKj8dsBMAaWeaAa4KTa/f+qZFKJtLM4cmFWWhBl+YdoHDIX+zBcNldO7go+jYrRVpSZFB8tvpYifKmbPjlrg6i1Ip2XHJ9NRgoxgsYgz8nVW8bEAtxDVx8Ur0c0yyBfZK096qTva3YEvB6OKp1FMFurCLC0xbloiwNIrTkBv6sqtOaNog4fsDoce34LEqVRid7k5c3j1/T3ju4iCGraOCxxQBeLH6USU/sTq2SCpSJF638M609+35Rcmmmgf/ewUOfnUDbOlypdbs2bZQObjh7FtVHwWZlg5m384KwO374PyNjJUWmRCfW2q0+9H4PBfXtqmiU4Fc1pIx+1xs/NmjURvdbFxlGwgu2KcyynEqf7iW5n7d7kDwSF9qjVIhdpRDFzsBlAOh6KYO3y4et03NoUOU4aDZf86ohFwhs6/4SyM75jWwGMLhU37PioDGGQoHodw9K15eKzcHFFCcggbk1YoynRLTYXHLTXIUPd5LYPnw5sW+876/tacLxozCRkcKFb1QzHoUM3ainlquANYa5di3IaBbKY0xmRHJPVDFvS5x5VoNN1lYMXwmkIxO/PMKJIOSe9exqzl2Xg7zVnP7TCisWEsGMApX3kSNdSzmqLKS7nNgyNs4lssVc2gR2F0RjqTe5W6HJIM7pVNJSzxLr65XxVc1al6C/WHRZAAhSBNlnwmhlt8LzUTcVsHa2bw8lLmXWS9AL3XVLbwO95vtTrz8SXX1UHJAdKjtbqYAr72o8oWpYVG9fHU1e4h+ufJZO+WbnoSxUXSxQYOovYH7E5o1UTRunIuyLkK1t8WxOJY9wneRoL0omWDmmhItLCbcNgu+jJCsQuHjdljYQUwCOpV+pgCGHP0vxVGspTBXwh647iz+nGo8Dwwvuhz3cEN2im7rmx2arjYCUgQBqw77Wx2A9e3M7vuaAEzM1bmUPBkCMPBQXE5E6TDB4GZ141xAkwY+hosZ67Pokvc/TnL0IK27uD/8XOwWcRMFh+YbL3zPFQNGY8yooKpRPl7ZghmEsJ8jfQ+pO5evvZv6gKBJW161/iWnjI4qGnLuWXmFmdKkXZVAJnhoPa3mQree2FNRHZCfNxvZ3k8HynqFAO3YyxRBcSHHBK/h8vWrtl2o1TUyDHXk8qqba89/jWqY2t5KHka9mCzwnWSBuRacKiPIoTomltdI4EsyqnWhdya9PMTySQUdL8XrJhdUa1h4UpT6n21jZuEoYOn3QsjiVusMBgaeg== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Q7lV0ctfdbqGzNz/gU9A+PORtmUHHIM+KhDj5AX7z4EIxAkZQIDvmSw3MzYhW7QMdkFnUMBoyNKZ//a7JbIDYKqeYZVFf26WAEiuUi+p6R1MRoOWALMs41E520E87MhM9N7qcF3N/gH5EIUzbwSUihbZFontTgY7HcS9mFvixabpy8KJ/kiIhs05nqUJOClacebqzkEAzuBemZ8Z3v9/upU4ZtcAno2I6fFXzHis6bJM1jmwlgVEo7VYNVdQyEjkZUcPK/bZfLSf3czucwMEmkrH1scVxKzHC5gdPuxpRDB38p58qIl3ON0n5hJNUW/6LWgcx68zwXuy4igycw0+vfaM8ku8SOjhINET1/w6RFG1w1HhFqjqjydDCcGkiJoT9O0m31WrsuqliBEdIw22gD8DHjfrUkLzncXmcxGzWqXGpCr8YVt+D8ZJwdX1ovoawJez9D0Ej8DOCSkdn19MeqxNR/M5EUD2kTfFmxfpA2s31zqgn6F2aV5AybA65KAurVLen5jMeUEh2j2vH1ZYzBh/pw87zI4p3kdCitcbMnKlkNvMg4N3rU4vrShpW/jVziVThRNIdiPlzz6AJgEoa/f9y6uXz2L4Qy3WLYMKQ1ZqBE7xrJ1DQl17ONPWeyLAOrDc/PRLRW7CGWXmLviJJHtFKxtoYY0ALUelERWZ+BobmxmKJCxItXAaO9LmVGYHH2XLYT6qapPR31uxoC0ldjImQU1CAn+Ter8xlfIuLfhyIAcWz0iBcQ9hLiQzD/aSUe8Te9hiINwvwBxmEjov8UTyt2cW2GzTaHbD/Tuu5txDVyp7V00YXUrNBVdkmA3eblLhpc/0gkaHCH1QG0MFifVLVqq1CbXHY3SS0P4JruMuNCuP1yBJ5xvCioONtaD+ijpvNidtnqfpnhyzH/JZYhzHwD6NR5a7aovu3mFTUbepvruUqdy6A62luVK9n21LfDGbXBU22lIEYeZrr9gw2fc5mrYWBZeZTEB+ASnbGjhAzQ/yOhhIMedJY393UAWmdtNdVboIChIUK/7V4mBsWjj3OFXUX+xt0NIsIWFCXAEZkyfUdPSqiplD7J8m3OFQvvOLogmkyzt0yohI1++AqrXH6nRR27NTlWn2ttGroE597+4QBj8tLZfbv1SghKB0XQpj69tl9KKnOLbloXJHrg== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: fe40a5e3-3a64-463d-e971-08dbc61b36fa X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2023 03:20:39.3096 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /Nfx8HTC1QPerFR71z5946NfhDqERJ76yvjAZF3OrR5DU7YQjypmVCBSZ+wd+9o4FHJO5VuHUVGg2qls0pfRYQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5391 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-05_18,2023-10-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 mlxlogscore=999 suspectscore=0 malwarescore=0 spamscore=0 bulkscore=0 mlxscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310060024 X-Proofpoint-GUID: 1CKExVxtulfT3Z_6BHhInhqEFdyED7GQ X-Proofpoint-ORIG-GUID: 1CKExVxtulfT3Z_6BHhInhqEFdyED7GQ X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on morse.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (morse.vger.email [0.0.0.0]); Thu, 05 Oct 2023 20:22:12 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778974758895953161 X-GMAIL-MSGID: 1778974758895953161 From: Joao Martins In an effort to minimize amount of TLB flushes, batch all PMD splits belonging to a range of pages in order to perform only 1 (global) TLB flush. Add a flags field to the walker and pass whether it's a bulk allocation or just a single page to decide to remap. First value (VMEMMAP_SPLIT_NO_TLB_FLUSH) designates the request to not do the TLB flush when we split the PMD. Rebased and updated by Mike Kravetz Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 92 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 88 insertions(+), 4 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 4ac521e596db..10739e4285d5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -27,6 +27,8 @@ * @reuse_addr: the virtual address of the @reuse_page page. * @vmemmap_pages: the list head of the vmemmap pages that can be freed * or is mapped from. + * @flags: used to modify behavior in vmemmap page table walking + * operations. */ struct vmemmap_remap_walk { void (*remap_pte)(pte_t *pte, unsigned long addr, @@ -35,9 +37,13 @@ struct vmemmap_remap_walk { struct page *reuse_page; unsigned long reuse_addr; struct list_head *vmemmap_pages; + +/* Skip the TLB flush when we split the PMD */ +#define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) + unsigned long flags; }; -static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) +static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start, bool flush) { pmd_t __pmd; int i; @@ -80,7 +86,8 @@ static int split_vmemmap_huge_pmd(pmd_t *pmd, unsigned long start) /* Make pte visible before pmd. See comment in pmd_install(). */ smp_wmb(); pmd_populate_kernel(&init_mm, pmd, pgtable); - flush_tlb_kernel_range(start, start + PMD_SIZE); + if (flush) + flush_tlb_kernel_range(start, start + PMD_SIZE); } else { pte_free_kernel(&init_mm, pgtable); } @@ -127,11 +134,20 @@ static int vmemmap_pmd_range(pud_t *pud, unsigned long addr, do { int ret; - ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK); + ret = split_vmemmap_huge_pmd(pmd, addr & PMD_MASK, + !(walk->flags & VMEMMAP_SPLIT_NO_TLB_FLUSH)); if (ret) return ret; next = pmd_addr_end(addr, end); + + /* + * We are only splitting, not remapping the hugetlb vmemmap + * pages. + */ + if (!walk->remap_pte) + continue; + vmemmap_pte_range(pmd, addr, next, walk); } while (pmd++, addr = next, addr != end); @@ -198,7 +214,8 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - flush_tlb_kernel_range(start, end); + if (walk->remap_pte) + flush_tlb_kernel_range(start, end); return 0; } @@ -297,6 +314,36 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, set_pte_at(&init_mm, addr, pte, mk_pte(page, pgprot)); } +/** + * vmemmap_remap_split - split the vmemmap virtual address range [@start, @end) + * backing PMDs of the directmap into PTEs + * @start: start address of the vmemmap virtual address range that we want + * to remap. + * @end: end address of the vmemmap virtual address range that we want to + * remap. + * @reuse: reuse address. + * + * Return: %0 on success, negative error code otherwise. + */ +static int vmemmap_remap_split(unsigned long start, unsigned long end, + unsigned long reuse) +{ + int ret; + struct vmemmap_remap_walk walk = { + .remap_pte = NULL, + .flags = VMEMMAP_SPLIT_NO_TLB_FLUSH, + }; + + /* See the comment in the vmemmap_remap_free(). */ + BUG_ON(start - reuse != PAGE_SIZE); + + mmap_read_lock(&init_mm); + ret = vmemmap_remap_range(reuse, end, &walk); + mmap_read_unlock(&init_mm); + + return ret; +} + /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to the page which @reuse is mapped to, then free vmemmap @@ -320,6 +367,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -368,6 +416,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, + .flags = 0, }; vmemmap_remap_range(reuse, end, &walk); @@ -419,6 +468,7 @@ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, + .flags = 0, }; /* See the comment in the vmemmap_remap_free(). */ @@ -628,11 +678,45 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) free_vmemmap_page_list(&vmemmap_pages); } +static int hugetlb_vmemmap_split(const struct hstate *h, struct page *head) +{ + unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; + unsigned long vmemmap_reuse; + + if (!vmemmap_should_optimize(h, head)) + return 0; + + vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); + vmemmap_reuse = vmemmap_start; + vmemmap_start += HUGETLB_VMEMMAP_RESERVE_SIZE; + + /* + * Split PMDs on the vmemmap virtual address range [@vmemmap_start, + * @vmemmap_end] + */ + return vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse); +} + void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) { struct folio *folio; LIST_HEAD(vmemmap_pages); + list_for_each_entry(folio, folio_list, lru) { + int ret = hugetlb_vmemmap_split(h, &folio->page); + + /* + * Spliting the PMD requires allocating a page, thus lets fail + * early once we encounter the first OOM. No point in retrying + * as it can be dynamically done on remap with the memory + * we get back from the vmemmap deduplication. + */ + if (ret == -ENOMEM) + break; + } + + flush_tlb_all(); + list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); From patchwork Fri Oct 6 03:20:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 149094 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp61521vqo; Thu, 5 Oct 2023 20:22:29 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEfP7kv8KfG4R4miXrYtaNjwMuOTDAQ38rxCTPCcZmG+G/8zEKuVJplKQeJLV1Ijr7M068r X-Received: by 2002:a17:903:18c:b0:1b7:e355:d1ea with SMTP id z12-20020a170903018c00b001b7e355d1eamr8789516plg.24.1696562548791; Thu, 05 Oct 2023 20:22:28 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696562548; cv=pass; d=google.com; s=arc-20160816; b=skg7Srre96lSJsZRjI7C1Qo509OBw2XcerMmdkcHnY0GMgY9WpIZH1YqDUs1UVG+30 8tM3vQGNIRKpA1Dysw7Hs2qChPMleqgnG0rOE6TZbtwWxTgC9kXlUCxJYLwggBDdjT15 MvG+EbOYNbUi00xnOa4tD6g45xotcNHfzgkQU6sANRwyNVY639nEzlIufo75rP4hr3FG qLD0CaCll0MVEWGpYCGK0ZxAo65mo/Gizj47U5v+GAoiDbC9oSKXSE8kdH3akZoBxKk3 nIucTvSOPiMMhfA4dN/QYoIVc9DUov+z1hZTo3yL6l1emqZ5jmiA7+SCstYECbpBgkA+ eV3A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=Tsc70UZvKWYWI+spZQiVZ3u2z8ZAo9dlRTusEOzzs6g=; fh=d5TJywUWx1H4J8STZuTxM2xyaY/BETP1UBu+8XkSxoA=; b=J6Tb7gyFXCfo11xi0u83TSinSuJNWf7C+5n9wE88FSsAlKvvLukcN1zc7D3tv6u+c2 0RVTMk/PaKAuzDF8tnq+VAZNOQOAvW1SuyKPIivFPdiQH8Hyo/9pcp47HZhm7GZgxWba hApw1R11z19E5tMQ0cjq2Hg9PfVdxQ6ULp3OvNoxm/wR4rnRYyUMxp/TQGlNWYG3Wqkx 4S0pygaxABhj0cDfEYdu/vdyn4j6nzs9BTnYCE+Taco1eODBUzyS6mKGo2weM30Jjfwb R1XhpdU8JtfcMjt494tCbqpGLmMXx6oXnX4zCYlomgow6uqm3m+Nc+eo7c8gQTh6xeJ+ qGBg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=olpdDtdz; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=g4Tuc5vx; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from fry.vger.email (fry.vger.email. [23.128.96.38]) by mx.google.com with ESMTPS id i12-20020a170902e48c00b001c62d935854si2645402ple.614.2023.10.05.20.22.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 20:22:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) client-ip=23.128.96.38; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=olpdDtdz; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=g4Tuc5vx; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.38 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id 1A2548087255; Thu, 5 Oct 2023 20:22:16 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229941AbjJFDVd (ORCPT + 18 others); Thu, 5 Oct 2023 23:21:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229876AbjJFDVY (ORCPT ); Thu, 5 Oct 2023 23:21:24 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 420C2EA for ; Thu, 5 Oct 2023 20:21:20 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3962O5IM013551; Fri, 6 Oct 2023 03:20:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=Tsc70UZvKWYWI+spZQiVZ3u2z8ZAo9dlRTusEOzzs6g=; b=olpdDtdzl/CwxMMvnqZEFjSQX+IquBx9RLp19IIjNUhzsJR7/nDQwPi7b7smEgQEl/Ew e3iaocS/eUd5VcsIAzDRPvUSyTsyrlOXPlNKO0JM666X3DQi2lS4YQpETHCMA5aod0zl O2tI3Fw5hiksBkI1Fe09JhpKPZaLDRceEtS26O7CtIIlqFnHizeHZ2MzQasgIoVi1Qo6 g5PmFCTktRdMuACOEGr8abul8hD512e6t7TL//f5SITVGRwHdai1sCzrSVElD17Cf201 rA2MKurRlVcB2piZAei1w7VmlvzgtOajt2xEs+5cnHh/VttLYGgJyfDjxvrX1VCS5WhB uw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3tebqe31sn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:45 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3960ZFQd000462; Fri, 6 Oct 2023 03:20:44 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam02lp2044.outbound.protection.outlook.com [104.47.56.44]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3tea49xw5j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=G5iPwe0XAUt0yYZMacMLd33X+fYyEigQzt+p2RCGfzPatrJjuxVbVSQ+vpjwv8Vz11SGQNECaqN+QbX01orMNPA2gDXmj1S6m/1NNYIVc6bE5uepHTFCrtI1yKrGUksq1EGCjGJK0gL3w9Dof+GRS5pcAsrqI4RiYqgSxNGQawbHeIwe/CbixTFjItoShPkELsz1GzclnkbCl2dF7Y28avBfUTESQemf/oAXRC667USw94DjBcos8Lyf5h2LIpvf1rWCukXFekRFEoANyghcGcsFpPUoZILonOS91GovU1F+h6zEpeDs2wbf+Cri8r/tVuvatsyw7Nnp1eanvIi1tQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Tsc70UZvKWYWI+spZQiVZ3u2z8ZAo9dlRTusEOzzs6g=; b=czXjKQHfK+InGMIsUV3Y2zoX6mebVFmVAqAvj4yJh06UgEHIaFdPdBqwj1AWB6IDKpamySrf5TNzNbC5ojB19a/huaetCVQjxkYX2VXdEcjCWeXL3dBCo9N6CQkaWmQgenwMzy4uiFZervuaKIhL7thZVpzaZ3irQBO6SE93qqcGmLTkjhc+LO/wPzedi6BI7AAZE7Z0hS7bjbNdPfAgQls/+n3oOWTJCBruGARtIdlXtttj3zrk3eSr0iwawV1E87AKoT78DNTAVSRmPRD/DCMRszibxoBjQT4BQruafkbaqWIjgAYYb+cb/olsFp1h1wedGwupoTS4gLem7r6Kyg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Tsc70UZvKWYWI+spZQiVZ3u2z8ZAo9dlRTusEOzzs6g=; b=g4Tuc5vx5yDq6klYbhgJdI2+UdFcQdljiponoS4TzqPh8+XBKLRGHFXN8N0ShHyjJDvG/2FNbGzYFjDfSdfAZS3sKAT9P77jwtdEkTVpHlfxee2sw6+QDdXxBADpCttCRiL30xuIiVNwrVLyqiXV0lv1oyYcAG6AClqJjhj3tDE= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS7PR10MB5391.namprd10.prod.outlook.com (2603:10b6:5:3ac::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.38; Fri, 6 Oct 2023 03:20:41 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2%6]) with mapi id 15.20.6838.033; Fri, 6 Oct 2023 03:20:41 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Konrad Dybcio , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v7 7/8] hugetlb: batch TLB flushes when freeing vmemmap Date: Thu, 5 Oct 2023 20:20:09 -0700 Message-ID: <20231006032012.296473-8-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231006032012.296473-1-mike.kravetz@oracle.com> References: <20231006032012.296473-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR03CA0357.namprd03.prod.outlook.com (2603:10b6:303:dc::32) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|DS7PR10MB5391:EE_ X-MS-Office365-Filtering-Correlation-Id: 5d80be4f-e16b-4ea5-1abf-08dbc61b383f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: W/cYXfL1LFwvYYhs9Ds/a8TPTpSdzs0VMGXU5UsO64Oo+Dc5jyRZ4eV4ymkdLdHZ60VZhdCHWgC0DEU1JvgyAeYQqiXeOjXLV13/9IAPIhfkd/awo4qnZ/3Q51iP9vNFt7HOUcCA7UrXZWNJqqBdp+7Qcq5AkR5yiA1VLS1vXmzhfTA7PtR7/FUwgHcNd0PDIdTnatqE22nqkP08GXIKN5eUDcmQoI/N8DNuypLLyHC2/0ZwROWiX5S4SDvQfdjD6OogxinIiH/7Aa3faVDAx8kjA+HeUns34JSJcfUyUJe0t8aFACUCuB2zPdCEW2tMGbhKV0zTbOuuIEN9jUZe/yf/43SwDAD1Z8gN/03KJDfjhdJwNjG8KoF1zHT2t3jJsPoNU1Fs+jqjSuZW+crktZBZrttMDh5l39Plu/9hjNOt8WTFBwbR9vEMtBpTBeJkYFf/xj2QI6QCea2L/nYonuqDvPo1ZR+dCZzCiZehtoFgZbGZuyQl0yFm5t7UNIAuYW+7yRHcsUx3JbSJRyD3xF+ytzCtN0XL0CX/vdqRU7qJ2AOs0YjXzduGm695Z4eA X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(366004)(376002)(136003)(346002)(230922051799003)(64100799003)(451199024)(186009)(1800799009)(6506007)(6666004)(6512007)(6486002)(478600001)(38100700002)(86362001)(2906002)(7416002)(316002)(41300700001)(2616005)(83380400001)(1076003)(107886003)(26005)(36756003)(44832011)(66946007)(66556008)(66476007)(5660300002)(54906003)(4326008)(8936002)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9tGa+2DSHFIMvqeG4oW3Uw3sRQ77YNnIrRFMLm7KF7KXi6PbpqP1pPUo9Zi0obEJraBICepfd4hY9jHdvl79wgs5EjKXnTe4b8mAQt9txuGskn5INTgsqM/2E222EnpK1ai/ppQAZKET3ed1OjXegZ5qXNpP3jQQrNKYgQDqUo9WQHwcrad/Oh9dfO4Wm+PUQId0xLS+OcZDI6yWli1GqA0M9mw+57Etq+J8ndeyLLymXg2RRPoMLCfwfu4zmn3ZNP8WV/QNAmrKaNaQyKs2hLBpPtkgfEXzWV3TJw2D4umzTq/Ro+5AWfrm4/L3jFfk4pJx9k0I7BihYUDmKORBi1ndLRtXa4Sh05FSQhk3UcA+lsB7rLrSWUSAQphSnTIHU5b3DY9QXMkVDsM2MF0cs/jvlocsGdbf3Ke7o+zN/J/kXcqP+EkeRd0cOI2Pifwzglre/crx+3SncpXiG2wPIxU9fVZuNMbNlbVB421SqtiG8PZUFAOVqK30jsGhrqDyiYlmULgCnP2dyyEu0YGE2BaXHD3idthMBqEFZDpQq2Uvd+HOOzBj1Cft1qoFToFG77jSXaceazxnv5NC0DJbBjSdgsMFayE1yk7ssLfkXLXw5B9OJTTv6CGrpiC7BkwrlR8PpuJRUlDqnTH6wsPPPcV6b+PQUZ8tno1ylvDXhZ5l5IcdPhvMl7L9I1yLUn4LXkVoUpevMLUPlGukqDCvnXmS3MPGZyHSSmhl6DfIvsiYSw+JyI8igpNXy/Jom8xhw5/489lN/fZ4i3bVWRQZUrLZ4UngZxb6UtIngdlBow2KZ9PoIVZ6TPLuYNpetSDseXTQ9SPQxASBjMw/RVgEFxTBJQKuQQYfM532H19+fYIcBK/QcROcmXtEGu0HvitDql4E/TIfXSLvFes16aQvaLl0ZDeGjJ594ck40CFLRfeWlyf0jzkjDoMaSYTRXPPgNlk/YjFZ72USLmwQx8VNiMkug3QuDDJwQmKToitzIa3qDQg2/6gBDSqv2FD288/C0H5XDJj63uCks7LLhHrH9HKdTp4ANnj1dhDmfand+LZaB6yVuK+AxyP+m2OzNOrxwNpxHJDbagGOhV1RPzzVCUaH41VS4ltqNJVVM6t6uAKrBu8Qw7HkWPbC8+/AcAsjYy2laGHZ7pXg6Cfw3vkEiJ+jeZ3ikrSAjlspJixzR+tGLCs4aj3SfUr2NLJBCwjIUQihGGH1Bf/bcdtzIG9sIdzhZDHaPriRxjg4JEL/5+72y1VNNdxfQ3/6BE5v5hGRGaFC1s164P+wcoFobwr2eVE81NBtqyeSyWxE+QQPRWYU9wGQcwhIcKjjdgMFvWsLgLiPWN1Z1Gx0q2Pm9g/cI51K+gratrxCGzCOnsNHlqKVYY/EEqapHDLFSW+dnbKnDAuG/+jydirSNO9zh7QWohP6SK0NHbei2PX+QJWEaiAvR3Cag/ZKvylvpOO4Xsc9u5G9U8TMvCMN8FSleiUSW3CKD5KQHFJqsIPF1WiC57r3rNVfD7YidCAloxTUB3nTL7TgqfEW84YD9yCJYyo8+eXwYhhSvpRF4AEvVLh+GG0MKES2d3+rDRI/rjPlLEIPGYmr8bdb54aSQYxswWzSqw== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: yLpf8DyXI//t1sGbnSBP6Ts/fSd7j2mwwpN4gOox/Z7mYjRWmHysTCCOkXY6AEZuaH/3VbvY1strOqm1TcdNbquJ2YZcH/0Qe6QalY6PKWV3do8cmYQcm8PmYx5KN0Zxt0/7oJtBVZ3uP3DdPuYdUSw2QdJg8S6S4TA+E5+nebIwqAIoakva+J/bBGMFDGDTVoy9rN1ouMMwenWroAw4Q0w+9j+YMFBayCID1mX/OZ5npllR5DKlt7cz8suHsHsEvRYkXpxhmO5D5kq2FFQeNsRNj2/LveuLNN1huUd8UstdBhV/kmb4vp1YikLaF4lqSMekVn0xtzdNxmEi7snqCMM/8GGOFqt1O2UmopkPYjlrYKr6fFmqeXxvCvLXn650ShKJ1byu43a5qdku9tn6xT48ZIl1HRIX9GEMLSWmZYP8P2OJKy1og60M/cUL16p6GZEAkooF5CuTLcE/i3nv61kmyQI54tsB6Zf96gl6NidjWadh9riAHvgHUsjJ+3RfVFycQz7cvpGcfuYdIJOTawjH/F0H2VMCyh5uN8id7aYKl4ZMeNouV76yxbBMlTA5pEKOpUvBDbhs3Jyp3VNW0BOf+51dDeQ74m/6sIPEOAJ/waxDRr+QWNWSx3edP+nIhymwUXXKatviZSwHxixi4lFWop6b/TriPteI8Vor5Xutw5Vt/riBA98x18e/PuE461sYKGe7uHMj6qh+HQyWHVJtOyiy6pxNjLccqrrmKNDQxpM+D9FO6O2uA8oT/vpJxZ2JimZ0Qyn9zAszFpGJeGelqh8uGIbIPoThGH8Hv4l/wSMTwUMfsCcbOS+ebLgHCOnt/q6RdhoV5IT/JUEQv+Yas+Nv7l/FKN8pYiRSboVlEhHHKkQOO+qltyyvTqf+f+CvP4rf3t6CthYx/YKLrZ1dRCDOmjacq0kJvcIHii5fzWclR4vCksifyQKDGRgMfCVqq/IorX5MOdnF5a6lcwJ2Di3ayE5FhMtXiNJkoLuy4D4TMoqiuAmG6lE2zck8HYdUyKaeZZvC4C2WpmYZpwOB/NRqAbGWSDDFRGVrEBJpy/8OXRlOCizTcj0/lCBflHonQMNlHXCs84hh4M9KNpnlUoNfo+g/49+Rik7GfHMiSnq2iUptU52I82DCrRQVgRKw71u5O8W4k4illTmojg== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5d80be4f-e16b-4ea5-1abf-08dbc61b383f X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2023 03:20:41.4226 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: D0RhtxQO8uLswloaQpO6+HggiAeSMBiUdm+T4jrIdafgVGUa4OeABoz3ZAJXTPiSz0eLDyxqGBDjuhP2FqyyUg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5391 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-05_18,2023-10-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 adultscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310060024 X-Proofpoint-GUID: -47XGJ4857WhJEnnl-UfNGV4jqLyXxaq X-Proofpoint-ORIG-GUID: -47XGJ4857WhJEnnl-UfNGV4jqLyXxaq X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 05 Oct 2023 20:22:16 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778974771450936320 X-GMAIL-MSGID: 1778974771450936320 From: Joao Martins Now that a list of pages is deduplicated at once, the TLB flush can be batched for all vmemmap pages that got remapped. Expand the flags field value to pass whether to skip the TLB flush on remap of the PTE. The TLB flush is global as we don't have guarantees from caller that the set of folios is contiguous, or to add complexity in composing a list of kVAs to flush. Modified by Mike Kravetz to perform TLB flush on single folio if an error is encountered. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 49 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 10739e4285d5..9df350372046 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -40,6 +40,8 @@ struct vmemmap_remap_walk { /* Skip the TLB flush when we split the PMD */ #define VMEMMAP_SPLIT_NO_TLB_FLUSH BIT(0) +/* Skip the TLB flush when we remap the PTE */ +#define VMEMMAP_REMAP_NO_TLB_FLUSH BIT(1) unsigned long flags; }; @@ -214,7 +216,7 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, return ret; } while (pgd++, addr = next, addr != end); - if (walk->remap_pte) + if (walk->remap_pte && !(walk->flags & VMEMMAP_REMAP_NO_TLB_FLUSH)) flush_tlb_kernel_range(start, end); return 0; @@ -355,19 +357,21 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end, * @reuse: reuse address. * @vmemmap_pages: list to deposit vmemmap pages to be freed. It is callers * responsibility to free pages. + * @flags: modifications to vmemmap_remap_walk flags * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, unsigned long reuse, - struct list_head *vmemmap_pages) + struct list_head *vmemmap_pages, + unsigned long flags) { int ret; struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_remap_pte, .reuse_addr = reuse, .vmemmap_pages = vmemmap_pages, - .flags = 0, + .flags = flags, }; int nid = page_to_nid((struct page *)reuse); gfp_t gfp_mask = GFP_KERNEL | __GFP_NORETRY | __GFP_NOWARN; @@ -629,7 +633,8 @@ static bool vmemmap_should_optimize(const struct hstate *h, const struct page *h static int __hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head, - struct list_head *vmemmap_pages) + struct list_head *vmemmap_pages, + unsigned long flags) { int ret = 0; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -640,6 +645,18 @@ static int __hugetlb_vmemmap_optimize(const struct hstate *h, return ret; static_branch_inc(&hugetlb_optimize_vmemmap_key); + /* + * Very Subtle + * If VMEMMAP_REMAP_NO_TLB_FLUSH is set, TLB flushing is not performed + * immediately after remapping. As a result, subsequent accesses + * and modifications to struct pages associated with the hugetlb + * page could be to the OLD struct pages. Set the vmemmap optimized + * flag here so that it is copied to the new head page. This keeps + * the old and new struct pages in sync. + * If there is an error during optimization, we will immediately FLUSH + * the TLB and clear the flag below. + */ + SetHPageVmemmapOptimized(head); vmemmap_end = vmemmap_start + hugetlb_vmemmap_size(h); vmemmap_reuse = vmemmap_start; @@ -651,11 +668,12 @@ static int __hugetlb_vmemmap_optimize(const struct hstate *h, * mapping the range to vmemmap_pages list so that they can be freed by * the caller. */ - ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, vmemmap_pages); - if (ret) + ret = vmemmap_remap_free(vmemmap_start, vmemmap_end, vmemmap_reuse, + vmemmap_pages, flags); + if (ret) { static_branch_dec(&hugetlb_optimize_vmemmap_key); - else - SetHPageVmemmapOptimized(head); + ClearHPageVmemmapOptimized(head); + } return ret; } @@ -674,7 +692,7 @@ void hugetlb_vmemmap_optimize(const struct hstate *h, struct page *head) { LIST_HEAD(vmemmap_pages); - __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages); + __hugetlb_vmemmap_optimize(h, head, &vmemmap_pages, 0); free_vmemmap_page_list(&vmemmap_pages); } @@ -719,19 +737,28 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l list_for_each_entry(folio, folio_list, lru) { int ret = __hugetlb_vmemmap_optimize(h, &folio->page, - &vmemmap_pages); + &vmemmap_pages, + VMEMMAP_REMAP_NO_TLB_FLUSH); /* * Pages to be freed may have been accumulated. If we * encounter an ENOMEM, free what we have and try again. + * This can occur in the case that both spliting fails + * halfway and head page allocation also failed. In this + * case __hugetlb_vmemmap_optimize() would free memory + * allowing more vmemmap remaps to occur. */ if (ret == -ENOMEM && !list_empty(&vmemmap_pages)) { + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); INIT_LIST_HEAD(&vmemmap_pages); - __hugetlb_vmemmap_optimize(h, &folio->page, &vmemmap_pages); + __hugetlb_vmemmap_optimize(h, &folio->page, + &vmemmap_pages, + VMEMMAP_REMAP_NO_TLB_FLUSH); } } + flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } From patchwork Fri Oct 6 03:20:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mike Kravetz X-Patchwork-Id: 149095 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a888:0:b0:403:3b70:6f57 with SMTP id x8csp61650vqo; Thu, 5 Oct 2023 20:22:52 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG6DbL3JomPjUDloJ0gnR/skxavqVdASYIJAdO/V02JaJnZlVyAuE/D2LSP0PChEFd1WB56 X-Received: by 2002:a05:6a00:2e84:b0:690:3a0f:4164 with SMTP id fd4-20020a056a002e8400b006903a0f4164mr8676364pfb.19.1696562571943; Thu, 05 Oct 2023 20:22:51 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696562571; cv=pass; d=google.com; s=arc-20160816; b=pUGlkClqLUGkwy0/f0RswV02GZWk7dIY8a6Hg/pVlge8HU77tD3jfG+D6DNqlxZvhp Nh9Txqj76m/dvyjtFocnLKGyUqdX/vgdtOVR5Y4TiTd/WM2DmY/CYLKlceOaKXgY7sTP 2uB++exy4xR45mJW3EVUNrg0Pu00GLdhcrL8GaaY2HdeGmbOBKfH1g0d1dKQrvlNISpK UI0ed8XcWR4zt9JupsXhW77b+d+LfhFhO1KS/mYPu/yYXvRDhdRxDbdd8oj9w2GxR+xM oJvZG/F+HKIdBR09cSYLW7XtLHGiWirHh0TsuZAitfl4K32Q/I/DojFzDcXU/W78kz2T CV1A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=ts++G+BYdmdUFqdAtNKLn3GW8JSwEPzR7WlqhqbkpgY=; fh=d5TJywUWx1H4J8STZuTxM2xyaY/BETP1UBu+8XkSxoA=; b=TIWUWlq501l7d5cGj4RfWosUjLm79p/fxWeuTEsJh839J6iIP8NR2DaFkSIv2tmC+i iWiN0tgzFCHDTF0T6V/BN5WgovjA80EKt1xYlGsO6L+kFA+f8+UY/JIIZ0L5F9a8P243 SPAHp7raDwgyWgkpJk9LiYaN/OK//bKvPUxNWZmFGujc10KOwojUUMl7oXPSrltV5L3y lw9OBj8eHjlqB8Zil0ccPtpL4vJVg7IMUMTqwRIAZI+vWKO/nb7i+qv8iSG458ALxw2T X4Hd+p8y4CelFpV/Eq1JDsIrW2PIGKqub+Cmf6qeGUglLR/VANAu2rbECPD4ti+dHG2N 9BYw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=W4L9Ypjy; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=DECk5jqG; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from fry.vger.email (fry.vger.email. [2620:137:e000::3:8]) by mx.google.com with ESMTPS id cn7-20020a056a00340700b0068a4ba92eafsi605957pfb.54.2023.10.05.20.22.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 20:22:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) client-ip=2620:137:e000::3:8; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2023-03-30 header.b=W4L9Ypjy; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=DECk5jqG; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:8 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by fry.vger.email (Postfix) with ESMTP id D473E808724C; Thu, 5 Oct 2023 20:22:42 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at fry.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230119AbjJFDWG (ORCPT + 18 others); Thu, 5 Oct 2023 23:22:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35496 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230003AbjJFDVs (ORCPT ); Thu, 5 Oct 2023 23:21:48 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53FD5D8 for ; Thu, 5 Oct 2023 20:21:42 -0700 (PDT) Received: from pps.filterd (m0333521.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3962Pdhg002406; Fri, 6 Oct 2023 03:20:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2023-03-30; bh=ts++G+BYdmdUFqdAtNKLn3GW8JSwEPzR7WlqhqbkpgY=; b=W4L9YpjywekoEZjjH0n4FHcRYLam6SSCpSinxXSFhbpAD2BBVvWOYIu8rFOzNdskjVjK YTE+Mut1YdouHqJJ9OArvuFyCAuh3wVynhNiHUERSMM0A2fu0X2d6MVLvOKxAf40KdNH ZcaY7egjo+yaX91/KxPl3+TScufb6Np9W5nMRk09ZqI0IDPDxIFWh1iX/dm2TN7K0aqJ icug3BoAQ0ssyvqUhCNjc8dH5xUiV96nNuW2y+lZQ/zeSKB3rbz0i1f9UhHPNBetO2b1 RmaULL/02sfLMy74XNEObMz74l/ZSVfI5NviQ0frJ9EciM9WGj4Os2UotRqfDgZ35CPU Kg== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3teakck2e2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:47 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3960toAS002971; Fri, 6 Oct 2023 03:20:46 GMT Received: from nam02-dm3-obe.outbound.protection.outlook.com (mail-dm3nam02lp2043.outbound.protection.outlook.com [104.47.56.43]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3tea49yq0h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 06 Oct 2023 03:20:46 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VSO9hTXC0YBw/ofp/+LUk16b+mwsW4t+kUFA8uCgYFPWyusK9En9SBgKazaWHwH5c07pdtc56lUb8wNrrWl/xyEra2C/c2LZ06jepVXhuDhNaIcdwet2jpXQm0qNjfIsQLJ9+nBT4j3AX+PDodNLEU4gjTHizjZRiAfhme+bct8sTm27mrVAyFivXZ3UeOTSX3J3Qftg2bscgDbNaxZu76coGfSUOd/UGGXQFQRvJ0aWTTz2qhEi+2NfXpIZ0ne0KP6GRNc4m0h1XsLZ84qK18v9V0Qr+m81IJY0Lg7bQiYezBYZAjajOfBYkM7UYl/2ADgR8upx3ordJXeaRwE+uA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ts++G+BYdmdUFqdAtNKLn3GW8JSwEPzR7WlqhqbkpgY=; b=ER6CjpJ+4Vs9KKv92IfWbfZeYFlWzUsfu1IHCoIP9SxjHgfSFNtKwpFU2w/sUkVGv41gridE2e/9/waTpC6cWb7GcrTMGX5d3eMZmrAa1KU71FgVX9RtTZ51NrOCd5ycWecwPhsxfgkIfcLSsH/Vw7vMuak/2wflK/KmD4lset6J3O1dsy11ZTJnFuVGAj9Z35oFrkxbozLj6z7M0sM1RKRE3R+Mk/iTmOFz+hNcVNo8wSQfEHjUnZcvAjUdT4RMK0HJUc9YZ7nv/a5843Trmz/bP605edVh0qI6ZilDxSX0RA4p9B+ayPil9tbUIRAIjNPBC6r0tUu5LdrMRrGj7A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ts++G+BYdmdUFqdAtNKLn3GW8JSwEPzR7WlqhqbkpgY=; b=DECk5jqGuq+T4ojIJKIEAZe3DxoCRzIX6wuem4Q41tLq15H0EGRX1gFntkJSZc9T3VcSHGnmhqgH0X75wQC0ms/I4iw5Rb/v0OJpeHHrOg9v9UYU53Zxv2pKsNkq3LF88SNBuDvPyzCcWll4wmN4Lbu5Sq1aaWJDNaBuDmnX68g= Received: from BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) by DS7PR10MB5391.namprd10.prod.outlook.com (2603:10b6:5:3ac::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.38; Fri, 6 Oct 2023 03:20:44 +0000 Received: from BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2]) by BY5PR10MB4196.namprd10.prod.outlook.com ([fe80::59f7:ec45:eb41:d8c2%6]) with mapi id 15.20.6838.033; Fri, 6 Oct 2023 03:20:44 +0000 From: Mike Kravetz To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , Joao Martins , Konrad Dybcio , Oscar Salvador , David Hildenbrand , Miaohe Lin , David Rientjes , Anshuman Khandual , Naoya Horiguchi , Barry Song <21cnbao@gmail.com>, Michal Hocko , Matthew Wilcox , Xiongchun Duan , Andrew Morton , Mike Kravetz Subject: [PATCH v7 8/8] hugetlb: batch TLB flushes when restoring vmemmap Date: Thu, 5 Oct 2023 20:20:10 -0700 Message-ID: <20231006032012.296473-9-mike.kravetz@oracle.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231006032012.296473-1-mike.kravetz@oracle.com> References: <20231006032012.296473-1-mike.kravetz@oracle.com> X-ClientProxiedBy: MW4PR04CA0283.namprd04.prod.outlook.com (2603:10b6:303:89::18) To BY5PR10MB4196.namprd10.prod.outlook.com (2603:10b6:a03:20d::23) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BY5PR10MB4196:EE_|DS7PR10MB5391:EE_ X-MS-Office365-Filtering-Correlation-Id: f4302c32-2a9a-4014-55c5-08dbc61b39f3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uDUhy576Vei0M6EPV4/DzehaHW8FWsqFhiFJDWZ7swelFl9XFE5xsthi6SkoOLIdxgjYYOVNc2oBa+daIl8tqfbQw26zqX5z1jIqn0P805kRepK0svoKTyajO/3tU+pP/N5keBF1XbKeH8XSpXSEB5pERCQUhbIGQ1KSSagAKNL8jMOoCqpzD4Nj79bxIu9hGXta0ZGmqsQhlzTCTWgihH0YUAHTNSsxTT2qj/DQxtdnOlW2owMV6J2xDChH6axCk3INhnVnAo6CM7kd6OpLxsUcp5a4+OHUuXDzJl4mB+cT7c3Cjvjx9PvQ6RPid96AWW4nuFXJixd8hAjumr0JBD/tbPHDZnO1INRhO0P81f3kxm8lhlUvuTtX4jXa5kapwA4G3x7moPvjxGyfIhctbpCZow+s11aHHOQ8aQSsff7OCQI/gl5xkUZoylmU34XrENPt51g7ZETKzeOa3hDp62K7yEXUvsW2hwvmUyyzzbnJsnURL6s6ILioe9LsnJAKsQDBmvmyF3IzU77gHfDaKi1ZD6mwUQrD3SEHsSs2rsGbluSfR/tgNHcAARKzaqVF X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR10MB4196.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(366004)(376002)(136003)(346002)(230922051799003)(64100799003)(451199024)(186009)(1800799009)(6506007)(6666004)(6512007)(6486002)(478600001)(38100700002)(86362001)(2906002)(7416002)(316002)(41300700001)(2616005)(83380400001)(1076003)(107886003)(26005)(36756003)(44832011)(66946007)(66556008)(66476007)(5660300002)(54906003)(4326008)(8936002)(8676002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: vkGV686udWlCGJwfKCCS7dwnslBcyl++oKGE0APluLZRGOQXYNFU475kPzKCs6vAiB/anYlKA3gDN248A53XPuyAwiQkjypkfV7mNdqNacU9lCkUO951+kSAz3wCBIBgfXZunLJok5vqXvY7LNzdYdfS5farhwhsUmJUpcbUY9dXSCfN1QPfThusVqM/opa3+BEqnx892OOwZPFWP/hKq/FK6rARvP33RgAAS8y3N0H3GRTzOFe1prXAdwj+9a4HxI1BQuJEcSSmmuP4t58ueff09joyNOi8lEb9+2bDRuUYCCOV+HkKdg6Pr1Zh0V2TD0hRiGAhP8VvM1k07l3vCmuKdfNvAN6le5Nj1RC+zqB70XlT/RzizSoygdIZbLHMc0g0bNx6TVR0djQjelT+xCL0+jGmf2A1j9PFMcGByNUOAdahh46TIXfUFG0+Uq0DL4X7MtRNKb4OzgYOz+1eNgfRTH11fkpwKmfOQcDEeslllwLF0yHbTdpFzy+t5ayCFO382xO234ymySqs1+l1e8ZQIjBMJg/KTlrAt59jNuYt0ywnoCUSExsZZkhQARCunh2ZILiBM6taCpHCtkHvsBBC7KFBg80gmSP9ig8leH0Z3xnF3+ePa9GugQnFwr8A3vPsxCJs+olOousSSJ/eI+49EH6KbP7WaOZeDb1sxeY9+aszMthJbZs7go7XFTBQ7gGeCteJpz/JUMKlvRJtnolGXrv9xZ9K7+Xqj1DphmpXeMM8rrQVMSHv8nk/ioLWHG5uSsK2vyedtZ4D6kAlPaGRryRQ046wK0tINmAZfg0S+DwS/aDLwgUm9vwzbyr2Yth8WpL+lBdAxA30rKhTokv4Qo5Qfe0V7+MAzvKpRKk+nIhao3pvUlkFtgcFEJWJoNAM3Kr+Hj1nxU8xbu0MQ394T2gyFKirUWPduWSc1YbKGPbxe88sTXO9d0K5f+W3SnYYwwNBNcjbaeFCvvyUQUNM5+Iu/giKgknphCwJdLjfQEhQ9Ixb5lvayhpD7AW27eEYo3zYUvLANo2K38827YFoP/46c4xqPUhwRRku1dgrgZcb3Zp0ZHgRB5ZORz7Ol7v3x70sls52hy9OWC/DNeQkw9tcuBfvDb4GCH62PFkVUm8mQg4cFwCySij0F/YW/3jsgOW/Q7infv/qqPu3QDRLMYQZ3qse5Hm8EJyeKFEIhbOwDqOjLNhLuERplkfJUpTDIB03I/ojXqStfzKhq6lL4kkKHOLiFFXOxRTAD6k9Wg4/9AqtQk/HU+cZzrsX0wJGvShb3Zy5qISMYXuRJ0gR6z+2Eqar0V8kWLynFC6VpIrEVOZIyKxQVc5AtMUhnu9VnD263ZJaLZHuKnF4aUQjFc/RGK2+fKrzK6ZDWQP9jqLFr2x/SmRijbMOH7dYykffVETMjqBnykC2kmqGeqRKtIfz237IN6/HglwhUXaaLcEnSntZnS5La8F7cJKD0NOCzoJ083BjEl7b6vyQ3gYdUHzGYlbDaG4e0Orddpt+tqgPieKZEyoBZ87GMs/aHfI3aV5zHu47v93DgcDQBCjxU2fnxrh2RFja2Ekc192R9Q1nJYhG3X08SpRNHfP+JQeg8U1eOoyK0lKdDIob4g== X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: sYWOesysSsb4uk/T4Ws9ERaH/CmViu4yJMGmLm+c9AxdhWRg0ncyzobQ+C3flwf6tWserEpAVLhtDDL679NdDnkSGIB0sRev4yaSoRYUIAfatXOocFDnwCrl7y+ZFt4pOIM5dW/oc2f0n8bTi1vpfuZKHAFXJvv/DEypx0ERn0FGPSp7vGgmgbzU3W9yQ0hwh/avxlkl9afWWt0DRC5y9L+oAWsYEtq2eswKJXTZtmVKruLWc8lv+FSdnpHgVIV1/5z5d6dAgMgB0Vp2XY3q7hmb/xk6QVHh5gVZcpa6GYDcLvmxXlfLHP9D/B97c3gDi7Z1GMmn/Mf9hqrYJS5LOnq5+ogL6mT+CMW54IKy4XxTPApseGCotLqqPrhUb4TNEK2sJNVGADswKeRTPRfUw5biVAb5kl90AY9IizTcjggtBuU64Verd7OHiptlXZ0to6+B/GZW8PKQPfut8y1WZRBrX7ANgRfRs3G//xDolKtk7KE0nkpoIn9XaLnhHqwbSksO+rXuTY2Hcia/1k1HvuxYQB1ZTcnlnDqQhy15xbwv8lUV/YjIXECNgwbjadi2JCst3GnJuhG7YLBs72IXuL0s0mlQFy4kH7R7ZhGY9rYdKKW9kiMxHRzJ/w6CDPok2kpcpgkYbuRToZOf22JzZmHBRay9xHqeHgX6t2KVhzHbALzShcxILt/44672ERu3FGVpKnG18F/ULD7E/n7i6mRnPnpc6IP/R1EdobjMhWDun965LBNq4CNEiFSmHqcRfiecWY/Q3rwOKUdE2EPq/vfYAmMrutAZldfcpK9iHJwz74I+nlkJYGJH9jj6QBsFXenmJvfXyITTI/4syjgRcOcJA1ailbM9t8FxKf/s+mFFYYLM7lahWeoxubzzWV37EL8jdhF8yTm/E7bEMyehc4wBO5EmcVFySAzn0/W8WRtH85gPf7OIyjDYzE6vlxIN+VTTuMKvvb7tSldb11hqwj9AP1bv8lXL+aAPThl2VNQQ6k9EMIV0RtAZEsOkSjvfNLXnjKoyJRyVK48obbF0aAq5hbmktAtZqaYXHXPfMz3weoTP1AXD3PYNhwAxD3aKVE/lMrnHK+ax2yXsOa3SZdteQ9jrR6UfEaw8GBMBokRyNOWDgvIDGFeqIehvCP8SWSJhDLQw7wQr3J29W9RO8Q== X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f4302c32-2a9a-4014-55c5-08dbc61b39f3 X-MS-Exchange-CrossTenant-AuthSource: BY5PR10MB4196.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Oct 2023 03:20:44.2980 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9QhQSvPVReA668Q/6dI4PtjJ+ySQzzt9C+CMWRMNeMaEZB4RFRjwNdZdVDABz8GJa1l4vwjdTA/UL28uXLIoQg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS7PR10MB5391 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.267,Aquarius:18.0.980,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-10-05_18,2023-10-05_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 mlxlogscore=999 mlxscore=0 spamscore=0 malwarescore=0 suspectscore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2309180000 definitions=main-2310060024 X-Proofpoint-GUID: hGyZmpWMRUcFCQu6hR3LLaKda783hnLa X-Proofpoint-ORIG-GUID: hGyZmpWMRUcFCQu6hR3LLaKda783hnLa X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on fry.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (fry.vger.email [0.0.0.0]); Thu, 05 Oct 2023 20:22:43 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778974795269850703 X-GMAIL-MSGID: 1778974795269850703 Update the internal hugetlb restore vmemmap code path such that TLB flushing can be batched. Use the existing mechanism of passing the VMEMMAP_REMAP_NO_TLB_FLUSH flag to indicate flushing should not be performed for individual pages. The routine hugetlb_vmemmap_restore_folios is the only user of this new mechanism, and it will perform a global flush after all vmemmap is restored. Signed-off-by: Joao Martins Signed-off-by: Mike Kravetz Reviewed-by: Muchun Song --- mm/hugetlb_vmemmap.c | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 9df350372046..d2999c303031 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -461,18 +461,19 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, * @end: end address of the vmemmap virtual address range that we want to * remap. * @reuse: reuse address. + * @flags: modifications to vmemmap_remap_walk flags * * Return: %0 on success, negative error code otherwise. */ static int vmemmap_remap_alloc(unsigned long start, unsigned long end, - unsigned long reuse) + unsigned long reuse, unsigned long flags) { LIST_HEAD(vmemmap_pages); struct vmemmap_remap_walk walk = { .remap_pte = vmemmap_restore_pte, .reuse_addr = reuse, .vmemmap_pages = &vmemmap_pages, - .flags = 0, + .flags = flags, }; /* See the comment in the vmemmap_remap_free(). */ @@ -494,17 +495,7 @@ EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); -/** - * hugetlb_vmemmap_restore - restore previously optimized (by - * hugetlb_vmemmap_optimize()) vmemmap pages which - * will be reallocated and remapped. - * @h: struct hstate. - * @head: the head page whose vmemmap pages will be restored. - * - * Return: %0 if @head's vmemmap pages have been reallocated and remapped, - * negative error code otherwise. - */ -int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +static int __hugetlb_vmemmap_restore(const struct hstate *h, struct page *head, unsigned long flags) { int ret; unsigned long vmemmap_start = (unsigned long)head, vmemmap_end; @@ -525,7 +516,7 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) * When a HugeTLB page is freed to the buddy allocator, previously * discarded vmemmap pages must be allocated and remapping. */ - ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse); + ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, vmemmap_reuse, flags); if (!ret) { ClearHPageVmemmapOptimized(head); static_branch_dec(&hugetlb_optimize_vmemmap_key); @@ -534,6 +525,21 @@ int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) return ret; } +/** + * hugetlb_vmemmap_restore - restore previously optimized (by + * hugetlb_vmemmap_optimize()) vmemmap pages which + * will be reallocated and remapped. + * @h: struct hstate. + * @head: the head page whose vmemmap pages will be restored. + * + * Return: %0 if @head's vmemmap pages have been reallocated and remapped, + * negative error code otherwise. + */ +int hugetlb_vmemmap_restore(const struct hstate *h, struct page *head) +{ + return __hugetlb_vmemmap_restore(h, head, 0); +} + /** * hugetlb_vmemmap_restore_folios - restore vmemmap for every folio on the list. * @h: hstate. @@ -557,7 +563,8 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, list_for_each_entry_safe(folio, t_folio, folio_list, lru) { if (folio_test_hugetlb_vmemmap_optimized(folio)) { - ret = hugetlb_vmemmap_restore(h, &folio->page); + ret = __hugetlb_vmemmap_restore(h, &folio->page, + VMEMMAP_REMAP_NO_TLB_FLUSH); if (ret) break; restored++; @@ -567,6 +574,8 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, list_move(&folio->lru, non_hvo_folios); } + if (restored) + flush_tlb_all(); if (!ret) ret = restored; return ret;