From patchwork Mon Apr 3 05:22:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 78354 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2079182vqo; Sun, 2 Apr 2023 22:38:54 -0700 (PDT) X-Google-Smtp-Source: AKy350ZjvQrqBP/Xg/x83FESe+VCeXkZB0g+4TGwc4W8Lq8YKTt5CDj8+KYMdWSv/HpREEO3m+BG X-Received: by 2002:a17:907:7f0d:b0:946:fa88:5901 with SMTP id qf13-20020a1709077f0d00b00946fa885901mr18518614ejc.63.1680500334722; Sun, 02 Apr 2023 22:38:54 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1680500334; cv=pass; d=google.com; s=arc-20160816; b=cwzJmQMETaWES+EmPvFDCtcsvMRHZhQZcE99hrnZrj57ujM5bbP+DriDk9+ESsTNu4 oV45OEpLrY1DNF/HLRg2Jfyz/xBlJ0mg3stavX/eej13gWmj57YcyvOCQs/z/Ct4Z0Ea ew7MHvMmfgFrJ/IYMraYvllgbwiVZCXgsGs5b8rrAucwcnQQqNwPSM8sTaMWiT+2hZJR gdlxEuCcN9QSYQed2hPr3b8pfnzzZYL2Jw3XpiYMnvpePbivLudZ0jm3mNJWrMQff3DZ f70rgUVyooad34xUmKd2CHyaSxGMzmbB2Hy3jdAlj8YFci8WPen0/uTCSPaLAHqLIp3w NFEg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=1N6qbt+qhUVgQy2JQLXyJCdS15y0cyZtppYubbAlJkY=; b=Vr605BEGpkWo70hSS2QCFEQZr/czVW284QiKs6AAk/FMZZV+WbngszkpiGKvlRAjbO 6f3TUBVpg//wiNRpGVvR6IFKbPdrIV6TFZ1uJZDvZ59ZLxoqncEKwo2o/KvgdA/EZccu fOcTATIJyuL1l/zCCy/vyFOOYkZunt1faGcNKoaA++w+Hc017V/IHY0yE5V8/A9/zhaq /5kSBIBvQocJH+ehZ1MlFlapy1QzpkodSy1atML3UHf4rf4nTMstsNYyLvw3zG7m5+Cl rQFAhhUKF5WIhHvLv6s8OvI+zj0MDryh3wH//wkQKbpZCE/LjxOjJKrrsfNo04vEGgnY vqww== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b="hOua9Es/"; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Hy2ugkgh; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id pk25-20020a170906d7b900b008ca8e21134esi6775155ejb.281.2023.04.02.22.38.29; Sun, 02 Apr 2023 22:38:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b="hOua9Es/"; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=Hy2ugkgh; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231559AbjDCFXp (ORCPT + 99 others); Mon, 3 Apr 2023 01:23:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231509AbjDCFXd (ORCPT ); Mon, 3 Apr 2023 01:23:33 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F14EB45E for ; Sun, 2 Apr 2023 22:23:14 -0700 (PDT) Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332Msls0031744; Mon, 3 Apr 2023 05:22:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=1N6qbt+qhUVgQy2JQLXyJCdS15y0cyZtppYubbAlJkY=; b=hOua9Es/E4R9NqhhOatqoqZs0Rk2TIVxG8ieeM+AIaeT8PTbTG4f6aG5P3XYQebQckMh Q7tqwvhoRemGGUJtPBiaXpXX4V/HUdTJ+9RvIUoIYlgUMuFpztsuaRrXhrTL6sm6eK3m j4qgPoHynHSCmDO0iGTwMEh+j7jopfyWrGGuErQ9vmx7vVm1JGrQvuvLcc7LRxohv3m/ 65SDlRj3p7Xl6yO04aSSGdpOBlVU11Dchg7rBGytx0iCmLZYnsHog+6cgqKr99MpN9P8 8idfndT1TjaLUodlXT8BYpXl1BwWbJWtWqL5Z7ArLAzN2eWJz2a8/TsglHor0XvlvMkS cQ== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppb71j7h3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:30 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332Pl6K014039; Mon, 3 Apr 2023 05:22:29 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3ppt3djufx-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:29 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cgT8+ZSkRpktst4jWPVexS1fiEbuJ1OI+a6c/rkVxC6U0BGuhmHoLA7Fy5+gxiZX1o7ove0JeJoNkyP8OSIJFmLdevl4jCBVtNUfNdDKuoj2ECLJ+6FCF3bKZ3+Kkjn1cA6ktMrwiPgCIuHj6c5y9GrpvbJ0lOouJatJ9oumUVvRQ2Cl2d2fYHH1GkWY1EF11S9apULhbUB5L9j3LKScKDPRfMg7GZhTgq4n2Gi5w/0yo1CAXMAQv/dN2peOduXGuyTxr9nvaVA8sJWFtnDAaDKnvcivTsdr8VYMDJX4VNrvIgCm6i1vQh3BDj6wC8fTHg7AwIvJcYgwEF2btEa/Bw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1N6qbt+qhUVgQy2JQLXyJCdS15y0cyZtppYubbAlJkY=; b=DPwVOGIGxn22GTFqi7mQAzNX0gJiWj1Ft/Ehqp5+9ArhjiEvq+iDd0yOR3XRsH+3mOZUndGU3hD9TZ6DeVst7kYoq1jIPSCyjp5YfdXxQwIqeod4lmKlXWuJrmHwFQ7O7jYdiO61UfwU4dWj5iSQYrp0YsVnl8LIErgGA7d4yHTtmxvgsPNHzMY7ruxi1HVuavqH3hkp1ESnZguUwUNaUdgIhO4VmdPwO4Kb0Ufk+7ShnYWZMB2doKdipX4gv/Hu0TRgMWXd6qYShY33iYmJsj9O7GHvTpXJxgn5WEwXDEAUdwwNLoXqPJUUDFo78opItjHn47P0p0mJW5Tyg/9z9Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1N6qbt+qhUVgQy2JQLXyJCdS15y0cyZtppYubbAlJkY=; b=Hy2ugkgh/AFwOS0PC/TBdVB5Hmafo3mRQA0a9UhGsOSgIVsD5vG3FifpRaW8hATHCe1RSAJym1FN+01vEBCVZis+ayoopfIy46YB66jSKtkNYeBirdwGDgrXy5cFlkcG9cy2UMkuAD2ibjM3YYTGa4FnwNPSXPjj5oiHDmwUK3E= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SJ0PR10MB6398.namprd10.prod.outlook.com (2603:10b6:a03:44a::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:27 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:27 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 1/9] huge_pages: get rid of process_huge_page() Date: Sun, 2 Apr 2023 22:22:25 -0700 Message-Id: <20230403052233.1880567-2-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR04CA0267.namprd04.prod.outlook.com (2603:10b6:303:88::32) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SJ0PR10MB6398:EE_ X-MS-Office365-Filtering-Correlation-Id: 45791c6b-8e75-4f74-8949-08db34036a2e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YD1Wo4q6FHinxM29vI1SsUONfNcIqT9VEQ0j15jqAUFM/+cbXdp4DWdXlZHyKcLQsbrFBn5s0nqALWiE6HEEvJccQu4cfVJ5l3y5ukUiX6qYZX2rtXREyE0L9R7bNEFOAZgnEO1Py+HdkN9qTQNxxRVKGfnQJ+Eeayz+XBlv/9G2GVVdtCGeXj8TWS83KxHhuecD1l14vYQJ6hSh7+/+79sMvvdlWO+x2XqIA/yZgo7SZQva0r8lVNRbb5WDvqHhtCQUmBk1PMtTAzu5U97KHYIgTbjVfSYxOJNuPVvfGCQBji5S2+Sg7hz3g3Hgw32e58YVw8oxAAI2q/JJV8qzwJ93gb8eOpKE/2g7WGkuZFT5VrEgUffc673O3l2tOjsOrDWPGeotzZLNOynELxAk4iYb2Zlp8fCwx42Wr+gzSmIKxRQFEbJaR27Ns51bn2IXToSs/4Xruj+Jf8zqFZnTZ/1VVhPNWKUA4eDpn7xWWsJcbi6Tpu9KWgtwe5JzeWgfJZorDWt8AHOg8BBX56IyM8SGGvHqy80Bv9dXP+9lPBCBGejQeml+FfAQEHsJEOihXo06jN0B+3lkDmvlxzH+I0xhu3fZclILX9tiszBa8DFXC30ePmmkoXDpdgUUfPMH X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(136003)(376002)(346002)(396003)(39860400002)(451199021)(2906002)(8936002)(38100700002)(41300700001)(7416002)(5660300002)(103116003)(36756003)(86362001)(478600001)(316002)(6486002)(966005)(107886003)(2616005)(1076003)(6512007)(6506007)(26005)(8676002)(4326008)(83380400001)(66476007)(66556008)(66946007)(186003)(26583001)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xLIrrbNyKGOhvDeQ72mC7nMogOYXdEbwUTXgTaGJnbwqAAdk5HV8hqRh1w1PYb0nFE3Z+T7Ujw2t2NcGqeW0+FTrzKQz6iv0QyDmy9MGkdmAuEbptn6Zm1zXRXGUogXxunRpf9R6bIQB+X1jRInCrpiX7SqnozmzKL4jPFEMwl5Zd2ssoOHpKhw82NI19yKK4irvB3oxUuPqy+3emd6igEiBGuYIG+tC/8GqHPFNLYFKSD+wkU62LC+cNpTCXABdhMMu1jvg0o9FKaveaFFDOHpfWuFBDAwyisYaMd8I6qZ7ZIRNS8vkHnBDmcF9aVNFcWxqYPuA5LQHb3T42kQrva+Pqy1RkIrk33VdMpcjFADhv7IoH9Qms/iM3phL43nPvZ/kpS0H4TBLg1odXUoOqEiIpbZxcIcJcpVT/aggxXoD+IAlBfJwzutINURmwBDdfLLesSZdxi6uEoAEzx8T/1NkPZIw5tZslAiphFbivg8mYnRLyCPpIBqkdebXTkKR4UtktxKaiYyta7KaWkfOv2PSCySB5MWlYpifmgNPUza/bEBW+FeAXQlGmxXsGYI2OCYVJH5ukTXKdDnmexTJfHTPjQARNH6pKHJVOYzO1h5sz5oHsF5dvCuOHRYgDzNZlZLYS2zy0Nc3iwn4omnDgPyF1gHzsESHyO9GWe/h7lPpJUCIDyitwvNHSh+vZUmMNfzGAPwwowV2Y1uLExv/wDPsxjpyK2hDXbbqzgIUeAXaZ9A7f6CetEc++lpacTbyT15ElW0uOIc+RG5x1WMlfs8e0qGHQKnHWliBdu60v+1o1megK83z8+blplS/H5iPK7eO49+xwizjhOFmWxJW3FNipsrooeLm344j1xrFgLa8yJ7HMg12xbXKo6Q+TMJOmaXddr6i+sGXjgE9YkBsle7I4JXK3Qegg12Zpc3KFVS/ANMcfLVgRqwhZBBA8ug6ewOZ0OD9ic2rZK7wayucB2u5gT9yvs0VCjJb74vkFVpAxquEJwJvHev9wPHABfrJra6spzS7xtRIF8VxxRi8eaLJIf/+w8AzemMfythu+O/Em4lw2s9HB9kLbDVpcDqDYRouG64k/IVEzFD/S0zVUETYOunPDIoVql+I1J39pJSu39ExyQFnHnDSRMxnkab/5UlW+eoTtRNhXcfaMASii1OFY8AKYvCXmLxAaa9wLyiVFLDvEHVrXr4jv9RQbtyRtUNuzPTgs3Jqf71jnRI8yVHQS00iHJddYDkE2YtoeXLGbFtY7490WjGWYMmUDwXBTmM9SWncqBi8MJp78wdH4DrMsDawTqYaZChHaUV2u1IfSW6giRWQLFW/BvqJ4PNeSjLQTStA5Ui0DC6XNcgatxrFnY6E2DFSCx1RHTDWdjNxgF1xwbPyPsp8zUNDcdZOzNI5r+AHQlKsB+PpCJHbl+vilfrxl3M0xEnqGm70SRx9VSlV3hyzbgxTD3Q/bsvft307mOHHW7O3/yQdwsuOQmBfNFHsTGmkbINSu3USC2Sv7BxXnGMmjvvKa4TL90KWKqXUg/xY4sfO80kLIwnY2ELSmWx6vNt4MMzThOvcVdyFUP6oRVenUXaylgGgrbaO X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: V3NALSOKYpCCpvh/qyzIZhyKL78AqK4EIAxQBe3+nkeExfNerw+O8GYUl4gmcSI9PV83/vqx0AJa1NJox8G4NXX3KDoGkbVZdvHqhxSDIb1MnXgzFDSwDCx27i/mQsrhqgMS/Pp3Wn2C4NcBzi8+C3Km0ZoyaYdQ00yOso539jEdaV8onZ64oQLO3F9qWW+NCHFxkSn4hQD8s2w+F3aKJNNkbGLxoo04yVduGatvpr+AQW6suF1agXbBlIVflUxJF8t9x9kM5S2bEkTZfOXCg5ub2SNejfy+YPmKRsCIPAB+03BbXliRRWnoNEwhzhaKQjPQjrKrOQVitx3UY08La1ZmEpi5+Pxfb8Mr9RLT6+GB/1gRO4bReFNKt/pUxWjOBO4FdzdCe/zQp84bt9FquOboByLpnz8Z3LML7vLHH3m90SsDUu7q3j0e0YFHeg5RxGxZ844qdEsiXCPUVxdH2UeCy5tJzfTDHQolIjaeZ3Bhj3gtoSO9I88bKndBsXM6MB+t99f7EwMmgm6ocqmrL5ulDO2XdmVpR78BfI9FhSLqaL/bRAjIoqwtGiYajhim8WBNscUbJs86bF+EznsXRjX3w7CQ+kD6LxJSnK5krHSdnguzUYvJhIHzCsPIOt6D7P1LNPQrQIvlwTPDdW/RSeQ6vAqmygaZup193b6vmQA+as9duYhJdtLPoZFxTfb6v36ctWnanMulzS9yx+/k+/1Hvu3V7lUviYgIqbNtaj2M58xfTAS5Y3fLw+b2zWersQW2Ffs7QeMS7FMkx8REuB/Xe7iC2M46I3hvjJmqctO7SUeRqkPMQujY/qhNri+MsVzZqIpBS51xOAdqs9D95ejv0xLfqofBW64CuFI6pLIART2Y7Dtl2mnGSuvPEZKVJEL+F4HbEKTK1FXxyg4jUvZW2bXKIzChhBMwyrm4Um+f1hfPx8WG/wOvKlDfojqGeO6SI7lrGxh0MN2EPhg6kIjTL6IEj7zhAxsDqHPv+REBnytAU9Mk7gNzNNYHjxYfIEg53oCgmfGsipiEdaLVukW1zXi+npMBsTtZSOqMyIZCwIoEFqgQSDTLUt2cy8nETOWhCNPxFJ/oB3z0M7mWJZWyphwdNpB/dG6RK5/3kl+nLJRN0JAbPgm6wUc2JMmAGT/Bse76zDdngZZScmgylI1m4mC4fPvuQ/3eM6YC/wU= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 45791c6b-8e75-4f74-8949-08db34036a2e X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:27.5865 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: QocwTopNBygm3UVFFOScbZPlsMlf7MXtjZkZkqwCu5BuBnY9oSMBt0zKthnkf45CgCdJ7F9e6AKy+TyaJX9fK1tThqOLe03lq9uozJEXAzw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB6398 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=999 mlxscore=0 adultscore=0 suspectscore=0 spamscore=0 malwarescore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-ORIG-GUID: qoiaRtD32JN3hRfot9KfEK_SPa1ULIEs X-Proofpoint-GUID: qoiaRtD32JN3hRfot9KfEK_SPa1ULIEs X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762132318933266849?= X-GMAIL-MSGID: =?utf-8?q?1762132318933266849?= process_huge_pages() (and the subsequent call to process_subpage()) translate between base-addr[index], struct page *, first when calling clear/copy_subpage() which then translates to the parameters needed in clear/copy_user_highpage(). There's no runtime cost in doing this, but it's unnecessary complexity for something that only has two users. Accordingly, fold process_huge_page() back in its callers. Link: https://lore.kernel.org/lkml/20220606202109.1306034-1-ankur.a.arora@oracle.com/ Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora --- mm/memory.c | 126 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 73 insertions(+), 53 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index f456f3b5049c..d54bc27a35ca 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5629,63 +5629,13 @@ EXPORT_SYMBOL(__might_fault); #endif #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_HUGETLBFS) -/* - * Process all subpages of the specified huge page with the specified - * operation. The target subpage will be processed last to keep its - * cache lines hot. - */ -static inline void process_huge_page( - unsigned long addr_hint, unsigned int pages_per_huge_page, - void (*process_subpage)(unsigned long addr, int idx, void *arg), - void *arg) -{ - int i, n, base, l; - unsigned long addr = addr_hint & - ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); - - /* Process target subpage last to keep its cache lines hot */ - might_sleep(); - n = (addr_hint - addr) / PAGE_SIZE; - if (2 * n <= pages_per_huge_page) { - /* If target subpage in first half of huge page */ - base = 0; - l = n; - /* Process subpages at the end of huge page */ - for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { - cond_resched(); - process_subpage(addr + i * PAGE_SIZE, i, arg); - } - } else { - /* If target subpage in second half of huge page */ - base = pages_per_huge_page - 2 * (pages_per_huge_page - n); - l = pages_per_huge_page - n; - /* Process subpages at the begin of huge page */ - for (i = 0; i < base; i++) { - cond_resched(); - process_subpage(addr + i * PAGE_SIZE, i, arg); - } - } - /* - * Process remaining subpages in left-right-left-right pattern - * towards the target subpage - */ - for (i = 0; i < l; i++) { - int left_idx = base + i; - int right_idx = base + 2 * l - 1 - i; - - cond_resched(); - process_subpage(addr + left_idx * PAGE_SIZE, left_idx, arg); - cond_resched(); - process_subpage(addr + right_idx * PAGE_SIZE, right_idx, arg); - } -} static void clear_gigantic_page(struct page *page, unsigned long addr, unsigned int pages_per_huge_page) { int i; - struct page *p; + struct page *p = page; might_sleep(); for (i = 0; i < pages_per_huge_page; i++) { @@ -5707,13 +5657,48 @@ void clear_huge_page(struct page *page, { unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + int i, n, base, l; if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { clear_gigantic_page(page, addr, pages_per_huge_page); return; } - process_huge_page(addr_hint, pages_per_huge_page, clear_subpage, page); + /* Process target subpage last to keep its cache lines hot */ + might_sleep(); + n = (addr_hint - addr) / PAGE_SIZE; + if (2 * n <= pages_per_huge_page) { + /* If target subpage in first half of huge page */ + base = 0; + l = n; + /* Process subpages at the end of huge page */ + for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { + cond_resched(); + clear_subpage(addr + i * PAGE_SIZE, i, (void *)page); + } + } else { + /* If target subpage in second half of huge page */ + base = pages_per_huge_page - 2 * (pages_per_huge_page - n); + l = pages_per_huge_page - n; + /* Process subpages at the begin of huge page */ + for (i = 0; i < base; i++) { + cond_resched(); + clear_subpage(addr + i * PAGE_SIZE, i, (void *)page); + } + } + /* + * Process remaining subpages in left-right-left-right pattern + * towards the target subpage + */ + for (i = 0; i < l; i++) { + int left_idx = base + i; + int right_idx = base + 2 * l - 1 - i; + + cond_resched(); + clear_subpage(addr + left_idx * PAGE_SIZE, left_idx, (void *)page); + cond_resched(); + clear_subpage(addr + right_idx * PAGE_SIZE, right_idx, (void *)page); + } } static void copy_user_gigantic_page(struct page *dst, struct page *src, @@ -5759,6 +5744,7 @@ void copy_user_huge_page(struct page *dst, struct page *src, .src = src, .vma = vma, }; + int i, n, base, l; if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { copy_user_gigantic_page(dst, src, addr, vma, @@ -5766,7 +5752,41 @@ void copy_user_huge_page(struct page *dst, struct page *src, return; } - process_huge_page(addr_hint, pages_per_huge_page, copy_subpage, &arg); + /* Process target subpage last to keep its cache lines hot */ + might_sleep(); + n = (addr_hint - addr) / PAGE_SIZE; + if (2 * n <= pages_per_huge_page) { + /* If target subpage in first half of huge page */ + base = 0; + l = n; + /* Process subpages at the end of huge page */ + for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { + cond_resched(); + copy_subpage(addr + i * PAGE_SIZE, i, &arg); + } + } else { + /* If target subpage in second half of huge page */ + base = pages_per_huge_page - 2 * (pages_per_huge_page - n); + l = pages_per_huge_page - n; + /* Process subpages at the begin of huge page */ + for (i = 0; i < base; i++) { + cond_resched(); + copy_subpage(addr + i * PAGE_SIZE, i, &arg); + } + } + /* + * Process remaining subpages in left-right-left-right pattern + * towards the target subpage + */ + for (i = 0; i < l; i++) { + int left_idx = base + i; + int right_idx = base + 2 * l - 1 - i; + + cond_resched(); + copy_subpage(addr + left_idx * PAGE_SIZE, left_idx, &arg); + cond_resched(); + copy_subpage(addr + right_idx * PAGE_SIZE, right_idx, &arg); + } } long copy_huge_page_from_user(struct page *dst_page, From patchwork Mon Apr 3 05:22:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 78358 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2081452vqo; Sun, 2 Apr 2023 22:46:27 -0700 (PDT) X-Google-Smtp-Source: AKy350aDfLxfaoiTJcrzK1oXyD3rpC0JBzbvssKWbOy7WVH2Ubx3HFFeBwrswlkqnKNyMA389HS5 X-Received: by 2002:a17:907:8d0d:b0:931:e0d2:7f5e with SMTP id tc13-20020a1709078d0d00b00931e0d27f5emr37071711ejc.22.1680500787204; Sun, 02 Apr 2023 22:46:27 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1680500787; cv=pass; d=google.com; s=arc-20160816; b=jHFjRq/5q8a8dlxKBZe9rDHHadOH5ILhQqlsRvnhC6fNQXF4MGEqt3yVJifCAmDccd nLsHCgVTNmXw2ajpz9TPKNxtA38mU9JqLdDstaw6qQUF/JYn6Ws6/JYPFhro0/dYYVVc /lrCUjot3wluUqA9Mf69jg4Q6loy7r236EsInUje6z9FgmmCXDIc/HTrT94vaeGHs9ZX 2GqAXsYEAMmMW1Icve/y/vW4pdi2XCyrak4t567D7KWQ1SCH9hFR05a0qlzFL8rP4v7g aCpYWH32jpaTdn+NvsRncmFtgSUHHfVIk4p0VyN8lW1un2EpDY5ZajS31R1vT4WAOMzI J1lQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=A+gbjMRQ73fBFl4pqWuQodz1X3vjS3hPwXVwaIKhxTY=; b=OzNoXKsa1motqdX9C83O7idfQe4Qp5PvrkffiI7rLqTViiIigBQuLfUCDXwEforvEP 0+NK0/mCUGXbhwI4j1u+8DNw2qvjyHPikTOCcLRIiUGQnItMDjzepoZ4vlzQ23lQ0uBs tJKSehwNDCE3+XMTjLlVzZTnK7gvpRiAp8UhOk23WOWcF4pDkORT1fg2/MUaEMCMaQiS jnjaZcSrVDWQrnkgQDjQpn+Tze49IfMIjhO6d9UbwqINW0gcJZkUCIlDJiFJwjIcoIrg 9FSzp2lFiOWNrFwhgE6FR5K5kvnlJgu8LKfO9LpQllEH+w4PtER3/+tOyQumVfpruf7S 3Xzg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=x12Br4NU; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gZnsbqt1; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y13-20020a170906524d00b008dffd95e783si1315133ejm.568.2023.04.02.22.46.03; Sun, 02 Apr 2023 22:46:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=x12Br4NU; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gZnsbqt1; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231538AbjDCFXh (ORCPT + 99 others); Mon, 3 Apr 2023 01:23:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231496AbjDCFXb (ORCPT ); Mon, 3 Apr 2023 01:23:31 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 103443C28 for ; Sun, 2 Apr 2023 22:23:13 -0700 (PDT) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332MwUai024750; Mon, 3 Apr 2023 05:22:33 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=A+gbjMRQ73fBFl4pqWuQodz1X3vjS3hPwXVwaIKhxTY=; b=x12Br4NU5hCMefRthWdz/zXcq/h36XvBZA48N62EX/+ton8v+E3dG+tJGyTl+HWC5CYa 3R/xWCSHtFlGQ0TvYHzP+HCDkeqyMDCmtxNcJQR3tjyg4OaIHP/X8EPGwKzmn589Z+kV o895JU7vuzXpwWg5ecYA+4wU6UsmXKKq+bxDCKRNHDsDFa5zq0WdqQr1LbybRdn3fvYW 4ZrSZb6OZF62Xjpl3O0pBAehieMZIDnAebyoLvZhTYbBSr2axYUAnyI4kMiwlHjZYivP S3Uv33xF03xhMj6Vm500pHUJHyY/WiCm00wEO6qKDvE0SmgD7pq4fwa+lX8OOJdeW+iB KA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppc7tt43u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:33 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332axZw001034; Mon, 3 Apr 2023 05:22:32 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2176.outbound.protection.outlook.com [104.47.57.176]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3ppt3eu93b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:32 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PK38zv7jQIUEySx9bR3gfkcP96Y/lsJNJ/2lLjaN4SCHIAK6YNGdOCeG9fWdPGIT94L3bSiq9gEePfG9musfn7D5ep+hELd6pHwAbBa76odzDVgwkeqQ+SW8jBwEeN2xPXuDROqVs7HzPbYgJTAXX1X716tXTJ87TTOCbzj1xkD56r+Bj6Rf4GT7P5smXznd3AsW18NMzKTpOD08wzNxuq8Is97dY1m1J9zyMqz0/+9LGhnY++qe4SKzB99y0rQlLyPwkX+958qqGNm4w8+ZtPueeeg3p+8itmMZUvUTY/u2V9DMXBSuqNA6vGThZR6ZjS1+85IOD16Ac2KIdJAf9g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=A+gbjMRQ73fBFl4pqWuQodz1X3vjS3hPwXVwaIKhxTY=; b=NqhwkBJ3T/wnZAMZD5mxvms9F0PXji9OnAwpss5ihUcnnVHufroYwY2SI+hRZbVsjR0jSkufPUAQvu18d4edzGm6DeZr4wBPoqxdJ5uRyNSz2FYrhHP0GBt/6gzXEu0OkH3Hxi45LqU++QB+w0LEd/7T54t0A1It0xYjpfXA1E+SUEGyiNFEQy/1mtr21GkCtsbVvldgJpwDrFdxKY4x9J+xT4PwIW8SAuJ5lXRkMfJQzQ+c3l8OsNWlzwFw3dO8g+wBMYSev8Mf/WrTRD4rMKYtNekUt89lFXejm5MlhRL3h8tbbaS5HfKCJgzs2uEa+Cn5RKsQjTQv5GdSZMwTnQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=A+gbjMRQ73fBFl4pqWuQodz1X3vjS3hPwXVwaIKhxTY=; b=gZnsbqt1OxjTPRecjJxa+fqCRJ3Tf4P/dVWfyJ0tpvTBW9djklyptSjocH+6t7VzT+SuNxkbV3bQtoHx7X/uusH3+bfjsYWMos/rh/KY3sd1HmkMdSoGV5K3Y9iCG4vjkt/OrGlrqY21ih96ObA3+mRhMdzUOf1nKO1Pxe0zUHU= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SJ0PR10MB6398.namprd10.prod.outlook.com (2603:10b6:a03:44a::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:30 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:30 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 2/9] huge_page: get rid of {clear,copy}_subpage() Date: Sun, 2 Apr 2023 22:22:26 -0700 Message-Id: <20230403052233.1880567-3-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR04CA0266.namprd04.prod.outlook.com (2603:10b6:303:88::31) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SJ0PR10MB6398:EE_ X-MS-Office365-Filtering-Correlation-Id: e6a7e217-89be-4ce1-1099-08db34036b92 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hyOmS+UFryA5XzOiwc0PreKDQKORHMZ/T0+Z6zKBBhDZP6xUWuulAkf4F2T7HUUdU+MbLk8dLNHOyL2/pLq4VDin1P+t99ziABeDAjdmUtX1fqbZ84ydFA8Mf1AmaTRgePtlIzAGMkOF9nWhaZWTCfBRwRn0ZxpB8BOIztkhN02ZeQSVOIAIBAqe7TRnDJMVcxrAjz2ty2RLMmro6KTgoYNlPz25jN1T5ZkSq3/1Jmt89NxonohqRT8D7thWSslUcsdVd/k7E6Obh1rRmKa8JCQgHQpgfPptdCzPNzs6eYf8YOJzQbouQi/hfGrb9KTXNQiujSt4ERvuEHxx6EnZzq7Vs2wwJjvdG/wu36MahN4XqTrsd3PwzipuzNQ/J9BLMBGwWdklHcbvi+06WDofvBWXZUvsPCgHp0qIjy2rYDsewD0JhaT+eRy0tPhDZzaft0LAIr0G4YD5vgMHIaOrCeV9vTpsxJgkUwG4OmV8w2Ui9Wed8wFtJmskmxZ0qT/W5/Qfd1plm3kBhy7AAkh23ze7XO90/lNBb1mrwVdCOg/5V2OT/NOsgN8YftfXYdoMvyFz7umRBPKisXNxQwF00zK/jVmyJcFQkDdRoOU33t84z2rmAxYdOCNndRRdyno34BdKCka0MJDv29/19dRKKQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(136003)(376002)(346002)(396003)(39860400002)(451199021)(2906002)(8936002)(38100700002)(41300700001)(7416002)(5660300002)(103116003)(36756003)(86362001)(478600001)(316002)(6486002)(107886003)(2616005)(1076003)(6512007)(6506007)(26005)(6666004)(8676002)(4326008)(83380400001)(66476007)(66556008)(66946007)(186003)(26583001)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: j0GKmuPQY6MndwSvByy3LbGQ10SgoBLdvUkrTeVi4lfI5ahCqdVRGXHtJH8IobDGgh3plyLEKF+ZDdwqCySZRx9YvhY7H8GplmD10B/hZ0TLwggl9lu9GJ2HUbsjD2G8mm3hSS7p0oo8cBJyInE/7qmZK8sHAgvNpeQI6tq6O5mCwa2IY/JbOaHxDEE+2R+In9N53UCFb81bNqT6y+SOr2+dCDCur5fbPKuiF5lXu+rnR4WH4bHALYa4P2VLg1hTFQyN1us4l8a8ZaKL3ohobCiNL256715/gI5O4j23+k6tyLhNfnKX9ey62NwBuBeOrt2456IcVU5bJxw+C4LHAnXuKkaA8ZKKPECZ5iVndpgQQwng5Zzj3c9PfUKX2Kzn9KorwsVl23SOY6i+x9TdmZmv/LDH/N1Vnt9hrFCsJLeAX3Aq3lpeldhiYcxFSFzi7LLM1/u/CvbCgoipZFsfX5xFf08M5ipHv5n92AvKfk18m5VWfFZ+ZsXfIatOY62ov91ggRrJElTA8YSGUUXbJMshpNJMc7ZXzqqM9/S5xFn0BKWf3rbZghJN71HHGP8so5B9o9GnBUoIragBNNRLaQrhXTPIyYTF7QIVcm5z0bKRyt+0/eTedLG3X2bzD6WPd4q3OKym31K5OH5gmZI2ilgA29Q1Z+2j8WoLb+6UJvuWOfLx27HER2w6GsqM8A2ekoMsau9FXnvh7SVkIMcyCttW521Mho5YRrv67yPdeyfV551e0cgujGmsKq//Xo12C3cixP/RFvqAVU01WbdHvezZBOwl//2cHKyUcdqHpxhgHTMwwZDnx8eBy17+3LQjJaZ58yqZyCKXnj+KQOVHlVJC34cK028zD0C+fcMpEh5aY8zrkR1JpEQnul3tVkiG/wXEalZX4CNliASaavRC3yD+pSwYnm3nLmV6bw0jykTjtTUMTSEzlxD8gnz30GxvVCwzt12YzVjXMoDZ/sjiRW8g53ChLGG1cSHHEznc8OyATgDQqksnyebOSGoL2gkAymU1RX9CbuzB+sh8UGf1jwTQ27te7v15Atl+YZiX582isRJsJH3gIIEod2/+Cxp0H5KgOJFr9jACfb2J1Gc2LywOm+wJrsCcJ/1nAZn6dKKxlMVA9twZfivlJozfF8I7h10yg/agx6Nec6RO/keO/Cqw9S8bq44U+rGrL2lxzxauLFmVJiESyGHAJAEFAXAsow/gUX+eeksXdREEPjVO9OW84aZNFr8oy6BdinpLegmw+8ImWpjB1SLVJT500o1A4WEJzevGvv4gGyunmmVy9bH6/w909/3zgiAFY7SBWCsqawIZyXkdnWPV2hEV0U8PSmpjp+V/qObAXsm1vrASPYO749JN13Oiot/j2PmkzUUKXDSEXNF5gUaTE0e2YiWUgTPKbrwUzoicqKaV/yKvzNK6jAd3ivrNLwtGCiZ1Z59XYi0DNvhuMaCg+7K5HVdDXwJzW8DCv7IbyJIxkaHJLu14MkIPG6Ithez8PgytJ8egQZhDJJdCl1JlALSylGtm7dArE/cs1IwAWL49QbDlMnRVu7xzae3LCyq9MavaFWIs04TvhpjnvehG1fz/qImr X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: kNquaCKMtyHag+0Yg+H3/LTeiyLlfuLyHzIqhWalNpR7d+qq3pLDIcdy/L4O1ApYlf75EQBlhv0LfBW16Vq0bDmHma2r5EAE/LY4JOhV0xeAR6K40oMhlDyCkw2fX0/1QaUzHBeX01GXyGMEdFtVQTXaPDnbFdY/YfTllVPDS0HlFEb/7WiU5EKf65zhOj4mnAt4vGxZaFrz9Hx07fU0J0y/0jistZ9SglfDcPdQ+L7rjYpdMuHi60Rl0qDbaGJ6nhU91sawHkT9oYPfvT/K5Zb2Ir6F0VQgki60jP+uc3/qtKQfdzYwnaWJ3RfOpZPXzqQSpm8oyTv0j3Fe5jQWDuzXGbSdu/exjYDiTyyBcjU2f2G42IvkGQLGQOJEHDfvSE1yZ2GZGedJlrJEC4FKSvMqEObOVG7HEU2rpy+pE1Ojhq5EtOMtL5uqn0Ap3oq1A9KgG2WY9LhVh3I33FKaJgklfDW8lXmsi4BVrOA/PBEVnv83h32Uhs2NNdc2ZfdfWMFf7Q1xJF6/7oUcNntsoRAvp+GgxYfd72CfQ+psLollm7d4C5R7Ww9Mq3lNnMML/YOjy8NDjvb0zBuGWd6WNudqs4ZLmyYCMCXiV5o4gJjEe5LdvFZdx5rGlE9ZRMRQKKt/Q385F8NhM2pxZvI18kzI0XfeOZE2CKZGzfBqPWrwzocg/SZ4bK8dZos9s07uUHtd7P/BQObfob9NXlzHW8u3/vavcoeAYH5CpQCDlqo05L/HrXdj577Kl/R8PRG7S3Y9ezxBoT0yefaOhGdNueQ1gw+qr/aR2Uh/J9bmsAQh1TlSHjiteOJD5/sn5b8x6og8UV8oteFFjgCgn9kc89XBkXDRBosn5M0FQV7jokEI6TUcxW0Nsf4TX/qljbhsYPlCz1J1XSMsLXXShJdbgTlHZp0XGDGmXU4bJbzPA466Bv33aqWv+IkbkcCLF5psiX5y/R23HgrbBYxueagRmn816GiJcsDdy/4clV/EwsqQtCoxZ5wKRUWO8t24PdEDIKj0l4gc1T4O5WbAo1nVkhSEbaRCR0CIeIwBbpSJ1/JaySB35lXZxjSa0tD8t3+LI3X9pi+Dim7YbHAkiPpl+uOdpjFz/idLxPlsRHNftrImTawF0jIrCNaAfg+vZ8dTtPR9eNTYm+fyEYMrjtmgp49NOM+ZJpQGrQu+y2DgRnc= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e6a7e217-89be-4ce1-1099-08db34036b92 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:29.9656 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SM97iqTUjbDjuwc9MF3gvVh6t3meW7PO7sazvOEB2EbZ3LrjlimRfaySfs9gjiCEeF2Ms5n40C8pz+khJZt2zTTP+KAzOJxWAbF7tcRazPM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB6398 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-ORIG-GUID: YPAMp7xC3Fv8wlTX6rHaL5Ti_UiHFRq4 X-Proofpoint-GUID: YPAMp7xC3Fv8wlTX6rHaL5Ti_UiHFRq4 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762132793435872354?= X-GMAIL-MSGID: =?utf-8?q?1762132793435872354?= clear/copy_subpage(): static void clear_subpage(unsigned long addr, int idx, void *arg) static void copy_subpage(unsigned long addr, int idx, void *arg) take as parameters: an index, a post indexing virtual address, and a base page * which is then resolved using the index. Instead just use clear/copy_user_highpage() directly. Signed-off-by: Ankur Arora --- mm/memory.c | 46 ++++++++++++---------------------------------- 1 file changed, 12 insertions(+), 34 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index d54bc27a35ca..6da97e6c7d21 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5645,13 +5645,6 @@ static void clear_gigantic_page(struct page *page, } } -static void clear_subpage(unsigned long addr, int idx, void *arg) -{ - struct page *page = arg; - - clear_user_highpage(page + idx, addr); -} - void clear_huge_page(struct page *page, unsigned long addr_hint, unsigned int pages_per_huge_page) { @@ -5674,7 +5667,7 @@ void clear_huge_page(struct page *page, /* Process subpages at the end of huge page */ for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { cond_resched(); - clear_subpage(addr + i * PAGE_SIZE, i, (void *)page); + clear_user_highpage(page + i, addr + i * PAGE_SIZE); } } else { /* If target subpage in second half of huge page */ @@ -5683,7 +5676,7 @@ void clear_huge_page(struct page *page, /* Process subpages at the begin of huge page */ for (i = 0; i < base; i++) { cond_resched(); - clear_subpage(addr + i * PAGE_SIZE, i, (void *)page); + clear_user_highpage(page + i, addr + i * PAGE_SIZE); } } /* @@ -5695,9 +5688,9 @@ void clear_huge_page(struct page *page, int right_idx = base + 2 * l - 1 - i; cond_resched(); - clear_subpage(addr + left_idx * PAGE_SIZE, left_idx, (void *)page); + clear_user_highpage(page + left_idx, addr + left_idx * PAGE_SIZE); cond_resched(); - clear_subpage(addr + right_idx * PAGE_SIZE, right_idx, (void *)page); + clear_user_highpage(page + right_idx, addr + right_idx * PAGE_SIZE); } } @@ -5719,31 +5712,12 @@ static void copy_user_gigantic_page(struct page *dst, struct page *src, } } -struct copy_subpage_arg { - struct page *dst; - struct page *src; - struct vm_area_struct *vma; -}; - -static void copy_subpage(unsigned long addr, int idx, void *arg) -{ - struct copy_subpage_arg *copy_arg = arg; - - copy_user_highpage(copy_arg->dst + idx, copy_arg->src + idx, - addr, copy_arg->vma); -} - void copy_user_huge_page(struct page *dst, struct page *src, unsigned long addr_hint, struct vm_area_struct *vma, unsigned int pages_per_huge_page) { unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); - struct copy_subpage_arg arg = { - .dst = dst, - .src = src, - .vma = vma, - }; int i, n, base, l; if (unlikely(pages_per_huge_page > MAX_ORDER_NR_PAGES)) { @@ -5762,7 +5736,8 @@ void copy_user_huge_page(struct page *dst, struct page *src, /* Process subpages at the end of huge page */ for (i = pages_per_huge_page - 1; i >= 2 * n; i--) { cond_resched(); - copy_subpage(addr + i * PAGE_SIZE, i, &arg); + copy_user_highpage(dst + i, src + i, + addr + i*PAGE_SIZE, vma); } } else { /* If target subpage in second half of huge page */ @@ -5771,7 +5746,8 @@ void copy_user_huge_page(struct page *dst, struct page *src, /* Process subpages at the begin of huge page */ for (i = 0; i < base; i++) { cond_resched(); - copy_subpage(addr + i * PAGE_SIZE, i, &arg); + copy_user_highpage(dst + i, src + i, + addr + i*PAGE_SIZE, vma); } } /* @@ -5783,9 +5759,11 @@ void copy_user_huge_page(struct page *dst, struct page *src, int right_idx = base + 2 * l - 1 - i; cond_resched(); - copy_subpage(addr + left_idx * PAGE_SIZE, left_idx, &arg); + copy_user_highpage(dst + left_idx, src + left_idx, + addr + left_idx*PAGE_SIZE, vma); cond_resched(); - copy_subpage(addr + right_idx * PAGE_SIZE, right_idx, &arg); + copy_user_highpage(dst + right_idx, src + right_idx, + addr + right_idx*PAGE_SIZE, vma); } } From patchwork Mon Apr 3 05:22:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 78353 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2076188vqo; Sun, 2 Apr 2023 22:29:12 -0700 (PDT) X-Google-Smtp-Source: AKy350Z8jd1SLOkAPFu5YkaJi1XGBLFQHGScJooHJxpQao2DQwE3rjZyP4R9n3hDGmQyh1eDAKjn X-Received: by 2002:a62:1ad7:0:b0:62d:b4ae:e48c with SMTP id a206-20020a621ad7000000b0062db4aee48cmr15796396pfa.23.1680499752323; Sun, 02 Apr 2023 22:29:12 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1680499752; cv=pass; d=google.com; s=arc-20160816; b=WrCjR+TQpzmH4wu2qbh0bAtJyWC/AHkYETrbEHEoAKhXdKjkQI4dGuKs/mFtgWs9o1 hj1zu4U5kvirZMsPYz3iR/QLn5xMdDI2177FgbCJa/eiYxMTuF8Ws4U79llJYEiOsQSP tQH7wuzvkeD8+q9i5s9d0EYR/+DncauEkA3pUbKsJF+ax1QRT/M+J10e8bfJ3BmjOr/R t2UwhQz80ECwCT5QfJVfGS8TbEhsl5k/SAXoRW6tEo768mD9VPFf0PpMvKvh2Do694ox fLqgWbn2qlGJttehAU6JPOHGgu7r/KRsYQb7U4X67/oD8GWv+wUNDL+2+F0Jq45rJh4h ymgg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=ohlWGuVeTFKX/ILJRWBRlxXLpQTPT1nzsZWWTdMkjxA=; b=lbn0Mtt+8uz9vK0/+YEv5lwXrc+R2RiFs1bm/teEq+W6VQc5Z3c9jjDl1ATXTNf0UD QmGbrStePZIAxOq2Ag4VFuZyY6bYJ2hIr3qC1JX5nupn/KiV2fOnM9tMuvIaEAu/Yb2B FeKDoc34e1/YkC7WngW1sAaNyOWe4qk2CTVi7PBn1i57qwIqeAmkdtR11rf2nk5REOCk NSQXk83ZMwfrTycLO//eQKaN24KaZ+HAmCBgRY6hoJ+9pdRIqZX6htBOwVYhlPUPAugu cDP4t7D/iaFQvjsEPDsSC7p7uKhWxbY0TAuEzw1+0PRwVnUjeKp//lWj+dB0Cufy+RNF TJVg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b="Umr8tG/b"; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gb4IdZuB; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v7-20020aa799c7000000b006254213ed4csi7892533pfi.91.2023.04.02.22.28.59; Sun, 02 Apr 2023 22:29:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b="Umr8tG/b"; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=gb4IdZuB; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231549AbjDCFXl (ORCPT + 99 others); Mon, 3 Apr 2023 01:23:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231520AbjDCFXc (ORCPT ); Mon, 3 Apr 2023 01:23:32 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E83FAF11 for ; Sun, 2 Apr 2023 22:23:14 -0700 (PDT) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332MU6pK007758; Mon, 3 Apr 2023 05:22:36 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=ohlWGuVeTFKX/ILJRWBRlxXLpQTPT1nzsZWWTdMkjxA=; b=Umr8tG/bSTqokt31/SwS8H6nVRyzLETUEd1pJhSfI38+SSBs9QM/3Kc43v9g1rLi1HWA CqJzKOSRWu7GYItYVXUu/o3vk5gcQ4/CChpYx2c1imXvcio0zt1QktC+wtdNcDRuaOAs oQaoeRmiro54Q8g/pKp2D0FzJlb4He0KZVcQppF9yJBJ/NosTL6gg4vg7ZgvMTJ6tCXF Hw+OgIiYdqoMs19OzftY2jdFYSxId9VzxdQufIPDPSdTyN5OXPY4I1WHFItiKkDanh7b eNP2QFmnc4lE0fh12qxcxnF93537u+cb2T2NEkFVQ0hjfvyKI7IAmTDgPA29v7YLqWeo wQ== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppb1dj5tg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:36 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332iZTo038319; Mon, 3 Apr 2023 05:22:35 GMT Received: from nam04-mw2-obe.outbound.protection.outlook.com (mail-mw2nam04lp2168.outbound.protection.outlook.com [104.47.73.168]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pptjptnh1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:35 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BTkekeb43YDGaytj8ibdOp8kbxQRmitOIpWZBjfAWkYAMpHeWKdgvyPNkf/GrJUDLUswOpW2FHWzE2Q7/RerKKsxPIqrkg+5TPC7CnkHhftMgdnCbAOcnO3RwFi3ydw5CIfap/7+1ItAVcC6bAph0XVZprTCBV0bxdfFDS9B4C7woqrB74DPbq8ISmkC+KQDiyEjMot9Ctjs4CobgYEQ2u5hsa+1H/z7CcbH5ruc7LvDU8aQRxSqc29hFxySL4o6AbolR/79vgALB9KSL/i2aMuCmfljQQUH03xKi6mqncnCJDd7cHoxh3Ktt9aS7oEOA6E1lbjFsUwFUnC0ALOBOw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ohlWGuVeTFKX/ILJRWBRlxXLpQTPT1nzsZWWTdMkjxA=; b=T3+lwwtbDWIuxosZQr7mbu1drV8m+YEFobi7qFxslRFa2qOUf44H4+Vl/iTljJ0JXfVyMW5lpRDgYwLPiOjZjZAdf8ee8d2McwO+CMuTN7kg9wa2yxChqSCmgHXzvDQ3N9b7mLhPlcSct9n3zMjkHK16BWLeVZvv/4wA1jOtuAopSAln9XzGYiTJ7XLdWWNNKg4hlmI2PymAAjuuxrWkzsK3pKx5CSPofIE5w80MRZdFgV5c5bxesJIMYHhgD4/rnjR5yBInsBGMsVRNxykffG401+2Aabh1PIw1t07M2h86eT7W3mh2mKqdSdvaB0CFRayL+hRB08yL8VAxCissVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ohlWGuVeTFKX/ILJRWBRlxXLpQTPT1nzsZWWTdMkjxA=; b=gb4IdZuB+7iEcqZpYLUj+q87gpc6O7iai0Da2r+tnrXyv7oVSoGzvjnVgQ5jKwNRxG/xVXEjazyXNqvXb11o91NvB3btsaVuLTDlU+abbadzphxZ31hERaxAaftrLrjB9wR0h6TJldYpg1+49Jqm5KuPAiNlVqoxiVTsixWkI9Y= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by PH7PR10MB7106.namprd10.prod.outlook.com (2603:10b6:510:268::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:32 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:32 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 3/9] huge_page: allow arch override for clear/copy_huge_page() Date: Sun, 2 Apr 2023 22:22:27 -0700 Message-Id: <20230403052233.1880567-4-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR03CA0130.namprd03.prod.outlook.com (2603:10b6:303:8c::15) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|PH7PR10MB7106:EE_ X-MS-Office365-Filtering-Correlation-Id: 01f37a97-c6f7-4797-e175-08db34036cda X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: g17nSWXUMfqWTm+Nexe+KttTvJn7LzjMkTOkxboC103/imibfWqRP+dRUAfy0MFiZ3jNho34I+Z5O8FmN3+0FXwMboZQuNpND9xd/2U0kbhed3oxKtcIp4ZFh52MnagA7VWrrRwHKIl/xpiEIZ12vK3J1DX0GnWPqWh6zgr2LMTQo9zWyWIAzJVhb12yWJytl4fhcgux6yreEErmcf8uvCi2b3niF+QVn7zLT4mduWYNbLPZzMThBR67aD2H1vR5GeQwB+RF9UbskebL7t1qCW9TWxSsMXqur96G7BDNzHVjpgLxJ7ql9od+TJP8M2VgHc3smjnMVdYu+3pZwb0bcnj5mYGillS9r3N6WudP+FX9Y2EapwbF1u1mTt8ntNF3wYUb+bjE3xD0wQC0jDYXYXkUO+zPcMMEHt5m6XQVhkika+qZR5KO0BxaPTmVMDL8wcHY+WPI19zyBmt4UNcxJV7J7jvCyoPwRw7m4a84JbiDFduLtwoamA3Yrmr0z9SNK3ZPKMSkmkvrWUQUks3/XvGAIqUwWZNVWPW4z4FKd5JguCbsoG/Z9py0JoQqvsuQ02F/29r558OAGgvOQzACYdo9GVkOh8bPQfIJmmjscAc= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(39860400002)(366004)(136003)(346002)(376002)(396003)(451199021)(2906002)(86362001)(5660300002)(7416002)(8936002)(66476007)(66556008)(66946007)(8676002)(41300700001)(36756003)(4326008)(316002)(478600001)(103116003)(6486002)(186003)(26005)(1076003)(6506007)(6512007)(2616005)(83380400001)(6666004)(107886003)(38100700002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: wlsYQxBY0Nisi8aeBxFdIVyAH4j2k7GBLV9PHGjuPzZYKNSHkfVq6//mQkp9vW5PW+R1Z6w/9s93iHnlLWjJAB3bQRbws17+WSQJND4Gfp3PvWRztXQWPMq8+qUW6tyAcNjZHf1OZnH1Cy3oMSRyLxI4kYk4wfCTroMT8ze/XtV9EWfq/qp/IjqEA/tPRrx+x5Vym+NwWEKG3UIvy5arWju7R8QKENBb2HAVdasmSZjAfAY5vp/E0iBBZb9KoCBuMfz743vKekPiT/mBe1h7zn/nVzXjty0x5wdo+QieQVpPTKFXKf2ha6jcwMMIpBZ5ML8pkoU+L5vXyhRMR4Izw8oNR2hqBNuuIo+4EGalOeK4SvGqIn0xFnmsj5MKXOgJBy9nQd4HR6ZIgKYrYGvsIuE6I/WE4t2ilh3JMBE2L/KoeJJtJ7hHaTIvLP1olsCQR14dSkpo47CoB3PbNyU8oTL0o7h0+gsGe0GageCqBcrxaUpQn867jMFxwdAhoUtXSzX6Tr/i4+SFJ/jvd9ibHI5AT6MKUp9K4q215mC7kKEbjB7M+jWXfQLsRLdcL/WTj037CA3mhYBi1FqRS0+gnEhOla/JL2FtudCOIxfPHEdcZp72HWiweEY3XKlO44LqSmHue/FRTjK7ZRznmCg2fBxFNx5xmALmkQI/wMwO1DCYPP+BGX7BCgMegE/qKeYRPpg/dNEcLJPcfGAonbamqehrkRW59fqafecZPLGh2F8bFotgqi1Cfp1lgojvzpTkq2XhWl+b3XRoDr+sfvsmBg4d/CtABr4lweIjsTgchSxPuz0Q6+tFiBuUOSoR5CFpQoLefbmrHvJEL/j2tE5goCBcxzyVl6iChc0+nBn4xW+AQFftwloS6W1RmVuVoXGjjI6Cm5FbxWr6l2H+Y/Kn3i6iap6XDVrz7R+eUc+18PpuAY6gFYGLs19L/bSxMxwTRVe+AWAssddLrKBWtsWpnJBawijj+p0+/dS0QIRO+VtqMue2qtYZPJhs5y07xGAOn0fRBvQVOi3kA8aQ5+CHNZCljyl4JZFEIsP3I6UenBNVfBzNmyIJKs7SNRoCYM9ljDzSFr/zV4xAG78VJpdWgoh0uOj+KYEkaEfqMdjpRh5bDBL4Vj0Jy/k2K2L/kPVGmJlWfqKR38q28UgPQk2Pp7CZJ81Q4Robaxou0AzCa043/vZl9CduJgc8UzQYr0ZC8wFLxyJjCUoaJVFKo/lqyswWZKCxWHSFTlbyISZD/qkZNX7v6qwn3yPqRhKSKsG8ul6gfa3YnS3ZRYokuJSmRRE3iPaiuL/W7eIVC8STW9jiEZf4ZBFBxrEIR+jckelBBTmsGwtBSnjI+feMtbOv0OBUNa4SkCROyMCP+DPg8eoRIm8bIKF/dZZ8HWP4OFVjoP8DSteky1ydF32A9awPOpyzk7VbIU5GRfvESJRXBYVcuy0b46mtXVPDgUJlhnwBuiyOBrYrZdqUiJUUmSHamGjlzUTfVfGMFdKzzkZai6tPg0GTykwCUG+8SiA6m7cNU53mVB/FpyArWZiRNS/Fh4JZ8mZAilp+HIZZwblQlgV/RKPZ6ptPlPjkDZAlB1C3 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 6e0FLCQ+TObGhDfhFPzRPTGMNfJoow+YEhi5Vx+XU3Ub0tJ7l5c6qDftuEAOc3SaPxFK1uzl/sz8J/g5gCAKslwZZWnpTM0IbM3ivxuPrYYrz2dv6CdfVwje/HI9xQayT3Nh65GVA3i/QEO111D36XV0uUAvBK2rontz27fxktQMhHloAoqw4k9on4JYju5AWM/9YzTeg0q3e6FuUgqYsYXoLmKzPZU9DavapHU71P2FczNhLxN9veKWYAPYETmYBvhEn6YzHGnLW/U2OKUYGQaNq7oYzNYCaf42SzCTBTXitW8eo3suuTdjqtVLHYiRLIM15JOUYQERKF/K/6WLHLCtoWp6dhkpXUlCiRkKigpPdSCt63i9G0BmLBQnjJ6yVXVK6BdUIZPcmg+b38aWQ6xl/u9M8jBil622+w27l9cvmszEB7hye34l0rXilTbe7ke/JvzglVPvTBiyVIna2F7zpmovGS56KCsJwF6ZZvnbh2qc9VnC0YNAWeoEBlz7Md9McaMTZJOHf8V9HFNqmuQfhDaiU3xXwEkcymafwQbHx+coINTBWxHUZbgDmzlNsPDVQI+yp7Cg9m2Ua6NRH6GtLsorr149K81AakdEKK6c1m2oG774e2Buasg+58kl/6+sYU2MghvWMs2cDVTd6vvT3Vwqvex70e5V9NTF5XiYV7DwmAtF8EF2/3HymWef9Fyqt3BF9gfxxz8OouZwxx6fvtUdUGR8FWf7nHjZvYmDGSRMe7alpM4448LGcgbL8MVIIy0Mx7P4CQngz8vGWVPYupevmV+4A24uV2xcZLdcKLOF8Fhfhbui/E09JJUZfcVdzVhdtFaaATrL2qwJ/bVqwvbWh21ZvQJKkmmgg9844iwM/gnrM2qYFzIUrZkuRKsWQmfpNVbcD9Vftl2mdoAjvNHJpfo3uPHC/IHtNhPsJP9kRRwiKjbLbm1c9gonpEplWRSwpgx34yf/0kzR9FrfcOsrhYVIvB1AjJTOqu+QOus6gPXqXa27eY/Hpu0fdo/fPTtPGGyNPf9yGRW+SZdxB4Rv60p16jWjHr0BJ0L2V3UVNjN2r9Ia0YNO4ynT0L/749d9GhbtxDZ4JrR0ScPUYwPgShHE2rPK3cNLJNYhl5ig4z8gZUrzvRE08ea8tq3z9csweudX6rPwbaIN7f8Z9n8Ex8mIgccvnOPIG88= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 01f37a97-c6f7-4797-e175-08db34036cda X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:32.1264 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rNJXMkL48RPF+xFdQXcLCh0so6eMyDa5HL9gezr45m7jf5XtCTvjoS1tnyCjf4dVMTs88RSIBfpiVA+VbXMR13gPNT6KLzT7YbKEoc9Y76Y= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB7106 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: ewEv5zCY3Gyz7aBBO5lxVpjQuKxkAKo8 X-Proofpoint-ORIG-GUID: ewEv5zCY3Gyz7aBBO5lxVpjQuKxkAKo8 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762131708190014832?= X-GMAIL-MSGID: =?utf-8?q?1762131708190014832?= clear/copy_huge_page() are constrained to operate on a page-at-a-time because they also handle the CONFIG_HIGHMEM case. Mark both __weak to allow for arch specific optimizations. Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora --- mm/memory.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6da97e6c7d21..9a6bce384616 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5645,8 +5645,12 @@ static void clear_gigantic_page(struct page *page, } } -void clear_huge_page(struct page *page, - unsigned long addr_hint, unsigned int pages_per_huge_page) +/* + * clear_huge_page(): does page-at-a-time clearing to handle the + * CONFIG_HIGHMEM case. + */ +__weak void clear_huge_page(struct page *page, + unsigned long addr_hint, unsigned int pages_per_huge_page) { unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); @@ -5712,9 +5716,13 @@ static void copy_user_gigantic_page(struct page *dst, struct page *src, } } -void copy_user_huge_page(struct page *dst, struct page *src, - unsigned long addr_hint, struct vm_area_struct *vma, - unsigned int pages_per_huge_page) +/* + * copy_user_huge_page(): does page-at-a-time copying to handle + * the CONFIG_HIGHMEM case. + */ +__weak void copy_user_huge_page(struct page *dst, struct page *src, + unsigned long addr_hint, struct vm_area_struct *vma, + unsigned int pages_per_huge_page) { unsigned long addr = addr_hint & ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); From patchwork Mon Apr 3 05:22:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 78357 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2080677vqo; Sun, 2 Apr 2023 22:43:54 -0700 (PDT) X-Google-Smtp-Source: AKy350YephTS5wsUgMgSODPOzTKicEQVeIFDYmOM9VUNGtR9uju/Yv2vLeNGjBl7L0afUgFDAT7T X-Received: by 2002:a17:906:9149:b0:932:4990:2563 with SMTP id y9-20020a170906914900b0093249902563mr35877426ejw.24.1680500634349; Sun, 02 Apr 2023 22:43:54 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1680500634; cv=pass; d=google.com; s=arc-20160816; b=KkdO7g016nh55/caw4bc67DnX3Kx1K/NUMYEycstfGjvEFJxrxCdFVIFQmW+GWTXRE 1FP3D2x3VST/pGDh3mU3iKd93JeDaqiZt66U3Kn8MmUkvYPOOPUR2YCKmsTrenf6GTx9 xf1Dzcsk0Py33FP1FoZuNrWz3TlH3zmyH/ke5XKyT+lPlYBeuzijsaMxV8Pspf9jtn4z BmfoICqIvn7AwPX/gp9vnPIoiRcHIz8zPl8YCptqFGyWm+I9Oa4PxPFVpeyH0nhEhAyI vjajwnADzljBobP+l3qtp6c5CifSSjVLmmAQK4Tm8YUQyBnL/duexOBGoq/rNi72wO4q Al4g== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=UjM07Xo3SboE2ZDIKJWFIgfW4cJRS+1QtuqFInHyTCg=; b=mW5k/+nnbfrNe/gxPfwqcikkXkembPbFrnWb6NN1dAy3IX4iQSB2wLDSc4fFuLnTNF eVi21QafBnOPnM6H1Jw70c8zvOnzReRcncVqFi0KxSSqqoQy3cRQsSuH+jjEIhlllTde 7+fKE4ruiRHB3McY8h7I65zal11MsTM6eAFODohZWbjeQ9pXXoWAIVxOatXZc1M+FDgQ XBcBmGZBxiehBh2dc44fPc1K5F5ejRQhLof1aiGmLaoeTHG5p3p2QTExl5x+TEhqLi5p wjQUuk++L27Ngzq9CkibyC35ssaY3gMKFvQnZvwcvEqlCx//apnMlzGmNNlnLf3NGyGE H1Pw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=1WOqz4d7; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=n6sXqHMa; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qt1-20020a170906ece100b0093b4e9918fdsi7146469ejb.154.2023.04.02.22.43.30; Sun, 02 Apr 2023 22:43:54 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=1WOqz4d7; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=n6sXqHMa; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231537AbjDCFXs (ORCPT + 99 others); Mon, 3 Apr 2023 01:23:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231524AbjDCFXe (ORCPT ); Mon, 3 Apr 2023 01:23:34 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 652E21987 for ; Sun, 2 Apr 2023 22:23:14 -0700 (PDT) Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332MM9Mx025427; Mon, 3 Apr 2023 05:22:38 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=UjM07Xo3SboE2ZDIKJWFIgfW4cJRS+1QtuqFInHyTCg=; b=1WOqz4d7LZ6Zo//oL3jSMkScTNg8MkljM3a2AvUGDRL9NhXxfqwq6za20/VP8Rt+BveG WwSOv61eHLje0XWyl7g/KxyPTeLKcPmtMUcgIFEEgyiUiabzc3qyqlLA54RWZwk39pE4 Hp/F/e5emxKhiPkwlKrNUnawXgj/aT8aC+HtKvsyMyIuomzQsNKONRa7mEGhzRgaOadU YcP+bAXBKdCxZwyep90VRVBKJgcF6Q6U/6PdONgClE5F0yFTtEzrqj/zun136DGpwSdE 5iAJV/qf2UUIN2E1INTUl4BZFQcOQHy9yDRoLrY9ic6rp5Q15hQE3qIvl0mEzTGdWpYg Pw== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppb1dj5th-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:38 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3333Uv6Y038324; Mon, 3 Apr 2023 05:22:37 GMT Received: from nam04-mw2-obe.outbound.protection.outlook.com (mail-mw2nam04lp2173.outbound.protection.outlook.com [104.47.73.173]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pptjptnhk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:37 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FwI3s+LsyusrVjnaFPv2Y/9rT3KrXGz3nEkFzYrxeiTsnBdSTjp/6CjUmfHcVNQieDLldaTL+BjEzNk0X9JyJ4l1YN8BII3sfw85XNRqRiWSTSmTdQhTpqngCJQ0svNGurkzEWjsQazl4ehJ3nNhHlZSXrvZ+4TdXiduLByVw03Djj815SpLdetgVS8KlFIV4MpzCDl+L18OxaR+6x6MD4oG2q2g4KF0RAh/KV0DkD6mrNJxZQ3N+uZHYvQMPOFCFBX1QpQ8X9tlxA/WJMY/laBaIzN594ktvMTX4TDTJIsqnNywPK83UCgOZBi8ImNrT7CA7Gr3hD1N8md4iNVjCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UjM07Xo3SboE2ZDIKJWFIgfW4cJRS+1QtuqFInHyTCg=; b=BB1EaJyKAaWcDrRfS5W9Dqz97Qe7zjeiSwxNWKq1MREshAbkEk7xGVK0S/gSgaoCtVtEyBg7cgi9mHSCrW6axUt/6LGd1tv0ijJVME5XVGuAaAY2UYRbY3Mp5ZvEX+zCnwdA0Z4OxXcFkn/DuNfcX7GRlMNDRBVadO8mLBNa9NabalGV7jM3wotuvGiIwdqJHRfLs35gXE//AFIgFzIby0IwPM4aNZhyM8A2Em9yICs1thqByfGQPx6I6862YADtWIG3VA1qDqNI8KZcVSlRVoFYYIrveTQAHZpIHSGU+gCz7YK+hm4QAS+XIjgSrvQ3h09H8p1RR0CGgWYEz/A+OA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UjM07Xo3SboE2ZDIKJWFIgfW4cJRS+1QtuqFInHyTCg=; b=n6sXqHMaO5hzCTlLFo8CTQFiIk8a/a/x85w1hx0YUnRws02gna4e7JWmCr7AwTxFAUcGU13b82WatIQDvsNyU1htOOfd01dEnx79PsbLdLp1N4hqBkf1NZflzM9bWMk8D+rLEwEI6MCa5uD0Ig8rZGn95VBl+udUA1Yjgc2lHxo= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by PH7PR10MB7106.namprd10.prod.outlook.com (2603:10b6:510:268::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:35 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:34 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 4/9] x86/clear_page: parameterize clear_page*() to specify length Date: Sun, 2 Apr 2023 22:22:28 -0700 Message-Id: <20230403052233.1880567-5-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR03CA0247.namprd03.prod.outlook.com (2603:10b6:303:b4::12) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|PH7PR10MB7106:EE_ X-MS-Office365-Filtering-Correlation-Id: 1f2f8e1c-02d5-4634-cd1d-08db34036e7e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WffUYm/xCiTWr3T6eBIlnT5ip5e6/1255qgOJCqYMQU1Dbqz8+vjgF8lRUmDYweMNPybA+AhbJC8VGgnyroj2NV+ZudsVlg0aD5+HVm6s7lY1xpFBcPx81m6eaiNCpWgYBPyc4EdRPadCeBjWWjroUUObsxOp+/UTim+ZivcayyGP3qW6bJli2hP94ymhS/DRz1dV6bhKgW9KGW3nI0oLs3E0/7sDrWUnUDHHbYDYzWssM0in0DKbX7ZULzgi+gp1nJxKR6Kvj8ne6/x3lI0WhjGpNRsILD9PXC+zG9jw4lWq6KqgQds2SjkAuttGRPx7zNe7zIX6qUF0qbYXfBgXSo0HOwYnSAGUx5TZ5DuJl04/NrIvZ3YBkjEGs3gTm4w0eM2MOSdBhjt9O7yKdSK9wS8QiDNKOjrmMWBNKGZc27eMDQHh6FhGT2AgIxcF3WLzk6Z6tbwJDFCqTYv5tGIS/jU9bPbaTzHdNwWOna37YZFx7m9uVaCjANMlLkWD5MhaFN4ammv7cxZ6nX6UaYJJH8gtEOx8fFixwGyI/ENpmt90OPowiHDnH2bAO7gIaQh X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(39860400002)(366004)(136003)(346002)(376002)(396003)(451199021)(2906002)(86362001)(5660300002)(7416002)(8936002)(66476007)(66556008)(66946007)(8676002)(41300700001)(36756003)(4326008)(316002)(478600001)(103116003)(6486002)(186003)(26005)(1076003)(6506007)(6512007)(2616005)(83380400001)(6666004)(107886003)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Yx05J4aPr+IqbvHOpvXk8ZEo8RheJNIi8S3M8yMX1PhG+wVxEpvFRvCkX/I0p/aPxICamVJ4bCIPuVwNf6SmJCbioX3TtahQIlZegGNjon9cYJEP4y3rFx6dfVYYUVxbZW3qaDH/JZmdvsHj4DRJhI+7Mxu3e5Wgyu2iTun++LGIjhljWSvPBv2bc+B9vk+C7s2Ncdpjkta7e6y+EG2jFAP3zCFCF7earEul2wJwUQ5neJq8Oy3Iip89LOrkdagaocVB20+XuUsKHky4tFgm4CL3JfV1oWVKQctLP/K4QqxcNmyLo430hxlmso4qu+PwnhdpZmsEKV3eMA/0uPwNypAWI2K0kb5S9qVWKnU2VuU4d/WW5/ci++QdEDYxIgh6ekqJubcNMWRhQR+hgJ+zOo9gVINXGrO4RQ46Vqg7+7ZF3b2P12qJyJg1IiHx3E1GHDfUM1IGGc+tkTD+1XvnaPCjSYD2SIH4m1oxNaSIcAJ6ja1/fPWbUApPJQnyjKkoUbdEHejG/p/Y8kD3OzWF//HAU5OWLyY8cOxCRbZav7jI0QSu1HioYDP1vKwYjfCBpf88NT5yBYjWB7QRKyTlfGAzgD5lGEn3fzJ6yK2kcK+Oony+p6KyoWg83AObNo/KEQV7/B/AIr5SY0fqlzog76J4xT5nv250wvaCkUQ2CIAgJSgFR2SSJfHkEUzfdQ7EY5jqrP9RyZm1paIASiB1QTFLq21SxvQWW8dNAwuV4hyDKUcIyZpOhmbAhPg3zeHAqWBPyW439F/azPXSm78lCW4Ah2gGrZwCuo1dpVQbiiARmZY6gxAUpO7VE/9hbwGZJ7wcmeN2JD3Hacr6rhlC/rie+3qnh4vDaUGPhP2iriTzArAzHCyELI4hvX2qhoBD9sih0zEhAS0FFlhr8x7eHA1XrLX+R/IsBRDxvv1aJavyBQAgCRsOO2nLJOkODbK1TyMKPsHfF9pNTzRSUDoEZqS2fBQ3pmstevurkwT6RaWVthxhnOQCnSFLbkVjy24fem7oiiPRoX99LaE44zLKCFSVEA2hxnm35ls09PXIylHTs7AXWXpNognsIGFOfXzX0RyRE4OrX0GsPhCx+OFe+zZEQH4176UFBpBhQvYfRg/WsPc2JDDkphILFJxzkg3+fQ5m3ON1M7Qo6T1PgepMhwyBAnJnjR/y4MvJcM97VvLS0H6os8zSDe+BV8+ULdGtoZeDlPuv0J7v68xzD3Q5QKcG/1fw0MfWQDjbzd13jQ00UhA5wDg+v3x22iMp37a8l8vkQ1lfljoq/P1sqB6JIzPTIcznG4wPe/yCQnV2YXWaG1THOA0i/Wd7Kfya2vkExOQZduwOTvCEeh4GoTR/All2o/4irtvv8WP7MLqB5YYrRN73sYR8fGz49+/+DZuWGm96U2uVK8KaI6xkubdmh/qK4k40IPZHAjrYNpglxONXvCg+dvGugdujKHPYBqJAZRKbqmOoB1BwK57egGt4pTNmKmZlNzXPa9peJ5w2ipXLJklEzzPljhvdet+vi5NBouyeNDhb+Xfm7+0Kk87s/WMGAB7sdATzrEBFRdUxqADA4/Y1hMDfVcI7KgIHE5iD X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 66vfYUq6go70SfEIgOxUSAfQFij07W6OPe7FvVXIA2JL9SaJN1lsQ0P9kN6Y4rTu7ndfTOxse2dvOohodIA0Lq1+wzaeA82Ic5CpC/V6zC9cqqwHTIzEt4EJZbn4TaS1p1XavDY7doWOFz/89T5/DxsEmB68sjeWtuEl4xtfHPR1x9uUr2kTTWu2U45MzhW92G+krl3TbfJadq3R+jiaLgtaUjmW0yCb7STlc3NJ8/LSUeAtQb7dcH2AtRct5XUpfIv0BLKvAtay+6yDMfkWGbB6OH/4FkSNpHGQkV/qiBV9Ni/COzfQls5S86Jv6U9MfmZ0QV1tTVbdaBf0NSwF3dGFWH/lorLy6zGkz0kboKyq+NXyzTVVTboASyNFjtT3HeF07ehHnItLhREr9jw8YXBE20fRKxnf+zbjuQuZZj6H0mgNslEyYe+5u/6UatAdn65rqksMKzGpe48rDrWOeYKHv9jNClhiOLZicGTVdsXajQZuc8gmIgec3pTJLz5EuCxm/vHmPEPWas4J5KLpqTbEIfuqmzl+zykPLLX7OIBWs3bWg7qQ+srAP955viutQkVSx0Pw9amMJTUR2zj0XoYFzcBsuNXAnp1Aw1u36iCQyFV/+IPGFLFni6+RRDuQiN1MoPj5qiu39aFIiA5FHQfo2wt122S/NbMr5U0DZg7rhhHk2s1NBg4MitwNQeXRhBhNddOm99i8Xa3/WcyX+00JmXUs9dgY+CAPKUI/+ixckPkqcAlISt3mN58Uo2xE9w1vmCvjMplz3v5GdYd791XzL7ykK11UZxfLU8Cu01DZiFO1wKzoR21vicK0lb57QbDmiy4iOyokuc+dVpijPTwZ3KI5qE7bFtmmzTetWq5oTNzhzQy3STJhv/RwZDLXSvaO01SLpBh/9iHHZNYDPkjPnTG3O7muGBArh/ySj1HTd9Ttioqx35dy8dKNIrBbDl8dItR61iLzacZ6lo/AcxJjlgRNrQgc61QTKOxIu0XGjJaYaRp2SRfWlFO/z6wJlT7z7LaQnMnm0MauNJ9rB+JoCzXJ0wD2EVopg8rkvF6aQFvbqRIsSzILzFk/IJjYh03ex/xil9SDxMYNTv3Yozh1NIG8UPsflZ71cNkitpIa996NfsTeaJXGrEsBaUDU59AYXH4oJylL+KV5RxnMAschuljt4yZ0dq24P4FBUAw= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1f2f8e1c-02d5-4634-cd1d-08db34036e7e X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:34.8234 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3va0khPeFN/HQs9xoXsiigLDI9gJ2EtiUo0uLwscOI4GgY64IrUkLy63zeq0OBaKYqG3GTbBxGYyrzeqRqpovULADgJ39+1Ezviu2Oqk0yY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB7106 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: rLVRmwNI7i5to-8xeaNRvZvYUnveTt8w X-Proofpoint-ORIG-GUID: rLVRmwNI7i5to-8xeaNRvZvYUnveTt8w X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762132633312794304?= X-GMAIL-MSGID: =?utf-8?q?1762132633312794304?= Change clear_page*() to take a length parameter. Rename to clear_pages_*(). Signed-off-by: Ankur Arora --- clear_page() is now defined in terms of clear_pages(). This means that all clear_page() callsites -- which are more numerous -- would use an additional register now. Opinions on if that's worth optimizing? --- arch/x86/include/asm/page_64.h | 18 ++++++++------ arch/x86/lib/clear_page_64.S | 45 +++++++++++++++++++++++++--------- 2 files changed, 43 insertions(+), 20 deletions(-) diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index cc6b8e087192..7ca3bd2448c1 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -39,22 +39,24 @@ extern unsigned long __phys_addr_symbol(unsigned long); #define __phys_reloc_hide(x) (x) -void clear_page_orig(void *page); -void clear_page_rep(void *page); -void clear_page_erms(void *page); +void clear_pages_orig(void *page, unsigned int length); +void clear_pages_rep(void *page, unsigned int length); +void clear_pages_erms(void *page, unsigned int length); static inline void clear_page(void *page) { + unsigned long length = PAGE_SIZE; /* * Clean up KMSAN metadata for the page being cleared. The assembly call * below clobbers @page, so we perform unpoisoning before it. */ - kmsan_unpoison_memory(page, PAGE_SIZE); - alternative_call_2(clear_page_orig, - clear_page_rep, X86_FEATURE_REP_GOOD, - clear_page_erms, X86_FEATURE_ERMS, + kmsan_unpoison_memory(page, length); + + alternative_call_2(clear_pages_orig, + clear_pages_rep, X86_FEATURE_REP_GOOD, + clear_pages_erms, X86_FEATURE_ERMS, "=D" (page), - "0" (page) + "0" (page), "S" (length) : "cc", "memory", "rax", "rcx"); } diff --git a/arch/x86/lib/clear_page_64.S b/arch/x86/lib/clear_page_64.S index ecbfb4dd3b01..6069acf6072f 100644 --- a/arch/x86/lib/clear_page_64.S +++ b/arch/x86/lib/clear_page_64.S @@ -2,6 +2,8 @@ #include #include #include +#include +#include /* * Most CPUs support enhanced REP MOVSB/STOSB instructions. It is @@ -13,18 +15,30 @@ /* * Zero a page. * %rdi - page + * %esi - page-length + * + * Clobbers: %rax, %rcx */ -SYM_FUNC_START(clear_page_rep) - movl $4096/8,%ecx +SYM_FUNC_START(clear_pages_rep) + movl %esi, %ecx xorl %eax,%eax + shrl $3,%ecx rep stosq RET -SYM_FUNC_END(clear_page_rep) -EXPORT_SYMBOL_GPL(clear_page_rep) +SYM_FUNC_END(clear_pages_rep) +EXPORT_SYMBOL_GPL(clear_pages_rep) -SYM_FUNC_START(clear_page_orig) +/* + * Original page zeroing loop. + * %rdi - page + * %esi - page-length + * + * Clobbers: %rax, %rcx, %rflags + */ +SYM_FUNC_START(clear_pages_orig) + movl %esi, %ecx xorl %eax,%eax - movl $4096/64,%ecx + shrl $6,%ecx .p2align 4 .Lloop: decl %ecx @@ -41,16 +55,23 @@ SYM_FUNC_START(clear_page_orig) jnz .Lloop nop RET -SYM_FUNC_END(clear_page_orig) -EXPORT_SYMBOL_GPL(clear_page_orig) +SYM_FUNC_END(clear_pages_orig) +EXPORT_SYMBOL_GPL(clear_pages_orig) -SYM_FUNC_START(clear_page_erms) - movl $4096,%ecx +/* + * Zero a page. + * %rdi - page + * %esi - page-length + * + * Clobbers: %rax, %rcx + */ +SYM_FUNC_START(clear_pages_erms) + movl %esi, %ecx xorl %eax,%eax rep stosb RET -SYM_FUNC_END(clear_page_erms) -EXPORT_SYMBOL_GPL(clear_page_erms) +SYM_FUNC_END(clear_pages_erms) +EXPORT_SYMBOL_GPL(clear_pages_erms) /* * Default clear user-space. From patchwork Mon Apr 3 05:22:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 78355 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2079780vqo; Sun, 2 Apr 2023 22:40:58 -0700 (PDT) X-Google-Smtp-Source: AKy350ZoW13tlZYoH5prn5fRtaPo1/Y1dNLIylL5ST80rD+pzU23+42TCpjHHqfIXVUfnplAWwHX X-Received: by 2002:a17:906:b303:b0:8b1:7fa:6588 with SMTP id n3-20020a170906b30300b008b107fa6588mr30431592ejz.12.1680500458486; Sun, 02 Apr 2023 22:40:58 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1680500458; cv=pass; d=google.com; s=arc-20160816; b=LTZU3UgforTc8T/HqhlcjMZPfapay23WsHG75VnWLJ2EMPzWiEsWJDOX58/nO+EWU2 6GMi5LqDB9omeTH07Ii9HtnqVyj0EtOsc/tqIfoY8F3e0lgqn2T+PInd0qXoq9BhuPTw pZj8/HZFArGC4CxmrEsK6FU9q5RY/RZEZCcLy6NqsqS/fpKkyUiw7vdWOYnfeRbOhZZS JjQB5tBvE7YleB871IIVTGpO41vKnjvVnPgJnoUsfU/16tWaXgoNmCkjE0IgqjuVaaMC AuW6lATNK/KSDaktBkcGe7500XYurd2NpNxLJRlyAXb3jjzI3mOepDZRL/wd1A4Pavmv LEkQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=gW/w4UBNoRyDr+fj9Sa/039P6gNwlfFSpI3amaMp3cc=; b=JDFUehcQaOq1Yt3ZEbbsWBKj3HZ2xE+u0hHG/ctZoNFqASqmVM2VjhFLeH97/2SFng GWzaIMVBAqiaAybBqRg5qF74Gu0W5bBDfnAUUSxd6cMm/DFLIan4riaQkYFXj0wEyIMj BiqDGkiMe9QBg2hDj8VgdX39+ioZ7zx+q9zomR6tApgNaiL7xuOogjVkLZUTcsluwh+E CLHZbP5ePbfYq2BiS3AKstT8koNk2zvMNap/k39ddSE6iwEBCS/5cXVTTLhtK1aIBWg8 k/PJRvzDREfEgulhtf/9nLgczh/haUJGcKEz/pyATVkYV2GhIA70vDBoKTUAudGW9TMM pjSA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b="Dn/H6HZm"; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=TUfpdY2t; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gr26-20020a170906e2da00b0092f97d1a8efsi5101672ejb.504.2023.04.02.22.40.35; Sun, 02 Apr 2023 22:40:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b="Dn/H6HZm"; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=TUfpdY2t; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231519AbjDCFXe (ORCPT + 99 others); Mon, 3 Apr 2023 01:23:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231511AbjDCFXa (ORCPT ); Mon, 3 Apr 2023 01:23:30 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10406AD04 for ; Sun, 2 Apr 2023 22:23:13 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332Le24N009559; Mon, 3 Apr 2023 05:22:40 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=gW/w4UBNoRyDr+fj9Sa/039P6gNwlfFSpI3amaMp3cc=; b=Dn/H6HZmJlLesKaHwksiUDif7P9mM1uzZUQbaX7ujbmfxhyibCNFgboF/dDK/CBCwqu0 h2QbvTUK+n1rQXsU6YoJwzJ1LeBU6ugQ+POVfi1CCeMxsxllbTsYWD4mCJJSd4lcySed MOhTtxMnW7vR5UWkfGDoSLAy057orTt5kyZ43n1A+GITTPyEPxd59Sn0HlHnShjkZovn CiVBxPJvoMsr/WIdfyKci22EVc/saxmgQzKI3WT4783X30rPMcYK1ZM2zE6DcY2WshJv fU/nC5drYb3LNlu8W+NyxEg94i7NPfGtMOMTBVpnPXnexiD1f9bR53BpGqZTlUJsa4ur ug== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppcnct4s8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:40 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 33358gdI028272; Mon, 3 Apr 2023 05:22:39 GMT Received: from nam04-mw2-obe.outbound.protection.outlook.com (mail-mw2nam04lp2170.outbound.protection.outlook.com [104.47.73.170]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3ppt243jdt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:39 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YgCfHXKDhVlODPaTrw5tanTx1s4DBtDjp10We2gX9y1WCAYc5JSVdLm9NcTQL/PsmaXGpDcUIeCsglKey1HO7oTR1GND7JPpIIFa3tvq28/LbNZnnAD2w5ST1cRFqRL9+z4q05CutuvCMbsqci9OFMTO+mzy8DR8C4/HgGiARxf6+ghonpShoLolKP8CbK2BTn2ineBgD0ZhNqUHTr+HQAEtjpTt0jmsyCX8fhAPPctTrLuEtOAqF4Ce1BXWAAeT7/T2jkJ4b2ZNfSc8y4yHG8frlq2YOvl3EpPNj27DeBzUyexe80aPe6hF3loFZSCwut8f52zxZ8K4PTjCgc8swg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gW/w4UBNoRyDr+fj9Sa/039P6gNwlfFSpI3amaMp3cc=; b=Y0Cm+uvbhKlFfPzOGyQzMU+r4HcyxXl57tCCL7aBTcn8OhTeR/sJeuS5jgLVAX/ydBhX8nCHgVckAb1BX4T8MC4gfJTH4xt3VMLjGeMf3oF9QOEVLjCNOW865vGKED4loLRu0s1QD/gn1yEAq0Yy3hiYts0EdHP5JzGMHhBV4Tfq+aLAMngaUo7aYlPRtY8PN6CCuCdKBDG7gZKm8uvB2q6M86Nx7RC9uZg+2SV7CDdgbqxnQp/ygjMxLakNJ9G+LHvP7EgQbRmXUA9U8B1axiCyzMvWREOi1NOvHl9jFDf9zH+KcgzWR7J8SWX0A38cH/sLqwOj+n1PBnLTCESf0A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gW/w4UBNoRyDr+fj9Sa/039P6gNwlfFSpI3amaMp3cc=; b=TUfpdY2tQMSA5eHHsYl5p6bY6EywNRdeKZjsmC2fKFyqoqO+MyeqKU6E7BcHFQ0J8vgYDFx3uq9YVTH/6+vuT7DPTFuVtRW+YzLijajZ7yuiAQdxIgZWgBaHA0ydlOzWuMdtsPUYtUc4G00QHExCXVdL3+MAevsGEf4VjuIuea4= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by PH7PR10MB7106.namprd10.prod.outlook.com (2603:10b6:510:268::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:37 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:37 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 5/9] x86/clear_pages: add clear_pages() Date: Sun, 2 Apr 2023 22:22:29 -0700 Message-Id: <20230403052233.1880567-6-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR04CA0289.namprd04.prod.outlook.com (2603:10b6:303:89::24) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|PH7PR10MB7106:EE_ X-MS-Office365-Filtering-Correlation-Id: 25fc5ff5-c6ed-4530-c68f-08db34036fd5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TyFwhO7FrdtV/0UFCTJWxkzDFsU/vNJIVTciCP1I/JqKVaSZYgA+1t8RsdQr870dbd/hjMNcGFzL+hEx/Kms9eDvv2is2SneGKgPGv4fM5ZTgLXXek3jyKFXemjKOcwa7X6MZF4TuGQ+FoAJee64573fiXl0S9D6dLEK0Sxck3FKaBWVBQmgVdDc/zwTZpUj5DxafWRRedqiiOnqiLBllWozYFkmZRublZsWyfMmEKgCU38csPOW/mcNX4p46/H40SB46t5li5D8zb6kU+XFVTF3oa/G9iYuC8QhbJpY7FmvoH5us+WB2t4dphASX1sriblXzX+5Mmgd3VMLYZXeCgSjEmm74pSXHXawL8bbMHmSUsNgHA7b3Fd6nrx403dkYoKCjI1vHk8efgs3xxitry4C5vVhwpEJfX4NpTUk2ysUL89IWfVxBpHOfBideFJEN1tUnJjNZKMnSWCUVOtAVFO+Ae+DP/vg0os05zAo/C/wYnTUBsGPlHmS3HAiPaCDYW9i1mSIU4E0tSS37mOtTPWSFz1xUrtNmc0ARDWfdfGqjDd+Ws/qY5iqJk6pKo5w X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(39860400002)(366004)(136003)(346002)(376002)(396003)(451199021)(2906002)(86362001)(5660300002)(7416002)(8936002)(66476007)(66556008)(66946007)(8676002)(41300700001)(36756003)(4326008)(316002)(478600001)(103116003)(6486002)(186003)(26005)(1076003)(6506007)(6512007)(2616005)(83380400001)(6666004)(107886003)(38100700002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: jOjgWgbM6lFjlTtRLSnNCkEMEwDZYEDYYjHrEMfAGp5ldXwtbaEOmYfQ6KGUUJwYfijNkHMz9ie3i5R9PugOFcFzQw54MuEYUS60CXM3NreNkbxzH9Sg8u3fSGv94wGVXne0Jpi+bixg3hl+5LTaKpY2TI7Cc4521nc+SD4X+yEuRGUkfmR43QuYIIjLiH8ku6FZN+V9E5jhcDY2bMk8T9MjtyNCniA0+bKZXsrqlNzJAUD8szfVoRnBlOKAsijKYrAuauxHrwEmunX5DzYpaVyyVxYHP4FQJ8Bw35TpY1fISlkrpb0hA1nNtQxIedW+750L0EnAFpU6XwDFo36uN7JqX4F8G2KdKG8vbA7aOzp+LUbt9zTBT9MzB8NMnEwDegPJfA8ygAjxZpw0FdKL+8qyl1bsj9zu1uh40tRAVsuoPwBVXoqQf01/j6xhNIeDr+NIEtwm0uiTJf8WA4tlVHO/BJ7lLaGQ38c8pk42oDtIrwUqb+cRv0W+dEnUdSWltfeeA9ql/KR2Ngl1OfaV4uWVN1y8FcKBwDiteJpQeiMjosyTCpV4GRQPDCaptbUNfIIBjbWwH2xFrPcySmznDmQHKVRQaAYb3Lyg/4AfSzj4nIUkrYMo5XLij4USuBPTDO6qWADgDW7eyB3fZyIwiO6ltyUDkUrc+B5vbRzqh01ayOoWnhAStXgTP8W/kt89CczJKDnZplLkFDXPeCMXeyCttXTLYFgrWj7zCLMTveHUBDqR66q2X5Fc97l31nyIWjfiHn4GLepBaz56KRDsHN4Aoo+Tjl4BxDmVYo9DUi1k8Hl1CvotE1aUe/7n9YBOdn6SXpDaSpaqN/Ovjn/DslDaG76Ssl9jP1i8WexH5XzcJi0vBxP7fz0E8ncZAGfMikQExkRGXM+tvBzoJhmxcVaA8icU6Qn7zPguA3g9H5RukupY1w/PKUre22FvU5xCOFY47je35d8zSlUJJ0LNpqsy7gvZcgeB3mYy5kp2JiCmjzXnt6Jw26nC2ml5RIjtEpcIH93U0ETqIuFhbbYYDLsxQ3Edla+cQEDSIav3kG/sPBoP4tzrgHFkPsPj0SqyXPsnrIVE0gwZAdMst/wfR2IZEixV8sHY2phU0yk5/+DCOrZdK1ClwCuR5Qr6xuE4RtGIcGY2MxuUl5BgIDtu9A550lCJ4/I271lt7edHM0yiP8rWUz2VX5pgAY+2NiIcy4UWhyQ27UplYyEAdQdNrrDeY56qS5KmA00FpclsawSobjfXQYnyVu67IQknsKPJvn3g8ZFRb3DKDBpt/87+w2ZQCE30uhNA81TlpjPNOuVdOwoDiSPRqPYgRdP9pOQmBEElvfRLl27tWxCvyX8pF+iSyb2YPblJjpQYE4EY6FSHS0BBC0ZH1DxfK89FldvQBU8LelasydHrg4vox2QiotEpZnWqQo72cWkL52UW5oLRpT6qUOJhEhHQy0os2XU+c6iNG/lkY+72O5pcWUCYu2YmexHVRgvbmfOLJjoHAkCAj9eU4SfWPYzQxdflbaDUGRm34eWgJ/E7ZfEJ0RkF1NUc5WwzzzQipahcXjT5IKYOmuGBJdPzyr/mpMp1QhVc X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: YLWSoUs2F+TjUZ9vlm7/o5qTYL00NfPC7EIcgf0lJhB9bywpH/JoKBV3IssJWP5Hn7VtDYJlpajko/SsVKZVzs/0XlZaoET8w/Hao9RXfyqCtMjS8n1GJnIm8vD15iC8ZoxK9GXfRptzv82puZjq6zPMP3lZlOTx0MM12sJgp4PtaxFiShTAyPPj+aEnid9y/FBNwp1BMrbDstIeBsU0AP8i1h6eTibAKytVL/c9dQQ4V4a57pD8rFlReQk3Tq5YvAofAe8jCGDtdCjW9rGQs4guwb7BlCXqbKxq9UXF89siHX4U2x5AX5qTGZPyUOPaQLzGwn+sQiyy0zzs5HuiITV4lNWEDymCUmgYBqY2kKhSx4stWYtAf6eUNozZA9127tTrQUw/0uSr0VrzTFksLi3oUFW+nk3zIW0k/1DVbPgVFxNGYRWiQLLT2vgptFGQxdD3iR+igqjOHLvaw+PeDmE6l42vYidp5VMWoaEoJETr91CWiTuQ3ByvBYbR8rMjZf9bqZRVGL9XxbSgGUYzfKBniMbFivytFyaWOukn8IyixkI26qecaY0cyF1pKCumIGoO1+EfUGybyOvYhUVfpisbxZG449B5s5b6PQbGu3nQtqBBh87UzOx1vfBgHWgrLzxTvUC+j6qWEoX4E5H84GS9rnJuFF5DxVwwYUv8PLfTMap24i6ZXoCp56Z6XYMEdwXQGXYjRwW46++//HL0Z4nA1GOcHvNiOI836I+rS6QzLs6RGcsGK3yK3u4TVCPMRS8LosIRI5j1Psfe/yx29c/SlVKvkTaF3FSzIr8L+TGQSm1D9cI1Si8NcF9CpIl5CV9gMvVsKZTHjZeurOfpJUvyv1s2DBW0XIOlzcLPOjLusnKHpzVv+97aB2lQCxhGTI2qg/i++qYRoC8s/2cwG6gYsN/dSAGiLZYB0wQRwHvSuZcOc6kj7mBoKB6WDpv5ObFD9MvU4Ua71KNhcP4CuV6Rbzv5H4MBCOwcMQfgS89ywpgMaNd6E6dJlCwpG1F886oEcE6sl2n7VUtA7ZElvAWMv6dye5A6RIKrIoFV5Jp/R6WGpHOgU8DF0Mvw/axvLFIpxoM4cfihGGRvjcm3cNFjKsV+fNre3xG3K0wJMY3bzy2OiFpWkq75V6JUFVVzvITfLw95X/echc8gKfibkTcWanAMreL7SjEes4z+KjM= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 25fc5ff5-c6ed-4530-c68f-08db34036fd5 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:37.1043 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lFy+bbT+hkUU2yGypQNLEk/z1osGbg4aI0nNHM5E1Q1+V/ApZpOwebI0gzc9Zj7p6CDcnbTKWS1iPhvjSH61bzfhFrgiE+iSeUGwwvzo1qM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: PH7PR10MB7106 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 malwarescore=0 adultscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: ajXacZ1gH2f6bdCwT9bIRPZ4J4mSVB7H X-Proofpoint-ORIG-GUID: ajXacZ1gH2f6bdCwT9bIRPZ4J4mSVB7H X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762132448597685865?= X-GMAIL-MSGID: =?utf-8?q?1762132448597685865?= Add clear_pages() and define the ancillary clear_user_pages(). Signed-off-by: Ankur Arora --- arch/x86/include/asm/page.h | 6 ++++++ arch/x86/include/asm/page_32.h | 6 ++++++ arch/x86/include/asm/page_64.h | 9 +++++++-- 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/page.h b/arch/x86/include/asm/page.h index d18e5c332cb9..03e3c69fc427 100644 --- a/arch/x86/include/asm/page.h +++ b/arch/x86/include/asm/page.h @@ -28,6 +28,12 @@ static inline void clear_user_page(void *page, unsigned long vaddr, clear_page(page); } +static inline void clear_user_pages(void *page, unsigned long vaddr, + struct page *pg, unsigned int nsubpages) +{ + clear_pages(page, nsubpages); +} + static inline void copy_user_page(void *to, void *from, unsigned long vaddr, struct page *topage) { diff --git a/arch/x86/include/asm/page_32.h b/arch/x86/include/asm/page_32.h index 580d71aca65a..3523d1150cfc 100644 --- a/arch/x86/include/asm/page_32.h +++ b/arch/x86/include/asm/page_32.h @@ -22,6 +22,12 @@ static inline void clear_page(void *page) memset(page, 0, PAGE_SIZE); } +static inline void clear_pages(void *page, unsigned int nsubpages) +{ + for (int i = 0; i < nsubpages; i++) + clear_page(page + i * PAGE_SIZE); +} + static inline void copy_page(void *to, void *from) { memcpy(to, from, PAGE_SIZE); diff --git a/arch/x86/include/asm/page_64.h b/arch/x86/include/asm/page_64.h index 7ca3bd2448c1..42f6c45206c1 100644 --- a/arch/x86/include/asm/page_64.h +++ b/arch/x86/include/asm/page_64.h @@ -43,9 +43,9 @@ void clear_pages_orig(void *page, unsigned int length); void clear_pages_rep(void *page, unsigned int length); void clear_pages_erms(void *page, unsigned int length); -static inline void clear_page(void *page) +static inline void clear_pages(void *page, unsigned int nsubpages) { - unsigned long length = PAGE_SIZE; + unsigned int length = nsubpages * PAGE_SIZE; /* * Clean up KMSAN metadata for the page being cleared. The assembly call * below clobbers @page, so we perform unpoisoning before it. @@ -60,6 +60,11 @@ static inline void clear_page(void *page) : "cc", "memory", "rax", "rcx"); } +static inline void clear_page(void *page) +{ + clear_pages(page, 1); +} + void copy_page(void *to, void *from); #ifdef CONFIG_X86_5LEVEL From patchwork Mon Apr 3 05:22:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 78359 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2082506vqo; Sun, 2 Apr 2023 22:50:07 -0700 (PDT) X-Google-Smtp-Source: AKy350ZpCahn6OCiwqeyNz6TrtVTCzQ6YBwGgfZUZl5EG/K085N8u334Dcw4mRw+luJ3f/1+FGk0 X-Received: by 2002:a17:906:5856:b0:930:1391:da7c with SMTP id h22-20020a170906585600b009301391da7cmr30708778ejs.60.1680501006958; Sun, 02 Apr 2023 22:50:06 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1680501006; cv=pass; d=google.com; s=arc-20160816; b=keKCp6EWsD+j2YW68AIJm9gFT3bVv6wSQbody0iPsAd2QI8AtRKUgg4WE4WJBP3EIE TgTRuzCJ2WoAir/sgAjAvRKzXlTa12uZsKidZsjHylhhZTp1BaBlYIS35POS8DAoSqf7 IhlHw0eOJTEodFbyev6GOY/UK27SVKLdixmd8/PJx1PvsBaanWtexYUgDAzn84bagjZ3 u+Nq1JT+RpCHL6ibYnqbpL/PidM+WsdD3RueoWlEe9LQTmvm49BJ6KOW11JSqjlM2hoY 6UVl9YmyuUCaQAGzY9/8WNAu2yC+CM4oXtLNdUkGYojFDPSJeLGIFH5pdZDweBJ8fUtU 8jXA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=j54SMO8aHZZZsgBi7r1t7d7dsJQz4KL14GWBankHCBc=; b=l7a9wlPJw9+G0iSWzSjPrCZL0FTpcb1aeo3VRQ8+g7kmX8ZQ4OwRwqZgjByYRl6hHz QagTZBXYwyW46g2pEydgJ0P2P8Z5UaO5HFrl0Chd4lNqzPihPM/Q2AwHd0MRc6BoPFgk p5pNHCgJhW8HISe1kzK8+YsVLK9SZ76Srvz3KzbyTW37ynwf7+BpAciXk8/XRFUmBaLo 3V+qs/vAuVjo4jNb+glKqM7CQSeS+4OE0y3rWSJxL5GNWu8GCmh2CS90btMdUa6MrsqM cuRYbQYrxSM47fPC0TO992Zz+C1aZX4koO9e3roUtUKrLFJ4KTsFONw2j+eupumLqQZm bayg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=lppQNTdG; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=rsOWDWhC; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ca21-20020aa7cd75000000b004aad91c6589si3770926edb.193.2023.04.02.22.49.43; Sun, 02 Apr 2023 22:50:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=lppQNTdG; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=rsOWDWhC; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231582AbjDCFXw (ORCPT + 99 others); Mon, 3 Apr 2023 01:23:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36312 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231527AbjDCFXe (ORCPT ); Mon, 3 Apr 2023 01:23:34 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F335BDF3 for ; Sun, 2 Apr 2023 22:23:14 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3330qGbo020222; Mon, 3 Apr 2023 05:22:45 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=j54SMO8aHZZZsgBi7r1t7d7dsJQz4KL14GWBankHCBc=; b=lppQNTdGZtE0ehbHr57NIjPQfjX3bwIRB/Y0grP86aC7vC6mTDe/8r5J/HiKIubZsiaK er7J1ohrWApuGNzCsHcCm+p2wIC9jAsy+Ipcih/ORquQ7dmNncTR1r9MQ0BAO6jT19Yk kTrPfLDvEwoiABFzDnh2ec1j8iagTtvpirk0VA8eO0+znd229SP31GlzZLGrgyiF36pA SME8AZFMRtq52+CgB8TYYt/VuiNJpiBtLuhAUgjj3I0r71nsO8ZKpO78InXHNBxggdmc mjVFWsF0U4Qlhl1FR15mUji7rWy6aJPYT59mNSqTrDKO9W/i+oHDvGlC+IeGxmXNX+mK VA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppcnct4sa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:45 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332NM3i028114; Mon, 3 Apr 2023 05:22:44 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2106.outbound.protection.outlook.com [104.47.55.106]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3ppt243jf8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HLx0Ec+40KiTSJJWUXk2QMZFsmqR1BwZ66ZexSTG1z+vBWlWGtzJ9Klhbi/mTiytvyOOvvReyiLgl9OPmsnFran9yjeTWVO2xjsQkx8x8SGFyfWCuz8Cm4DSJGx5alwRI1jOt7YjSTMInSZEKX2YluOuKGY5SA9nbjMkHEkcJtOp5BpbT6+QSkcyg1YJ6a6GS7nE07VpsJEwhYvmpWBzOh+TEAUZNwe71W6i3MBfA9PXmpxkbmyHvcm1jwF5TUcwZZqzd+hjA8FPGS/aRhwxOkLHHissxs4x7AR12/Y3Pl51O2c9e+qoePjWJNttCsEUAXtrx9HUZ8BFZmbVwC6sDg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=j54SMO8aHZZZsgBi7r1t7d7dsJQz4KL14GWBankHCBc=; b=FbVPuUSqf8qDw6A2V1Czu3Mk993iYLy/sO2KUwpSVAEaRHq38bB03fsnQMxLO8cP8PmfbSXcUb5IedaoOwP6aCeqrIx1cwm9e3O6krjRXrpIJHRjRGNUKwR4vETIReNaZg7CMjU6OueXRfLA4wx7ssR6wijmWc3FWq57cqBghdHR3U9PuP+DpqqbiZJZnlvsRa4zCoCJc2NFDPJitoA4Nvicb93XDemsVpEWMzPGRW9gV0MgfysxhlwUfIsnu+5SVX+kdLhJEAgkCHEwmQP7S+iAWDC/SC8Kyv95QNJeOGMwHOBZ6aqtWSd7EMfOJEuBsD+ZzBL55OYfp3pRXFi05w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=j54SMO8aHZZZsgBi7r1t7d7dsJQz4KL14GWBankHCBc=; b=rsOWDWhCY3QbgShl49gkoaEKpfnDdPghQYRs6hg1QOE2tNKSKNifGD5K9mq78qxsJXh3IaGsm42DIfNA/YP3LcXqaMxl+SeFWbYBiBqojXbvpOq0bjVONt4oo2PINsxwN9MZDTPwscNunHaEcCRAttbma2evFksAWXu6hHLGp80= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SA1PR10MB6365.namprd10.prod.outlook.com (2603:10b6:806:255::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:39 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:39 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 6/9] mm/clear_huge_page: use multi-page clearing Date: Sun, 2 Apr 2023 22:22:30 -0700 Message-Id: <20230403052233.1880567-7-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR03CA0315.namprd03.prod.outlook.com (2603:10b6:303:dd::20) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SA1PR10MB6365:EE_ X-MS-Office365-Filtering-Correlation-Id: 2abb81d4-735e-4473-77db-08db3403715f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oUltraz0xtWPZn3Uj5/EngPUTJ+kcPgAYtWU08DjJwocetxXw/OPferKiVhzJd8mOl9VLy1c1I55KK0+WcTM0ayPI4vMsv6IAEyq8PbSrSpzj19Xafahz0UQWziyt5l7mLeUrnjL7gGMZZkBM3bx0ZQsD3SqJcyr4aLpkV01fso/hA4RJrWPfUDhXFaCEf4tIYchu/z0Mpuv//pzhDlx4iIi+lL1jVTthamtqS+G0oLrSMKFmECuk+QfQpc7NG28fMv3sb5z4MVzmG1Gs/VVkvi5d5fuZPnBlUlyKjWod6Bp/ERAsiyYhvimBj8CvMx/dxGm0LlLu//SPn4dy9789PhYrjlA3dSFJPepQ+UvlvgSe3zL3jyvfRfwvatbiCEhjjo0xjjvRcXBQLwx4tdtpytvm6UVP7fYmSfuHwMEX4Kvt2NlSGv8+sKM+FPBdri4JwqoSrSOYWhoDPNULI5zj5ojVbTWyzvbIcxD4y5ts/Lzq5MSUC9u+lXgCDVKvOSHUEdI+z+jGOQd6YZV74YP2E0Q90CXeF11U8kwEzP2QadtryhUZhCPJkbfEGomI+f8GTansnfpMDpYjkguVVwnJ69BeFTNPAInqigCtIjRFe0= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(346002)(136003)(396003)(376002)(39860400002)(451199021)(86362001)(36756003)(103116003)(2906002)(2616005)(83380400001)(186003)(6512007)(6506007)(1076003)(26005)(6486002)(107886003)(6666004)(4326008)(8676002)(66946007)(66556008)(66476007)(41300700001)(5660300002)(7416002)(38100700002)(316002)(478600001)(30864003)(8936002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: LDwZaoxviroa5oZKWcjLk4+gKaJkaTgn1Tv7iWksudtQqRwBOXztjUo6ZBsYygygzZLMbpeW7v0BZm9oM3CjDfo2VQzMPKO2P+Oca2nF5RU7W4jrvcgLzNAnrWxbvzmulCvyAJUtiNSIL3EwH8hdru03fUFxhb199R6Z5KWOrU0+BZW3XbUGisZvYsBd4waNZcrzKNq6SGaTLaKYjWFB2YTOVzQxHcgvUf4QLAjkYKWLu2Ke2KE2219HPA1YxGgswDUHVPL+iqNpH5HPLkUfzGUUEPKgyl2b1k/9EppRYA9KeOfb1UfnAp77+dl4ZrZ3+m5KNpqGz8TTl3BbUWvAxZHOMHuhdPtI9+cx+ruLYcKewx7zb71lq7Mu7FN4JCfjomZIG1xuFEyk/938QyVCnOJAQYLr+dU+d/sOqWcmX+ACIXgQDBCSvv5icmXODcTH+3GlMHjA2DqgLNG/C7VQ2HV6fmEmR45GljiKGctMAxW8G+C6WhEBWneoWcZfLp/lceRDNTilqpgfJ8lP1pXPFOskFsJE3H5vQFv4mqpn43kQXNyBRRUbXkA/HNxF5m3DtRhuXqU4sFeWUH1zlz8RB4CiWvE1N3q6ZLfbB92N4CLw1k5AWZLZekidy73cHsNjySTyn57PCbEnR9sAJLNnOSV0TwEYShihHJP255vNTPO+pO65Y5es5kjwMtVOaZi9mAUDF/Tuu5+w9At5EQu846YDeKlBj6QxZmkezFr/SC3dKlp+81ovQA0z8+yyV/zj7lWwcZ9Qvutcc/CkQOPPS5e3IwBnzWT/ehgCf5HWFgj1eIqTkvDzZHa2tX3yTaNTJ2H3CJy7zms+YShFDm4i1NgWmfZty98kWs8kdbOvCpaivhSMgepE3mZAP1CiGF64rBWRZeUwZHYWD+GWXTsYXLv1RvXDJgATqEOfdcH2KKEh0EBpfaGdzLodNdiHvTUl9iMItgj2ayB6QwFRNAF2Os9AjRbbJY3D8tZzVQeT7FBcWetQRz8MNtmEyiQPZbWm7n5NvVk4htMRSlZAT8GlQw9iEsEqp3Egg3Ce8Bg0HOT6wx2suTrOB/ZXl4lOhN4DSOikKu012PW1i42wsNa3Gxr7U9aCDQXp6o6cFlN1UaTJ6XaEKJ0MHrwWF9dj7NTVVGNbO8QktpIXMJYi6WryCOMOHydJhJBW+o4klpAyylVicmUAv1usOu4dTOpS3W5A4dBaDSpingMzIZRsBYrFhR2VjtNBgg4OY83BzuqCWA8s/FDwgL9qWZarYHFBtWJyGNKTq9b1cDjzaguKfkxOIp7PS6GHqDvOHQGGgyNeC+SB5gHyKCMMFCSLKO1/l2NTtxYIbSpj/sQ6WOyhFpsTOWTgikcuAYXEMl3JrdACXzE9KaEQ1w87IzA2wXb+3vvljBpBdSsZAqhjmUKWrwJ7HJKLixyLdhH8UcnvyL/Q0q6MHtP5tBhpqSlrBjEiQHUFA2cxirpbGHcqtI0wYIuoJVmhszh9COk+O9DdZ2iWlvYvKj9+bCWty6+7fihtqDo8pg6IEO9V0QRpLayLgrdM80j11ai7ArMrw2ld82qrvIY4PEomVxvUeMTa1U/ypICh X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: jC5u5HHWWcM+U3N7pE9x+2Vpo1XxWleztbM+IsfBIR/ncJHRueiqIggpefANnBAaM+E0I0ExyL2fcXtb7pVfDjwvX8r6MzYI09oCbCoeg/zDheswn5f1dHRM0LNaxUoiMgtvQvMVnw6DF4Dt9fvATXmOq8QdBMjfVDYsrsQYl+txcqQmJCpMTwxYT0+ZaD6P3iCSyA7vKKllzgumCTQKqAGn+dXDXUgaWn2zFN+L7FYUfke6/NhVxH+pkpXsYToQbED9RB9j49Rz1gORr25/5ec9u2iLhTCjyvmWqay6DuNRT15NhwOJaOltskdE4Sptn+BIfONNqQ53sNsd1smJ0jhxl8+DAgFtIg4IkE2l8ROgznedsbqtZSWsqNGZ3gFvYLg9TaPhXJLSpxOqREkLP5exKWm3888r0qD0Vu3lbKzF5REQKnNvAm7T/oEfwuObt/ltZLheiui6HT10q8SoHgc+5XzEi5r7iZE6EZqFKd6zNpXHjSILMwSnFkyxjAQEwZ2UFvZWDlA9zCSK5+pRaQX36BplOQLqIQzQJCA2+UA+01bCiTV7e3Afsdu0OyuGAKsGrgTXjjfhmWZZ1WTd8As186F9ar5yZ4f8RqDE5quMhuw7fuOhVBc8N0Za8Yi19PNdzsOiXVAqQ5OHF/qpJnprzKbYzK1VdCgh5dxujI+0n9TeOYpZaoIGwvh4Gq7M8QWXJeXBd+9uBfh33aHBEoNz/wksaLqfjKsDl+ZY44Dladv3P4ag7ifiCrpV7Uo4lv4EvnffyaTunT4xi8gFVsAgvB9TzXQeR/w0Ln7QD5f6vYx49GflwMumf4B8ZqYYMYex15Njrs9V1XwErxNkpeeYzEmifaywlpGu9YZaNRRy4WQP/h2DuojZIWOmrnr/NBa9puV3ImDG2nVQ07kb1P7D2Be5K+ckmiZlIf3QCfpf4WcE1iqHd0MIMyJ69+n0KSAQf0HKW14xrbLwidIYIq+HyUtSJlMjr+a/SwetyznwByKFmxgYcFdTi1QDEsYczsOd569b3oPfjZJtmpK8fbHUt0vfSGF5IMnNnHmcOrCoLAjhNuc62++zr4480ve3vzSOLtKGfU5w7RfakQv/WMiLmE0Yw6MZafPGMCsI8h/zeNBZO95rpEvLjZrzYWhYtpTjfEOWeMDSGWNXukl1L2G/wvTeW85leIpkhn2A07Q= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2abb81d4-735e-4473-77db-08db3403715f X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:39.7274 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2MdWZq9BlrhY8Ydt9i6R0oplwyjsNqivR+6F0aE66sPzJMWuB23Eqm7AdmO16PfdzWUtDeCbR2o1BUC6B+Hr0dFbh1MJm3y04n2f/uhib4U= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6365 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 malwarescore=0 adultscore=0 bulkscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: OzYmjE-GglT-w3DRsHWC62p0xNTxrbmF X-Proofpoint-ORIG-GUID: OzYmjE-GglT-w3DRsHWC62p0xNTxrbmF X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762133023864031184?= X-GMAIL-MSGID: =?utf-8?q?1762133023864031184?= clear_pages_rep(), clear_pages_erms() use string instructions internally. These, unlike a MOV loop, allow us to explicitly advertise the region-size to the processor. Thus, clearing in multi-page chunks means we can specify the real region sizes (or close to it) which is good for two reasons: - region-size can serve as a hint to current (some AMD Zen models) and possibly future uarchs which can use this hint to avoid polluting one or more levels of the dcache. - string instructions are typically microcoded, and would be cheaper if amortized across larger regions. We also execute fewer loop iterations (ex. a cond_resched() check for each page but those instructions are likely free.) clear_huge_page() now clears in three sections: the local neighbourhood of the faulting address (faulting page, and four surrounding pages), and its left and right regions. The local neighbourhood is cleared last to keep its cachelines hot. Performance == Use mmap(MAP_HUGETLB) to demand fault a 64GB region (on the local NUMA node): Icelakex (Platinum 8358, ucode=0xd0002c1, no_turbo=1): mm/clear_huge_page x86/clear_huge_page change (GB/s) (GB/s) pg-sz=2MB 8.76 11.82 +34.93% pg-sz=1GB 8.99 12.18 +35.48% On Icelakex we continue to allocate cachelines: pg-sz=2MB: - 701,951,397 L1-dcache-loads # 47.985 M/sec ( +- 19.22% ) (69.23%) - 3,239,403,770 L1-dcache-load-misses # 691.17% of all L1-dcache accesses ( +- 19.25% ) (69.24%) + 194,318,641 L1-dcache-loads # 17.905 M/sec ( +- 19.07% ) (69.25%) + 3,238,878,229 L1-dcache-load-misses # 2480.93% of all L1-dcache accesses ( +- 19.25% ) (69.26%) pg-sz=1GB: - 532,232,051 L1-dcache-loads # 37.378 M/sec ( +- 19.25% ) (69.23%) - 3,224,574,249 L1-dcache-load-misses # 909.02% of all L1-dcache accesses ( +- 19.25% ) (69.24%) + 22,587,703 L1-dcache-loads # 2.150 M/sec ( +- 19.38% ) (69.25%) + 3,223,143,697 L1-dcache-load-misses # 21478.37% of all L1-dcache accesses ( +- 19.25% ) (69.25%) Milan (EPYC 7J13, ucode=0xa0011a9, boost=0): mm/clear_huge_page x86/clear_huge_page change (GB/s) (GB/s) pg-sz=2MB 12.24 17.54 +43.30% pg-sz=1GB 17.98 37.24 +107.11% Milan uses a threshold ~32MB for eliding cacheline allocation, so we see a dropoff in cacheline-allocations for pg-sz=1GB: pg-sz=2MB: - 2,495,566,569 L1-dcache-loads # 476.417 M/sec ( +- 0.04% ) (33.38%) - 1,079,711,798 L1-dcache-load-misses # 43.28% of all L1-dcache accesses ( +- 0.01% ) (33.37%) + 2,235,310,058 L1-dcache-loads # 610.770 M/sec ( +- 0.02% ) (33.37%) + 1,089,602,355 L1-dcache-load-misses # 48.73% of all L1-dcache accesses ( +- 0.01% ) (33.37%) pg-sz=1GB: - 2,417,846,489 L1-dcache-loads # 679.753 M/sec ( +- 0.01% ) (33.38%) - 1,075,531,869 L1-dcache-load-misses # 44.49% of all L1-dcache accesses ( +- 0.01% ) (33.35%) + 31,159,378 L1-dcache-loads # 18.119 M/sec ( +- 3.27% ) (33.46%) + 14,692,358 L1-dcache-load-misses # 48.21% of all L1-dcache accesses ( +- 3.12% ) (33.46%) Signed-off-by: Ankur Arora --- Fuller perf stats for context: # Icelakex, baseline (mm/clear_huge_page), region-sz=64g, pg-sz=2mb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge 1' (3 runs): 21,945.59 msec task-clock # 2.999 CPUs utilized ( +- 19.25% ) 34 context-switches # 2.324 /sec ( +- 20.38% ) 3 cpu-migrations # 0.205 /sec ( +- 19.25% ) 198,152 page-faults # 13.546 K/sec ( +- 19.29% ) 56,513,364,885 cycles # 3.863 GHz ( +- 19.25% ) (38.44%) 2,583,719,806 instructions # 0.07 insn per cycle ( +- 19.24% ) (46.14%) 585,212,952 branches # 40.005 M/sec ( +- 19.23% ) (53.83%) 562,164 branch-misses # 0.14% of all branches ( +- 19.23% ) (61.53%) 282,621,312,162 slots # 19.320 G/sec ( +- 19.25% ) (69.22%) 11,048,627,225 topdown-retiring # 3.8% Retiring ( +- 19.22% ) (69.22%) 34,358,400,894 topdown-bad-spec # 11.5% Bad Speculation ( +- 19.57% ) (69.22%) 2,231,092,499 topdown-fe-bound # 0.8% Frontend Bound ( +- 19.25% ) (69.22%) 246,679,210,776 topdown-be-bound # 84.0% Backend Bound ( +- 19.21% ) (69.22%) 701,951,397 L1-dcache-loads # 47.985 M/sec ( +- 19.22% ) (69.23%) 3,239,403,770 L1-dcache-load-misses # 691.17% of all L1-dcache accesses ( +- 19.25% ) (69.24%) 11,475,685 LLC-loads # 784.475 K/sec ( +- 19.23% ) (69.25%) 793,272 LLC-load-misses # 10.36% of all LL-cache accesses ( +- 19.23% ) (69.25%) 17,821,045 L1-icache-load-misses # 0.00% of all L1-icache accesses ( +- 19.51% ) (30.77%) 693,339,354 dTLB-loads # 47.397 M/sec ( +- 19.33% ) (30.76%) 637,811 dTLB-load-misses # 0.14% of all dTLB cache accesses ( +- 19.09% ) (30.75%) 131,922 iTLB-load-misses # 0.00% of all iTLB cache accesses ( +- 19.59% ) (30.75%) 7.31681 +- 0.00177 seconds time elapsed ( +- 0.02% ) # Icelakex, multi-page (x86/clear_huge_page), region-sz=64g, pg-sz=2mb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge 1' (3 runs): 16,276.28 msec task-clock # 2.999 CPUs utilized ( +- 19.24% ) 27 context-switches # 2.488 /sec ( +- 19.25% ) 3 cpu-migrations # 0.276 /sec ( +- 19.25% ) 196,935 page-faults # 18.146 K/sec ( +- 19.25% ) 41,906,597,608 cycles # 3.861 GHz ( +- 19.24% ) (38.44%) 729,479,932 instructions # 0.03 insn per cycle ( +- 19.38% ) (46.14%) 133,969,095 branches # 12.344 M/sec ( +- 19.35% ) (53.84%) 412,818 branch-misses # 0.46% of all branches ( +- 18.97% ) (61.54%) 209,574,316,961 slots # 19.311 G/sec ( +- 19.24% ) (69.24%) 4,933,512,982 topdown-retiring # 2.3% Retiring ( +- 19.24% ) (69.24%) 20,272,641,267 topdown-bad-spec # 9.4% Bad Speculation ( +- 19.51% ) (69.24%) 837,421,487 topdown-fe-bound # 0.4% Frontend Bound ( +- 19.24% ) (69.24%) 190,089,232,476 topdown-be-bound # 88.0% Backend Bound ( +- 19.19% ) (69.24%) 194,318,641 L1-dcache-loads # 17.905 M/sec ( +- 19.07% ) (69.25%) 3,238,878,229 L1-dcache-load-misses # 2480.93% of all L1-dcache accesses ( +- 19.25% ) (69.26%) 10,560,508 LLC-loads # 973.081 K/sec ( +- 19.23% ) (69.26%) 724,884 LLC-load-misses # 10.28% of all LL-cache accesses ( +- 17.15% ) (69.26%) 14,378,070 L1-icache-load-misses # 0.00% of all L1-icache accesses ( +- 19.13% ) (30.75%) 185,562,230 dTLB-loads # 17.098 M/sec ( +- 19.74% ) (30.74%) 617,978 dTLB-load-misses # 0.51% of all dTLB cache accesses ( +- 18.72% ) (30.74%) 112,509 iTLB-load-misses # 0.00% of all iTLB cache accesses ( +- 19.76% ) (30.74%) 5.42697 +- 0.00152 seconds time elapsed ( +- 0.03% ) # Icelakex, baseline (mm/clear_huge_page), region-sz=64g, pg-sz=1gb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64 --huge 2' (3 runs): 21,361.22 msec task-clock # 2.999 CPUs utilized ( +- 19.25% ) 23 context-switches # 1.615 /sec ( +- 18.95% ) 3 cpu-migrations # 0.211 /sec ( +- 19.25% ) 701 page-faults # 49.230 /sec ( +- 19.27% ) 54,981,958,487 cycles # 3.861 GHz ( +- 19.25% ) (38.44%) 2,012,625,953 instructions # 0.05 insn per cycle ( +- 19.25% ) (46.14%) 470,264,509 branches # 33.026 M/sec ( +- 19.25% ) (53.83%) 194,801 branch-misses # 0.06% of all branches ( +- 18.88% ) (61.53%) 274,966,507,627 slots # 19.311 G/sec ( +- 19.25% ) (69.22%) 10,555,137,650 topdown-retiring # 3.8% Retiring ( +- 19.04% ) (69.22%) 21,206,785,918 topdown-bad-spec # 7.8% Bad Speculation ( +- 18.13% ) (69.22%) 1,094,597,329 topdown-fe-bound # 0.4% Frontend Bound ( +- 19.25% ) (69.22%) 244,462,123,545 topdown-be-bound # 88.0% Backend Bound ( +- 19.33% ) (69.22%) 532,232,051 L1-dcache-loads # 37.378 M/sec ( +- 19.25% ) (69.23%) 3,224,574,249 L1-dcache-load-misses # 909.02% of all L1-dcache accesses ( +- 19.25% ) (69.24%) 2,318,195 LLC-loads # 162.804 K/sec ( +- 19.35% ) (69.25%) 206,737 LLC-load-misses # 13.44% of all LL-cache accesses ( +- 18.30% ) (69.25%) 4,950,866 L1-icache-load-misses # 0.00% of all L1-icache accesses ( +- 19.26% ) (30.77%) 531,299,560 dTLB-loads # 37.313 M/sec ( +- 19.24% ) (30.76%) 2,811 dTLB-load-misses # 0.00% of all dTLB cache accesses ( +- 17.25% ) (30.75%) 26,355 iTLB-load-misses # 0.00% of all iTLB cache accesses ( +- 19.58% ) (30.75%) 7.12187 +- 0.00190 seconds time elapsed ( +- 0.03% ) # Icelakex, multi-page (x86/clear_huge_page), region-sz=64g, pg-sz=1gb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64 --huge 2' (3 runs): 15,764.52 msec task-clock # 2.999 CPUs utilized ( +- 19.25% ) 17 context-switches # 1.618 /sec ( +- 20.47% ) 3 cpu-migrations # 0.285 /sec ( +- 19.25% ) 700 page-faults # 66.614 /sec ( +- 19.22% ) 40,560,984,582 cycles # 3.860 GHz ( +- 19.25% ) (38.45%) 79,578,792 instructions # 0.00 insn per cycle ( +- 19.24% ) (46.15%) 13,872,134 branches # 1.320 M/sec ( +- 19.23% ) (53.85%) 119,492 branch-misses # 1.29% of all branches ( +- 18.80% ) (61.55%) 202,854,573,160 slots # 19.304 G/sec ( +- 19.25% ) (69.25%) 3,982,417,725 topdown-retiring # 2.0% Retiring ( +- 19.25% ) (69.25%) 13,523,424,635 topdown-bad-spec # 6.8% Bad Speculation ( +- 18.69% ) (69.25%) 18,661,431 topdown-fe-bound # 0.0% Frontend Bound ( +- 19.28% ) (69.25%) 185,884,147,789 topdown-be-bound # 91.3% Backend Bound ( +- 19.28% ) (69.25%) 22,587,703 L1-dcache-loads # 2.150 M/sec ( +- 19.38% ) (69.25%) 3,223,143,697 L1-dcache-load-misses # 21478.37% of all L1-dcache accesses ( +- 19.25% ) (69.25%) 1,777,675 LLC-loads # 169.169 K/sec ( +- 19.60% ) (69.25%) 126,583 LLC-load-misses # 10.77% of all LL-cache accesses ( +- 19.82% ) (69.25%) 3,333,729 L1-icache-load-misses # 0.00% of all L1-icache accesses ( +- 19.49% ) (30.75%) 19,999,517 dTLB-loads # 1.903 M/sec ( +- 19.38% ) (30.75%) 1,833 dTLB-load-misses # 0.01% of all dTLB cache accesses ( +- 17.72% ) (30.75%) 34,066 iTLB-load-misses # 0.00% of all iTLB cache accesses ( +- 19.09% ) (30.75%) 5.25624 +- 0.00176 seconds time elapsed ( +- 0.03% ) # Milan, baseline (mm/clear_huge_page), region-sz=64g, pg-sz=2mb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge 1' (3 runs): 5,241.76 msec task-clock # 1.000 CPUs utilized ( +- 0.08% ) 10 context-switches # 1.909 /sec ( +- 8.82% ) 1 cpu-migrations # 0.191 /sec 65,636 page-faults # 12.530 K/sec ( +- 0.00% ) 12,730,694,768 cycles # 2.430 GHz ( +- 0.08% ) (33.31%) 36,709,243 stalled-cycles-frontend # 0.29% frontend cycles idle ( +- 24.07% ) (33.32%) 37,520,225 stalled-cycles-backend # 0.29% backend cycles idle ( +- 9.87% ) (33.34%) 874,896,010 instructions # 0.07 insn per cycle # 0.05 stalled cycles per insn ( +- 1.23% ) (33.36%) 199,308,386 branches # 38.049 M/sec ( +- 0.84% ) (33.38%) 441,428 branch-misses # 0.22% of all branches ( +- 4.68% ) (33.38%) 2,495,566,569 L1-dcache-loads # 476.417 M/sec ( +- 0.04% ) (33.38%) 1,079,711,798 L1-dcache-load-misses # 43.28% of all L1-dcache accesses ( +- 0.01% ) (33.37%) 50,936,391 L1-icache-loads # 9.724 M/sec ( +- 1.29% ) (33.35%) 284,407 L1-icache-load-misses # 0.56% of all L1-icache accesses ( +- 4.60% ) (33.33%) 546,596 dTLB-loads # 104.348 K/sec ( +- 6.14% ) (33.31%) 231,897 dTLB-load-misses # 42.08% of all dTLB cache accesses ( +- 4.27% ) (33.29%) 6 iTLB-loads # 1.145 /sec ( +- 72.65% ) (33.29%) 34,065 iTLB-load-misses # 262038.46% of all iTLB cache accesses ( +- 44.88% ) (33.29%) 18,237,487 L1-dcache-prefetches # 3.482 M/sec ( +- 12.84% ) (33.29%) 5.23915 +- 0.00421 seconds time elapsed ( +- 0.08% ) # Milan, multi-page (x86/clear_huge_page), region-sz=64g, pg-sz=2mb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge 1' (3 runs): 3,655.71 msec task-clock # 0.999 CPUs utilized ( +- 0.13% ) 7 context-switches # 1.913 /sec ( +- 8.25% ) 1 cpu-migrations # 0.273 /sec 65,636 page-faults # 17.934 K/sec ( +- 0.00% ) 8,879,727,514 cycles # 2.426 GHz ( +- 0.13% ) (33.26%) 5,733,380 stalled-cycles-frontend # 0.06% frontend cycles idle ( +-170.04% ) (33.28%) 42,012,302 stalled-cycles-backend # 0.47% backend cycles idle ( +- 24.51% ) (33.31%) 214,672,610 instructions # 0.02 insn per cycle # 0.28 stalled cycles per insn ( +- 1.71% ) (33.34%) 42,298,268 branches # 11.557 M/sec ( +- 1.28% ) (33.36%) 267,936 branch-misses # 0.62% of all branches ( +- 7.80% ) (33.37%) 2,235,310,058 L1-dcache-loads # 610.770 M/sec ( +- 0.02% ) (33.37%) 1,089,602,355 L1-dcache-load-misses # 48.73% of all L1-dcache accesses ( +- 0.01% ) (33.37%) 48,725,812 L1-icache-loads # 13.314 M/sec ( +- 0.25% ) (33.37%) 231,227 L1-icache-load-misses # 0.47% of all L1-icache accesses ( +- 13.20% ) (33.37%) 280,655 dTLB-loads # 76.685 K/sec ( +- 13.33% ) (33.37%) 151,028 dTLB-load-misses # 44.02% of all dTLB cache accesses ( +- 6.64% ) (33.35%) 15 iTLB-loads # 4.099 /sec ( +- 6.67% ) (33.32%) 121,208 iTLB-load-misses # 865771.43% of all iTLB cache accesses ( +- 2.74% ) (33.29%) 18,702,209 L1-dcache-prefetches # 5.110 M/sec ( +- 12.51% ) (33.27%) 3.66065 +- 0.00461 seconds time elapsed ( +- 0.13% ) # Milan, baseline (mm/clear_huge_page), region-sz=64g, pg-sz=1gb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge 2' (3 runs): 3,544.20 msec task-clock # 0.996 CPUs utilized ( +- 0.21% ) 5 context-switches # 1.406 /sec ( +- 6.67% ) 1 cpu-migrations # 0.281 /sec 227 page-faults # 63.819 /sec ( +- 0.15% ) 8,609,810,964 cycles # 2.421 GHz ( +- 0.21% ) (33.30%) 77,420,424 stalled-cycles-frontend # 0.90% frontend cycles idle ( +- 20.55% ) (33.33%) 25,197,541 stalled-cycles-backend # 0.29% backend cycles idle ( +- 1.09% ) (33.35%) 658,146,061 instructions # 0.08 insn per cycle # 0.16 stalled cycles per insn ( +- 0.04% ) (33.38%) 154,867,131 branches # 43.539 M/sec ( +- 0.04% ) (33.41%) 167,531 branch-misses # 0.11% of all branches ( +- 5.19% ) (33.41%) 2,417,846,489 L1-dcache-loads # 679.753 M/sec ( +- 0.01% ) (33.38%) 1,075,531,869 L1-dcache-load-misses # 44.49% of all L1-dcache accesses ( +- 0.01% ) (33.35%) 12,835,321 L1-icache-loads # 3.609 M/sec ( +- 0.41% ) (33.33%) 55,282 L1-icache-load-misses # 0.43% of all L1-icache accesses ( +- 1.98% ) (33.30%) 23,287 dTLB-loads # 6.547 K/sec ( +- 15.61% ) (33.29%) 1,333 dTLB-load-misses # 4.48% of all dTLB cache accesses ( +- 1.26% ) (33.29%) 3 iTLB-loads # 0.843 /sec ( +- 33.33% ) (33.29%) 231 iTLB-load-misses # 11550.00% of all iTLB cache accesses ( +- 6.14% ) (33.29%) 170,608,062 L1-dcache-prefetches # 47.965 M/sec ( +- 0.84% ) (33.29%) 3.55776 +- 0.00738 seconds time elapsed ( +- 0.21% ) # Milan, multi-page (x86/clear_huge_page), region-sz=64g, pg-sz=1gb Performance counter stats for 'taskset -c 15 bench/pf-test --sz 64g --huge 2' (3 runs): 1,718.27 msec task-clock # 0.999 CPUs utilized ( +- 0.08% ) 6 context-switches # 3.489 /sec ( +- 14.70% ) 1 cpu-migrations # 0.581 /sec 227 page-faults # 132.000 /sec ( +- 0.15% ) 4,176,107,493 cycles # 2.428 GHz ( +- 0.08% ) (33.19%) 2,675,797 stalled-cycles-frontend # 0.06% frontend cycles idle ( +- 0.34% ) (33.25%) 147,394,527 stalled-cycles-backend # 3.53% backend cycles idle ( +- 8.80% ) (33.31%) 12,779,784 instructions # 0.00 insn per cycle # 13.14 stalled cycles per insn ( +- 0.09% ) (33.37%) 2,428,829 branches # 1.412 M/sec ( +- 0.08% ) (33.42%) 63,460 branch-misses # 2.61% of all branches ( +- 3.48% ) (33.46%) 31,159,378 L1-dcache-loads # 18.119 M/sec ( +- 3.27% ) (33.46%) 14,692,358 L1-dcache-load-misses # 48.21% of all L1-dcache accesses ( +- 3.12% ) (33.46%) 2,556,688 L1-icache-loads # 1.487 M/sec ( +- 0.89% ) (33.46%) 21,148 L1-icache-load-misses # 0.84% of all L1-icache accesses ( +- 0.25% ) (33.41%) 6,114 dTLB-loads # 3.555 K/sec ( +- 12.76% ) (33.35%) 1,742 dTLB-load-misses # 33.73% of all dTLB cache accesses ( +- 21.79% ) (33.29%) 45 iTLB-loads # 26.167 /sec ( +- 7.52% ) (33.23%) 90 iTLB-load-misses # 210.94% of all iTLB cache accesses ( +- 21.20% ) (33.17%) 257,942 L1-dcache-prefetches # 149.993 K/sec ( +- 9.84% ) (33.17%) 1.72042 +- 0.00139 seconds time elapsed ( +- 0.08% ) --- arch/x86/mm/hugetlbpage.c | 49 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index 5804bbae4f01..4294b77c4f18 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -148,6 +148,55 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, return hugetlb_get_unmapped_area_topdown(file, addr, len, pgoff, flags); } + +/* + * This is used on all !CONFIG_HIGHMEM configurations. + * + * CONFIG_HIGHMEM, falls back to the __weak version. + */ +#ifndef CONFIG_HIGHMEM +static void clear_contig_region(struct page *page, unsigned long vaddr, + unsigned int npages) +{ + clear_user_pages(page_address(page), vaddr, page, npages); +} + +void clear_huge_page(struct page *page, + unsigned long addr_hint, unsigned int pages_per_huge_page) +{ + unsigned long addr = addr_hint & + ~(((unsigned long)pages_per_huge_page << PAGE_SHIFT) - 1); + const long pgidx = (addr_hint - addr) / PAGE_SIZE; + const int first_pg = 0, last_pg = pages_per_huge_page - 1; + const int width = 2; /* pages cleared last on either side */ + int sidx[3], eidx[3]; + int i, n; + + if (pages_per_huge_page > MAX_ORDER_NR_PAGES) + return clear_contig_region(page, addr, pages_per_huge_page); + + /* + * Neighbourhood of the fault. Cleared at the end to ensure + * it sticks around in the cache. + */ + n = 2; + sidx[n] = (pgidx - width) < first_pg ? first_pg : (pgidx - width); + eidx[n] = (pgidx + width) > last_pg ? last_pg : (pgidx + width); + + sidx[0] = first_pg; /* Region to the left of the fault */ + eidx[0] = sidx[n] - 1; + + sidx[1] = eidx[n] + 1; /* Region to the right of the fault */ + eidx[1] = last_pg; + + for (i = 0; i <= 2; i++) { + if (eidx[i] >= sidx[i]) + clear_contig_region(page + sidx[i], + addr + sidx[i] * PAGE_SIZE, + eidx[i] - sidx[i] + 1); + } +} +#endif /* CONFIG_HIGHMEM */ #endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_X86_64 From patchwork Mon Apr 3 05:22:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 78362 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2087480vqo; Sun, 2 Apr 2023 23:03:11 -0700 (PDT) X-Google-Smtp-Source: AKy350YDalsXS3rKIE0JajXi7thu1LcVxMqsO0hl+EFgxeefQr7T5yQxE4PCbaWS0DQabs5w2SKz X-Received: by 2002:a17:902:ec91:b0:19e:72c5:34df with SMTP id x17-20020a170902ec9100b0019e72c534dfmr47329416plg.52.1680501791550; Sun, 02 Apr 2023 23:03:11 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1680501791; cv=pass; d=google.com; s=arc-20160816; b=NuioL9anf3RcIy7qUtC6G6fHM6xOiqZPd58nr6/2cfeBuGkZEFzXDzB9GthaC5mHMB 3pf5DR5syJDD6lvwx+j3ik2gE6Jht2t9SGhOYCJshRdjIjU/YU6/swhSUX4qyXfge3VM r1RAcbP7xpmGDfHh9xgunsU+XQPDRkhvPN2UW0gRt9vDkWQq/wgAwz1rcy1R3ldAzJiE 0UMIWzk44kY72AYGU3T23Ih10OglDmICfCSsKIMU3oWpYFBaMjQoJUDjgZu4puhf258z Xe+lzBsSsCTMKRikYDaT0Zc52699OTuEUFlqMh+2qGBxmUc9w+00cI4E5BoIMF6nsMMr rErA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=AWtO26WrUxcC8Dr9AfFmU8r0+3TZoPF8bVvYSgph62Q=; b=pMcSe6n9n7X4Wvf8v2erJDud2xImsaVuw5hpOz09UWC93PebKa4g51t6IX2WNilUT4 HzgEqbrb0MVpvFB7zxnN79mnYlDVo6+TmDl6XuDce2cv1N7LcCny+9aS3sur3HcyL/S0 jJpZoik2U7UPleI77Tvp1FRW7/5McxG+eCqu0+PNgIx5qjOkw2kRZiQmgENK2I5vOnMj yULedmGsEksYNgwMe9oOzcRRAKJXdKRLK3iLGg917Ij0bWNcGjj1RAMKwSHS66DclaS+ RnIjMD9oj69JMkircYyD9lU98u8KRaw9Z/Sv1H2x6szKfQe98NMMTsHoio2MePZZ6jrZ Ntsg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=QzWiclfT; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=pc5h1iDO; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e4-20020a63d944000000b0050bd4b86169si7998301pgj.414.2023.04.02.23.02.54; Sun, 02 Apr 2023 23:03:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=QzWiclfT; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=pc5h1iDO; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231598AbjDCFYI (ORCPT + 99 others); Mon, 3 Apr 2023 01:24:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231536AbjDCFXh (ORCPT ); Mon, 3 Apr 2023 01:23:37 -0400 Received: from mx0a-00069f02.pphosted.com (mx0a-00069f02.pphosted.com [205.220.165.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9D893A99 for ; Sun, 2 Apr 2023 22:23:18 -0700 (PDT) Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332NIh1B013542; Mon, 3 Apr 2023 05:22:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=AWtO26WrUxcC8Dr9AfFmU8r0+3TZoPF8bVvYSgph62Q=; b=QzWiclfTa84OJnoMiqC/7ngDJfU7OilahzGorX6/iDroF+VOt9o5E7YvMa0voGN9GMGi L/T5pVXsk7jDRfAs7svSsRk7spEwhTQFau+963l3607okU487LO2MlMNcIqeid/QU+z1 JIAipmHfsYHs8URW3bpfWRiu2GNX8IT78Gb2nICU7cJLQBTPET8ijOcuhWy89YwgmqiJ GaelT/lv+yXyj+22lS1X1citlyXcHod7ULtxhQgX+WrIQVy6d75z3/GB/mMeHtkmX43g vGQd0ImiCp3A6cAsFjE7cP5HpE5c8fnv8GK7sJjkIv+N2ALnV8OES7PI1C20nQaIC4+V xQ== Received: from phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta03.appoci.oracle.com [138.1.37.129]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppcgaj5a1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:48 +0000 Received: from pps.filterd (phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332i7pt013932; Mon, 3 Apr 2023 05:22:47 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2107.outbound.protection.outlook.com [104.47.55.107]) by phxpaimrmta03.imrmtpd1.prodappphxaev1.oraclevcn.com (PPS) with ESMTPS id 3ppt3djuqb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:47 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=VMfTjRopugbMrhSsq/HHWJBSJZik/77H9g6QRhukxMC6LFF5yXoM6dkaDhJBX+3CkJtQ7TrzImxT6D6QeN1lPrni9zrpdXN+jbevC8kAg0jmWgw7k2X5IWjf3jnQnTGr/51Ap6iqVgEhDNnbbnhJyoSknL4UXc4UpXKOkCVe4b/eTpDtVBvqU63bhDeb9YNqMz6w5d/jcwV0vtRjlMMfnG8cKbhmmhXGQidA8vixZZJZEa+pmqBxMfWGW4H1gw0CFcJysCGvFy9E+CivKfp92KBsNZuRLVVhvS7wZKKWEJnc+EZHSgCDzvv46S5HpDPdcYxcWV/IkEyc6hqLYZ5+/g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AWtO26WrUxcC8Dr9AfFmU8r0+3TZoPF8bVvYSgph62Q=; b=fpG7HBxAZfSNIkJ23oXjW8pfBDsAos6ZuomHiV3sgcsGHdvuPyqbuiVgNfFloS7Hyv34uLNInFhbYMVxYz4+DNEG4rSQrG51yPpjg20NH/6QGETJOVlL5FRIZb0kc8JFIHX78IvBNN1EpGYhq5SNyZ/sxo7GvHo/10Vn2M2S43qgsjiVc1TyCUo9XSjq9PQQzcjW6u6KC/f6PU+AGi2OB065AWblwpSmYGvpxazSFyCaJT21oCR60Bst8kWSzOSH1kTw00NVdlFAn2IiRqbIYLYTUMPX1birWvpyxi4GmEW8wG4Hsij9Nu7tto9t3bdeezeetfr8BTjIVXCqXRFyTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AWtO26WrUxcC8Dr9AfFmU8r0+3TZoPF8bVvYSgph62Q=; b=pc5h1iDO5I63MTP/JVPka1qQH1fKok4RVjVA60P+OIRd7FyTXOAlx5KDtGE1ooBdQzg880bOxouff4eRuD5IB2IdLVINdht+W/suw6OhBNdx8cckUIppCxX4ntmfeGyZJqaZR3WgeIIi9FZJ2QU13TuIJybzKbjvzEkh3dGyiQg= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SA1PR10MB6365.namprd10.prod.outlook.com (2603:10b6:806:255::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:45 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:45 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 7/9] sched: define TIF_ALLOW_RESCHED Date: Sun, 2 Apr 2023 22:22:31 -0700 Message-Id: <20230403052233.1880567-8-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR03CA0166.namprd03.prod.outlook.com (2603:10b6:303:8d::21) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SA1PR10MB6365:EE_ X-MS-Office365-Filtering-Correlation-Id: 0187daa4-2b8a-47ea-8b48-08db340372f9 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: n+2pEnKRx9o049wD/LZjcRBZ5dXhBClqW6Shrj5mkMgFH/k2q6ngnOXQKb7FtFuMSSNO1In8SUP693EX4t+kl/1xHdXBvGqkGrsZmlgnzA0zSm6TGbMHf/S6/SDxYHqrwoMkVwY/wx8thwDWtg1A7YZlMg818SO1ttX44/7KYfMS2YeM4uaCO+wPSVWsFegPV1rLysE4oDM4Qwsqh3qMyVZDuhy7CT3wWyRD1Hur6hb+LGo3VlZIW9ybe7GVaIs6SWXhBGr3mH32eQRDGCtQdcTbzzOJvxXFOitwQmeezuvQGoEIbOKw7QbghRm1VWU7X2vO2NPlxkDJrQCRezyjslVb/SGrxwjiTdxNsVHL2rpaovryQROCol8+K3ixyUuH1SQJh2omXaSqlyCLQSs+PY3cVvoPctamXrNZER+prq36KY2TTZhDeDnQ8WfLRZOL1QZTvb+E8Bod58GvDNhgXXSeeqsbPjAOaFNWaJQmL3qhIuK/YbgJFxoNvwfi/mrtilrhsfoP3z3bsr3eQcXxjRJnFZ4PnoL1CDLT/xeT5x1Kdje/mJHkQ9za4OHGZ8Pd X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(346002)(136003)(396003)(376002)(39860400002)(451199021)(86362001)(36756003)(103116003)(2906002)(2616005)(83380400001)(186003)(6512007)(6506007)(1076003)(26005)(6486002)(107886003)(6666004)(4326008)(8676002)(66946007)(66556008)(66476007)(41300700001)(5660300002)(7416002)(38100700002)(316002)(478600001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ahJtsEbR0Pu035xSPsiACxf6wPqKzntHhxV8lwYFiKOJ5IXr7E3AwfLMJCJ+kD5ILdKgds1RrLPz4pca2iteRIzAxmCKACDLf7bsLn74fffJqKInGBId2mmgcXwhOK4cg6dvJ8Q1Z2Thd3x99lymP1mu3U+vRuwuzqysWUQJY8L0CFDpG9zLMnvtHphNvXhu6lc75UmDaGBDo/h7FQeWv0OKKFp5NemLeo55/2Uhv128LExDWyyVlKAS0JX+4DadL/3g8/ultQ5Nr80NO/I6E31N5i45QBC8KaI52Kc73Z4zxjfyjjt5aPOS9WXusKpCiYNJmmFeZ5EXeBFARu5MNMWeKjsEpp6rnctQjsKixYeOSbCzsjivtv/FkcP4gf5w80BuKzGqjtL98xfTSGS1U9LXwLtq1zbnf0CsivCtONCG5ckAsNtqv5dG03yJ0mZhAh2ctgUZaPTyQqqRtQNA5jhNexXOnrNSGRx8WHx5f1930m0u0npTOAc6WnflCeCQk7pKsTZEbbzI7+v3L0sGFq/T6cOj5MbOfRw5O567vSGWkerDpJ0VTxFYNbUEgwgr5LNnPNAVMRfZoOjA9vxwRox9JSIFUpAaKXn1O7m+zrnuDjmU7Gj0hHs6aE7L4cYm6QHqNxaBYrz7B+aIppR/T2nMmL/f9ZZ2ZPnm2V57d1WZLziQlqvM1izLb1pcJXGr/1YhPSWDJ9Hn6e1VAlRua4e23/umZc9IO5WkQ+Ib0h9o13UEcqxyM28eP/q0eIGfCHK9zLR1WGsNiqZMs0/Llb/oMoqv4UVt1CN75XS5lbbuhC+CjdgHlYvwcQ7bff7WuIBOYc48PNJNVMt6Uv0Ir4xa288WrIqTWADdVAXqPgcgicS/6fHrRR/gwTopnxor+Whi3+OIFdJz7shHnlnlg/erYINnKNJAR7Oc89V02mDDK7agTqFLz4vau8PCSWEVG8oVIpLeVSY9XsB+b0M1Kd+L7y0l4B5ySMFI8ULxij1r+CUC258sXurQKqG0/zkk2uMckRvJa4evuDFyTH1XyjAEfCLqV1eUmK19xUl2gjNarNb5I68lWpkLhx/Y5NRjxjCbTuhTz3hb1eKB5qGZk/fsAgeW0zCkAKb9hSGXg46aV7SNKQN2SGIMguc2kruriOcbwW2wXqGSr+jNjCSXowHvfGTeV2Hrw6wyTekFPiFwYZxBQWnBYcDEDumJ+9WyEYl64CSAeKQqq27PX96S6necIp3E3stRrCbg6zNa6jNP7zvQjpWXA9wYS+ur5Zge1oCzIVFabrqtzAAX2HfPW4cSW1aKk9relzp+U1Z9zi6X+leCNqjd45IgMBal0FrOJ8vrCSXPAEZeKWccv4a8M2f7Fb9HOxawJVSHWfYX/gVqCfauk7rdJI5ice4DEGynXw+iEscDnwPzmX9GMtYLpXpfdUI8wHG8fvVkFajEzI5v+dDw8OLOCMhEt4uKXwRwP4sAyK6EYWL3ChofGkDrB2GBc3VtFu+dlk67XuHnOI9yApRfofROHoSUJTWxtJK6ZoyQvYEqmTixpl2SQVg+dQlSKpa4HkId99GyWUkNlUaw+arC8biS4dIav3rKwjiL X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: 3r6ZE2sDejkM8UbH77cxDVrIGoYh+cGi9iyshzrkFpfEcbAwfl9Me7pPIyWWZZ/I1JIHpyzvQiGnb/IturQjewasADuxN6tKFrZqhuL1rXKqYJMvvqO54Vv01HuYeHw9v1mS3p3CynkSAnzWjgABRIfQdm7pgLq6PboArgi9bauSun/VWrhVk1qr4GpoGWHEg8kEo8lA3yLaWDX+Z4C2ZaPJYyHGOjXPbF3NvUrOyjk+0GShQN+2PsXBQVf4y4yvw7iltqT12DEb3Dp5raC4Ckzx9oOvNkK9UcVhkAWMw/A7u2lu+sUStulM+Ij7E7F8KyOekq5Mf2+Hc/tquCL0yqx5KmmxBQmA104R63YLinuXqe0V1EDmttVRKsTyK6+tSYSgzwlOugSxdIsOguPg6PI5bYzXO0cLN4yD0XDubdNxta3YjJlP/y50IY1ys5Sw52oRWJK24pWlXeRlwIs3fUuHKnkTNS4s9FD28/zb2z/VlgHI/vV0TVZELnlSxWnI89vrcbO37ZOXppDGNHBLXMwnVcBhnVwtdhG8iZHl+t6Q5JAPpIH883OhWs+ZA6cwkvmV9g0fg1ljwz1OqN0e6EZJANanebwfQe8xMG6Qt+SVLFr0WDdVV+dMSIODOdglfTAj1B4aJtQ0Pf6gAKFaW79hxqt++xtU9w6RSNYh5626h6pZfS81ZIUnPUFf6KULjMKI9+4vGMk9zr4LYnG8YQTLHXLJaik1EKHVLrag3iMHc1UrZMl3w48vnCAngyxljjUrMQYe6gchcxRUgN/PhnzfilydRE2BAIENFjwKoMBl76NUM85px6K0YwIbmZFNW4p+K7dteu7ZV1pDaXr05PTJ3JvV75U6lywb8vxU0aGQCIdjYRUqUonXRlVkReGw/qZzyGJhRz5TsUDb5iS3U4GbzDgIHtW3swFZ7ho9vSJy6Ls7P4dvfldvrErdXlPxERhsR9yBnwxklZkaCcts/KDNVs6TowdRLJgtawie3zjIO8I7dx87JNgDbm/Cci6Ub6OtCOxDbKvP68jJx8ZVb1Dp5Sa3lm4DlMdsWE/Ny4P83lW2RAOe78xafB2aRvH5Jlo909rIID9EJdA8dC51AybwmDtn5VteCXJmP5ec/1O3CqJjs/xKKvGbGkGnGmjWukJolvVpPIsopYFl9Y8fC1D78ex4+WSkQKA6eByvNTc= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0187daa4-2b8a-47ea-8b48-08db340372f9 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:45.0440 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: H02fg1kyYkymzqYus/XlB3WnUpEHO0gj3BRTGe3t9rTYEGIfgBgvzuwIjlyrWRV5JaeGqodX5vMi6AG69uauyrbUGbThIjkeWKF4QojsDlY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6365 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxlogscore=627 mlxscore=0 adultscore=0 suspectscore=0 spamscore=0 malwarescore=0 phishscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: 3n25f6C0Vp3I69V4G6C_LKVTj8U4YkXk X-Proofpoint-ORIG-GUID: 3n25f6C0Vp3I69V4G6C_LKVTj8U4YkXk X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762133846608972662?= X-GMAIL-MSGID: =?utf-8?q?1762133846608972662?= Define TIF_ALLOW_RESCHED to allow threads to mark themselves as allowing rescheduling in the irqexit path with PREEMPTION_NONE/_VOLUNTARY. This is meant to be used for long running tasks where it is not convenient to periodically call cond_resched(). Suggested-by: Linus Torvalds Signed-off-by: Ankur Arora --- arch/x86/include/asm/thread_info.h | 2 ++ include/linux/sched.h | 29 +++++++++++++++++++++++++++++ 2 files changed, 31 insertions(+) diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thread_info.h index f1cccba52eb9..8c18b9eaeec4 100644 --- a/arch/x86/include/asm/thread_info.h +++ b/arch/x86/include/asm/thread_info.h @@ -100,6 +100,7 @@ struct thread_info { #define TIF_BLOCKSTEP 25 /* set when we want DEBUGCTLMSR_BTF */ #define TIF_LAZY_MMU_UPDATES 27 /* task is updating the mmu lazily */ #define TIF_ADDR32 29 /* 32-bit address space on 64 bits */ +#define TIF_RESCHED_ALLOW 30 /* can reschedule if needed */ #define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME) #define _TIF_SIGPENDING (1 << TIF_SIGPENDING) @@ -122,6 +123,7 @@ struct thread_info { #define _TIF_BLOCKSTEP (1 << TIF_BLOCKSTEP) #define _TIF_LAZY_MMU_UPDATES (1 << TIF_LAZY_MMU_UPDATES) #define _TIF_ADDR32 (1 << TIF_ADDR32) +#define _TIF_RESCHED_ALLOW (1 << TIF_RESCHED_ALLOW) /* flags to check in __switch_to() */ #define _TIF_WORK_CTXSW_BASE \ diff --git a/include/linux/sched.h b/include/linux/sched.h index 63d242164b1a..1e7536e6d9ce 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -2229,6 +2229,35 @@ static __always_inline bool need_resched(void) return unlikely(tif_need_resched()); } +/* + * Define this in common code to avoid include hell. + */ +static __always_inline bool resched_allowed(void) +{ +#ifndef TIF_RESCHED_ALLOW + return false; +#else + return unlikely(test_tsk_thread_flag(current, TIF_RESCHED_ALLOW)); +#endif +} + +static inline void allow_resched(void) +{ + /* + * allow_resched() allows preemption via the irqexit context. + * To ensure that we stick around on the current runqueue, + * disallow migration. + */ + migrate_disable(); + set_tsk_thread_flag(current, TIF_RESCHED_ALLOW); +} + +static inline void disallow_resched(void) +{ + clear_tsk_thread_flag(current, TIF_RESCHED_ALLOW); + migrate_enable(); +} + /* * Wrappers for p->thread_info->cpu access. No-op on UP. */ From patchwork Mon Apr 3 05:22:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 78361 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2084220vqo; Sun, 2 Apr 2023 22:55:09 -0700 (PDT) X-Google-Smtp-Source: AKy350bV5Xuf/gE3n9cwI8XVpEgGKOrnpTAd3t0tvDBya4Ha1C7MxXtFmlXMm8NGMuDxjMg+l/XK X-Received: by 2002:aa7:c989:0:b0:502:243b:26f8 with SMTP id c9-20020aa7c989000000b00502243b26f8mr32038127edt.8.1680501309183; Sun, 02 Apr 2023 22:55:09 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1680501309; cv=pass; d=google.com; s=arc-20160816; b=xk7LHoYmPHXgzGpKV+QBCfmEWoeY6tqBXugyMCmq+ZFStvhMixHkVKBt6UL6UiN9Cz MMUw9NJCymoe6B8Of9eBEla+eHI+pHMps2sNH3FfvZHLJjsA8Al4tPR/V+HDKbXenEFd r+xXu3cvaUfLvYBLVE4JUQapp3hmRFB3yZisdFU6Bj2d6V7MdzP6odD+rJ4KrHq/dQTo MgzmS9T7SByG6YZ968831mikCRMF/TH4ZG7PvqGXVaEw1m1j2ngaLBssBC9VPFjF65wI oQFd5loWpqu0iFruCowI9d2aMVnREPFW43kb7zWyNZIKaVF8KcyXkBruRRGuZ6+AdiA/ AKTg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=ZJIcErO6gCpLr53xeyAlFo9Dh6PeuItoHvFCmPpopeE=; b=Ui22pv6FygHnad51SH2HezEdsY5UCbofa7a30V/Bgais+m5iC3AHWhnVmtIfNd0zVe aO3mPvQxND+i+1tDkROWwTqHVX8DOk82Gj1zH/xNiQmgXQtpyafsg14b3Gb8xcN6k+90 wTVGW2y+2hoxMreTQZrqoSPR17F/rnI87rgiQVB4PL8YfjO2qnFKEnmWMPgeUQleT4P+ p5c++TxfwILyPkUWSUXWrPMewmpzvzGMZ+QoAKMI3Ps1I4TC19U8vkkbYhLAWYtMnFfD hOBYF1+TeC4AD2iDiVa+DUYLuND1KoXeyprC5yK84FuHE6axdoBCWva666LuMRgbNu9v CZgw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=yq99iavo; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oeU9coHO; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ca21-20020aa7cd75000000b004aad91c6589si3770926edb.193.2023.04.02.22.54.44; Sun, 02 Apr 2023 22:55:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=yq99iavo; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=oeU9coHO; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231364AbjDCFYB (ORCPT + 99 others); Mon, 3 Apr 2023 01:24:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231530AbjDCFXf (ORCPT ); Mon, 3 Apr 2023 01:23:35 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43B3E19A9 for ; Sun, 2 Apr 2023 22:23:15 -0700 (PDT) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332MwUaj024750; Mon, 3 Apr 2023 05:22:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=ZJIcErO6gCpLr53xeyAlFo9Dh6PeuItoHvFCmPpopeE=; b=yq99iavokHx7dDj/ZMnuuCRoekfZAZl5KaNeqSlK04pxvsWqQEpkSwpc1RyYEgxtWazh Gsk6VAIjRbXsQsJvEZsBrs3oX3QqLbZ2SPxiisC8ksJVzMZVLdj58OiuSPV0AcyG/xJ8 rqB3jyVZg6gclK9PgSQCa9LOGwiC8Y8LUsYRcuzBZaKlpJKb1VOnJlK8b0hDRZLJaIGA OitEEg/3grDjfXy7YcCVOoIjOdJUjdgIm9O/Je7YwyqIi2UUoDjfmpxtgXPhcAIAGamf HbVqFcet6GodoGhyvmgvGTSjVmrrXUX6hKAdLlMKnafPhuwmDjRmCsZYNvtoHZNimaae 7w== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppc7tt446-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:56 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332iZTr038319; Mon, 3 Apr 2023 05:22:55 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2105.outbound.protection.outlook.com [104.47.55.105]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pptjptnpk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:55 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E3Fk8fHQFXbxUx4L75SPpKVNndtsTtMxpzFepb+u/SyfuZRHRuwZXA7348ZJc2ayXY4grV+WTTynN0Sj4XYwi3sUPK9ZD4pzVCX6s2T6AU7UdRJ5g70rX76EzB3deevgEXX1Y+dxPXHXv8mkF4KhzIC3cguMUrE2CeBI6WLIuOo7iBqWHTkVJxQaEGiayAkvL2z+/xhQnGcqzZf5fjUYl0m1a4CJQ2gvrZeZd1rdhSIZmqN/w0MUdZWAWFvXBKlQPXpE6PCkpt9mks9cgkcOoExZUpvmImin60rFapctfnfZ5hjZjoHggN7OYEv3JA7cEtPnZ8kBuzx+R8TtgfsAOQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZJIcErO6gCpLr53xeyAlFo9Dh6PeuItoHvFCmPpopeE=; b=GYFfn4lWX/o6HACFvYBHIa6oyUp7J85S9A3QB5ybIBWriZiTwRivris9+CAD5QpEVXkXs+nF2CUhCalO0RdtUC7kdsy2lOCUFVB0goFe2CLjSGeaX5uDHAhTmc/8DpXHx8JOols44WpXEbouSu/1dSQsfmuyITMRhEWmA/jJ62qF82c18YpjkYeC2zt8FzkyMJTXdr3zT2uuy3PbOpd+7nCmbemZm62CJ5SXvEbCCDWiGjo0ZwaK1YIDkG6iJnZ/dipOsm/obuQTIqDjdXZQO77/8TMUJlcfI/ZD3EllKQPvM8AzLulCrmD9g2Rfh4eAaqWEIHBYkr79Okk1e0tPbg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZJIcErO6gCpLr53xeyAlFo9Dh6PeuItoHvFCmPpopeE=; b=oeU9coHOBgncybcyMmLuFEfDVxbWPi66XZWaOd/VM2sTKXVYT0B4NEQWTwpLvhaMMzT8CXY/COf8sInHJ24ll0CH5yYH/COMqcLNBBWT4r+Zrbw397BI1VHK3r3qrS4dFF/BcM8IQKy3xB8MD75L0wCT0j9uM6zAVVTfH6hIcHI= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SA1PR10MB6365.namprd10.prod.outlook.com (2603:10b6:806:255::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:48 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:47 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 8/9] irqentry: define irqentry_exit_allow_resched() Date: Sun, 2 Apr 2023 22:22:32 -0700 Message-Id: <20230403052233.1880567-9-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR03CA0078.namprd03.prod.outlook.com (2603:10b6:303:b6::23) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SA1PR10MB6365:EE_ X-MS-Office365-Filtering-Correlation-Id: 1bae760d-8984-4fbf-13ef-08db34037643 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /JMEgRLdVtjfsAbqIduYdVtaKETVJDnsWFbwxXSnLwTHw4Yb9b8Y5huwV6UPRr8sSAwGlLLN2IdBDFqxJ7tZkqTna5OlDEkJHe0vl7anwNP2EV1gxtPGt9ywVb30esqGvOAt/Nj/AMpIWY2FTtrRUhyFTBpaAIXqtbRnH+ErC0dajgKPDAK6ED6J8LSWdAwEaYOiSNpj+FFxVNo2XMuxfBEIO7OLFhJAmXxaBCUc5THRjQHI6BNRKHRnsE3PuXLk8ZO1+0Jr/HMzPhO9FcRyn5fnDKJ/Q6o1WDFVBCusNGfY8EmY/gI27codm129pJV0jzx88rSgCCgQhe99CwI13yW3Nvi3LZOB58PIhXHYL+9NMKoTCa+IX6ShBMc12pPjI12xflIMSvl93fA9+N+sGyAe2Srkh0alfuCH3clTBtqJkQ6kvnQkcFmMYEZpaS7D6f0NQjkzUIaBCuFVOnAjm18Bb5HjKDPdm2KU44XHrchWlIRWK63hQW0b6XexPUlUYo1d7mbx/ii4AsWfoDR4eGrzg0h29rarn3Wf918MLz4uETNmLgas8eVuMxmOG6VI X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(346002)(136003)(396003)(376002)(39860400002)(451199021)(86362001)(36756003)(103116003)(2906002)(2616005)(83380400001)(186003)(6512007)(6506007)(1076003)(26005)(6486002)(107886003)(6666004)(4326008)(8676002)(66946007)(66556008)(66476007)(41300700001)(5660300002)(7416002)(38100700002)(316002)(478600001)(8936002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Aq/HuEd+N7dXNtUvPz7JE4EYv/PgBz8EgC+c9JG9OA57ctbp12SGWCp9Lu2dXGfhRPb0G57ZWij0+Qrb1dIoWUT4/q7MuD8bsbPCzsBQhiJhrTaF79fvvFT5zeBUgTvKDvCYQuxMyx4l7yw2jxullhaHVYJqhV8xZs8GgVjXUxNU+ItWlJXf8eHW7yA9tT3T+8wVqvLNcdFtjcTYbmnmfI3tXo+K/0UqKIM+wA2ZtlDNwqirpbB/yR/RDlSGg367GenXLCo7rD5Vr4+G5NLz3iDMitJYMGndeuG+af8XzMViIC2mC8pEFZ0ojColm2FxHG4gR2lrgdl0n6o5XckhLQ2SEvHhmqgCWbOeb7bLmoR4XOASez4Lp66G9OKuHu60kbEUm2bazWHXaIPuZ5RIoKHD8TFNc2LTkPvXi3fnWIXQ8Nu+D+P8FsVPzJwedOVi/kHsJHuRQvD6es7+jq/4dBcPewO01PJ8RIHD3czOP4D42UHDIvkM6oSj6bGKcYwdo8BdP+7qHM0V1iZLNUilYdGn4vHitf0szzK07yUojt0UifoAd4yFKUwUEU/dm9jyr0oWH09a4M6LPeQL6cEoOtO5JU7V7egCWX243h2AT8qJWnkSlqYR2RMs5b5bx7nDiJYzpmW4kSMKKSBIC3iSUhJKRJGw3yCeJtHidZHBT8Rq5KTH/5N3dfH98R3EW94tgpVdiPMnrYqABR3zkDUZSVkt8sbUwuCqe6e5TIOgDSXO/I/J7epPO4HAP1xOhKRbIpfGjSNvEgL8uTO4L4Eqt4beHw2LkdpunSQc2F2sLuffIgDD3T7WF3OHqjudSB50oFW/HDzZhbpR33lOOq4BAxAgpZTFSMhBHIRLR2H6FecQhbCFctFfvb6nWvB17O+oy5MtPTJ7W5zffMkZ2LTS6R9p+E+CLkF3t/xtUl4KtYrqNaW0QxGwicZH0yvIJWpy2PBWpCdomOzqns8qd1De975GM8TFsb4u1AktJNOQnLoyKKmHZ/npviHAuPVoOxxZcIDR7vTa9nU+LjbaAXiebwZbVYnc1tzg8OqGjptmmngOcHK0pVU3e8uatkEuBejWMc+FvCDARdqyHEuCKTfcGGrZoJxjE0MG6V3RIY7139eOX67xXru145aHIvOwk1P5FD4iYjbjPuKwaPljGnrwChNO0fUL8sA+gvDkHZSyrvNO2Y4tDiKwWny9wpOsoSFhlogZi0yn0XjiZfsrANt6CLrq6sAX18V7dRY+C21tQU75fPNsLMokIgbhRCickICel1TKsxVdEdx3WjrUk1CPPnR/CgtSHUn6BQiWZIzUIRpVAzbcbw0/fD4oJQN73iwCkDCsFYm1sPkjrInKP+C/WjI4jq7ZVO4QZm79lWvgDeaMXRd43Olz6gNkGi4mOFbNFQZpKrIeY5DYLLWtWyCIJYY2tHvOJcUttgg4hCByoMKKD7CSxwQ5Xent+9EwJdobL6L/iU9QI1HXkjXnv0okKV22DLe1hrt+p8QlXc6ziSO0LQutYGsUXImbwkEpVDO22cQl9aDhj+b6AtUBh6coJXFBZUV0Ub20Hbuy4qhmkneP0sVy5/0qra4pfKwEkgOL X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: Cbnia/eFO/wgtFWUA7cTl+DzNH4JwlXTsRuR2KXvyvnnt0dQE++1XfH6ZEV/uN8kNb7CLsKFmptrmLcG8pTYgvQFnmfJdrzarrtCvBVymrrLbHMVXyoN2Y0KAXpd6+SCJubhawJcqg9jyI8z2WGrazBLD2D0jahyIwF2xVrsa5N+ZgG5XjNTdHPY5wpUmjvR3my4u/UgK3nSp/k13wIhvpTPSxWbkUqSouirxIahMY1/sYrvcWL9BmGXkrrlTkuelwr5hXiKGB4rcF16LY8ZRxkvaWZJSstQK9fRb2e79HlkBZ5O0MycnqA9bMt6tGzpcQra0lV5kpKav4/ypKgDLmK1ieQ4+pKEhIPfMSzbL1OXbQtoAYjJgAW97bDcbhCho0U4nclIjN9uwbo/Cju3LpFm5zZxDDsVWhIqwn2Ur4NIF/qk3uiQ1JYfU449EZkzUPQKdfPX3Q/L69NQ8jZ8pxxsDpheEUsdzlARblva8p/rSXYr9e2oSwwZFwwR3HrmecsOJTDwzSMhMbPHRQsPwX92Tbeq25wKh4v/SXxs44SowvUHRDLTBDVV6lTJs6dV5fkWcZf6WEPKnL1P/JTc6/62ArsIR3ypQXrvQZMy1hMPS04o6zHqgRNlZ1Nwb0ZHTA0h+sUEAVh2BeEiUS9NZUx0hdwtjMEYl501cL6NafwLZTivngAMBtQvIfm8hBx2HdjbrXLhxMHYYRjmfjswWhg9IgSdnVLbM8YBH+P242AwbOQdOOoAtaud/iI9wnqkfJkIuOEfVivmqVnMKz1bRPwS6SldOTkY0+lnzLOWPLpssQtOEZzUDPIMb5wkBiH+weR8b1w+Cr5eRElKpCK5CYIC4xzMxvs2s8bfezvsxQd2MACCa37ciHqOdIdJ4vyxTDWJ4zy5toJsKc6wi1i8mR96DKWX8UUdWOD+ZWUvKE4ieR0QPq1mym6fQhGHwIaZd05KGAS6DSd8GgtyruE7HLufinVjrDFyD9qNnJiPnqfX4fnOyPX5hzy1Z03sh7VJnJ8SSVxwfDHggkuoBmM+iOrTMTqk2Fx0ro17oxi5MyU7ZQrK+TYjvSqOASCL/eeW67R8w+CMKgot1hGzZaqr0onuvxOAYeBJEOXUj7umwLTts6xWWJWjdBFmT+q+DiCyW04ejvSJrjzOQmEl4pPRcy55KBS5LhT0rmQgmDicDdg= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1bae760d-8984-4fbf-13ef-08db34037643 X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:47.8596 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tDyBYPmBhG9E/ZhXTWDy8nevhTrDYQ2xwFEfFIflBN0TDmISNcR80y883HbWBEqB1/K9lCEqqji9S+qnPrpEqO6lNqvE6ujFguDmRECddm4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6365 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=879 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-ORIG-GUID: f-a-V6JSgdGnspaNV5qnhzyl8asrVf1g X-Proofpoint-GUID: f-a-V6JSgdGnspaNV5qnhzyl8asrVf1g X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762133340774572507?= X-GMAIL-MSGID: =?utf-8?q?1762133340774572507?= Allow threads marked TIF_ALLOW_RESCHED to be rescheduled in irqexit. This is only necessary under !preempt_model_preemptible() for which we reuse the same logic as irqentry_exit_code_resched(). Signed-off-by: Ankur Arora --- kernel/entry/common.c | 8 ++++++++ kernel/sched/core.c | 36 +++++++++++++++++++++--------------- 2 files changed, 29 insertions(+), 15 deletions(-) diff --git a/kernel/entry/common.c b/kernel/entry/common.c index be61332c66b5..f1005595ebe7 100644 --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -390,6 +390,9 @@ void raw_irqentry_exit_cond_resched(void) preempt_schedule_irq(); } } + +void irqentry_exit_allow_resched(void) __alias(raw_irqentry_exit_cond_resched); + #ifdef CONFIG_PREEMPT_DYNAMIC #if defined(CONFIG_HAVE_PREEMPT_DYNAMIC_CALL) DEFINE_STATIC_CALL(irqentry_exit_cond_resched, raw_irqentry_exit_cond_resched); @@ -431,6 +434,11 @@ noinstr void irqentry_exit(struct pt_regs *regs, irqentry_state_t state) instrumentation_begin(); if (IS_ENABLED(CONFIG_PREEMPTION)) irqentry_exit_cond_resched(); + /* + * We care about this clause only in the dynamic !preemptible case. + */ + if (unlikely(!preempt_model_preemptible() && resched_allowed())) + irqentry_exit_allow_resched(); /* Covers both tracing and lockdep */ trace_hardirqs_on(); diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 0d18c3969f90..11845a91b691 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -6500,6 +6500,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) * - explicit schedule() call * - return from syscall or exception to user-space * - return from interrupt-handler to user-space + * - return from interrupt-handler with the task having set + * TIF_RESCHED_ALLOW * * WARNING: must be called with preemption disabled! */ @@ -8597,28 +8599,32 @@ EXPORT_SYMBOL(__cond_resched_rwlock_write); * SC:preempt_schedule * SC:preempt_schedule_notrace * SC:irqentry_exit_cond_resched + * SC:irqentry_exit_allow_resched * * * NONE: - * cond_resched <- __cond_resched - * might_resched <- RET0 - * preempt_schedule <- NOP - * preempt_schedule_notrace <- NOP - * irqentry_exit_cond_resched <- NOP + * cond_resched <- __cond_resched + * might_resched <- RET0 + * preempt_schedule <- NOP + * preempt_schedule_notrace <- NOP + * irqentry_exit_cond_resched <- NOP + * irqentry_exit_allow_resched <- irqentry_exit_allow_resched * * VOLUNTARY: - * cond_resched <- __cond_resched - * might_resched <- __cond_resched - * preempt_schedule <- NOP - * preempt_schedule_notrace <- NOP - * irqentry_exit_cond_resched <- NOP + * cond_resched <- __cond_resched + * might_resched <- __cond_resched + * preempt_schedule <- NOP + * preempt_schedule_notrace <- NOP + * irqentry_exit_cond_resched <- NOP + * irqentry_exit_allow_resched <- irqentry_exit_allow_resched * * FULL: - * cond_resched <- RET0 - * might_resched <- RET0 - * preempt_schedule <- preempt_schedule - * preempt_schedule_notrace <- preempt_schedule_notrace - * irqentry_exit_cond_resched <- irqentry_exit_cond_resched + * cond_resched <- RET0 + * might_resched <- RET0 + * preempt_schedule <- preempt_schedule + * preempt_schedule_notrace <- preempt_schedule_notrace + * irqentry_exit_cond_resched <- irqentry_exit_cond_resched + * irqentry_exit_allow_resched <- NOP */ enum { From patchwork Mon Apr 3 05:22:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ankur Arora X-Patchwork-Id: 78360 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2083082vqo; Sun, 2 Apr 2023 22:51:46 -0700 (PDT) X-Google-Smtp-Source: AKy350YqmNus1h7A3d/g244XQJnNeQIO86qaWxyf7rVLwMUtfpvxLlFeBApFNPf1fPw9WhdXJX2b X-Received: by 2002:a17:906:1685:b0:930:660d:8f92 with SMTP id s5-20020a170906168500b00930660d8f92mr35526677ejd.52.1680501105932; Sun, 02 Apr 2023 22:51:45 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1680501105; cv=pass; d=google.com; s=arc-20160816; b=ZOnoGIOwzFZMh0L/5PdZp42iXCwORn4gDczirzQbGfGuIeF/amUQEwQ8qnfNJcWv7e 5XPS0jQPPx8bR+kFfWjNFakQflm0uJGATnw0BTCd9NoQudN3pUOMJh/4ZKH8halZV8cj fts3fhy8154pzCiZe2dcO1BMkLaDOg9u9SwZYgNeUBZXsQMGFskoSJ7DV/J7Pz2KhNwi xPQZtABHMYtkf5CnyYlrFgs695d/SN166JhKHII8khjFEuOTbUI4BMryuHXadW5vUz+6 RarnNlM+gqZyCVl47py1JIubLuzz0NFCfl0R7ZCYxLdijVi3e2XAX43rMP6qSn6x4t2u dYrQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:dkim-signature; bh=4gImNjdBdcopMpLrPUI6c2EyH13eycBU+Tw9ntgOP6Q=; b=DVPPG5hKYIRxia7JoFFwyXGXedc8SPrw5Rmt+yMrqHZLIBjei0TRAtFkrn8Wq3lyyx QeKEzDHs4rbNYHAOLRpS/gL1ag8aTGdokcpwlszYwJKscNN53PsZ0kcPNHbycYd8IhN9 Dx1+AgGohMc0bs2vXXdxx/UQ5ZwGyJrdn123W8iivvPvTaEEsJsPeLCHCoEiN4WvKUxn jrLCdSY0G9juVzx4hS/VDPaOgx9IhjOLC6Mz5HoJEnAhfGGjOEnNVjBCy73mORmBE9z9 javh5gbaHGXtwBbp9t7t5rniJwIRCvRI1+W4k5x79pfKoMS4A2pChFrIiun5RWKGva/n LtSA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=VAXqkuIL; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=hrxFOJGx; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bx14-20020a170906a1ce00b0093defbd6293si6263722ejb.1050.2023.04.02.22.51.22; Sun, 02 Apr 2023 22:51:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@oracle.com header.s=corp-2022-7-12 header.b=VAXqkuIL; dkim=pass header.i=@oracle.onmicrosoft.com header.s=selector2-oracle-onmicrosoft-com header.b=hrxFOJGx; arc=pass (i=1 spf=pass spfdomain=oracle.com dkim=pass dkdomain=oracle.com dmarc=pass fromdomain=oracle.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=oracle.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231593AbjDCFYE (ORCPT + 99 others); Mon, 3 Apr 2023 01:24:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231534AbjDCFXg (ORCPT ); Mon, 3 Apr 2023 01:23:36 -0400 Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FF7B5FD2 for ; Sun, 2 Apr 2023 22:23:15 -0700 (PDT) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 332ML952010090; Mon, 3 Apr 2023 05:22:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2022-7-12; bh=4gImNjdBdcopMpLrPUI6c2EyH13eycBU+Tw9ntgOP6Q=; b=VAXqkuILTv1fDmRnW2ObaZ5Y3mpmAS5G9sXXZ18ssmMK2aENP3rweY+kmyUosGNQ91ki 9fXkBRil/1wOZlhtaU+QMCN2UYI1MgdRzJUh14rfk5kpsSE3o8ZQC/r5LW/DbF5dTqp3 nvQBeK2nINQTlk2bji6xt8VMjh6ToTPKO8lo7F5nBYswLftjl/wWPCqoJZhf4DA3xlMb Ls4UMQsaP11TpQwNUwnK9h2pPcYfvN8MkKF+pWZBd323QnZ2R3F8VsbCGpKeNsRgn/cP kSniV524vjnqeP+QpNA5PN0BivXSAqtrW7Zh+0vfk0oVeu7gfq6N2Mk/AdSb9Oj82LB1 ug== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3ppcnct4sf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:57 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3332iZTs038319; Mon, 3 Apr 2023 05:22:56 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2105.outbound.protection.outlook.com [104.47.55.105]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3pptjptnpk-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 03 Apr 2023 05:22:56 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=n0ihLWkM/rbL+bJdATRKrvBavYnXp/Zp2f062BDmqwkm9RJckaDHIllhye01eOecD3q6fhbh2R6D96F5miAW0g59PxH2xog9kboGfq72EZm7zqynDB1SeoIeWESZ5QqFgvXn55vXSxXWvR8cQqWuGhFiGL2mtkyldqBuC5udI1X3UhT3CrUpnZ7OISs3bYNRHOwipnicgdAEsEakEEA2GZqYhNqJT6s4GQ0dmslb5ZfvitXlkPnoT4lsiFgs8kNPeAFgkGvQDcUBNkQ8W/TmXBPY9DmK4ZVbXYtXlHy7UM51xPpK/Sobsa9ILXBO0FDMV1ryteKaTvyHQQwRPAsXMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4gImNjdBdcopMpLrPUI6c2EyH13eycBU+Tw9ntgOP6Q=; b=lv91NjVV6ySkt8VungP3ENIpnfjso/xqzrEihbXLS3HQFet6kHAB34nZ+kG5h9YwT371pN/wInALslUkqbhc3VbvtUn21wZP0ABP14lki5Kh54Ig5kAkmLdkGY3sHd9gHNzcra1QUbEAvK+ugrLhim3J5JBYHPqac9+6e6i63lSvuIh649pFOF1/+QGFtf1Vxflwx0J5H/xSudzhso9QSiY4qSnBg6le3ClLoWlHBUdtW1fhwkQ0sO+jlKrp2w0OT2O0xfd94yG+wVjlGv4JNSDKy/DIU0hGntHtmCcqCDIx6FYzG2VldaZI0+K1YZbc+YC4Y987qRAgz2sfYRuO5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4gImNjdBdcopMpLrPUI6c2EyH13eycBU+Tw9ntgOP6Q=; b=hrxFOJGxUlMf3BtPCEx23f6FMeCKATpfuY9g9kFZjJ2AC7RwqC3kWnXghkSFwEme0Xfu2cYNr6aHsf3KBJHATzZsxw4BLqUbK7gwiukjVUwX9kwczch5q9mrja97kQaMFip1GOyqAbYb16i0k6T6d2psLJx7ZshYY3RuYxMyoOY= Received: from CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) by SA1PR10MB6365.namprd10.prod.outlook.com (2603:10b6:806:255::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6254.33; Mon, 3 Apr 2023 05:22:50 +0000 Received: from CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6]) by CO6PR10MB5409.namprd10.prod.outlook.com ([fe80::986d:1a90:f60a:53b6%8]) with mapi id 15.20.6254.029; Mon, 3 Apr 2023 05:22:50 +0000 From: Ankur Arora To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, x86@kernel.org Cc: torvalds@linux-foundation.org, akpm@linux-foundation.org, luto@kernel.org, bp@alien8.de, dave.hansen@linux.intel.com, hpa@zytor.com, mingo@redhat.com, juri.lelli@redhat.com, willy@infradead.org, mgorman@suse.de, peterz@infradead.org, rostedt@goodmis.org, tglx@linutronix.de, vincent.guittot@linaro.org, jon.grimm@amd.com, bharata@amd.com, boris.ostrovsky@oracle.com, konrad.wilk@oracle.com, ankur.a.arora@oracle.com Subject: [PATCH 9/9] x86/clear_huge_page: make clear_contig_region() preemptible Date: Sun, 2 Apr 2023 22:22:33 -0700 Message-Id: <20230403052233.1880567-10-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20230403052233.1880567-1-ankur.a.arora@oracle.com> References: <20230403052233.1880567-1-ankur.a.arora@oracle.com> X-ClientProxiedBy: MW4PR03CA0003.namprd03.prod.outlook.com (2603:10b6:303:8f::8) To CO6PR10MB5409.namprd10.prod.outlook.com (2603:10b6:5:357::14) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CO6PR10MB5409:EE_|SA1PR10MB6365:EE_ X-MS-Office365-Filtering-Correlation-Id: e955a7c0-ed1c-47a0-8465-08db340377bb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EbL+kGjbKMFebho9UeIYDAfwzB792wIeFWpV1vh9sGs83OIYJEydEXYIHVSMhBvRHES+eVIMNRRr79X+Vu9k8NJR0nTHtqmipzbeIt/N45SxqXEE1HkqOpsw35vLkg0hhw4O11CfW6Tjm1Ydw0iNXnYAc9VaVf2vx31yaRT3MtC3LY+qkreaWxsx6ArAKfEdpYrr5JZ2yEYvL5X/7o7u7fAh0E307GMLx5c/ZRqoeAJxdaDo/6vIJYLIaFrciCZq3+24PUK85Ulc3aEp2TVDoaw1q5isZ1T5Ytm21/id2qmRrT0wVgQFgQB0mK4yaMtQUdiBRVyTIdujMCX4S0M2Fry1pLdqhHLMbbr7sRB9lMtrmfTdEnQbnWRXLQuKvJgoFH8i0BzHwW01jgtEW4jXvUh5PQTSAyP6JXXIaXCcSUu3UmB7yiOzj9CCqY2gCqpfQTmDM2VME3yA1oDPVWfEQWGsCCIc5MzSRdzQiK1sqASy817NjtfCyQ2s+Wt1V1vtPzH7qYYJIMTo0p58LbCJI4mCdnCVnEuhJaVOqWD3zjz2ObXyXDEGwnTeU0C7l6hFRXZiCMCHkicNe/I/DAvZp1hR7NG5/8NW8iL7xvfIW10= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CO6PR10MB5409.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(366004)(346002)(136003)(396003)(376002)(39860400002)(451199021)(86362001)(36756003)(103116003)(2906002)(2616005)(83380400001)(186003)(6512007)(6506007)(1076003)(26005)(6486002)(107886003)(6666004)(4326008)(8676002)(66946007)(66556008)(66476007)(41300700001)(5660300002)(7416002)(38100700002)(316002)(478600001)(8936002)(14583001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 7VYrq0RaXD3bgzxE/BXFfDzqbIwOC0DYaCtJqLH5lNpBA63lLnaZWuLJ3tPilkdsy+ZJFMRFm14jYuksDPqGV3NhE+nFW31Gx0vaEQLgYG5DtB1/4/LWNZFKZGBitzCcqxJY57b/BuuxWh4BCdhvg+TjWkuLbzXdsBSWEVOqlYMYmHTLUwXSAPMPzDA/+pKObtOnrr6tHfd4lOc3IdhkyvzaW4lWQD4t/cffS/Ml1DwEx4UrywlbU1Hj/kHXjuyT8VqiKXq0ai9DOg/0HYrndsVHXos9ka/sueJbJl6yD/YRpPi6+cgLSv1v67QlGzejM5H6e0HXF83YfzsXJ96/y37iRufWvXtmOAmanWK2iPi394NaCS3+viVN2qgCB1VFj8qX1hXLn9GWU+KcmiDpo0x/kWRg/SKal4I3Z1vl3qSBuHi1yTPjq8RQzYQfDSwVPGAk7ZqhbIG8BZzoiKsNLiAEgN2vsScLdmHgOiYhN2csxq3lIUNqtes7FGzCU7SkCtYlaK/ua2uzW7M+B1nwETsBpBB2/iont+xIF0Bwbt0ze9z6YaCAjZiIsnDXlK3hZxzBYWZF03ihM20VNuhStgF0JpzrWWKcyiR1JheGP5boZkKHOQufFHpTkjpHWmXF63eXYLTtAzXw1zdOZKpjDGG2huHj9+xoUaM7IIbryj2eRy36xVwSOtpyLorUAztf8Apf6RVje4ZqegPnqIop15cCy5soXu3hfWIQwzulUZCd+WMCQbyfLzHkIJ3FReZTaJablAyOkhqT+26m4mahzcyLVz0U/Vc92pDcE11fGqxQTiVb8ThmjW33PvV/cT6wN+aTCPVYMNzyBjkTXlGpnqsX88hI2FTB53OX1ov08ltA+WmN2SmF6Xz3DUbDRxBkyhHo/GMufWTcWlPjKMiPap1V8VPG95D8w/WyuDEp4BXi5QgFgzd9jVjP0+2no3bSIIXyzkmV3yRpZitM8bH0hmZVJdg5UhdZJBCmD+4poXmAvgeTTmfM0xdlLXqhtbcVARD/MDq7/yhZ9vfbioALR7EjaILBzKIfbrgESsznglPrE5FOo624UjsAYCSeBvyE4Oc42zVx4XZ9SQKAOGQ1BfKGEqN915UW0NpKeSAWqD4CAqif0sv/EEHkxFa71mfJ1cyIplzf5HHgFKFAPrU/+hewPly9TKqOjQVdi+aH0hj8mW9KP/Yc1czmdf4XlgZGp0XvDmfEa4h/SW6QlXs74DUtt1XSa3rSXbGpkUjQcl5yp+q0GGDnI7bbrcVWvnpPcO/KHwBfrQLfxcaVMDTZtvW3EDxyYqlRG3knO5MHmgN37sUf2t6aaEIiYjle6HGXVS35McEoImJlH/k0Vd3cJZWJUUgnb1PIbteM0l3KFT8GAf2vbZBv8jtN/mueAIluhCi8xNFqirWbvdJc5W0oMJuB1XeHB9LxwU3VfZk6Zy5lj6Cm1dL2aMww+S8vQS7aBj1ylgqT02YOlm+G1ovPao3R2e+hBgmqyseJ9pXjE3yqu9govtzFf3ANq6lsGcvx5Rt/cc0rRSGNAYPlyPN79zPXp/56H8nI5DmgYmnRHjQMofC2oVFr6/CwF2ekJkW1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-ExternalHop-MessageData-0: zr7CdQo1TrKnvgmfFDz3j1d2Anwbz6bLRpTr+VoRl9+nwyQaWpYiomWGBkNGmt0NUftprS8cjk3KoubFfszkqIss2U3xLl9O6CpJQs0ZdlGA9yX9VHUFIgD91tdvLaJ+LPHJylUYK0YrB3ii/R+LmJ8XN/KssKle/J8x7VYaAvnE/nAWfVk+aP7sLnt6DqQxvqA/ldn26Rsm11Xw8JOxrBpnO4hDetLXuztPR89cD2IttP5GzoZaHkB6H8qQwrS5BNuLkY0543Qew2FB3sYTKsLFt6D4b69uQIDKRaE/VoTA53ToVyf3cJjUDXeScPzVYnayT4IdHbrETHe72U3No/NUP4lSfr0GMjKiRCIh2Q6tX1c78RLaeuwIG3tuGToMY22D1tZvgxlae5U/D22tvdBbB5pCj0zBEVSO25PVshq+dYHgJD9dJwtVKc3cZGeKjbbh23Doj0wFK4PPJTX2InbSVk/s6TNgqhVauNFnTIovP40U+Emme9r43WH6gc07gcyzOsx7STmKWRZdcZQ5TRJdO7Yr2tH3iE2BEJXvT67wHGPoadidRC7Dl2zqplJa7cAu5KkVotwPFUPxNsy5EFeXET9/2BUGbzwTASTVVtqwruC4k7ObwzevzvVNm6Kv6+QeSjJvih1Di8YILSSF8G0HuNGLwwWfSlzKip+PONfJp3oAMYzAITu/QxdpsbbzeAAt+4BdxoqGs02NBfAue1MyxOheYQbZlfgaS6N6MG2006KlSFOgu+oAK5/wzsfOXxmzhmp3mSXm9SIgqHgQ8oRs8biJ3hjoOemDPfOQqFsMqEcfv6uJP+e3Z075BjKjyCvXCnmsrPcwe5O4Wa/lQ4lNXDnrz/R35bFHcjkXtl3nQiYI8B3AZ64+aJ/bXhcC2ECr6RwEMb4TwLSCL9OJJ90CxxTYukwfcX4Poc6ome553Z6UX/SpohcMJ/sR7t7D5D3Qj6QlqWFVVG61hYcu/Kcanazh+NBaxSTchoElgq1ntzk5HecvAJ9Nzr3rglZA/3QaD9CD3iOLsmMOqmXkyMdqYIhXyd9YYGo1aLYS7hsoMIXo6DwFL8Et/B7wM2Gdukn2RONgDs8vvxKEUSvw8K5gmy6ZrqqqeQm68rykVCvGS5cvD6HkrHxk7d5DlhnLaV0L3FaPhzXcaJDKZDBeFyKye0K7ju3oztBdh7LZ9uI= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e955a7c0-ed1c-47a0-8465-08db340377bb X-MS-Exchange-CrossTenant-AuthSource: CO6PR10MB5409.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Apr 2023 05:22:50.3049 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: pToPetuEUS3W64nx5gmBwlvkqB4VZHXRQ5V0Ff95DELUVJ0dIXKXQ2JbahRL8UPalC6BtBeRJbS50q2c4MrN1LLDBTrGYYyWX/4rkAerLAk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA1PR10MB6365 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.942,Hydra:6.0.573,FMLib:17.11.170.22 definitions=2023-04-03_02,2023-03-31_01,2023-02-09_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 mlxscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2303200000 definitions=main-2304030039 X-Proofpoint-GUID: JqiodZDSs7DsfTtPyIchBXU6qHB8E9Ma X-Proofpoint-ORIG-GUID: JqiodZDSs7DsfTtPyIchBXU6qHB8E9Ma X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762133127430514585?= X-GMAIL-MSGID: =?utf-8?q?1762133127430514585?= clear_contig_region() can be used to clear up to a huge-page (2MB/1GB) chunk Allow preemption in the irqentry_exit path to make sure we don't hold on to the CPU for an arbitrarily long period. Performance: vm-scalability/case-anon-w-seq-hugetlb mmaps an anonymous hugetlb-2mb region, and then writes sequentially to the region, demand faulting pages on the way. This test, with a CONFIG_VOLUNTARY config shows the effects of this change: stime drops (~18% on Icelakex, ~5% on Milan), while the utime goes up (~15% on Icelakex, ~13% on Milan.) *Icelakex* mm/clear_huge_page x86/clear_huge_page change (mem=4GB/task, tasks=128) stime 293.02 +- .49% 239.39 +- .83% -18.30% utime 440.11 +- .28% 508.74 +- .60% +15.59% wall-clock 5.96 +- .33% 6.27 +-2.23% + 5.20% *Milan* mm/clear_huge_page x86/clear_huge_page change (mem=1GB/task, tasks=512) stime 490.95 +- 3.55% 466.90 +- 4.79% - 4.89% utime 276.43 +- 2.85% 311.97 +- 5.15% +12.85% wall-clock 3.74 +- 6.41% 3.58 +- 7.82% - 4.27% The drop in stime is due to REP; STOS being more efficient for bigger extents. The increase in utime is due to cache effects of that change: mm/clear_huge_page() clears page-at-a-time, while narrowing towards the faulting page; while x86/clear_huge_page only optimizes for cache locality in the local neighbourhood of the faulting address. This effect on utime is visible via the increased L1-dcache-load-misses and LLC-load* and an increased backend boundedness for perf user-stat --all-user on Icelakex. The effect is slight but given the heavy cache pressure generated by the test, shows up in the drop in user IPC: - 9,455,243,414,829 instructions # 2.75 insn per cycle ( +- 14.14% ) (46.17%) - 2,367,920,864,112 L1-dcache-loads # 1.054 G/sec ( +- 14.14% ) (69.24%) - 42,075,182,813 L1-dcache-load-misses # 2.96% of all L1-dcache accesses ( +- 14.14% ) (69.24%) - 20,365,688 LLC-loads # 9.064 K/sec ( +- 13.98% ) (69.24%) - 890,382 LLC-load-misses # 7.18% of all LL-cache accesses ( +- 14.91% ) (69.24%) + 9,467,796,660,698 instructions # 2.37 insn per cycle ( +- 14.14% ) (46.16%) + 2,369,973,307,561 L1-dcache-loads # 1.027 G/sec ( +- 14.14% ) (69.24%) + 42,155,621,201 L1-dcache-load-misses # 2.96% of all L1-dcache accesses ( +- 14.14% ) (69.24%) + 22,116,300 LLC-loads # 9.588 K/sec ( +- 14.20% ) (69.24%) + 1,355,607 LLC-load-misses # 10.29% of all LL-cache accesses ( +- 15.49% ) (69.25%) Given the fact that the stime improves for all loads using this path, while the utime drop is load dependent add this change. Signed-off-by: Ankur Arora --- arch/x86/mm/hugetlbpage.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/arch/x86/mm/hugetlbpage.c b/arch/x86/mm/hugetlbpage.c index 4294b77c4f18..c8564b0552e5 100644 --- a/arch/x86/mm/hugetlbpage.c +++ b/arch/x86/mm/hugetlbpage.c @@ -158,7 +158,17 @@ hugetlb_get_unmapped_area(struct file *file, unsigned long addr, static void clear_contig_region(struct page *page, unsigned long vaddr, unsigned int npages) { + might_sleep(); + + /* + * We might be clearing a large region. + * Allow rescheduling. + */ + allow_resched(); clear_user_pages(page_address(page), vaddr, page, npages); + disallow_resched(); + + cond_resched(); } void clear_huge_page(struct page *page,