From patchwork Thu Nov 16 01:29:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 165644 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b909:0:b0:403:3b70:6f57 with SMTP id t9csp2924261vqg; Wed, 15 Nov 2023 17:29:53 -0800 (PST) X-Google-Smtp-Source: AGHT+IH0TlHhZVj1f7GFC+yt6C3k5F95BVFglmYrjSqvIKmjsPYeIh0Vo+41hQFUFS8wnDpjUo9n X-Received: by 2002:a05:6a20:734f:b0:187:958b:993b with SMTP id v15-20020a056a20734f00b00187958b993bmr379517pzc.10.1700098193677; Wed, 15 Nov 2023 17:29:53 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700098193; cv=none; d=google.com; s=arc-20160816; b=brH/83R3xJbYXiKu2jT4aD1u+qRPgAbFCW+KKANn2qsaXvRlT3BqJonTeopeJG994s eRl+CeqwxLYNTdj8nO3rAxdPUvMSXxD+2XuDsB2ZIk/y5ER0EGijG67e5xPKogVt6rGm f3ZBoQKk1WXO7ZgXuPXqIzu0/BU6J1RTFExnu49E8chM54b4Sur7+Lw9DZgxOBNKJUZQ aZ3z1zOK7arNM/Vp3PEv7K0srq9tthnPfcyrXpH2Yr/IxIJSno3zvV0D4rLjl4PCjM6z pXxrZYiu/7UGZxeut5B3a6R3x6bEhrtBo3dqXL3lRAFhf2fSpui92WAqFWfJAgdAoMAW Kpyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; fh=o+FJD7UTEFgAOwGBZppViFyhRpBzPGRfoO/6xzRNnMs=; b=IiCm2i9edsf76efr7lp3soa5HvgGInD5wyzcOhZVgC5ceNrQcavxJKy2KSFi24PX70 sMgfW5R/CUYwuGPKTupPgtZmkXMD3EP9rXGXOD5HFxnhgQKTHuRIzuOEXXx47iO5So4r FinI6Gde4+lv1QPWzKox8atk3VRpb+mTbAE2dRwCeC9mITqgVkGuvTmhVgYiaZwuxk8s dULf58VlNIvGKRR/yd6u8WjhrBfKJsocciKqV8NNnPPEJPt3RL5uXCKk91LAOLd5MFid KJmYQSUV2ZIubKvGAMWIlmfxJWhS1eBKfeU78MIoAnL4mDsuHfb+CuYtIFapR9Bg9hPL qGfA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MqOjT7Fc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from snail.vger.email (snail.vger.email. [2620:137:e000::3:7]) by mx.google.com with ESMTPS id e2-20020a656882000000b005be09a118d3si11394189pgt.143.2023.11.15.17.29.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) client-ip=2620:137:e000::3:7; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MqOjT7Fc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:7 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 67E8A81121EB; Wed, 15 Nov 2023 17:29:52 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344514AbjKPB3p (ORCPT + 28 others); Wed, 15 Nov 2023 20:29:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50658 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1344487AbjKPB3f (ORCPT ); Wed, 15 Nov 2023 20:29:35 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A70101BF for ; Wed, 15 Nov 2023 17:29:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1700098164; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; b=MqOjT7Fci5yrpWa33B2q5gKSg1vCmiVYiKxFz5mY56b/A3vh18DQRUnsJ3ZOA6Ppv34ngf SdtNNqi1wVt7JMBgHWV2fY/TX7MTBfYJAWMdeuS65/M/BkEoLjNB8lWBQpo907QZKdwokz RK8KXwvkkeq/f4Vggm/dtwXytIv5S9k= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-175-WEUvw7azPfWEV7Xuhjq34Q-1; Wed, 15 Nov 2023 20:29:23 -0500 X-MC-Unique: WEUvw7azPfWEV7Xuhjq34Q-1 Received: by mail-qt1-f199.google.com with SMTP id d75a77b69052e-420c7034fddso915891cf.0 for ; Wed, 15 Nov 2023 17:29:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700098163; x=1700702963; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/m4Nmkq0cr3qM8eU/2c8RM1XZmbD+yMVfcEDigP2nYM=; b=DV0nlCQpnC2ux22Ay1mhabMQ5MnA5ZlP6AMDXiqLB9O4LG3tVmpAWcV7CU+v+kp4WS JXOCNUIBolIEOhtPTuFUsXgDo26ZH1aUwMg2+Xb3V6hIhD7+PrvO35Q7uVewUEYXKfhS xxqgRCJBT/Z3+p1NaSw3I+nadYR0/XQOnSS27dnUnhASmeEZOEZXQZPBTukLlPefjALc E4Ud/4Kg+ImdvXJYUinFjyiLEtjbDlrB9tLnaR2jZ766FjmTOuz0tlSYPOglyXj+eicp AmvWzy0WlPdKk68s8pbwQpvV36GOB8ACljJ6aQzPl3b9cHPtInUce6LucpeBpsS0c+0J P2eg== X-Gm-Message-State: AOJu0YygplLJBAkFgFsQPSr046zRK//umVntqA7h0xxk12sSxC/E8PJZ eWDdJsuqBhB1PRmWLQ7uJDSFt1dhZ0ya+BwAQh6kSJ3+lGvq2NoO5Wq36KI2P0Hj1KLwltPJQnL EisMjOgxCsWzkXYdT6VCx7BboG1QwGIfDfk9H1CDUAceKCEvIAu2/pSJxOfClNRP76jczcG8N7p 1RFZX8Rg== X-Received: by 2002:ac8:1496:0:b0:421:9fe5:6dc1 with SMTP id l22-20020ac81496000000b004219fe56dc1mr6484684qtj.3.1700098163039; Wed, 15 Nov 2023 17:29:23 -0800 (PST) X-Received: by 2002:ac8:1496:0:b0:421:9fe5:6dc1 with SMTP id l22-20020ac81496000000b004219fe56dc1mr6484654qtj.3.1700098162561; Wed, 15 Nov 2023 17:29:22 -0800 (PST) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id c24-20020ac85198000000b0041e383d527esm3922598qtn.66.2023.11.15.17.29.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Nov 2023 17:29:22 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Mike Kravetz , "Kirill A . Shutemov" , Lorenzo Stoakes , Axel Rasmussen , Matthew Wilcox , John Hubbard , Mike Rapoport , peterx@redhat.com, Hugh Dickins , David Hildenbrand , Andrea Arcangeli , Rik van Riel , James Houghton , Yang Shi , Jason Gunthorpe , Vlastimil Babka , Andrew Morton Subject: [PATCH RFC 07/12] mm/gup: Refactor record_subpages() to find 1st small page Date: Wed, 15 Nov 2023 20:29:03 -0500 Message-ID: <20231116012908.392077-8-peterx@redhat.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20231116012908.392077-1-peterx@redhat.com> References: <20231116012908.392077-1-peterx@redhat.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.2 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, RCVD_IN_DNSWL_BLOCKED,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL, SPF_HELO_NONE,SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Wed, 15 Nov 2023 17:29:52 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1782682163448510062 X-GMAIL-MSGID: 1782682163448510062 All the fast-gup functions take a tail page to operate, always need to do page mask calculations before feeding that into record_subpages(). Merge that logic into record_subpages(), so that we always take a head page, and leave the rest calculation to record_subpages(). Signed-off-by: Peter Xu --- mm/gup.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 424d45e1afb3..69dae51f3eb1 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2763,11 +2763,14 @@ static int __gup_device_huge_pud(pud_t pud, pud_t *pudp, unsigned long addr, } #endif -static int record_subpages(struct page *page, unsigned long addr, - unsigned long end, struct page **pages) +static int record_subpages(struct page *head, unsigned long sz, + unsigned long addr, unsigned long end, + struct page **pages) { + struct page *page; int nr; + page = nth_page(head, (addr & (sz - 1)) >> PAGE_SHIFT); for (nr = 0; addr != end; nr++, addr += PAGE_SIZE) pages[nr] = nth_page(page, nr); @@ -2804,8 +2807,8 @@ static int gup_hugepte(pte_t *ptep, unsigned long sz, unsigned long addr, /* hugepages are never "special" */ VM_BUG_ON(!pfn_valid(pte_pfn(pte))); - page = nth_page(pte_page(pte), (addr & (sz - 1)) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pte_page(pte); + refs = record_subpages(page, sz, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2870,8 +2873,8 @@ static int gup_huge_pmd(pmd_t orig, pmd_t *pmdp, unsigned long addr, pages, nr); } - page = nth_page(pmd_page(orig), (addr & ~PMD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pmd_page(orig); + refs = record_subpages(page, PMD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2914,8 +2917,8 @@ static int gup_huge_pud(pud_t orig, pud_t *pudp, unsigned long addr, pages, nr); } - page = nth_page(pud_page(orig), (addr & ~PUD_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pud_page(orig); + refs = record_subpages(page, PUD_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio) @@ -2954,8 +2957,8 @@ static int gup_huge_pgd(pgd_t orig, pgd_t *pgdp, unsigned long addr, BUILD_BUG_ON(pgd_devmap(orig)); - page = nth_page(pgd_page(orig), (addr & ~PGDIR_MASK) >> PAGE_SHIFT); - refs = record_subpages(page, addr, end, pages + *nr); + page = pgd_page(orig); + refs = record_subpages(page, PGDIR_SIZE, addr, end, pages + *nr); folio = try_grab_folio(page, refs, flags); if (!folio)