From patchwork Wed Feb 14 20:44:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 201126 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7300:bc8a:b0:106:860b:bbdd with SMTP id dn10csp1502928dyb; Wed, 14 Feb 2024 13:13:05 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVpZfs/bg9Kh/HpOR5AG3SPkdkVrAiD6/FHcFmwNKYNuxpCtlkJyrPF+HIEV2OhBJGlFUk2LKkaHsASgLT2rQmPtFr2Ow== X-Google-Smtp-Source: AGHT+IHOJh2dd9LkG7M8Dng75NSleSbYbxkv4Y7BC7AO2NPK9RmL4IicttG/z0Ow2K3OQ2L7LON5 X-Received: by 2002:a05:6a21:1799:b0:19e:bf18:61fe with SMTP id nx25-20020a056a21179900b0019ebf1861femr10058pzb.11.1707945185520; Wed, 14 Feb 2024 13:13:05 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1707945185; cv=pass; d=google.com; s=arc-20160816; b=HRFrVYMYAwcNYiQ2ttZUG+vuKsRCB52OnhfkgyESvUeRmB76Q3q83/U6ErUfKifeuD cGP7GSOmnq8KQ5H16l2cFhs8ACqsPjt7GyhABEZZa6r0Eb8ZkhxlLXV/9VzMOv5xY4zJ dkPrMsM7htZwbcqO+4hxKmS8u/HFxg9hdIOkUs8Mi3lM/JNNPPuIlJgbhiS460JgCh3y 0dvkO7180EIlx/Dcqebn0IIZTAh8Yh6yVwSC/zLXDJMht8Rh/BPlD0gI6etuurkG6RN1 wI+ir2OmSQa8dgxvDNmQ23B1IGeligfJ+Lxcu+rtWXMBIQKq1et8cllohiC9l/keHSsv 8h9A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:list-unsubscribe :list-subscribe:list-id:precedence:references:in-reply-to:message-id :date:subject:cc:to:from:dkim-signature; bh=4Q1ph86PH1Crd3Eqr5OMYEFX/2u3faALVbtCzM66/MM=; fh=Jt2A3ytOmSNzpS6wOdFjY92dCuvx3EaqusiR9DRwmKE=; b=dW74964nXYmI/iujY10b4NFT+UNJvust2OzlY+6jd/TmJnIk3qrFRobUhOl+BPfLpQ LAJP6CmA4R3vu+DI0mqiHzkJcWi2t5VpeE8KwTPRw5UY8ToEvuhO2BDvmSo+3Tyz3j3w 7hUt2l2IHmdoKLj9vF+//3sPQrw1KNpgG7KN36IDXnbcBE8Ph8cDdlvKu6Hhna+Y7nd7 fkRJi+BvzBa8GZ2VZX1ApTcqGeulUIHfTEMElimv2yWJdF8v6l3p5t1qCAmsakTHIZWF 1nGBrEgvnYz9IMakDkx18l93KVjapo9PEB7yxnW4OO66uD4BN+fqOe2i6CqEPiQCHxdM bJ0Q==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Idqvrsgu; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-65951-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65951-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Forwarded-Encrypted: i=2; AJvYcCUZLLeZuFDoXq5Ebg3Nj8FkilDwksTJHggOUZaxcwObBT8MnGYAp6A+BRSZhocZG2xWa6Q2UKbr8qcSTY+TgYQZy04D+g== Received: from sy.mirrors.kernel.org (sy.mirrors.kernel.org. [147.75.48.161]) by mx.google.com with ESMTPS id y189-20020a6364c6000000b005ca5616facesi4195132pgb.38.2024.02.14.13.13.05 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Feb 2024 13:13:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-65951-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) client-ip=147.75.48.161; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=Idqvrsgu; arc=pass (i=1 spf=pass spfdomain=redhat.com dkim=pass dkdomain=redhat.com dmarc=pass fromdomain=redhat.com); spf=pass (google.com: domain of linux-kernel+bounces-65951-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.48.161 as permitted sender) smtp.mailfrom="linux-kernel+bounces-65951-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sy.mirrors.kernel.org (Postfix) with ESMTPS id 177D7B276EC for ; Wed, 14 Feb 2024 20:46:22 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id A83DE1419BE; Wed, 14 Feb 2024 20:45:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="Idqvrsgu" Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCAD81419A4 for ; Wed, 14 Feb 2024 20:45:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.133.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707943507; cv=none; b=oz4cKB94B5qEHLaRVa+ismCBobiHUlvM1JCXbAyrXREv5bZg4wwZT3/x3ZKfzaIcYbFOys9A6TtYo8rUVHqGJapMLHcE6RqFoPEIKljZSNQIQmz90AujJYRcN5VLTCmDEk77p1ca4TLEtYoo82XbCuI8zYFf96A817ClnX5ubXU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1707943507; c=relaxed/simple; bh=klZ3/8gE/hLexWRn9Rfgs4Jd0fYkSsj1v6sMZOE0W70=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=okkVJqIpNBZpJV5jGh/k9pwO3YxQm4kpA6iNxjLzJQDSNz4VhdkousShmePLMN0zQ5sAfJWUuSjuuSrGaybpsYUpZ29aGHeLpBE1BLi4WJmTgTmXEJIsyQOkz1XVGJzk31nw0Tf2JlysC6WbZ0BhtDP+uCq9/zgAotrqN7ynPWw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=Idqvrsgu; arc=none smtp.client-ip=170.10.133.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1707943505; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4Q1ph86PH1Crd3Eqr5OMYEFX/2u3faALVbtCzM66/MM=; b=IdqvrsgunK4/c/ezi9TsDMpswAVhVcXptWySqJV5aEqIQXCePWwu6527P763Dx76ELhwwC iR8RkMTO0IuZrqnfK/q8h3F+1JsOY6Di1QZs5BUq/+Kfu3oErH6uZUpF2zI7TMOPtzvoSp W7A73Q12+FhiW81CFWbWaz1hcMcIjqo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-670--0HI9YGROZaGy4w0dg4ihQ-1; Wed, 14 Feb 2024 15:45:01 -0500 X-MC-Unique: -0HI9YGROZaGy4w0dg4ihQ-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 864D985A589; Wed, 14 Feb 2024 20:45:00 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.174]) by smtp.corp.redhat.com (Postfix) with ESMTP id ECCE01C06532; Wed, 14 Feb 2024 20:44:56 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Catalin Marinas , Yin Fengwei , Michal Hocko , Will Deacon , "Aneesh Kumar K.V" , Nick Piggin , Peter Zijlstra , Michael Ellerman , Christophe Leroy , "Naveen N. Rao" , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Arnd Bergmann , linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-s390@vger.kernel.org Subject: [PATCH v3 05/10] mm/mmu_gather: pass "delay_rmap" instead of encoded page to __tlb_remove_page_size() Date: Wed, 14 Feb 2024 21:44:30 +0100 Message-ID: <20240214204435.167852-6-david@redhat.com> In-Reply-To: <20240214204435.167852-1-david@redhat.com> References: <20240214204435.167852-1-david@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.7 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1790910330917260474 X-GMAIL-MSGID: 1790910330917260474 We have two bits available in the encoded page pointer to store additional information. Currently, we use one bit to request delay of the rmap removal until after a TLB flush. We want to make use of the remaining bit internally for batching of multiple pages of the same folio, specifying that the next encoded page pointer in an array is actually "nr_pages". So pass page + delay_rmap flag instead of an encoded page, to handle the encoding internally. Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- arch/s390/include/asm/tlb.h | 13 ++++++------- include/asm-generic/tlb.h | 12 ++++++------ mm/mmu_gather.c | 7 ++++--- 3 files changed, 16 insertions(+), 16 deletions(-) diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index d1455a601adc..48df896d5b79 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -25,8 +25,7 @@ void __tlb_remove_table(void *_table); static inline void tlb_flush(struct mmu_gather *tlb); static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, - int page_size); + struct page *page, bool delay_rmap, int page_size); #define tlb_flush tlb_flush #define pte_free_tlb pte_free_tlb @@ -42,14 +41,14 @@ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, * tlb_ptep_clear_flush. In both flush modes the tlb for a page cache page * has already been freed, so just do free_page_and_swap_cache. * - * s390 doesn't delay rmap removal, so there is nothing encoded in - * the page pointer. + * s390 doesn't delay rmap removal. */ static inline bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, - int page_size) + struct page *page, bool delay_rmap, int page_size) { - free_page_and_swap_cache(encoded_page_ptr(page)); + VM_WARN_ON_ONCE(delay_rmap); + + free_page_and_swap_cache(page); return false; } diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 129a3a759976..2eb7b0d4f5d2 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -260,9 +260,8 @@ struct mmu_gather_batch { */ #define MAX_GATHER_BATCH_COUNT (10000UL/MAX_GATHER_BATCH) -extern bool __tlb_remove_page_size(struct mmu_gather *tlb, - struct encoded_page *page, - int page_size); +extern bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, + bool delay_rmap, int page_size); #ifdef CONFIG_SMP /* @@ -462,13 +461,14 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) static inline void tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size) { - if (__tlb_remove_page_size(tlb, encode_page(page, 0), page_size)) + if (__tlb_remove_page_size(tlb, page, false, page_size)) tlb_flush_mmu(tlb); } -static __always_inline bool __tlb_remove_page(struct mmu_gather *tlb, struct page *page, unsigned int flags) +static __always_inline bool __tlb_remove_page(struct mmu_gather *tlb, + struct page *page, bool delay_rmap) { - return __tlb_remove_page_size(tlb, encode_page(page, flags), PAGE_SIZE); + return __tlb_remove_page_size(tlb, page, delay_rmap, PAGE_SIZE); } /* tlb_remove_page diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 604ddf08affe..ac733d81b112 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -116,7 +116,8 @@ static void tlb_batch_list_free(struct mmu_gather *tlb) tlb->local.next = NULL; } -bool __tlb_remove_page_size(struct mmu_gather *tlb, struct encoded_page *page, int page_size) +bool __tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, + bool delay_rmap, int page_size) { struct mmu_gather_batch *batch; @@ -131,13 +132,13 @@ bool __tlb_remove_page_size(struct mmu_gather *tlb, struct encoded_page *page, i * Add the page and check if we are full. If so * force a flush. */ - batch->encoded_pages[batch->nr++] = page; + batch->encoded_pages[batch->nr++] = encode_page(page, delay_rmap); if (batch->nr == batch->max) { if (!tlb_next_batch(tlb)) return true; batch = tlb->active; } - VM_BUG_ON_PAGE(batch->nr > batch->max, encoded_page_ptr(page)); + VM_BUG_ON_PAGE(batch->nr > batch->max, page); return false; }