From patchwork Wed Feb 1 16:01:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marcelo Tosatti X-Patchwork-Id: 51451 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp366754wrn; Wed, 1 Feb 2023 08:12:33 -0800 (PST) X-Google-Smtp-Source: AK7set8BRNwrlnMKlVdhtllQukxCXY7EFiegVoUX/pOBtOx/Ovk6yZf5hFBL1SEEJQRrme5m5hos X-Received: by 2002:a05:6402:1292:b0:4a3:43c1:8437 with SMTP id w18-20020a056402129200b004a343c18437mr2112838edv.11.1675267953648; Wed, 01 Feb 2023 08:12:33 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1675267953; cv=none; d=google.com; s=arc-20160816; b=tC/gAElLpc7OrEVsHPQCPalLf/yR9o2vmg8jJlUHjLOwbzGYt3rquIBPZ0xodcvzxR AWgMWJIlzg+bz8gbhRkX+OFpWbhrHUX+6groz68x1BygcwokhGs6t17wF5euBHkVrZQD 5P3XqZrXKHoQDDFs5Z8cIpWSK19rnu5DcPFfArWkOFjYh1D6lbwnD6d/GCoGrX6MPui3 A70Y3FN0R7iNTdWbURyBk0zpEghHokxhZFThGL+lG2QCZZ1Vg1yCnWk2eqDrA9vbFhWk TXFDEbDjgab4aCB8SalBss1vISw4rh1BuhlZJGp02oVDuUSE/vwRbi0VVRmwjZqRHXkT 6m4A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-disposition:mime-version:message-id :subject:cc:to:from:date:dkim-signature; bh=AiUOCdYZiZ0EhpszTzQvWhcI8sDqoZDTm+/tkKX62Oo=; b=0bNFtEPZnAKYor8FDbFI9z0UQfrfTup74PgbCQaE7MVTxWoQlm5Ty1Q46dIhcsEDkx rkFX4CNj49+FCDWZy2JK3DVyIELt3odxQaT8y0+PRfTTqlknKqL5Ydax5MmPc7Z+A5yi 1WMxsAhUYrH+l4WZUfa+hoH5fHHcliNDqJ0/xh1z/0Q8XGiupOVS0juFi5wY3dzpSP/8 PNxZV42YKw89CGBqoUy7FfCh9ptMUbmjxdb0DXZBMCNwZrSqG133dw3+EOswmQiN/Fix G0rlrGWl+fXSVxNpgayWwKvItV05u3IQuLX7NNJFlcyj0+0Y2d1i//r+aF68/SvcSf0n m74Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="IHazJrw/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gj31-20020a170907741f00b0087bda90a805si17211011ejc.488.2023.02.01.08.12.10; Wed, 01 Feb 2023 08:12:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b="IHazJrw/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232884AbjBAQEq (ORCPT + 99 others); Wed, 1 Feb 2023 11:04:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231665AbjBAQEn (ORCPT ); Wed, 1 Feb 2023 11:04:43 -0500 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A23F3E098 for ; Wed, 1 Feb 2023 08:03:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1675267438; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type; bh=AiUOCdYZiZ0EhpszTzQvWhcI8sDqoZDTm+/tkKX62Oo=; b=IHazJrw/sgERUoR+odTrp9nC17nT5gw6xNF07ZsZKd2Jk5HPHmNxQvwISTwKhAQPDfnQ2c 4gGRRo7eP/Gc1tiDJEYaF6LRe48pFhtWpcAUn3Z3bC3+2QR1sFfa/dr+BIlalp2uVOYs0v c5apCbJsOrnbIQuUhFN8J4MuukKbfxg= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-25-hY-NQHINNpGkAjV3mvyciA-1; Wed, 01 Feb 2023 11:03:54 -0500 X-MC-Unique: hY-NQHINNpGkAjV3mvyciA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8E1DE38149AB; Wed, 1 Feb 2023 16:03:53 +0000 (UTC) Received: from tpad.localdomain (ovpn-112-2.gru2.redhat.com [10.97.112.2]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 9741D404BEC4; Wed, 1 Feb 2023 16:03:52 +0000 (UTC) Received: by tpad.localdomain (Postfix, from userid 1000) id 30599403CC75B; Wed, 1 Feb 2023 13:01:47 -0300 (-03) Date: Wed, 1 Feb 2023 13:01:47 -0300 From: Marcelo Tosatti To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Cc: Alexander Viro , Matthew Wilcox , Christoph Hellwig , Jens Axboe , Frederic Weisbecker , Leonardo Bras , Yair Podemsky , P J P Subject: [PATCH v3] fs/buffer.c: update per-CPU bh_lru cache via RCU Message-ID: MIME-Version: 1.0 Content-Disposition: inline X-Scanned-By: MIMEDefang 3.1 on 10.11.54.1 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1756645769465130880?= X-GMAIL-MSGID: =?utf-8?q?1756645769465130880?= umount calls invalidate_bh_lrus which IPIs each CPU that has non empty per-CPU buffer_head cache: on_each_cpu_cond(has_bh_in_lru, invalidate_bh_lru, NULL, 1); This interrupts CPUs which might be executing code sensitive to interferences. To avoid the IPI, free the per-CPU caches remotely via RCU. Two bh_lrus structures for each CPU are allocated: one is being used (assigned to per-CPU bh_lru pointer), and the other is being freed (or idle). Signed-off-by: Marcelo Tosatti --- v3: fix CPU hotplug v2: fix sparse warnings (kernel test robot) diff --git a/fs/buffer.c b/fs/buffer.c index d9c6d1fbb6dd..0c54ffe9fd62 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -1203,7 +1203,21 @@ struct bh_lru { struct buffer_head *bhs[BH_LRU_SIZE]; }; -static DEFINE_PER_CPU(struct bh_lru, bh_lrus) = {{ NULL }}; + +/* + * Allocate two bh_lrus structures for each CPU. bh_lru points to the + * one that is currently in use, and the update path does + * (consider cpu->bh_lru = bh_lrus[0]). + * + * cpu->bh_lrup = bh_lrus[1] + * synchronize_rcu() + * free bh's in bh_lrus[0] + */ +static unsigned int bh_lru_idx; +static DEFINE_PER_CPU(struct bh_lru, bh_lrus[2]) = {{{ NULL }}, {{NULL}}}; +static DEFINE_PER_CPU(struct bh_lru __rcu *, bh_lrup); + +static DEFINE_MUTEX(bh_lru_invalidate_mutex); #ifdef CONFIG_SMP #define bh_lru_lock() local_irq_disable() @@ -1245,16 +1259,19 @@ static void bh_lru_install(struct buffer_head *bh) return; } - b = this_cpu_ptr(&bh_lrus); + rcu_read_lock(); + b = rcu_dereference(per_cpu(bh_lrup, smp_processor_id())); for (i = 0; i < BH_LRU_SIZE; i++) { swap(evictee, b->bhs[i]); if (evictee == bh) { + rcu_read_unlock(); bh_lru_unlock(); return; } } get_bh(bh); + rcu_read_unlock(); bh_lru_unlock(); brelse(evictee); } @@ -1266,28 +1283,32 @@ static struct buffer_head * lookup_bh_lru(struct block_device *bdev, sector_t block, unsigned size) { struct buffer_head *ret = NULL; + struct bh_lru *lru; unsigned int i; check_irqs_on(); bh_lru_lock(); + rcu_read_lock(); + + lru = rcu_dereference(per_cpu(bh_lrup, smp_processor_id())); for (i = 0; i < BH_LRU_SIZE; i++) { - struct buffer_head *bh = __this_cpu_read(bh_lrus.bhs[i]); + struct buffer_head *bh = lru->bhs[i]; if (bh && bh->b_blocknr == block && bh->b_bdev == bdev && bh->b_size == size) { if (i) { while (i) { - __this_cpu_write(bh_lrus.bhs[i], - __this_cpu_read(bh_lrus.bhs[i - 1])); + lru->bhs[i] = lru->bhs[i - 1]; i--; } - __this_cpu_write(bh_lrus.bhs[0], bh); + lru->bhs[0] = bh; } get_bh(bh); ret = bh; break; } } + rcu_read_unlock(); bh_lru_unlock(); return ret; } @@ -1381,35 +1402,54 @@ static void __invalidate_bh_lrus(struct bh_lru *b) b->bhs[i] = NULL; } } -/* - * invalidate_bh_lrus() is called rarely - but not only at unmount. - * This doesn't race because it runs in each cpu either in irq - * or with preempt disabled. - */ -static void invalidate_bh_lru(void *arg) -{ - struct bh_lru *b = &get_cpu_var(bh_lrus); - - __invalidate_bh_lrus(b); - put_cpu_var(bh_lrus); -} bool has_bh_in_lru(int cpu, void *dummy) { - struct bh_lru *b = per_cpu_ptr(&bh_lrus, cpu); + struct bh_lru *b; int i; - + + rcu_read_lock(); + b = rcu_dereference(per_cpu(bh_lrup, cpu)); for (i = 0; i < BH_LRU_SIZE; i++) { - if (b->bhs[i]) + if (b->bhs[i]) { + rcu_read_unlock(); return true; + } } + rcu_read_unlock(); return false; } +/* + * invalidate_bh_lrus() is called rarely - but not only at unmount. + */ void invalidate_bh_lrus(void) { - on_each_cpu_cond(has_bh_in_lru, invalidate_bh_lru, NULL, 1); + int cpu, oidx; + + mutex_lock(&bh_lru_invalidate_mutex); + cpus_read_lock(); + oidx = bh_lru_idx; + bh_lru_idx++; + if (bh_lru_idx >= 2) + bh_lru_idx = 0; + + /* Assign the per-CPU bh_lru pointer */ + for_each_online_cpu(cpu) + rcu_assign_pointer(per_cpu(bh_lrup, cpu), + per_cpu_ptr(&bh_lrus[bh_lru_idx], cpu)); + synchronize_rcu_expedited(); + + for_each_online_cpu(cpu) { + struct bh_lru *b = per_cpu_ptr(&bh_lrus[oidx], cpu); + + bh_lru_lock(); + __invalidate_bh_lrus(b); + bh_lru_unlock(); + } + cpus_read_unlock(); + mutex_unlock(&bh_lru_invalidate_mutex); } EXPORT_SYMBOL_GPL(invalidate_bh_lrus); @@ -1422,8 +1462,10 @@ void invalidate_bh_lrus_cpu(void) struct bh_lru *b; bh_lru_lock(); - b = this_cpu_ptr(&bh_lrus); + rcu_read_lock(); + b = rcu_dereference(per_cpu(bh_lrup, smp_processor_id())); __invalidate_bh_lrus(b); + rcu_read_unlock(); bh_lru_unlock(); } @@ -2920,15 +2962,25 @@ void free_buffer_head(struct buffer_head *bh) } EXPORT_SYMBOL(free_buffer_head); +static int buffer_cpu_online(unsigned int cpu) +{ + rcu_assign_pointer(per_cpu(bh_lrup, cpu), + per_cpu_ptr(&bh_lrus[bh_lru_idx], cpu)); + return 0; +} + static int buffer_exit_cpu_dead(unsigned int cpu) { int i; - struct bh_lru *b = &per_cpu(bh_lrus, cpu); + struct bh_lru *b; + rcu_read_lock(); + b = rcu_dereference(per_cpu(bh_lrup, cpu)); for (i = 0; i < BH_LRU_SIZE; i++) { brelse(b->bhs[i]); b->bhs[i] = NULL; } + rcu_read_unlock(); this_cpu_add(bh_accounting.nr, per_cpu(bh_accounting, cpu).nr); per_cpu(bh_accounting, cpu).nr = 0; return 0; @@ -3021,7 +3073,7 @@ EXPORT_SYMBOL(__bh_read_batch); void __init buffer_init(void) { unsigned long nrpages; - int ret; + int ret, cpu; bh_cachep = kmem_cache_create("buffer_head", sizeof(struct buffer_head), 0, @@ -3029,6 +3081,11 @@ void __init buffer_init(void) SLAB_MEM_SPREAD), NULL); + cpus_read_lock(); + for_each_online_cpu(cpu) + rcu_assign_pointer(per_cpu(bh_lrup, cpu), per_cpu_ptr(&bh_lrus[0], cpu)); + cpus_read_unlock(); + /* * Limit the bh occupancy to 10% of ZONE_NORMAL */ @@ -3037,4 +3094,7 @@ void __init buffer_init(void) ret = cpuhp_setup_state_nocalls(CPUHP_FS_BUFF_DEAD, "fs/buffer:dead", NULL, buffer_exit_cpu_dead); WARN_ON(ret < 0); + ret = cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "fs/buffer:online", + NULL, buffer_cpu_online); + WARN_ON(ret < 0); }