From patchwork Wed Jan 18 15:26:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 45352 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2416898wrn; Wed, 18 Jan 2023 08:02:51 -0800 (PST) X-Google-Smtp-Source: AMrXdXuzoV1Z8BVkFoIb5XU8CyPsZCqhDVvbc94oZS8rFlkTQ1zo9ZGk+/eKz8fyUk2cO5YmA2u3 X-Received: by 2002:a17:906:6d99:b0:870:58ae:842e with SMTP id h25-20020a1709066d9900b0087058ae842emr6673845ejt.24.1674057771198; Wed, 18 Jan 2023 08:02:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1674057771; cv=none; d=google.com; s=arc-20160816; b=NRtDCrpLf8LUya23vQW1saa1RtzusTFTG50TjLWdwPcRlERkmXaihOYnB6sSAnwh4U FGOh1cUbWT1d9aLJc8Eq0WAqJQfZSQ9070/RTNsbMe0McGLzm/8QQQ0rGhutJCx2u4Nt dyl39fX9mpQKiKjLuEiKl2neY4nArhHV3y+sy3dNA+PuLvtqQPXmgUL2BqmZAyKQ7f2P mIMdWLHD5EdCErjodRTdwxBbGEXKeMlODiegWiXrGk6DbayCrB5mAPRouta0VAiiiHJP xAcJ9BR41RH1h0QD82FA3UDbzAVpvAS5M5C77B/OTYGacEYrdpJLtT5cbYaNVMcRv9Yl r4bA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=oh1p6MQJBk1v+dZi/ao7mkvsZxgunqwy2zNpz53Nk9Q=; b=KlUN5ytdQrtaeI/+8yM48lZYmGl359l/th90R41tsW15chdYCbMifBmekGStvchngG M1j5P5JWIGxBR7kMjAN4qDRczCZj5rMLFSuezfUUZ3fzbh9S8E4Jb0C11YmpiimOv2Lf Vnpt2ldXWYijajP5BPzlTwOFLLaem7XAqc1RlgBKgKETeszNMkhWiwWL6osVpfwsufld c6SaGYm0kHrVZN9y6tT8chFKHKBBYtg8mVqB1xts1FcJcnYvuuYDgxm8vywKqgEjX4EJ L++KnjWTOvX7nQB8xS4OEWHZYZiD0cZbbAOvd0rriI/XmIjVUIGzxoXBFAHaDMRvnGCf 58MQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=gCR4Eo2N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hq25-20020a1709073f1900b008672552b4c9si23785636ejc.38.2023.01.18.08.02.23; Wed, 18 Jan 2023 08:02:51 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=gCR4Eo2N; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230007AbjARPzb (ORCPT + 99 others); Wed, 18 Jan 2023 10:55:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230380AbjARPyM (ORCPT ); Wed, 18 Jan 2023 10:54:12 -0500 X-Greylist: delayed 887 seconds by postgrey-1.37 at lindbergh.monkeyblade.net; Wed, 18 Jan 2023 07:51:05 PST Received: from out0.migadu.com (out0.migadu.com [IPv6:2001:41d0:2:267::]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9651B3A86B; Wed, 18 Jan 2023 07:51:04 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1674055580; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oh1p6MQJBk1v+dZi/ao7mkvsZxgunqwy2zNpz53Nk9Q=; b=gCR4Eo2NDzSBH5kJqEcQmic2x/rt7h3W3vJR5nVRP0pmoNtZthy5Ij3dVYCPtKxDOe7rze Yh04QjizoCgCMeHeV0/Bm35tiKLymzlFhAj6NEaLjWKkO456Lkff5M+9K+PUFJ5NI7tTnV OgFXO41CZsefCw4Xqk0Fh2eC0EAiUkI= From: Kent Overstreet To: linux-kernel@vger.kernel.org Cc: Kent Overstreet , linux-aio@kvack.org, linux-fsdevel@vger.kernel.org Subject: [PATCH 2/2] fs/aio: obey min_nr when doing wakeups Date: Wed, 18 Jan 2023 10:26:03 -0500 Message-Id: <20230118152603.28301-2-kent.overstreet@linux.dev> In-Reply-To: <20230118152603.28301-1-kent.overstreet@linux.dev> References: <20230118152603.28301-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.8 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_LOW,SPF_HELO_PASS, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1755376801360391288?= X-GMAIL-MSGID: =?utf-8?q?1755376801360391288?= I've been observing workloads where IPIs due to wakeups in aio_complete() are ~15% of total CPU time in the profile. Most of those wakeups are unnecessary when completion batching is in use in io_getevents(). This plumbs min_nr through via the wait eventry, so that aio_complete() can avoid doing unnecessary wakeups. Signed-off-by: Kent Overstreet Cc: Benjamin LaHaise ki_ctx; struct aio_ring *ring; struct io_event *ev_page, *event; - unsigned tail, pos, head; + unsigned tail, pos, head, avail; unsigned long flags; /* @@ -1157,6 +1162,10 @@ static void aio_complete(struct aio_kiocb *iocb) ctx->completed_events++; if (ctx->completed_events > 1) refill_reqs_available(ctx, head, tail); + + avail = tail > head + ? tail - head + : tail + ctx->nr_events - head; spin_unlock_irqrestore(&ctx->completion_lock, flags); pr_debug("added to ring %p at [%u]\n", iocb, tail); @@ -1177,8 +1186,18 @@ static void aio_complete(struct aio_kiocb *iocb) */ smp_mb(); - if (waitqueue_active(&ctx->wait)) - wake_up(&ctx->wait); + if (waitqueue_active(&ctx->wait)) { + struct aio_waiter *curr, *next; + unsigned long flags; + + spin_lock_irqsave(&ctx->wait.lock, flags); + list_for_each_entry_safe(curr, next, &ctx->wait.head, w.entry) + if (avail >= curr->min_nr) { + list_del_init_careful(&curr->w.entry); + wake_up_process(curr->w.private); + } + spin_unlock_irqrestore(&ctx->wait.lock, flags); + } } static inline void iocb_put(struct aio_kiocb *iocb) @@ -1294,7 +1313,9 @@ static long read_events(struct kioctx *ctx, long min_nr, long nr, struct io_event __user *event, ktime_t until) { - long ret = 0; + struct hrtimer_sleeper t; + struct aio_waiter w; + long ret = 0, ret2 = 0; /* * Note that aio_read_events() is being called as the conditional - i.e. @@ -1310,12 +1331,34 @@ static long read_events(struct kioctx *ctx, long min_nr, long nr, * the ringbuffer empty. So in practice we should be ok, but it's * something to be aware of when touching this code. */ - if (until == 0) - aio_read_events(ctx, min_nr, nr, event, &ret); - else - wait_event_interruptible_hrtimeout(ctx->wait, - aio_read_events(ctx, min_nr, nr, event, &ret), - until); + aio_read_events(ctx, min_nr, nr, event, &ret); + if (until == 0 || ret < 0 || ret >= min_nr) + return ret; + + hrtimer_init_sleeper_on_stack(&t, CLOCK_MONOTONIC, HRTIMER_MODE_REL); + if (until != KTIME_MAX) { + hrtimer_set_expires_range_ns(&t.timer, until, current->timer_slack_ns); + hrtimer_sleeper_start_expires(&t, HRTIMER_MODE_REL); + } + + init_wait(&w.w); + + while (1) { + w.min_nr = min_nr - ret; + + ret2 = prepare_to_wait_event(&ctx->wait, &w.w, TASK_INTERRUPTIBLE) ?: + !t.task ? -ETIME : 0; + + if (aio_read_events(ctx, min_nr, nr, event, &ret) || ret2) + break; + + schedule(); + } + + finish_wait(&ctx->wait, &w.w); + hrtimer_cancel(&t.timer); + destroy_hrtimer_on_stack(&t.timer); + return ret; }