From patchwork Thu Apr 6 13:20:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pavel Begunkov X-Patchwork-Id: 7816 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1021861vqo; Thu, 6 Apr 2023 06:21:50 -0700 (PDT) X-Google-Smtp-Source: AKy350Y7T5XYZ6GudHEJpgx04HHxjuYWF9uxSFfzOeQKIeUK5uFzNKrUpaG1K2iTKuaWW4SJg8W0 X-Received: by 2002:aa7:9e92:0:b0:62d:7f6:cf1 with SMTP id p18-20020aa79e92000000b0062d07f60cf1mr8570212pfq.3.1680787310595; Thu, 06 Apr 2023 06:21:50 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680787310; cv=none; d=google.com; s=arc-20160816; b=JcP+O7ccfKV6DfXZ1xWOI5SEa7xMhHHkQm2iq9e9UZUffVD5xutpLKN+MssS61ISin oWRvE9lVvKsovyXBwRD6K7vXBm+SEGBLoDX0r2ETu41JIttRP4hy7L2NQLcJUJYKFDPk zuitypX2cfMR3stt8zTCJIETDeZIBl++UMmGAsSd4dzjAc/zGwPonjJ0V/28J6e3jxPD X3Tc60V4095BLc9/EpmIp44890ohLoVw+IsXg62OE2tSMDIwTcdcUdmpFxxevmzveyxq nqktCp2POvLNC/8ZVnAJ7zOTz9Q67j6R7+QtcJ7s0qyxuSJ0NspYk0PN5ZTMxwvTH/QW q9sA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=bQnJmNTlCd7Gp0yJXnxtAGpYcfqiysBD6ZHlqPg+cT4=; b=jeirPiecVW9WEEO302A+RqcLQqi5oNCz6Zw2wKXlj0jhQOGPgKFYW3lzEM06iTMAbe zmHtn6zW9L8WppJiUd1T5VKCipr1aAGfVt8g8oz+7qes5JupBz8Q8IxyyeylkuI8LCls B32NoR43MLMpZoKXiBk8Y/s++fNFaFrxtkoPv9C3A6lVkaKQZbDFs2mIuXlD2rsk6vm8 nf0gYLpRvTGBN6WUc1ZK18V2yUpZWp39Sfu7Lzu5eLyFwrKaopdx7m9iOWfhhwSgHNcz xFGQSTxp1+qx+neanh9WZaARC6y05s1Q17nbKKbp04UJWo02QDDdUm9STDJIgQUL7EnK 5B2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=exUjrwim; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w189-20020a6382c6000000b00513fd21728csi1303130pgd.463.2023.04.06.06.21.37; Thu, 06 Apr 2023 06:21:50 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20210112 header.b=exUjrwim; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238300AbjDFNVE (ORCPT + 99 others); Thu, 6 Apr 2023 09:21:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238083AbjDFNU5 (ORCPT ); Thu, 6 Apr 2023 09:20:57 -0400 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B03B9ED7; Thu, 6 Apr 2023 06:20:25 -0700 (PDT) Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-930e691e34eso108810266b.0; Thu, 06 Apr 2023 06:20:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1680787223; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=bQnJmNTlCd7Gp0yJXnxtAGpYcfqiysBD6ZHlqPg+cT4=; b=exUjrwimKbVYwFi6ACeSjkaV6qQ0gaUUI8LUhOFpD+Fb5H79XdpADBGsggkysr/VXo iXvsPS5AcVPyuo5XVVrrxQ/o/8KDDdI5QOvl/1XaSoXUwfUj6e61S3Q6ARJwj8t0ST0k k5IJSr+EHMZLlCUvcQqfaq+7jw6g2MTvb7F8eyNyS/bdzcZdoeDm10ul1c26HlVBB1QN 3t77rSdaPl6WEnFpIvrMYlqcUI6hipLKS9uRVO0LYuDlXbuFYkc8h1mct4i++Tt4hgna 0Enchjt+/IfTdro4Chbr1yIVFlwjf6Vfk4Zs41uzwfIKyAHyuAA4X6ARs4+fYZhzy5Md lPMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680787223; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=bQnJmNTlCd7Gp0yJXnxtAGpYcfqiysBD6ZHlqPg+cT4=; b=Yu6hi3QkzhgSr5bgmSfpX+QNeEV+0schqx6JzJuYdzrgYdHbbpSH1t1phBYBm+UWQ5 tgBlq0R6st+JVIjkaQQjgDVSrY16noWnwRNKRGJiHUQTLYNcX2fXcPPRB12ElX8E6Ad0 bfZwoZGcmhycw3WETrGmL6Q2nOz8qDtKkfHW7h+vCxcCeZGSmaWtbSoQCRlTKpx8Izwv m8fVVw9JdfBEFf5I8ZPVREw2x3WFJNZ0cq5ocXRHXn9KCfsYg7frMDicuGB49tMWQyKZ FCgbrB/rTSPOjcBgKy+Mv8zOGZIe3P+WSEh7ke+K2MyoiLAqPyFo0+b3Zcwzw6jcv1jR bXMQ== X-Gm-Message-State: AAQBX9d6rexyqpHwEdSGAi0dx7VNaagZqU1ukkYLVQm75RmuvFQg1b2M v5hxt4lifjlvHvVn+MU8xcC4CsvB5z8= X-Received: by 2002:a05:6402:6d9:b0:502:22fe:ef3c with SMTP id n25-20020a05640206d900b0050222feef3cmr4678466edy.41.1680787223198; Thu, 06 Apr 2023 06:20:23 -0700 (PDT) Received: from 127.com ([2620:10d:c092:600::2:a638]) by smtp.gmail.com with ESMTPSA id m20-20020a509994000000b0050470aa444fsm312732edb.51.2023.04.06.06.20.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Apr 2023 06:20:23 -0700 (PDT) From: Pavel Begunkov To: io-uring@vger.kernel.org Cc: Jens Axboe , asml.silence@gmail.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 0/8] optimise resheduling due to deferred tw Date: Thu, 6 Apr 2023 14:20:06 +0100 Message-Id: X-Mailer: git-send-email 2.40.0 MIME-Version: 1.0 X-Spam-Status: No, score=-0.2 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762433234632671412?= X-GMAIL-MSGID: =?utf-8?q?1762433234632671412?= io_uring extensively uses task_work, but when a task is waiting every new queued task_work batch will try to wake it up and so cause lots of scheduling activity. This series optimises it, specifically applied for rw completions and send-zc notifications for now, and will helpful for further optimisations. Quick testing shows similar to v1 results, numbers from v1: For my zc net test once in a while waiting for a portion of buffers I've got 10x descrease in the number of context switches and 2x improvement in CPU util (17% vs 8%). In profiles, io_cqring_work() got down from 40-50% of CPU to ~13%. There is also an improvement on the softirq side for io_uring notifications as io_req_local_work_add() doesn't trigger wake_up() as often. System wide profiles show reduction of cycles taken by io_req_local_work_add() from 3% -> 0.5%, which is mostly not reflected in the numbers above as it was firing off of a different CPU. v2: Remove atomics decrements by the queueing side and instead carry all the info in requests. It's definitely simpler and removes extra atomics, the downside is touching the previous request, which might be not cached. Add a couple of patches from backlog optimising and cleaning io_req_local_work_add(). Pavel Begunkov (8): io_uring: move pinning out of io_req_local_work_add io_uring: optimie local tw add ctx pinning io_uring: refactor __io_cq_unlock_post_flush() io_uring: add tw add flags io_uring: inline llist_add() io_uring: reduce scheduling due to tw io_uring: refactor __io_cq_unlock_post_flush() io_uring: optimise io_req_local_work_add include/linux/io_uring_types.h | 3 +- io_uring/io_uring.c | 131 ++++++++++++++++++++++----------- io_uring/io_uring.h | 29 +++++--- io_uring/notif.c | 2 +- io_uring/notif.h | 2 +- io_uring/rw.c | 2 +- 6 files changed, 110 insertions(+), 59 deletions(-)