From patchwork Thu Apr 6 20:47:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Sebastian Andrzej Siewior X-Patchwork-Id: 80541 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1313313vqo; Thu, 6 Apr 2023 14:32:39 -0700 (PDT) X-Google-Smtp-Source: AKy350azH57Fg5PqDGnF/W6uxMKbXMWow43wamt1ldraOdS3xjw4zkuxFCBLJCFW1BJg3MX8mAqf X-Received: by 2002:a17:902:f687:b0:19a:b427:230a with SMTP id l7-20020a170902f68700b0019ab427230amr660546plg.63.1680816759664; Thu, 06 Apr 2023 14:32:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680816759; cv=none; d=google.com; s=arc-20160816; b=0a+9tn1S0VLQfKGhvEpW5hljQ3uOWj+6SoTY9cZ/oDn/M+FRD7T+mVguzSGrNYHQdE ZfXOl5VYh00uhsznE170eBeFbQo04ktZ87RAqdIRoukftjSL6qXRCRhFF5ZwjvSMti0Q 2hIKvbsLi9uxtGXNDa8Z13AlIlughIzbloMtTr/U7ydYxeeX2nQZrSfBFuJNla+saluv 7+8WvvtqvuSpVG2BaacK21zI9E/jtPTCQymCm0UG47GqssiJq0TT9Aa4v+EZTFwoPfQ7 TvlUoQL6jEnRg0qComotEcYymnr/S14ZJG5TxQ5J6fjmS7EeyVNegqxmy/lYiwmj7ETG VL6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:in-reply-to:content-transfer-encoding :content-disposition:mime-version:references:message-id:subject:cc :to:from:dkim-signature:dkim-signature:date; bh=4nfE2RKzKiS1tWZ/hTXfIEE3f8A6Qg1pMEXWmyuRQ0M=; b=Z4sBXW399WFQf9Xl5caUASuJ3c3O+zrZo8YhOz+NwAgj7WiTtMhTi0sa4n4emqn19U y6jwsJlAdgWwVS0ZYTwjlk2yx+iTDgd029u0cZK+DpLux1jUSeBk/gN0E4JDFAGjIw+Y VVGYHML6gVPlucD+ndlXNUyyg5BAbgTeVWut2yqfca1Wi2QTAf+lOqHXdLuKjFfYxJeA e8cHQW4wJ7RVy1RsTGrowY6mRDWRtbuz3jhKsPQCEpt1f5etDvC+i3B9PHsT6gDT3X/z N3BBsz4FKhOWWZwLL0MA/1OCKYkAT9Cm4H5PpsYJ4v0FRsQwfwlH3JxQ5yCehOxmOjy7 cwLA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=asdcemsX; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=C6OnuerO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 206-20020a6302d7000000b00513f1464b0esi2022650pgc.618.2023.04.06.14.32.27; Thu, 06 Apr 2023 14:32:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linutronix.de header.s=2020 header.b=asdcemsX; dkim=neutral (no key) header.i=@linutronix.de header.s=2020e header.b=C6OnuerO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=linutronix.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240260AbjDFUt2 (ORCPT + 99 others); Thu, 6 Apr 2023 16:49:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239779AbjDFUtM (ORCPT ); Thu, 6 Apr 2023 16:49:12 -0400 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0334CAF0F for ; Thu, 6 Apr 2023 13:47:24 -0700 (PDT) Date: Thu, 6 Apr 2023 22:47:21 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1680814042; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4nfE2RKzKiS1tWZ/hTXfIEE3f8A6Qg1pMEXWmyuRQ0M=; b=asdcemsX5gQAeB9ZmYCyRtnrQRO/JHCniKJHJch8lrqeeCCoDHL1pqGgf314kbXED35vmK dgMRclkyGfVOr066VOEeMgZlYXfIOaIQ5mdzA/GjutbEYywBo32Z4w3mvlyaOxNxEN/YgK RFOW/8oqAWMnIzdQKUZpYaCedm96SJU9QNoLbh73zdppjFwxCiAhh/ullocwUEyauGMqJs ORj8MpSTWfmHqPgbAsxvnUDdlp8GNl+HEoSe/XgGKHjlJV9SHLpb4VZT6szkSuBF1wwLmk C114EQZ3tRa6ztigZstEcJ44ETmWACN0aJRDPeKQXStTYLteVdkskW+lE5UWog== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1680814042; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4nfE2RKzKiS1tWZ/hTXfIEE3f8A6Qg1pMEXWmyuRQ0M=; b=C6OnuerOp+NhYe6FwXhKS+hSfZaUXaOZ3J7i6FpT5gYn12KiqNgYUeg03BeMGZeZZihPYx D5BPAbdZxQvXyxBg== From: Sebastian Andrzej Siewior To: linux-kernel@vger.kernel.org Cc: "Eric W. Biederman" , Ben Segall , Christian Brauner , Daniel Bristot de Oliveira , Dietmar Eggemann , Ingo Molnar , Juri Lelli , Mel Gorman , Oleg Nesterov , Peter Zijlstra , Steven Rostedt , Thomas Gleixner , Valentin Schneider , Vincent Guittot , Linus Torvalds Subject: [PATCH v4] signal: Let tasks cache one sigqueue struct. Message-ID: <20230406204721.A6lSYL7A@linutronix.de> References: <20230406194004.KP1K6FwO@linutronix.de> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20230406194004.KP1K6FwO@linutronix.de> X-Spam-Status: No, score=-2.5 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1762458842170572897?= X-GMAIL-MSGID: =?utf-8?q?1762464114629199433?= The sigqueue caching originated in the PREEMPT_RT tree. A few of the applications, that were ported to Linux, were ported from OS-9. Sending notifications from one task to another via a signal was a common communication model there and so the applications are heavy signal users. Removing the allocation reduces the critical path, avoids locks and so lowers the maximal latency of the task while sending a signal. After posting the first version a discussion came up whether it wouldn't make sense to have this caching unconditionally and not restricted to PREEMPT_RT only. The sigqueue flagged SIGQUEUE_PREALLOC is used by the POSIX timer code. It is allocated on initialisation and reused during the lifetime of the timer. The sigqueue is freed once the timer is deleted. The POSIX timer sigqueue has its own caching and unique lifetime pattern and therefore does not fit for the generic caching. In the common case the signal is dequeued and freed in collect_signal(). At this point, the 'current' task receives the signal and its sighand_struct::siglock is held. __sigqueue_alloc() is used to allocate a new sigqueue. The task_struct passed as argument is the task that will receive the signal. Its sighand_struct::siglock is acquired (except for SIGQUEUE_PREALLOC allocation which is ignored). Use a cached sigqueue before allocating a new one. As a result of this pattern, the task sending a signal will use the cache from the task that will receive the signal which in turn caches the signal. The numbers of system boot followed by an allmod kernel build: Out of 333216 allocations, 194876 (~58%) were served from the cache. From all free invocations, 4212 were in a path were caching is not done and 329002 sigqueue were cached. Cache the struct sigqueue in collect_signal() and reuse it for the allocation. Rely on sighand_struct::siglock locking for cache handling which is held during allocation and free. The cache is cleaned once sighand_struct is freed. Signed-off-by: Sebastian Andrzej Siewior --- v3…v4: Repost with Brauner and Torvalds on Cc. v2…v3: Wrote the patch and its description from scratch without looking at the previous version. include/linux/sched/signal.h | 2 + kernel/fork.c | 11 +++++++++ kernel/signal.c | 48 +++++++++++++++++++++++++++++++++++++++---- 3 files changed, 57 insertions(+), 4 deletions(-) --- a/include/linux/sched/signal.h +++ b/include/linux/sched/signal.h @@ -22,6 +22,7 @@ struct sighand_struct { refcount_t count; wait_queue_head_t signalfd_wqh; struct k_sigaction action[_NSIG]; + struct sigqueue *sigqueue_cache; }; /* @@ -349,6 +350,7 @@ extern int send_sig(int, struct task_str extern int zap_other_threads(struct task_struct *p); extern struct sigqueue *sigqueue_alloc(void); extern void sigqueue_free(struct sigqueue *); +extern void sigqueue_free_cached_entry(struct sigqueue *q); extern int send_sigqueue(struct sigqueue *, struct pid *, enum pid_type); extern int do_sigaction(int, struct k_sigaction *, struct k_sigaction *); --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1661,6 +1661,7 @@ static int copy_sighand(unsigned long cl RCU_INIT_POINTER(tsk->sighand, sig); if (!sig) return -ENOMEM; + sig->sigqueue_cache = NULL; refcount_set(&sig->count, 1); spin_lock_irq(¤t->sighand->siglock); @@ -1677,7 +1678,17 @@ static int copy_sighand(unsigned long cl void __cleanup_sighand(struct sighand_struct *sighand) { if (refcount_dec_and_test(&sighand->count)) { + struct sigqueue *sigqueue = NULL; + signalfd_cleanup(sighand); + spin_lock_irq(&sighand->siglock); + if (sighand->sigqueue_cache) { + sigqueue = sighand->sigqueue_cache; + sighand->sigqueue_cache = NULL; + } + spin_unlock_irq(&sighand->siglock); + + sigqueue_free_cached_entry(sigqueue); /* * sighand_cachep is SLAB_TYPESAFE_BY_RCU so we can free it * without an RCU grace period, see __lock_task_sighand(). --- a/kernel/signal.c +++ b/kernel/signal.c @@ -432,7 +432,18 @@ static struct sigqueue * return NULL; if (override_rlimit || likely(sigpending <= task_rlimit(t, RLIMIT_SIGPENDING))) { - q = kmem_cache_alloc(sigqueue_cachep, gfp_flags); + + if (!sigqueue_flags) { + struct sighand_struct *sighand = t->sighand; + + lockdep_assert_held(&sighand->siglock); + if (sighand->sigqueue_cache) { + q = sighand->sigqueue_cache; + sighand->sigqueue_cache = NULL; + } + } + if (!q) + q = kmem_cache_alloc(sigqueue_cachep, gfp_flags); } else { print_dropped_signal(sig); } @@ -447,14 +458,43 @@ static struct sigqueue * return q; } -static void __sigqueue_free(struct sigqueue *q) +static bool sigqueue_cleanup_accounting(struct sigqueue *q) { if (q->flags & SIGQUEUE_PREALLOC) - return; + return false; if (q->ucounts) { dec_rlimit_put_ucounts(q->ucounts, UCOUNT_RLIMIT_SIGPENDING); q->ucounts = NULL; } + return true; +} + +static void __sigqueue_free(struct sigqueue *q) +{ + if (!sigqueue_cleanup_accounting(q)) + return; + kmem_cache_free(sigqueue_cachep, q); +} + +void sigqueue_free_cached_entry(struct sigqueue *q) +{ + if (!q) + return; + kmem_cache_free(sigqueue_cachep, q); +} + +static void sigqueue_cache_or_free(struct sigqueue *q) +{ + struct sighand_struct *sighand = current->sighand; + + if (!sigqueue_cleanup_accounting(q)) + return; + + lockdep_assert_held(&sighand->siglock); + if (!sighand->sigqueue_cache) { + sighand->sigqueue_cache = q; + return; + } kmem_cache_free(sigqueue_cachep, q); } @@ -594,7 +634,7 @@ static void collect_signal(int sig, stru (info->si_code == SI_TIMER) && (info->si_sys_private); - __sigqueue_free(first); + sigqueue_cache_or_free(first); } else { /* * Ok, it wasn't in the queue. This must be