From patchwork Tue May 9 16:56:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91666 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3033045vqo; Tue, 9 May 2023 09:59:19 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5KO/HoMJMGVOdClVtYPbH41sC/fax6DXfjJmfppABHNIzaAh87RCrSNbfltI/KKG3QZh1u X-Received: by 2002:a17:903:41c2:b0:1ac:750e:33d9 with SMTP id u2-20020a17090341c200b001ac750e33d9mr10331882ple.16.1683651558743; Tue, 09 May 2023 09:59:18 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683651558; cv=none; d=google.com; s=arc-20160816; b=lPgT4icCiqauJPgFHkuWK2Fd85leyiQTUoW7D91M7L0db3s8B2stjf3Y88LCU1QPEB gjcWZ7zBDNvdJXoNUzJQ3mSOEPkvv6ZgNZTlQZRKSJ6iisJ+ujpzBI3D2aDC2x9ArfMG lC9fT1IhC1qFRpJ67QYkfMu5y6ttc11T3t2MuaYJvexNwEvcN2cx3mDWg0QxjF9ilw0L ag8+XtyGvmU4qkdAaS0R+MpH7zjCXZJqL21cTmO7f1/AKaBS2Q2KKDTCdGrphkBTpEq3 iMB9FpPsE1dPP+CtZ5uTwZGi6vfb3RU/F/eU1GDy1oISXZAvykR0NTB3pUVDfAYh7Jic lN0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mk2GM0+dzxVCnTybx12Btfro6vDgTHvnILBWigNj4c8=; b=udC2uAkXgK5YPUz1sLLI+8ZIUFlJnNir9l3yv+8jMSDrhFb0W2himqxZqrEP9aIeAQ hN2lIR+j9ODVFnsVdK/i9xxoXVJkbdePC5QdHb8xoekOXeCjq4mgxtnm9GAeDfpVFYyY L4lZ7s8L0EBZ8+YENVrcTsMm4tFI3clWNgKUZ+kfjrs+NmWCFFplc8dP03oohiQudQjt bJ2eTwpPOls+9Bm6pPIedrCdASw7mh9OA9KWooZtovDtmrVsj0HBa1r8SHr76dRODK6m f+LBIAxsPn7Q+6VMXeBm5+BjY+G9dR+SDQinHxlfxOVYUEeu07V9pOdDnMDx39zYlwqT X+Jg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=gvv06Efl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e7-20020a17090301c700b001aae90dbcc1si2037238plh.225.2023.05.09.09.59.06; Tue, 09 May 2023 09:59:18 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=gvv06Efl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234728AbjEIQ5U (ORCPT + 99 others); Tue, 9 May 2023 12:57:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229549AbjEIQ5O (ORCPT ); Tue, 9 May 2023 12:57:14 -0400 Received: from out-4.mta1.migadu.com (out-4.mta1.migadu.com [95.215.58.4]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 765A149EB for ; Tue, 9 May 2023 09:57:11 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651429; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mk2GM0+dzxVCnTybx12Btfro6vDgTHvnILBWigNj4c8=; b=gvv06Eflh4cSDF3nvay9HJM/D+7UHiIBgSqj7ssD0zLDoqIlYSxw9l1oY17SVUZrK3Vf42 LpQMzG48+j/L6sANVmqlFcGlIj8Ig7PqI9jibpFr9xJNtXIt8bjU+VZFJiOgnKqSEWGJ49 SPTS/U6/Tr/fe9ohzRHhEAZUr9KkMOU= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Miguel Ojeda , Nick Desaulniers , Kent Overstreet Subject: [PATCH 01/32] Compiler Attributes: add __flatten Date: Tue, 9 May 2023 12:56:26 -0400 Message-Id: <20230509165657.1735798-2-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765436616777332598?= X-GMAIL-MSGID: =?utf-8?q?1765436616777332598?= From: Kent Overstreet This makes __attribute__((flatten)) available, which is used by bcachefs. Signed-off-by: Kent Overstreet Cc: Miguel Ojeda (maintainer:COMPILER ATTRIBUTES) Cc: Nick Desaulniers (reviewer:COMPILER ATTRIBUTES) Signed-off-by: Kent Overstreet --- include/linux/compiler_attributes.h | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/include/linux/compiler_attributes.h b/include/linux/compiler_attributes.h index e659cb6fde..e56793bc08 100644 --- a/include/linux/compiler_attributes.h +++ b/include/linux/compiler_attributes.h @@ -366,4 +366,9 @@ */ #define __fix_address noinline __noclone +/* + * gcc: https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html#index-flatten-function-attribute + */ +#define __flatten __attribute__((flatten)) + #endif /* __LINUX_COMPILER_ATTRIBUTES_H */ From patchwork Tue May 9 16:56:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91686 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3043999vqo; Tue, 9 May 2023 10:13:02 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7rzfLHLKJXyqIsHeINs/m/hOfpWs3M87jIA55HgQbrkbatb5LQfkWpao25N12h2U0ax3KK X-Received: by 2002:a05:6a20:258c:b0:f5:c6d5:adf5 with SMTP id k12-20020a056a20258c00b000f5c6d5adf5mr19186188pzd.40.1683652382318; Tue, 09 May 2023 10:13:02 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652382; cv=none; d=google.com; s=arc-20160816; b=gD0IqXADrMwTx2zi6AZM3OwHbTZ8KS+kkpsN+3/3USODxwZ6LQWivZrOOp5qNFcOYm w824pbIYoB3fLvZ7JgJPFnkGXk4MC3rdp2fwgZCNeGZBxHfLWD8AYh2lIlLo2x4+IEhc yC3orSitt9KHGaXSL+jKGAj4koOfUqsT1GA+EgzHx+Im0hL0apbCbLzezSJZQGkGDjfn Oq11IUdhktbjEzH4YxDUiai8I+MrGTVor1lChOEzbt1ho2wnhzME9OBZwLMzKacU+wSj f3YKmkNv9nrymmvccmsz49rOEGJ7wsx/VNLdpYAg9A3j0CUL5S0z0YMQ7bArsLGVrEeN Ku8w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ZW9aTms42T46/dSvQYESAhkOAQjGvYQLSdyAX7h+W6s=; b=I5WCK1xoLYsShKDSjbEjGT7U7bafeKjoPrcRgS4NUx45zD2KkJylL8edmG1bAl8Vz5 C/WVGDPum7AAn6rq5jEDbaoxhlNoFX9/SB+kGbu7y6Mn5f7/ZTXDgVaFBayB6K18+bqt lbEYJMNU/FBxMRTeT8lXcdw+0pdewMhCBFs5C4eRrwG8j+CpjzZyxMZZAm/cBBgmgmVQ V6pWciaOiqPbyHZ/Zg3g2RIhIbCn0pVr1BNCr+lztT8efoj0IjIEZIKPm9MF8NgWb90V 7qu0BCtPpjF9crZ1uMv8pietX2DMDsoIUUWvH5XRdPpsdj28xbTgswJH8rNEX3Jiyukf Y+3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=oPneXlUL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h70-20020a638349000000b0051ba2478f52si1908097pge.524.2023.05.09.10.12.49; Tue, 09 May 2023 10:13:02 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=oPneXlUL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229637AbjEIQ5Y (ORCPT + 99 others); Tue, 9 May 2023 12:57:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234588AbjEIQ5R (ORCPT ); Tue, 9 May 2023 12:57:17 -0400 Received: from out-46.mta1.migadu.com (out-46.mta1.migadu.com [95.215.58.46]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31E7B2126 for ; Tue, 9 May 2023 09:57:12 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651431; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZW9aTms42T46/dSvQYESAhkOAQjGvYQLSdyAX7h+W6s=; b=oPneXlUL5Qng7vxH10AO/hWFlZlWqRuLgPD4Xj67JKkDqbHoCKWj8RiBBZ2u0qdoMotROK 8MIYFSyqk4psf12fOel+CldcoNyYMIAcEhhPk6a/CXVe/Z2fCA404A36ou5D++qDPkcHzF AkOhN+4hig+zKd5aX1NbEdZ3+BlhFSY= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng Subject: [PATCH 02/32] locking/lockdep: lock_class_is_held() Date: Tue, 9 May 2023 12:56:27 -0400 Message-Id: <20230509165657.1735798-3-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437480400215127?= X-GMAIL-MSGID: =?utf-8?q?1765437480400215127?= From: Kent Overstreet This patch adds lock_class_is_held(), which can be used to assert that a particular type of lock is not held. Signed-off-by: Kent Overstreet Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng --- include/linux/lockdep.h | 4 ++++ kernel/locking/lockdep.c | 20 ++++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h index 1023f349af..e858c288c7 100644 --- a/include/linux/lockdep.h +++ b/include/linux/lockdep.h @@ -339,6 +339,8 @@ extern void lock_unpin_lock(struct lockdep_map *lock, struct pin_cookie); #define lockdep_repin_lock(l,c) lock_repin_lock(&(l)->dep_map, (c)) #define lockdep_unpin_lock(l,c) lock_unpin_lock(&(l)->dep_map, (c)) +int lock_class_is_held(struct lock_class_key *key); + #else /* !CONFIG_LOCKDEP */ static inline void lockdep_init_task(struct task_struct *task) @@ -427,6 +429,8 @@ extern int lockdep_is_held(const void *); #define lockdep_repin_lock(l, c) do { (void)(l); (void)(c); } while (0) #define lockdep_unpin_lock(l, c) do { (void)(l); (void)(c); } while (0) +static inline int lock_class_is_held(struct lock_class_key *key) { return 0; } + #endif /* !LOCKDEP */ enum xhlock_context_t { diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index 50d4863974..e631464070 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -6487,6 +6487,26 @@ void debug_check_no_locks_held(void) } EXPORT_SYMBOL_GPL(debug_check_no_locks_held); +#ifdef CONFIG_LOCKDEP +int lock_class_is_held(struct lock_class_key *key) +{ + struct task_struct *curr = current; + struct held_lock *hlock; + + if (unlikely(!debug_locks)) + return 0; + + for (hlock = curr->held_locks; + hlock < curr->held_locks + curr->lockdep_depth; + hlock++) + if (hlock->instance->key == key) + return 1; + + return 0; +} +EXPORT_SYMBOL_GPL(lock_class_is_held); +#endif + #ifdef __KERNEL__ void debug_show_all_locks(void) { From patchwork Tue May 9 16:56:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91691 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3046177vqo; Tue, 9 May 2023 10:16:35 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7QhMh94nH0iS1ZAevMh6LlrVSGa4L33w3xWfFjKT8jPUNOF9ieFC9nrxBOUjkiXEQTBNQH X-Received: by 2002:a05:6a00:a21:b0:62a:d752:acc5 with SMTP id p33-20020a056a000a2100b0062ad752acc5mr19972275pfh.32.1683652595519; Tue, 09 May 2023 10:16:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652595; cv=none; d=google.com; s=arc-20160816; b=rXw4o6GwXUOXly0lHc+zuStSKEIQS5WUrUTRkOiZNpTeZJjeaj9zLWBlNDk1ALXe/N WNmHizkhnG3Chrvs5YSv51dSeW9BDqfDj3Us6pEZ1xpxp7Io6CpokOYPO/JguynfGejv weKWouHMAc9luZmmMUdPqx6YbeceMTjsBggM7tSYWaAGgal5vKts4W2YgGzg8F/l9Pk7 uGMPZBdBqfPNOPn2Dm7ufW0aTTz65n/0AyGPH9juOO0SlmKUzVovbxBa7f5vLmUdc3a9 CBsqImv7qWZ33iMOvrY2DJOClIGLPe/aLR4Rp6gUeZyOpT5UyoTjvc00+kpBtdle8HIq KClg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=7HXn4Ai3HCc0Fe34KxBMMS4Cno5NneBVwyeI+DfOSZ4=; b=06DB4dgjkeNr1mAHDXm2Le2PuACu/tfTpvoKO9YtBWUmVb2y0GhIp8mdycVJFHdOvU YyF5s8XgR5Zsc8tY3Be88vae7UCg3ABLGwENkxANriyTO5/FchzDg5zR0ine4pkGhHb2 afA5wZkN+KNWJxdJRKfDk7B3EXjt33m62f+dYFxH9bR7cRGQoBhXn4VxTFeOxCwpzDtI t7MpP61SorN0RkfT870wgZWAD3bbXhrcc2dc+Ey3z8TZGe517G7CA3iiiasTOWmQU8+0 XrfSXHkp7wFJZtFEP8DNSk3YktdwFRfsZByAeDEdDlDBcZUrwYiSy8TBfkGDOzBq8wgL rlxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=aTfDXF9V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b191-20020a6334c8000000b005303b08eb16si2014984pga.631.2023.05.09.10.16.22; Tue, 09 May 2023 10:16:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=aTfDXF9V; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234836AbjEIQ50 (ORCPT + 99 others); Tue, 9 May 2023 12:57:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41438 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234704AbjEIQ5R (ORCPT ); Tue, 9 May 2023 12:57:17 -0400 Received: from out-30.mta1.migadu.com (out-30.mta1.migadu.com [IPv6:2001:41d0:203:375::1e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C94B449C for ; Tue, 9 May 2023 09:57:14 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651432; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7HXn4Ai3HCc0Fe34KxBMMS4Cno5NneBVwyeI+DfOSZ4=; b=aTfDXF9VG6rJ5VBr/J1ipGa3nA3Clu74t98FmSTVJUFgFARXPIpxAfp1Ovgk+6ZiaUeFjD bH5CLS4lIqmGRgG9Baix4APh2l4XUo2b6AeytH8ospRhzCz+fIjSQqKysb/odZULWx+TVC bs/XLATI6OhaENIAq7yonjl8pTvBAKc= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng Subject: [PATCH 03/32] locking/lockdep: lockdep_set_no_check_recursion() Date: Tue, 9 May 2023 12:56:28 -0400 Message-Id: <20230509165657.1735798-4-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437703729458418?= X-GMAIL-MSGID: =?utf-8?q?1765437703729458418?= This adds a method to tell lockdep not to check lock ordering within a lock class - but to still check lock ordering w.r.t. other lock types. This is for bcachefs, where for btree node locks we have our own deadlock avoidance strategy w.r.t. other btree node locks (cycle detection), but we still want lockdep to check lock ordering w.r.t. other lock types. Signed-off-by: Kent Overstreet Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng --- include/linux/lockdep.h | 6 ++++++ include/linux/lockdep_types.h | 2 +- kernel/locking/lockdep.c | 26 ++++++++++++++++++++++++++ 3 files changed, 33 insertions(+), 1 deletion(-) diff --git a/include/linux/lockdep.h b/include/linux/lockdep.h index e858c288c7..f6cc8709e2 100644 --- a/include/linux/lockdep.h +++ b/include/linux/lockdep.h @@ -665,4 +665,10 @@ lockdep_rcu_suspicious(const char *file, const int line, const char *s) } #endif +#ifdef CONFIG_DEBUG_LOCK_ALLOC +void lockdep_set_no_check_recursion(struct lockdep_map *map); +#else +static inline void lockdep_set_no_check_recursion(struct lockdep_map *map) {} +#endif + #endif /* __LINUX_LOCKDEP_H */ diff --git a/include/linux/lockdep_types.h b/include/linux/lockdep_types.h index d22430840b..506e769b4a 100644 --- a/include/linux/lockdep_types.h +++ b/include/linux/lockdep_types.h @@ -128,7 +128,7 @@ struct lock_class { u8 wait_type_inner; u8 wait_type_outer; u8 lock_type; - /* u8 hole; */ + u8 no_check_recursion; #ifdef CONFIG_LOCK_STAT unsigned long contention_point[LOCKSTAT_POINTS]; diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c index e631464070..f022b58dfa 100644 --- a/kernel/locking/lockdep.c +++ b/kernel/locking/lockdep.c @@ -3024,6 +3024,9 @@ check_deadlock(struct task_struct *curr, struct held_lock *next) if ((next->read == 2) && prev->read) continue; + if (hlock_class(next)->no_check_recursion) + continue; + /* * We're holding the nest_lock, which serializes this lock's * nesting behaviour. @@ -3085,6 +3088,10 @@ check_prev_add(struct task_struct *curr, struct held_lock *prev, return 2; } + if (hlock_class(prev) == hlock_class(next) && + hlock_class(prev)->no_check_recursion) + return 2; + /* * Prove that the new -> dependency would not * create a circular dependency in the graph. (We do this by @@ -6620,3 +6627,22 @@ void lockdep_rcu_suspicious(const char *file, const int line, const char *s) warn_rcu_exit(rcu); } EXPORT_SYMBOL_GPL(lockdep_rcu_suspicious); + +#ifdef CONFIG_DEBUG_LOCK_ALLOC +void lockdep_set_no_check_recursion(struct lockdep_map *lock) +{ + struct lock_class *class = lock->class_cache[0]; + unsigned long flags; + + raw_local_irq_save(flags); + lockdep_recursion_inc(); + + if (!class) + class = register_lock_class(lock, 0, 0); + if (class) + class->no_check_recursion = true; + lockdep_recursion_finish(); + raw_local_irq_restore(flags); +} +EXPORT_SYMBOL_GPL(lockdep_set_no_check_recursion); +#endif From patchwork Tue May 9 16:56:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91680 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3041271vqo; Tue, 9 May 2023 10:09:04 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4k11/UhgCvH02hHYS4xo2ddfHfAAqivQff9FiVO4WMiqwbk5W2ZPUb9AGAJPBEsDxhbDW+ X-Received: by 2002:a05:6a00:23d3:b0:63d:4752:4da3 with SMTP id g19-20020a056a0023d300b0063d47524da3mr18445129pfc.25.1683652143967; Tue, 09 May 2023 10:09:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652143; cv=none; d=google.com; s=arc-20160816; b=PBigjWIxA6S6yhTWfiV61nWWhTYXyYurgjLkTBXYr9CLJj5PuED1saq3mre8TTiuSP YH3Hlew7mIvQqXUVddDwP+Q+tjdVHS/hTlB8G4COs7bOoDy4QKGUSYgQ2NtS5mFR3uU0 sjnuwhyt4dMKGZKgg8MHih8kp+Le4m6NHg4OjfOoyGCnlk8NAHEh0jqwEUr5Z3u7ZOmn NkhMusI3y7N6OX+3Qll46ONYD6VqsEIh088MKBAqC8iUns9l8olrFRlmm63ngdeNYgLm bvvlJdZkXIABSDs+maI5jk6ToVpGo3P7vokwEbOvi6w5upGhH1bwzCgExNMf9BEl3Hkt VQYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=JCJ8O3gHGzWSTUHunad7bv+VVr2JiKI12XxqWHg6IAY=; b=RdT9Ecrbh/xrwLYegOU2OhtS7m4vObaQebuINTl0kG7swfsz1cTAynJRW7sR3OUxDe UdtAciRQFXGb3Eq2g4JEgAwjOZoRqqTsC4Gt5RLmZBL7d+N4P/c0d4xW7+Uu4AJOFO8y 1w53i4plZArg5g9AELPUTSEtfKzA7sWlZLnIMfF7BLcjn2GFQee/MlgV09YlFBFIvyLq 1KyQCE7UO55y+UrjyjmzqDTGIF81gPI+n8MlM1SvDSU21FLx+6nwIR+i73fJ5K6ThBW2 VIV4zZIKQO+sEy0+9W7pbmalRO7y9izL27B7Jt9YWRjEXvEyrvYXhyuz5MK0plgpl7Hk wYpA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=c6uoAbGw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e7-20020a056a0000c700b0063b7341250esi2818233pfj.277.2023.05.09.10.08.51; Tue, 09 May 2023 10:09:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=c6uoAbGw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234871AbjEIQ53 (ORCPT + 99 others); Tue, 9 May 2023 12:57:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234792AbjEIQ5U (ORCPT ); Tue, 9 May 2023 12:57:20 -0400 Received: from out-6.mta1.migadu.com (out-6.mta1.migadu.com [IPv6:2001:41d0:203:375::6]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E490549DF for ; Tue, 9 May 2023 09:57:15 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651433; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JCJ8O3gHGzWSTUHunad7bv+VVr2JiKI12XxqWHg6IAY=; b=c6uoAbGw+kIQtL2uJI+I0BOHRY0NnACngniDCwUtBCIYVVXafCcrPI0niIDSnuwEMW1R3s HSggkTdUBc1awkRf/U1VNummMQrRq655gzvRDPfzkJovY840lATyB0ZsFPjIRjem1OIeMP XQ5pFSjpLP7jVKkKUr5ZwagkfUZYhkA= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet , Peter Zijlstra , Ingo Molnar , Will Deacon , Waiman Long , Boqun Feng Subject: [PATCH 04/32] locking: SIX locks (shared/intent/exclusive) Date: Tue, 9 May 2023 12:56:29 -0400 Message-Id: <20230509165657.1735798-5-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437230164126990?= X-GMAIL-MSGID: =?utf-8?q?1765437230164126990?= From: Kent Overstreet New lock for bcachefs, like read/write locks but with a third state, intent. Intent locks conflict with each other, but not with read locks; taking a write lock requires first holding an intent lock. Signed-off-by: Kent Overstreet Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng --- include/linux/six.h | 210 +++++++++++ kernel/Kconfig.locks | 3 + kernel/locking/Makefile | 1 + kernel/locking/six.c | 779 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 993 insertions(+) create mode 100644 include/linux/six.h create mode 100644 kernel/locking/six.c diff --git a/include/linux/six.h b/include/linux/six.h new file mode 100644 index 0000000000..41ddf63b74 --- /dev/null +++ b/include/linux/six.h @@ -0,0 +1,210 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#ifndef _LINUX_SIX_H +#define _LINUX_SIX_H + +/* + * Shared/intent/exclusive locks: sleepable read/write locks, much like rw + * semaphores, except with a third intermediate state, intent. Basic operations + * are: + * + * six_lock_read(&foo->lock); + * six_unlock_read(&foo->lock); + * + * six_lock_intent(&foo->lock); + * six_unlock_intent(&foo->lock); + * + * six_lock_write(&foo->lock); + * six_unlock_write(&foo->lock); + * + * Intent locks block other intent locks, but do not block read locks, and you + * must have an intent lock held before taking a write lock, like so: + * + * six_lock_intent(&foo->lock); + * six_lock_write(&foo->lock); + * six_unlock_write(&foo->lock); + * six_unlock_intent(&foo->lock); + * + * Other operations: + * + * six_trylock_read() + * six_trylock_intent() + * six_trylock_write() + * + * six_lock_downgrade(): convert from intent to read + * six_lock_tryupgrade(): attempt to convert from read to intent + * + * Locks also embed a sequence number, which is incremented when the lock is + * locked or unlocked for write. The current sequence number can be grabbed + * while a lock is held from lock->state.seq; then, if you drop the lock you can + * use six_relock_(read|intent_write)(lock, seq) to attempt to retake the lock + * iff it hasn't been locked for write in the meantime. + * + * There are also operations that take the lock type as a parameter, where the + * type is one of SIX_LOCK_read, SIX_LOCK_intent, or SIX_LOCK_write: + * + * six_lock_type(lock, type) + * six_unlock_type(lock, type) + * six_relock(lock, type, seq) + * six_trylock_type(lock, type) + * six_trylock_convert(lock, from, to) + * + * A lock may be held multiple types by the same thread (for read or intent, + * not write). However, the six locks code does _not_ implement the actual + * recursive checks itself though - rather, if your code (e.g. btree iterator + * code) knows that the current thread already has a lock held, and for the + * correct type, six_lock_increment() may be used to bump up the counter for + * that type - the only effect is that one more call to unlock will be required + * before the lock is unlocked. + */ + +#include +#include +#include +#include + +#define SIX_LOCK_SEPARATE_LOCKFNS + +union six_lock_state { + struct { + atomic64_t counter; + }; + + struct { + u64 v; + }; + + struct { + /* for waitlist_bitnr() */ + unsigned long l; + }; + + struct { + unsigned read_lock:27; + unsigned write_locking:1; + unsigned intent_lock:1; + unsigned waiters:3; + /* + * seq works much like in seqlocks: it's incremented every time + * we lock and unlock for write. + * + * If it's odd write lock is held, even unlocked. + * + * Thus readers can unlock, and then lock again later iff it + * hasn't been modified in the meantime. + */ + u32 seq; + }; +}; + +enum six_lock_type { + SIX_LOCK_read, + SIX_LOCK_intent, + SIX_LOCK_write, +}; + +struct six_lock { + union six_lock_state state; + unsigned intent_lock_recurse; + struct task_struct *owner; + struct optimistic_spin_queue osq; + unsigned __percpu *readers; + + raw_spinlock_t wait_lock; + struct list_head wait_list[2]; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +}; + +typedef int (*six_lock_should_sleep_fn)(struct six_lock *lock, void *); + +static __always_inline void __six_lock_init(struct six_lock *lock, + const char *name, + struct lock_class_key *key) +{ + atomic64_set(&lock->state.counter, 0); + raw_spin_lock_init(&lock->wait_lock); + INIT_LIST_HEAD(&lock->wait_list[SIX_LOCK_read]); + INIT_LIST_HEAD(&lock->wait_list[SIX_LOCK_intent]); +#ifdef CONFIG_DEBUG_LOCK_ALLOC + debug_check_no_locks_freed((void *) lock, sizeof(*lock)); + lockdep_init_map(&lock->dep_map, name, key, 0); +#endif +} + +#define six_lock_init(lock) \ +do { \ + static struct lock_class_key __key; \ + \ + __six_lock_init((lock), #lock, &__key); \ +} while (0) + +#define __SIX_VAL(field, _v) (((union six_lock_state) { .field = _v }).v) + +#define __SIX_LOCK(type) \ +bool six_trylock_##type(struct six_lock *); \ +bool six_relock_##type(struct six_lock *, u32); \ +int six_lock_##type(struct six_lock *, six_lock_should_sleep_fn, void *);\ +void six_unlock_##type(struct six_lock *); + +__SIX_LOCK(read) +__SIX_LOCK(intent) +__SIX_LOCK(write) +#undef __SIX_LOCK + +#define SIX_LOCK_DISPATCH(type, fn, ...) \ + switch (type) { \ + case SIX_LOCK_read: \ + return fn##_read(__VA_ARGS__); \ + case SIX_LOCK_intent: \ + return fn##_intent(__VA_ARGS__); \ + case SIX_LOCK_write: \ + return fn##_write(__VA_ARGS__); \ + default: \ + BUG(); \ + } + +static inline bool six_trylock_type(struct six_lock *lock, enum six_lock_type type) +{ + SIX_LOCK_DISPATCH(type, six_trylock, lock); +} + +static inline bool six_relock_type(struct six_lock *lock, enum six_lock_type type, + unsigned seq) +{ + SIX_LOCK_DISPATCH(type, six_relock, lock, seq); +} + +static inline int six_lock_type(struct six_lock *lock, enum six_lock_type type, + six_lock_should_sleep_fn should_sleep_fn, void *p) +{ + SIX_LOCK_DISPATCH(type, six_lock, lock, should_sleep_fn, p); +} + +static inline void six_unlock_type(struct six_lock *lock, enum six_lock_type type) +{ + SIX_LOCK_DISPATCH(type, six_unlock, lock); +} + +void six_lock_downgrade(struct six_lock *); +bool six_lock_tryupgrade(struct six_lock *); +bool six_trylock_convert(struct six_lock *, enum six_lock_type, + enum six_lock_type); + +void six_lock_increment(struct six_lock *, enum six_lock_type); + +void six_lock_wakeup_all(struct six_lock *); + +void six_lock_pcpu_free_rcu(struct six_lock *); +void six_lock_pcpu_free(struct six_lock *); +void six_lock_pcpu_alloc(struct six_lock *); + +struct six_lock_count { + unsigned read; + unsigned intent; +}; + +struct six_lock_count six_lock_counts(struct six_lock *); + +#endif /* _LINUX_SIX_H */ diff --git a/kernel/Kconfig.locks b/kernel/Kconfig.locks index 4198f0273e..b2abd9a5d9 100644 --- a/kernel/Kconfig.locks +++ b/kernel/Kconfig.locks @@ -259,3 +259,6 @@ config ARCH_HAS_MMIOWB config MMIOWB def_bool y if ARCH_HAS_MMIOWB depends on SMP + +config SIXLOCKS + bool diff --git a/kernel/locking/Makefile b/kernel/locking/Makefile index 0db4093d17..a095dbbf01 100644 --- a/kernel/locking/Makefile +++ b/kernel/locking/Makefile @@ -32,3 +32,4 @@ obj-$(CONFIG_QUEUED_RWLOCKS) += qrwlock.o obj-$(CONFIG_LOCK_TORTURE_TEST) += locktorture.o obj-$(CONFIG_WW_MUTEX_SELFTEST) += test-ww_mutex.o obj-$(CONFIG_LOCK_EVENT_COUNTS) += lock_events.o +obj-$(CONFIG_SIXLOCKS) += six.o diff --git a/kernel/locking/six.c b/kernel/locking/six.c new file mode 100644 index 0000000000..5b2d92c6e9 --- /dev/null +++ b/kernel/locking/six.c @@ -0,0 +1,779 @@ +// SPDX-License-Identifier: GPL-2.0 + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#ifdef DEBUG +#define EBUG_ON(cond) BUG_ON(cond) +#else +#define EBUG_ON(cond) do {} while (0) +#endif + +#define six_acquire(l, t) lock_acquire(l, 0, t, 0, 0, NULL, _RET_IP_) +#define six_release(l) lock_release(l, _RET_IP_) + +struct six_lock_vals { + /* Value we add to the lock in order to take the lock: */ + u64 lock_val; + + /* If the lock has this value (used as a mask), taking the lock fails: */ + u64 lock_fail; + + /* Value we add to the lock in order to release the lock: */ + u64 unlock_val; + + /* Mask that indicates lock is held for this type: */ + u64 held_mask; + + /* Waitlist we wakeup when releasing the lock: */ + enum six_lock_type unlock_wakeup; +}; + +#define __SIX_LOCK_HELD_read __SIX_VAL(read_lock, ~0) +#define __SIX_LOCK_HELD_intent __SIX_VAL(intent_lock, ~0) +#define __SIX_LOCK_HELD_write __SIX_VAL(seq, 1) + +#define LOCK_VALS { \ + [SIX_LOCK_read] = { \ + .lock_val = __SIX_VAL(read_lock, 1), \ + .lock_fail = __SIX_LOCK_HELD_write + __SIX_VAL(write_locking, 1),\ + .unlock_val = -__SIX_VAL(read_lock, 1), \ + .held_mask = __SIX_LOCK_HELD_read, \ + .unlock_wakeup = SIX_LOCK_write, \ + }, \ + [SIX_LOCK_intent] = { \ + .lock_val = __SIX_VAL(intent_lock, 1), \ + .lock_fail = __SIX_LOCK_HELD_intent, \ + .unlock_val = -__SIX_VAL(intent_lock, 1), \ + .held_mask = __SIX_LOCK_HELD_intent, \ + .unlock_wakeup = SIX_LOCK_intent, \ + }, \ + [SIX_LOCK_write] = { \ + .lock_val = __SIX_VAL(seq, 1), \ + .lock_fail = __SIX_LOCK_HELD_read, \ + .unlock_val = __SIX_VAL(seq, 1), \ + .held_mask = __SIX_LOCK_HELD_write, \ + .unlock_wakeup = SIX_LOCK_read, \ + }, \ +} + +static inline void six_set_owner(struct six_lock *lock, enum six_lock_type type, + union six_lock_state old) +{ + if (type != SIX_LOCK_intent) + return; + + if (!old.intent_lock) { + EBUG_ON(lock->owner); + lock->owner = current; + } else { + EBUG_ON(lock->owner != current); + } +} + +static inline unsigned pcpu_read_count(struct six_lock *lock) +{ + unsigned read_count = 0; + int cpu; + + for_each_possible_cpu(cpu) + read_count += *per_cpu_ptr(lock->readers, cpu); + return read_count; +} + +struct six_lock_waiter { + struct list_head list; + struct task_struct *task; +}; + +/* This is probably up there with the more evil things I've done */ +#define waitlist_bitnr(id) ilog2((((union six_lock_state) { .waiters = 1 << (id) }).l)) + +static inline void six_lock_wakeup(struct six_lock *lock, + union six_lock_state state, + unsigned waitlist_id) +{ + if (waitlist_id == SIX_LOCK_write) { + if (state.write_locking && !state.read_lock) { + struct task_struct *p = READ_ONCE(lock->owner); + if (p) + wake_up_process(p); + } + } else { + struct list_head *wait_list = &lock->wait_list[waitlist_id]; + struct six_lock_waiter *w, *next; + + if (!(state.waiters & (1 << waitlist_id))) + return; + + clear_bit(waitlist_bitnr(waitlist_id), + (unsigned long *) &lock->state.v); + + raw_spin_lock(&lock->wait_lock); + + list_for_each_entry_safe(w, next, wait_list, list) { + list_del_init(&w->list); + + if (wake_up_process(w->task) && + waitlist_id != SIX_LOCK_read) { + if (!list_empty(wait_list)) + set_bit(waitlist_bitnr(waitlist_id), + (unsigned long *) &lock->state.v); + break; + } + } + + raw_spin_unlock(&lock->wait_lock); + } +} + +static __always_inline bool do_six_trylock_type(struct six_lock *lock, + enum six_lock_type type, + bool try) +{ + const struct six_lock_vals l[] = LOCK_VALS; + union six_lock_state old, new; + bool ret; + u64 v; + + EBUG_ON(type == SIX_LOCK_write && lock->owner != current); + EBUG_ON(type == SIX_LOCK_write && (lock->state.seq & 1)); + + EBUG_ON(type == SIX_LOCK_write && (try != !(lock->state.write_locking))); + + /* + * Percpu reader mode: + * + * The basic idea behind this algorithm is that you can implement a lock + * between two threads without any atomics, just memory barriers: + * + * For two threads you'll need two variables, one variable for "thread a + * has the lock" and another for "thread b has the lock". + * + * To take the lock, a thread sets its variable indicating that it holds + * the lock, then issues a full memory barrier, then reads from the + * other thread's variable to check if the other thread thinks it has + * the lock. If we raced, we backoff and retry/sleep. + */ + + if (type == SIX_LOCK_read && lock->readers) { +retry: + preempt_disable(); + this_cpu_inc(*lock->readers); /* signal that we own lock */ + + smp_mb(); + + old.v = READ_ONCE(lock->state.v); + ret = !(old.v & l[type].lock_fail); + + this_cpu_sub(*lock->readers, !ret); + preempt_enable(); + + /* + * If we failed because a writer was trying to take the + * lock, issue a wakeup because we might have caused a + * spurious trylock failure: + */ + if (old.write_locking) { + struct task_struct *p = READ_ONCE(lock->owner); + + if (p) + wake_up_process(p); + } + + /* + * If we failed from the lock path and the waiting bit wasn't + * set, set it: + */ + if (!try && !ret) { + v = old.v; + + do { + new.v = old.v = v; + + if (!(old.v & l[type].lock_fail)) + goto retry; + + if (new.waiters & (1 << type)) + break; + + new.waiters |= 1 << type; + } while ((v = atomic64_cmpxchg(&lock->state.counter, + old.v, new.v)) != old.v); + } + } else if (type == SIX_LOCK_write && lock->readers) { + if (try) { + atomic64_add(__SIX_VAL(write_locking, 1), + &lock->state.counter); + smp_mb__after_atomic(); + } + + ret = !pcpu_read_count(lock); + + /* + * On success, we increment lock->seq; also we clear + * write_locking unless we failed from the lock path: + */ + v = 0; + if (ret) + v += __SIX_VAL(seq, 1); + if (ret || try) + v -= __SIX_VAL(write_locking, 1); + + if (try && !ret) { + old.v = atomic64_add_return(v, &lock->state.counter); + six_lock_wakeup(lock, old, SIX_LOCK_read); + } else { + atomic64_add(v, &lock->state.counter); + } + } else { + v = READ_ONCE(lock->state.v); + do { + new.v = old.v = v; + + if (!(old.v & l[type].lock_fail)) { + new.v += l[type].lock_val; + + if (type == SIX_LOCK_write) + new.write_locking = 0; + } else if (!try && type != SIX_LOCK_write && + !(new.waiters & (1 << type))) + new.waiters |= 1 << type; + else + break; /* waiting bit already set */ + } while ((v = atomic64_cmpxchg_acquire(&lock->state.counter, + old.v, new.v)) != old.v); + + ret = !(old.v & l[type].lock_fail); + + EBUG_ON(ret && !(lock->state.v & l[type].held_mask)); + } + + if (ret) + six_set_owner(lock, type, old); + + EBUG_ON(type == SIX_LOCK_write && (try || ret) && (lock->state.write_locking)); + + return ret; +} + +__always_inline __flatten +static bool __six_trylock_type(struct six_lock *lock, enum six_lock_type type) +{ + if (!do_six_trylock_type(lock, type, true)) + return false; + + if (type != SIX_LOCK_write) + six_acquire(&lock->dep_map, 1); + return true; +} + +__always_inline __flatten +static bool __six_relock_type(struct six_lock *lock, enum six_lock_type type, + unsigned seq) +{ + const struct six_lock_vals l[] = LOCK_VALS; + union six_lock_state old; + u64 v; + + EBUG_ON(type == SIX_LOCK_write); + + if (type == SIX_LOCK_read && + lock->readers) { + bool ret; + + preempt_disable(); + this_cpu_inc(*lock->readers); + + smp_mb(); + + old.v = READ_ONCE(lock->state.v); + ret = !(old.v & l[type].lock_fail) && old.seq == seq; + + this_cpu_sub(*lock->readers, !ret); + preempt_enable(); + + /* + * Similar to the lock path, we may have caused a spurious write + * lock fail and need to issue a wakeup: + */ + if (old.write_locking) { + struct task_struct *p = READ_ONCE(lock->owner); + + if (p) + wake_up_process(p); + } + + if (ret) + six_acquire(&lock->dep_map, 1); + + return ret; + } + + v = READ_ONCE(lock->state.v); + do { + old.v = v; + + if (old.seq != seq || old.v & l[type].lock_fail) + return false; + } while ((v = atomic64_cmpxchg_acquire(&lock->state.counter, + old.v, + old.v + l[type].lock_val)) != old.v); + + six_set_owner(lock, type, old); + if (type != SIX_LOCK_write) + six_acquire(&lock->dep_map, 1); + return true; +} + +#ifdef CONFIG_LOCK_SPIN_ON_OWNER + +static inline int six_can_spin_on_owner(struct six_lock *lock) +{ + struct task_struct *owner; + int retval = 1; + + if (need_resched()) + return 0; + + rcu_read_lock(); + owner = READ_ONCE(lock->owner); + if (owner) + retval = owner->on_cpu; + rcu_read_unlock(); + /* + * if lock->owner is not set, the mutex owner may have just acquired + * it and not set the owner yet or the mutex has been released. + */ + return retval; +} + +static inline bool six_spin_on_owner(struct six_lock *lock, + struct task_struct *owner) +{ + bool ret = true; + + rcu_read_lock(); + while (lock->owner == owner) { + /* + * Ensure we emit the owner->on_cpu, dereference _after_ + * checking lock->owner still matches owner. If that fails, + * owner might point to freed memory. If it still matches, + * the rcu_read_lock() ensures the memory stays valid. + */ + barrier(); + + if (!owner->on_cpu || need_resched()) { + ret = false; + break; + } + + cpu_relax(); + } + rcu_read_unlock(); + + return ret; +} + +static inline bool six_optimistic_spin(struct six_lock *lock, enum six_lock_type type) +{ + struct task_struct *task = current; + + if (type == SIX_LOCK_write) + return false; + + preempt_disable(); + if (!six_can_spin_on_owner(lock)) + goto fail; + + if (!osq_lock(&lock->osq)) + goto fail; + + while (1) { + struct task_struct *owner; + + /* + * If there's an owner, wait for it to either + * release the lock or go to sleep. + */ + owner = READ_ONCE(lock->owner); + if (owner && !six_spin_on_owner(lock, owner)) + break; + + if (do_six_trylock_type(lock, type, false)) { + osq_unlock(&lock->osq); + preempt_enable(); + return true; + } + + /* + * When there's no owner, we might have preempted between the + * owner acquiring the lock and setting the owner field. If + * we're an RT task that will live-lock because we won't let + * the owner complete. + */ + if (!owner && (need_resched() || rt_task(task))) + break; + + /* + * The cpu_relax() call is a compiler barrier which forces + * everything in this loop to be re-loaded. We don't need + * memory barriers as we'll eventually observe the right + * values at the cost of a few extra spins. + */ + cpu_relax(); + } + + osq_unlock(&lock->osq); +fail: + preempt_enable(); + + /* + * If we fell out of the spin path because of need_resched(), + * reschedule now, before we try-lock again. This avoids getting + * scheduled out right after we obtained the lock. + */ + if (need_resched()) + schedule(); + + return false; +} + +#else /* CONFIG_LOCK_SPIN_ON_OWNER */ + +static inline bool six_optimistic_spin(struct six_lock *lock, enum six_lock_type type) +{ + return false; +} + +#endif + +noinline +static int __six_lock_type_slowpath(struct six_lock *lock, enum six_lock_type type, + six_lock_should_sleep_fn should_sleep_fn, void *p) +{ + union six_lock_state old; + struct six_lock_waiter wait; + int ret = 0; + + if (type == SIX_LOCK_write) { + EBUG_ON(lock->state.write_locking); + atomic64_add(__SIX_VAL(write_locking, 1), &lock->state.counter); + smp_mb__after_atomic(); + } + + ret = should_sleep_fn ? should_sleep_fn(lock, p) : 0; + if (ret) + goto out_before_sleep; + + if (six_optimistic_spin(lock, type)) + goto out_before_sleep; + + lock_contended(&lock->dep_map, _RET_IP_); + + INIT_LIST_HEAD(&wait.list); + wait.task = current; + + while (1) { + set_current_state(TASK_UNINTERRUPTIBLE); + if (type == SIX_LOCK_write) + EBUG_ON(lock->owner != current); + else if (list_empty_careful(&wait.list)) { + raw_spin_lock(&lock->wait_lock); + list_add_tail(&wait.list, &lock->wait_list[type]); + raw_spin_unlock(&lock->wait_lock); + } + + if (do_six_trylock_type(lock, type, false)) + break; + + ret = should_sleep_fn ? should_sleep_fn(lock, p) : 0; + if (ret) + break; + + schedule(); + } + + __set_current_state(TASK_RUNNING); + + if (!list_empty_careful(&wait.list)) { + raw_spin_lock(&lock->wait_lock); + list_del_init(&wait.list); + raw_spin_unlock(&lock->wait_lock); + } +out_before_sleep: + if (ret && type == SIX_LOCK_write) { + old.v = atomic64_sub_return(__SIX_VAL(write_locking, 1), + &lock->state.counter); + six_lock_wakeup(lock, old, SIX_LOCK_read); + } + + return ret; +} + +__always_inline +static int __six_lock_type(struct six_lock *lock, enum six_lock_type type, + six_lock_should_sleep_fn should_sleep_fn, void *p) +{ + int ret; + + if (type != SIX_LOCK_write) + six_acquire(&lock->dep_map, 0); + + ret = do_six_trylock_type(lock, type, true) ? 0 + : __six_lock_type_slowpath(lock, type, should_sleep_fn, p); + + if (ret && type != SIX_LOCK_write) + six_release(&lock->dep_map); + if (!ret) + lock_acquired(&lock->dep_map, _RET_IP_); + + return ret; +} + +__always_inline __flatten +static void __six_unlock_type(struct six_lock *lock, enum six_lock_type type) +{ + const struct six_lock_vals l[] = LOCK_VALS; + union six_lock_state state; + + EBUG_ON(type == SIX_LOCK_write && + !(lock->state.v & __SIX_LOCK_HELD_intent)); + + if (type != SIX_LOCK_write) + six_release(&lock->dep_map); + + if (type == SIX_LOCK_intent) { + EBUG_ON(lock->owner != current); + + if (lock->intent_lock_recurse) { + --lock->intent_lock_recurse; + return; + } + + lock->owner = NULL; + } + + if (type == SIX_LOCK_read && + lock->readers) { + smp_mb(); /* unlock barrier */ + this_cpu_dec(*lock->readers); + smp_mb(); /* between unlocking and checking for waiters */ + state.v = READ_ONCE(lock->state.v); + } else { + EBUG_ON(!(lock->state.v & l[type].held_mask)); + state.v = atomic64_add_return_release(l[type].unlock_val, + &lock->state.counter); + } + + six_lock_wakeup(lock, state, l[type].unlock_wakeup); +} + +#define __SIX_LOCK(type) \ +bool six_trylock_##type(struct six_lock *lock) \ +{ \ + return __six_trylock_type(lock, SIX_LOCK_##type); \ +} \ +EXPORT_SYMBOL_GPL(six_trylock_##type); \ + \ +bool six_relock_##type(struct six_lock *lock, u32 seq) \ +{ \ + return __six_relock_type(lock, SIX_LOCK_##type, seq); \ +} \ +EXPORT_SYMBOL_GPL(six_relock_##type); \ + \ +int six_lock_##type(struct six_lock *lock, \ + six_lock_should_sleep_fn should_sleep_fn, void *p) \ +{ \ + return __six_lock_type(lock, SIX_LOCK_##type, should_sleep_fn, p);\ +} \ +EXPORT_SYMBOL_GPL(six_lock_##type); \ + \ +void six_unlock_##type(struct six_lock *lock) \ +{ \ + __six_unlock_type(lock, SIX_LOCK_##type); \ +} \ +EXPORT_SYMBOL_GPL(six_unlock_##type); + +__SIX_LOCK(read) +__SIX_LOCK(intent) +__SIX_LOCK(write) + +#undef __SIX_LOCK + +/* Convert from intent to read: */ +void six_lock_downgrade(struct six_lock *lock) +{ + six_lock_increment(lock, SIX_LOCK_read); + six_unlock_intent(lock); +} +EXPORT_SYMBOL_GPL(six_lock_downgrade); + +bool six_lock_tryupgrade(struct six_lock *lock) +{ + union six_lock_state old, new; + u64 v = READ_ONCE(lock->state.v); + + do { + new.v = old.v = v; + + if (new.intent_lock) + return false; + + if (!lock->readers) { + EBUG_ON(!new.read_lock); + new.read_lock--; + } + + new.intent_lock = 1; + } while ((v = atomic64_cmpxchg_acquire(&lock->state.counter, + old.v, new.v)) != old.v); + + if (lock->readers) + this_cpu_dec(*lock->readers); + + six_set_owner(lock, SIX_LOCK_intent, old); + + return true; +} +EXPORT_SYMBOL_GPL(six_lock_tryupgrade); + +bool six_trylock_convert(struct six_lock *lock, + enum six_lock_type from, + enum six_lock_type to) +{ + EBUG_ON(to == SIX_LOCK_write || from == SIX_LOCK_write); + + if (to == from) + return true; + + if (to == SIX_LOCK_read) { + six_lock_downgrade(lock); + return true; + } else { + return six_lock_tryupgrade(lock); + } +} +EXPORT_SYMBOL_GPL(six_trylock_convert); + +/* + * Increment read/intent lock count, assuming we already have it read or intent + * locked: + */ +void six_lock_increment(struct six_lock *lock, enum six_lock_type type) +{ + const struct six_lock_vals l[] = LOCK_VALS; + + six_acquire(&lock->dep_map, 0); + + /* XXX: assert already locked, and that we don't overflow: */ + + switch (type) { + case SIX_LOCK_read: + if (lock->readers) { + this_cpu_inc(*lock->readers); + } else { + EBUG_ON(!lock->state.read_lock && + !lock->state.intent_lock); + atomic64_add(l[type].lock_val, &lock->state.counter); + } + break; + case SIX_LOCK_intent: + EBUG_ON(!lock->state.intent_lock); + lock->intent_lock_recurse++; + break; + case SIX_LOCK_write: + BUG(); + break; + } +} +EXPORT_SYMBOL_GPL(six_lock_increment); + +void six_lock_wakeup_all(struct six_lock *lock) +{ + struct six_lock_waiter *w; + + raw_spin_lock(&lock->wait_lock); + + list_for_each_entry(w, &lock->wait_list[0], list) + wake_up_process(w->task); + list_for_each_entry(w, &lock->wait_list[1], list) + wake_up_process(w->task); + + raw_spin_unlock(&lock->wait_lock); +} +EXPORT_SYMBOL_GPL(six_lock_wakeup_all); + +struct free_pcpu_rcu { + struct rcu_head rcu; + void __percpu *p; +}; + +static void free_pcpu_rcu_fn(struct rcu_head *_rcu) +{ + struct free_pcpu_rcu *rcu = + container_of(_rcu, struct free_pcpu_rcu, rcu); + + free_percpu(rcu->p); + kfree(rcu); +} + +void six_lock_pcpu_free_rcu(struct six_lock *lock) +{ + struct free_pcpu_rcu *rcu = kzalloc(sizeof(*rcu), GFP_KERNEL); + + if (!rcu) + return; + + rcu->p = lock->readers; + lock->readers = NULL; + + call_rcu(&rcu->rcu, free_pcpu_rcu_fn); +} +EXPORT_SYMBOL_GPL(six_lock_pcpu_free_rcu); + +void six_lock_pcpu_free(struct six_lock *lock) +{ + BUG_ON(lock->readers && pcpu_read_count(lock)); + BUG_ON(lock->state.read_lock); + + free_percpu(lock->readers); + lock->readers = NULL; +} +EXPORT_SYMBOL_GPL(six_lock_pcpu_free); + +void six_lock_pcpu_alloc(struct six_lock *lock) +{ +#ifdef __KERNEL__ + if (!lock->readers) + lock->readers = alloc_percpu(unsigned); +#endif +} +EXPORT_SYMBOL_GPL(six_lock_pcpu_alloc); + +/* + * Returns lock held counts, for both read and intent + */ +struct six_lock_count six_lock_counts(struct six_lock *lock) +{ + struct six_lock_count ret = { 0, lock->state.intent_lock }; + + if (!lock->readers) + ret.read += lock->state.read_lock; + else { + int cpu; + + for_each_possible_cpu(cpu) + ret.read += *per_cpu_ptr(lock->readers, cpu); + } + + return ret; +} +EXPORT_SYMBOL_GPL(six_lock_counts); From patchwork Tue May 9 16:56:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91674 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3040569vqo; Tue, 9 May 2023 10:08:06 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7fdiRFLFiW+VwxWff1bnTq91d66GJW+QVCUD8sj230EXPLEXyfgU2YVeE67xDKjHalHb+K X-Received: by 2002:a05:6a20:12cb:b0:f0:dedb:83b8 with SMTP id v11-20020a056a2012cb00b000f0dedb83b8mr3152098pzg.19.1683652086071; Tue, 09 May 2023 10:08:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652086; cv=none; d=google.com; s=arc-20160816; b=AwaHaLC0P7MZXpZhgZtiDlhX9ZrsUg8Y4R8s8GYUR+dASoBkYYRVrStpa3NjbhzOcf 9r2bQcAlr/s8RhqeHwbApXObG1O0pb7grA1Wi1FGhYtnHes4kLmAGJKltLCC8pvqQgMP t0D988XrGVijdoYjeFWBvRNN1HM/pVkEpl91a4tJagGyjce6Rdpj25p6Tq3/5iiUgPKC AShgTXr1Wo+Zim3pO2a7Kpjaurd27w33qtv6jpTi9GZkZvpuYyenm2/AtKxngVrR8kJl ogKWVjDpm6R6Stndo+G951J4SoJikwqZKJQfB6GDJai2L17Hgig+UHxIS3gV65FIcCQB F0jg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NMx7O3Ijlmo8XgCqd75OZDA70sC2El+hm/txmwADigM=; b=KVM7yEFiY5tdfgMCmlzeEEzQqTPRePWc4f2gttSnEMFJINi7ZJoX4qeTr5ppY1F69L kJylWK4F9SqjbBCgnmRyocbjh8z46SHdvUvwtkQ0zsHdj2Q8XKYTxXOSGVsh8xViWL2D fbTeA+hh56MpVemk5IriGrkWj9OcAvlgZkjGe8OQIbKXu0Mm3HxcP0mjrsWyKeDAd7so JrxXXiDWENYIlhMlfG/iEpkC9flwEjZsF0cvHOfZWrB8ow24OMmZ7QQ8pn47pcX3pRqO 7xYJ0vnoOW+P+w6Z32gBvbpSELPF2gkfQl4E+iK5uwfBYaY5ion+qkyFaV+hBwZQ7YOW rikg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=QOYjXAVn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a6-20020a631a46000000b00524ca1b89f0si1798667pgm.596.2023.05.09.10.07.53; Tue, 09 May 2023 10:08:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=QOYjXAVn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234596AbjEIQ5e (ORCPT + 99 others); Tue, 9 May 2023 12:57:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234808AbjEIQ5Y (ORCPT ); Tue, 9 May 2023 12:57:24 -0400 Received: from out-42.mta1.migadu.com (out-42.mta1.migadu.com [95.215.58.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B2344EC5 for ; Tue, 9 May 2023 09:57:16 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651434; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NMx7O3Ijlmo8XgCqd75OZDA70sC2El+hm/txmwADigM=; b=QOYjXAVnD0h4G8wOSjSoV+jU0bZjhj4Su/rVUvjNuHn/D49MiqA89G3KDobfUgHt8sj7B9 PAMKYmGN4x6g7SBnVXrc5Mtk32EXr7Hnp6g5a3d5LyIzgInLfBbJrvUK+XMaSBKOYKMe7n WYd+zugQph5ei1C/uPydSbPvSrp7EA0= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 05/32] MAINTAINERS: Add entry for six locks Date: Tue, 9 May 2023 12:56:30 -0400 Message-Id: <20230509165657.1735798-6-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437169509978619?= X-GMAIL-MSGID: =?utf-8?q?1765437169509978619?= SIX locks are a new locking primitive, shared/intent/exclusive, currently used by bcachefs but available for other uses. Mark them as maintained. Signed-off-by: Kent Overstreet --- MAINTAINERS | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index c6545eb541..3fc37de3d6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -19166,6 +19166,14 @@ S: Maintained W: http://www.winischhofer.at/linuxsisusbvga.shtml F: drivers/usb/misc/sisusbvga/ +SIX LOCKS +M: Kent Overstreet +L: linux-bcachefs@vger.kernel.org +S: Supported +C: irc://irc.oftc.net/bcache +F: include/linux/six.h +F: kernel/locking/six.c + SL28 CPLD MFD DRIVER M: Michael Walle S: Maintained From patchwork Tue May 9 16:56:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91685 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3043777vqo; Tue, 9 May 2023 10:12:43 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6Q3myEdJT4Ia9CsedVTJlAPs5xrz0dbkvfNxoeMJP0a0gAH5ztCBgY6yGw9tBeLaVScTwM X-Received: by 2002:a05:6a00:cd5:b0:646:f971:b179 with SMTP id b21-20020a056a000cd500b00646f971b179mr4389819pfv.16.1683652362906; Tue, 09 May 2023 10:12:42 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652362; cv=none; d=google.com; s=arc-20160816; b=vjmSoWAQrK24jSiFJqFpnikuUjhmZaYTcGhfKiK01Ge61V84lJsVtfGT8XtPzOhmyf y2SnYh4SoQeKKOncC6mOf32K0SdsegtBJ4maSDttC2M0ueTbMOeT5UI8sjnoaCYyHyow xSIgxAPrPE9MzvmsI7mKpvRNCNQKkgYyg528e/d5/2RY3WKAbxP8LFqfi5SNyMKu+F9c de5vFyq+0msUQGvLUkUqfnl21YucYDYw7p5lck+XrZz3/84oOienFbKaNUrVYC08TO/n nC5KGzarBd0tXFc8DGzpCBLiPL6cGps99FzG3UJBwn0CbxLAw5d8NI9GMGr3vjuh3stW rczA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=P5i0UV4XUiq/O8euXN5wZ8WzR/MP84BiMxFHt/1Czt4=; b=e8ggl33AACoYVePa0s88qRhV9zo7H+mcSiSmpfOV8oZPhzRaRKVLXDx9mSxajxc5aO XV+9fSpE84gJaiZTn+QbGD0pchkh4TMNYgRQawXBj71oJRdTawl4Os1sHvqQFP5NnsOX lJCWlJcNggPWQNu7T2tDn30sls3CEQxg9NPUztwQF67aUMZEf2yDSFqINfOqXGM0yzrH NO6KArnkEXYhZV55o98pxVOhmgR2QJ8YYT7oqWZQXHrnKDsa452AmDsdBnqpScuD9nZj 83vOz3E8Nxpm28RPUY13G5EkMwuoHPi0mA1L2wadc11j0BO+c1+R/TiZCozwgI+puLkc IYTw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=iW5mq41x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c63-20020a621c42000000b0063b8f0a6f51si2910110pfc.117.2023.05.09.10.12.28; Tue, 09 May 2023 10:12:42 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=iW5mq41x; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234908AbjEIQ5v (ORCPT + 99 others); Tue, 9 May 2023 12:57:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41976 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234837AbjEIQ50 (ORCPT ); Tue, 9 May 2023 12:57:26 -0400 Received: from out-61.mta1.migadu.com (out-61.mta1.migadu.com [95.215.58.61]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A5A92716 for ; Tue, 9 May 2023 09:57:17 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651435; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P5i0UV4XUiq/O8euXN5wZ8WzR/MP84BiMxFHt/1Czt4=; b=iW5mq41x5yiJn7gfvLTGaqQQdNpqEW8JmuzeTOK8Sfw4u+jcMap56jXivpM0QqNJ8IMBqF 3re0jfAERex9kaSEuHDeD0891yfxuIr+YxKFkBDI5xLP7ONOmVUqy+vtWc7eEx3mg/+NBS BLTetoZLlSSytF0SvKiLN6RhTyaI7o8= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet , Jan Kara , "Darrick J . Wong" Subject: [PATCH 06/32] sched: Add task_struct->faults_disabled_mapping Date: Tue, 9 May 2023 12:56:31 -0400 Message-Id: <20230509165657.1735798-7-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437460163050516?= X-GMAIL-MSGID: =?utf-8?q?1765437460163050516?= From: Kent Overstreet This is used by bcachefs to fix a page cache coherency issue with O_DIRECT writes. Also relevant: mapping->invalidate_lock, see below. O_DIRECT writes (and other filesystem operations that modify file data while bypassing the page cache) need to shoot down ranges of the page cache - and additionally, need locking to prevent those pages from pulled back in. But O_DIRECT writes invoke the page fault handler (via get_user_pages), and the page fault handler will need to take that same lock - this is a classic recursive deadlock if userspace has mmaped the file they're DIO writing to and uses those pages for the buffer to write from, and it's a lock ordering deadlock in general. Thus we need a way to signal from the dio code to the page fault handler when we already are holding the pagecache add lock on an address space - this patch just adds a member to task_struct for this purpose. For now only bcachefs is implementing this locking, though it may be moved out of bcachefs and made available to other filesystems in the future. --------------------------------- The closest current VFS equivalent is mapping->invalidate_lock, which comes from XFS. However, it's not used by direct IO. Instead, direct IO paths shoot down the page cache twice - before starting the IO and at the end, and they're still technically racy w.r.t. page cache coherency. This is a more complete approach: in the future we might consider replacing mapping->invalidate_lock with the bcachefs code. Signed-off-by: Kent Overstreet Cc: Jan Kara Cc: Darrick J. Wong Cc: linux-fsdevel@vger.kernel.org --- include/linux/sched.h | 1 + init/init_task.c | 1 + 2 files changed, 2 insertions(+) diff --git a/include/linux/sched.h b/include/linux/sched.h index 63d242164b..f2a56f64f7 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -869,6 +869,7 @@ struct task_struct { struct mm_struct *mm; struct mm_struct *active_mm; + struct address_space *faults_disabled_mapping; int exit_state; int exit_code; diff --git a/init/init_task.c b/init/init_task.c index ff6c4b9bfe..f703116e05 100644 --- a/init/init_task.c +++ b/init/init_task.c @@ -85,6 +85,7 @@ struct task_struct init_task .nr_cpus_allowed= NR_CPUS, .mm = NULL, .active_mm = &init_mm, + .faults_disabled_mapping = NULL, .restart_block = { .fn = do_no_restart_syscall, }, From patchwork Tue May 9 16:56:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91668 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3037267vqo; Tue, 9 May 2023 10:04:15 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ46iEmh0RHd//2IRlhsxbofYIybENBGFPgHxclUpyk5rPFplewlFMIugvtEN9upxtF48Utr X-Received: by 2002:a05:6a00:10d3:b0:641:3ca2:1aec with SMTP id d19-20020a056a0010d300b006413ca21aecmr19245813pfu.27.1683651854711; Tue, 09 May 2023 10:04:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683651854; cv=none; d=google.com; s=arc-20160816; b=TCjpL6JrXhaOLzJ7qI5tzIq3Di59pEKbm5yWS6hytuZOHyx9MAY6ZQCD6XrGylHmUX wXTQT9zykhafyjdbOwQtS8qCFEcwrfCSXkF86NK96ED1kOHmcSa5sy2+jCAryN60ti5/ iVCH9lEqUucrNOX7zO6MSw6vymAAyfurKRXCD8Y+5hjDfSAu/Hv/633lNF+u1meoVNIK VdQqoVW3675PXFSA0YKhJQsXY4eXx3OXiPpG+keQIbnMvikTPhUWuQxQDw3lv5EAF1Ux NDSlz8mR4xyRFl5hSrJzsDa0wBT3GVKeOmioEPTNx+gsJ4cYHlChwgUSLXnRPy8mOW6r zRqQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SY6fXe4pdNkKgId+Z2KcjRoSaHIjUmN9p3hLtNQFQOM=; b=p13RtM2RMcXsrQZB7XvxBMceopg6qUCSkx+8h3SUbAxl91QOBhpwMSVojK+Ca7kPQ+ 6VAfynKUzvYNajl3WA/3c/+rjTBZWc3q4zNZygD+ipcoUouvdEX5QUadB6yBCWTtlrR/ r5z2n3E0B4WVOmK7ySPU58qdbX4vGTULw/rDTeQ1KOWV+WwNY3J18ugqK9KyK0TakpY3 9cK7fYElH75qwFfx7keoVPGLSfwYFj/maKTZBGtYL+nmYMLa+QklVZQ2A0e7XhyqL4Nt 0MtZsXSMCcvbHcpvBewzXgCpNYVzHErRAZIGyZo6tbreiEs9a7BqlhAnrIIUpMXXHt8E 9SYQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=E343AoWg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t1-20020a63b241000000b0051b1aede2b0si1949720pgo.341.2023.05.09.10.03.57; Tue, 09 May 2023 10:04:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=E343AoWg; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234879AbjEIQ6I (ORCPT + 99 others); Tue, 9 May 2023 12:58:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234880AbjEIQ5a (ORCPT ); Tue, 9 May 2023 12:57:30 -0400 Received: from out-8.mta1.migadu.com (out-8.mta1.migadu.com [IPv6:2001:41d0:203:375::8]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 131F64C0E for ; Tue, 9 May 2023 09:57:18 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651436; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SY6fXe4pdNkKgId+Z2KcjRoSaHIjUmN9p3hLtNQFQOM=; b=E343AoWginlmYA0XNC7JmuE92IbwRHmxwqnCWVBaeQu8jZ05EtSZ5jzp35SoalYgdAyVi2 mGxFIQ5f5mTuLhPjrCK1pmI755V80kOd3XnWhwPE3Gk3/WusCJYaQLjuq0KStuhsVRV+f0 42/iLxS5+Ar2E2XYNjjpc2Zygm9+MFY= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , linux-mm@kvack.org Subject: [PATCH 07/32] mm: Bring back vmalloc_exec Date: Tue, 9 May 2023 12:56:32 -0400 Message-Id: <20230509165657.1735798-8-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765436927259414832?= X-GMAIL-MSGID: =?utf-8?q?1765436927259414832?= From: Kent Overstreet This is needed for bcachefs, which dynamically generates per-btree node unpack functions. Signed-off-by: Kent Overstreet Cc: Andrew Morton Cc: Uladzislau Rezki Cc: Christoph Hellwig Cc: linux-mm@kvack.org Acked-by: Lorenzo Stoakes Nacked-by: Lorenzo Stoakes Signed-off-by: Kent Overstreet Nacked-by: Andy Lutomirski # for x86 --- include/linux/vmalloc.h | 1 + kernel/module/main.c | 4 +--- mm/nommu.c | 18 ++++++++++++++++++ mm/vmalloc.c | 21 +++++++++++++++++++++ 4 files changed, 41 insertions(+), 3 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 69250efa03..ff147fe115 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -145,6 +145,7 @@ extern void *vzalloc(unsigned long size) __alloc_size(1); extern void *vmalloc_user(unsigned long size) __alloc_size(1); extern void *vmalloc_node(unsigned long size, int node) __alloc_size(1); extern void *vzalloc_node(unsigned long size, int node) __alloc_size(1); +extern void *vmalloc_exec(unsigned long size, gfp_t gfp_mask) __alloc_size(1); extern void *vmalloc_32(unsigned long size) __alloc_size(1); extern void *vmalloc_32_user(unsigned long size) __alloc_size(1); extern void *__vmalloc(unsigned long size, gfp_t gfp_mask) __alloc_size(1); diff --git a/kernel/module/main.c b/kernel/module/main.c index d3be89de70..9eaa89e84c 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -1607,9 +1607,7 @@ static void dynamic_debug_remove(struct module *mod, struct _ddebug_info *dyndbg void * __weak module_alloc(unsigned long size) { - return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, - GFP_KERNEL, PAGE_KERNEL_EXEC, VM_FLUSH_RESET_PERMS, - NUMA_NO_NODE, __builtin_return_address(0)); + return vmalloc_exec(size, GFP_KERNEL); } bool __weak module_init_section(const char *name) diff --git a/mm/nommu.c b/mm/nommu.c index 57ba243c6a..8d9ab19e39 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -280,6 +280,24 @@ void *vzalloc_node(unsigned long size, int node) } EXPORT_SYMBOL(vzalloc_node); +/** + * vmalloc_exec - allocate virtually contiguous, executable memory + * @size: allocation size + * + * Kernel-internal function to allocate enough pages to cover @size + * the page level allocator and map them into contiguous and + * executable kernel virtual space. + * + * For tight control over page level allocator and protection flags + * use __vmalloc() instead. + */ + +void *vmalloc_exec(unsigned long size, gfp_t gfp_mask) +{ + return __vmalloc(size, gfp_mask); +} +EXPORT_SYMBOL_GPL(vmalloc_exec); + /** * vmalloc_32 - allocate virtually contiguous memory (32bit addressable) * @size: allocation size diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 31ff782d36..2ebb9ea7f0 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -3401,6 +3401,27 @@ void *vzalloc_node(unsigned long size, int node) } EXPORT_SYMBOL(vzalloc_node); +/** + * vmalloc_exec - allocate virtually contiguous, executable memory + * @size: allocation size + * + * Kernel-internal function to allocate enough pages to cover @size + * the page level allocator and map them into contiguous and + * executable kernel virtual space. + * + * For tight control over page level allocator and protection flags + * use __vmalloc() instead. + * + * Return: pointer to the allocated memory or %NULL on error + */ +void *vmalloc_exec(unsigned long size, gfp_t gfp_mask) +{ + return __vmalloc_node_range(size, 1, VMALLOC_START, VMALLOC_END, + gfp_mask, PAGE_KERNEL_EXEC, VM_FLUSH_RESET_PERMS, + NUMA_NO_NODE, __builtin_return_address(0)); +} +EXPORT_SYMBOL_GPL(vmalloc_exec); + #if defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA32) #define GFP_VMALLOC32 (GFP_DMA32 | GFP_KERNEL) #elif defined(CONFIG_64BIT) && defined(CONFIG_ZONE_DMA) From patchwork Tue May 9 16:56:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91690 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3046046vqo; Tue, 9 May 2023 10:16:24 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ79bvr+7l5FnZbss67K4e0M17nt8O+FAKMoZugTO89zmRb10s7pFRafUEOtpeTnEkdrBkSl X-Received: by 2002:a05:6a20:158a:b0:f4:1acb:8b20 with SMTP id h10-20020a056a20158a00b000f41acb8b20mr19813763pzj.3.1683652583815; Tue, 09 May 2023 10:16:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652583; cv=none; d=google.com; s=arc-20160816; b=YAzhmpiSG3diE4rItDWMurruI96zp+0xD+HWDvPPNkMcO+IuTjTqv06Kg8Atk2v+Hi JtGZLwi7RhiVCLfZt7TGRhUyJeRrzd+oad18iOrVU+Qsouz9HwlrLR8bqU3Wcwg0vST0 g/iM06r+uFG57Nz7jhxfdYI5V8/GuNFGTxzn0AlbK097IgqQHIM2Xsvfl0zfknL3IQv+ 2+dcPG9a1f0FjI0/2jXXJ2R19n7OrcKYWNLtZ634c7GS0Kdb6Tm8UwgctRqleZdLRP0k g93ZDj8BpAGAd2WHfUnpvd9bDypTI2fujHXGVwh9zM05lThtYGhnunoIYguSsPmPSylg cxbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3fmI1GsqhKc/VjkbsmLsO1lPQkBpUyPyeZNLX2gLM4A=; b=uPBeSA22v9ZCQ93xwIMnqLFp76Q4xjxI3xYoKk5fBDvZm6jEGexCkWyRd4pFwKnbVN fL1J+8rZDbHet5rK4PrAm5c8gGDmJmytbCyCuowAamciDtK07s16Nt5I5Jc4+DDKuqdG 42MghXUY229/KEGRB2OVtBj4s+uC9jIb4vaL8EBXX/q3fMCnxxojpKJnVo20ovARf7mN w5eajznhyh9K1B8qwpIYBUhikNrJCQKxCYrCglwWs6xBAXR4RbTqmYrQvgLNFXz460kT T4LG7uuBMlfO3FK3zST4jNufZdCd3A09JRvQf3exaxtDd7w6xNuaKunTrEVkINxX1sai /wWw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=HxbPD30d; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z7-20020a633307000000b0052c73483bbfsi1800962pgz.830.2023.05.09.10.16.11; Tue, 09 May 2023 10:16:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=HxbPD30d; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234963AbjEIQ6M (ORCPT + 99 others); Tue, 9 May 2023 12:58:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229557AbjEIQ5b (ORCPT ); Tue, 9 May 2023 12:57:31 -0400 Received: from out-17.mta1.migadu.com (out-17.mta1.migadu.com [95.215.58.17]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 74A46E71 for ; Tue, 9 May 2023 09:57:19 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651437; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3fmI1GsqhKc/VjkbsmLsO1lPQkBpUyPyeZNLX2gLM4A=; b=HxbPD30d9PS44l4BFocC7+4W53wukqnTMPxPn1K1Dbc9Y/EsxCrjkf4QSlJqxIFFlazBMT 6SVHA9CXL8xpVGqkipFpzj13Ez5vDhN2yCAlYAdQpq1Cff2u9KY26LppoRnsIuiZKmD0eD klJFD5WmmOeWr9WGDK7h91hyWwK5z5I= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet , Alexander Viro , Christian Brauner Subject: [PATCH 08/32] fs: factor out d_mark_tmpfile() Date: Tue, 9 May 2023 12:56:33 -0400 Message-Id: <20230509165657.1735798-9-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437691526396484?= X-GMAIL-MSGID: =?utf-8?q?1765437691526396484?= From: Kent Overstreet New helper for bcachefs - bcachefs doesn't want the inode_dec_link_count() call that d_tmpfile does, it handles i_nlink on its own atomically with other btree updates Signed-off-by: Kent Overstreet Cc: Alexander Viro Cc: Christian Brauner Cc: linux-fsdevel@vger.kernel.org --- fs/dcache.c | 12 ++++++++++-- include/linux/dcache.h | 1 + 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/fs/dcache.c b/fs/dcache.c index 52e6d5fdab..dbdafa2617 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -3249,11 +3249,10 @@ void d_genocide(struct dentry *parent) EXPORT_SYMBOL(d_genocide); -void d_tmpfile(struct file *file, struct inode *inode) +void d_mark_tmpfile(struct file *file, struct inode *inode) { struct dentry *dentry = file->f_path.dentry; - inode_dec_link_count(inode); BUG_ON(dentry->d_name.name != dentry->d_iname || !hlist_unhashed(&dentry->d_u.d_alias) || !d_unlinked(dentry)); @@ -3263,6 +3262,15 @@ void d_tmpfile(struct file *file, struct inode *inode) (unsigned long long)inode->i_ino); spin_unlock(&dentry->d_lock); spin_unlock(&dentry->d_parent->d_lock); +} +EXPORT_SYMBOL(d_mark_tmpfile); + +void d_tmpfile(struct file *file, struct inode *inode) +{ + struct dentry *dentry = file->f_path.dentry; + + inode_dec_link_count(inode); + d_mark_tmpfile(file, inode); d_instantiate(dentry, inode); } EXPORT_SYMBOL(d_tmpfile); diff --git a/include/linux/dcache.h b/include/linux/dcache.h index 6b351e009f..3da2f0545d 100644 --- a/include/linux/dcache.h +++ b/include/linux/dcache.h @@ -251,6 +251,7 @@ extern struct dentry * d_make_root(struct inode *); /* - the ramfs-type tree */ extern void d_genocide(struct dentry *); +extern void d_mark_tmpfile(struct file *, struct inode *); extern void d_tmpfile(struct file *, struct inode *); extern struct dentry *d_find_alias(struct inode *); From patchwork Tue May 9 16:56:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91693 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3046332vqo; Tue, 9 May 2023 10:16:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4B7ZEVlbyNFXKY+KNmv/P/XvUPEz3SwYKe5mMnPOksiAyOnwJKn3KwekBgufXrz/6uJH95 X-Received: by 2002:a17:903:2451:b0:1ab:142a:3dd7 with SMTP id l17-20020a170903245100b001ab142a3dd7mr16885722pls.68.1683652608536; Tue, 09 May 2023 10:16:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652608; cv=none; d=google.com; s=arc-20160816; b=X4iPx7sFfAhQ/Wo4l1noadhYK6GQg+WcuQSpD+wfiSOAxRXUWileDot+ISZthHcmlC ibQ6V8uPzJTC/KprJlm3kWeJIGPTOHD3s6gNuPosB5OXzpOkQg2RrZfD8eije4kZiJBU 2DlXEF21HnUFqD6Ip/DKF8Gt7Ow5iH767BPd4Fo/YAy+U5m+6AAwfiNVEc4C1FY6ghKm Bgc+O5R/f9W+Gi3SkgWzUdd9q6PaddRZQ+8YQEvChAjsrFOck+86h7oDoUerYXA2VdSn 0+wB0485kVaeZO95egMM3EySUyocD8wV0+mMP+rDgbSfsuF+crE5mqQ0APL5Rdgx6j/d /e8A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/eEGjYouYQ7iaZLNOVNtoto5kM+J4djsghn1OaVu7vs=; b=JT8GdJ+QHul8VkDPY0PfkMJGnWzvSb1zjlzy4+ulkEe1XPOXb1znBi98tDMMoO8syS J500pN9820kJfDlQPq7zyfSvarhJRj4jv0Pq7gJarnDKoCPH49lQqYpx3IW1Bj33YEmN fVlC0GwFIM3bBDEd99af6RrXDQK/bsKKsT+EceHLrClOh54PueshEwqA2+x4awy3hPto mVLehVUolU6izLvWUMp9AbBWROZu5+UGDHEILvGd9+pbJXt5/rISRH6BE6KkNRQ1wLLz jm+NyHGwaCDsMW37J4oa8A8qGzjMYafHmtT5YJFCpBRznqnn64e0e0sYPuSY4rIVNew6 QsvA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Li1A9oMV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z7-20020a633307000000b0052c73483bbfsi1800962pgz.830.2023.05.09.10.16.36; Tue, 09 May 2023 10:16:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Li1A9oMV; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229664AbjEIQ6W (ORCPT + 99 others); Tue, 9 May 2023 12:58:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229539AbjEIQ5s (ORCPT ); Tue, 9 May 2023 12:57:48 -0400 Received: from out-27.mta1.migadu.com (out-27.mta1.migadu.com [IPv6:2001:41d0:203:375::1b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AA584486 for ; Tue, 9 May 2023 09:57:20 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651439; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/eEGjYouYQ7iaZLNOVNtoto5kM+J4djsghn1OaVu7vs=; b=Li1A9oMVL5H+4ABVWOa/qXbmjoZ3SFYys6CVbgJMR6AEj6aRIewZzsTnOivgBM+vgvd0sR z8F2iTs2lJ0sNRombbYIDq2texP8GzEjWWA/R7gEEkxwVlS36I4S9u2K3D9FRaMqndkjgG 6fqR0yEbyle22UucUC6Zb7BQrsGrrII= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , linux-block@vger.kernel.org, Jens Axboe , Kent Overstreet Subject: [PATCH 09/32] block: Add some exports for bcachefs Date: Tue, 9 May 2023 12:56:34 -0400 Message-Id: <20230509165657.1735798-10-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437718099567606?= X-GMAIL-MSGID: =?utf-8?q?1765437718099567606?= From: Kent Overstreet - bio_set_pages_dirty(), bio_check_pages_dirty() - dio path - blk_status_to_str() - error messages - bio_add_folio() - this should definitely be exported for everyone, it's the modern version of bio_add_page() Signed-off-by: Kent Overstreet Cc: linux-block@vger.kernel.org Cc: Jens Axboe Signed-off-by: Kent Overstreet --- block/bio.c | 3 +++ block/blk-core.c | 1 + block/blk.h | 1 - include/linux/blkdev.h | 1 + 4 files changed, 5 insertions(+), 1 deletion(-) diff --git a/block/bio.c b/block/bio.c index fd11614bba..1e75840d17 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1159,6 +1159,7 @@ bool bio_add_folio(struct bio *bio, struct folio *folio, size_t len, return false; return bio_add_page(bio, &folio->page, len, off) > 0; } +EXPORT_SYMBOL(bio_add_folio); void __bio_release_pages(struct bio *bio, bool mark_dirty) { @@ -1480,6 +1481,7 @@ void bio_set_pages_dirty(struct bio *bio) set_page_dirty_lock(bvec->bv_page); } } +EXPORT_SYMBOL_GPL(bio_set_pages_dirty); /* * bio_check_pages_dirty() will check that all the BIO's pages are still dirty. @@ -1539,6 +1541,7 @@ void bio_check_pages_dirty(struct bio *bio) spin_unlock_irqrestore(&bio_dirty_lock, flags); schedule_work(&bio_dirty_work); } +EXPORT_SYMBOL_GPL(bio_check_pages_dirty); static inline bool bio_remaining_done(struct bio *bio) { diff --git a/block/blk-core.c b/block/blk-core.c index 42926e6cb8..f19bcc684b 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -205,6 +205,7 @@ const char *blk_status_to_str(blk_status_t status) return ""; return blk_errors[idx].name; } +EXPORT_SYMBOL_GPL(blk_status_to_str); /** * blk_sync_queue - cancel any pending callbacks on a queue diff --git a/block/blk.h b/block/blk.h index cc4e8873df..cc04dc73e9 100644 --- a/block/blk.h +++ b/block/blk.h @@ -259,7 +259,6 @@ static inline void blk_integrity_del(struct gendisk *disk) unsigned long blk_rq_timeout(unsigned long timeout); void blk_add_timer(struct request *req); -const char *blk_status_to_str(blk_status_t status); bool blk_attempt_plug_merge(struct request_queue *q, struct bio *bio, unsigned int nr_segs); diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 941304f174..7cac183112 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -867,6 +867,7 @@ extern const char *blk_op_str(enum req_op op); int blk_status_to_errno(blk_status_t status); blk_status_t errno_to_blk_status(int errno); +const char *blk_status_to_str(blk_status_t status); /* only poll the hardware once, don't continue until a completion was found */ #define BLK_POLL_ONESHOT (1 << 0) From patchwork Tue May 9 16:56:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91683 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3042952vqo; Tue, 9 May 2023 10:11:27 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5PHKstF2d/06OBdBOmC3KgbgRAMxFGO/vN7P32+nhFeVJV0gVGp+aO2S0Ng+RBDZAzn846 X-Received: by 2002:a05:6a20:7489:b0:f6:4c57:265d with SMTP id p9-20020a056a20748900b000f64c57265dmr18123116pzd.1.1683652287237; Tue, 09 May 2023 10:11:27 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652287; cv=none; d=google.com; s=arc-20160816; b=S7ged3vFjcNDvNF19VeXHTKn1ZKDBDXP/gLOn02km88M6Kky6Dt9fgfmYHn/3upCQX LYyKBSYI57BwlLo2YSw6AjuxL92fP5cQbR1cHJNVOStsh4WSqjSL0lEH4ceMAAOSQXXG 26iKgD/QOJ/o3wit+92nlD148nC1yUPz8i/mnX4qf6CJEnEXd8ijhBASyScK0NiYbi7t UchY0nzRRoLeZktctH8MgFctOvvoM3LTStCvDxnD7x/0xs8UImNhWoXMTTCVbaxa1/DA qFL1w4FDxlQt0lQdw9x0MSJzS9AqJSFtORrxC7XtooU4KuQ6eOksg5yOIPmUCzrl68eN QtUg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=t8R8a1McseA9yHEzYOxyXGPVlvlW3Oa2+9zEDFiMSek=; b=gezmV4+ILUqJNnPqpJecNA2bbSzNetMcTSPtopnZ/oMvfR5L+oS/IFkTHpC5zabU2U tmewpGoE0nJHfIYNjy/JV2yBwSc+JSeJx21+7cyRMb3g2qYtY+UIhxUiU1Ek811n5y7D 4SJP7d0oXDfllai+EnB/efQmRQRFGBAQ/g5/xBc0Sor4NRvRFvKSKujyx1CL0Zx4EsJr VnD1BaM4SM3eyC8BHn7pbTEdG917p7PDs0+bhSM3r6tnon59NoStQTmT6E3j/Qo1C2xW ejMVcck2CwVX8QaDUN2GYNS7rCiYgKBYmw3MjxIYR53evibBREAwQCoVEnMWVfxOW1AH pJvQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=kdwBwCR3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k184-20020a636fc1000000b00508eeafc9ecsi1962139pgc.176.2023.05.09.10.11.09; Tue, 09 May 2023 10:11:27 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=kdwBwCR3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235038AbjEIQ6Y (ORCPT + 99 others); Tue, 9 May 2023 12:58:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234838AbjEIQ6C (ORCPT ); Tue, 9 May 2023 12:58:02 -0400 Received: from out-62.mta1.migadu.com (out-62.mta1.migadu.com [IPv6:2001:41d0:203:375::3e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9E322126 for ; Tue, 9 May 2023 09:57:21 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651439; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t8R8a1McseA9yHEzYOxyXGPVlvlW3Oa2+9zEDFiMSek=; b=kdwBwCR3xVaP5Jo+IaCVgWgyW4Ce3kcbutp/EEyLJxQ81ZP1QgFIFCBVpPfLZYqpcUSZjS ZqoNakyb6ZKW9/hKVySDM9SU66Z1JNKNd2OyiIQSAZbFpvoIj95fSHFiMJ0M4veDRt2U3L Xb8o/Gae/L7EKhGQ8mj+UEpQfOAFNs4= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Jens Axboe , linux-block@vger.kernel.org Subject: [PATCH 10/32] block: Allow bio_iov_iter_get_pages() with bio->bi_bdev unset Date: Tue, 9 May 2023 12:56:35 -0400 Message-Id: <20230509165657.1735798-11-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437380745106382?= X-GMAIL-MSGID: =?utf-8?q?1765437380745106382?= bio_iov_iter_get_pages() trims the IO based on the block size of the block device the IO will be issued to. However, bcachefs is a multi device filesystem; when we're creating the bio we don't yet know which block device the bio will be submitted to - we have to handle the alignment checks elsewhere. Thus this is needed to avoid a null ptr deref. Signed-off-by: Kent Overstreet Cc: Jens Axboe Cc: linux-block@vger.kernel.org --- block/bio.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index 1e75840d17..e74a04ea14 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1245,7 +1245,7 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) struct page **pages = (struct page **)bv; ssize_t size, left; unsigned len, i = 0; - size_t offset, trim; + size_t offset; int ret = 0; /* @@ -1274,10 +1274,12 @@ static int __bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter) nr_pages = DIV_ROUND_UP(offset + size, PAGE_SIZE); - trim = size & (bdev_logical_block_size(bio->bi_bdev) - 1); - iov_iter_revert(iter, trim); + if (bio->bi_bdev) { + size_t trim = size & (bdev_logical_block_size(bio->bi_bdev) - 1); + iov_iter_revert(iter, trim); + size -= trim; + } - size -= trim; if (unlikely(!size)) { ret = -EFAULT; goto out; From patchwork Tue May 9 16:56:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91677 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3040928vqo; Tue, 9 May 2023 10:08:35 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ709y5QpFXCPNNEMD0oEhRfmn8I7KAH/TYlGvk1zMpW0lLqs9y5Rs4PeVhEaPT+XAlTaQ2M X-Received: by 2002:a05:6a20:841b:b0:f2:7da5:f27e with SMTP id c27-20020a056a20841b00b000f27da5f27emr19251594pzd.21.1683652114915; Tue, 09 May 2023 10:08:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652114; cv=none; d=google.com; s=arc-20160816; b=NyJ2ydv/Uzyuqq9+wFmybR0o1eEOpId7L+sFOeKjGjzTytV8v5aGwi4xVCCM6GID1q rRsUTo/2DjHNgFCkRPzaQdLUbUzRk+agfr56i21UiM65o/97DHXougQMkCg+8L3cEwr/ 3EaMAtARtEIMAX8qMqaWDS7EqgjH6881gp0yhufPkp1AzTsjcakQV0Sw5RBjbTm7Dj3e 9DSUlzxRwbG6nqBpgAjlsOnUVs5eRcfAibQkqXuJPbsulFEermHaAibs4YQdsk4kaU+V ShyVcd3P4j24nMlwWNXNOx5VmjDQBVQtbp1j5UOnyhCW20+pmRb8l5s2OhfhPqK9OhIS Ezew== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KkRPMSjK8YKASg8ZAP2GYnCrc/m4pRzmXY3p1j1RS/g=; b=PoXN3mOh3H9kIV+N4ohN+S/na8OWw0jPQKlxLbscgAQaXDX5CrVcKPxdirlPxx5zs+ 0f6JoXJObUHKXgnk2GC8Fr2z2OtH7LF5gbHHlX1eMp56HmbeScfI4fAwMyLTRHYdrmjV zXp51WTzyIdowxedsE3vEDIFi3Y/wuhj/+W3+0n9tVvUwe2HCae5f7MjpMW+Vt5TtlqZ 7BE9E8RpnatoN0CSxjdtlvPJbe5gUd3yjpvWoFv91a4O9rgO2ctlSvMF8h+UHgQSj324 iYWpD0St7q7Paupp06HYt0FCk2uDOcSEE8pJrMlF7HSkhC6aYF30GD8GiWIpvUXa14xW xQGQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="e6zAL/14"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m12-20020a633f0c000000b0050fad17282dsi1861660pga.217.2023.05.09.10.08.22; Tue, 09 May 2023 10:08:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="e6zAL/14"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234914AbjEIQ63 (ORCPT + 99 others); Tue, 9 May 2023 12:58:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234909AbjEIQ6C (ORCPT ); Tue, 9 May 2023 12:58:02 -0400 Received: from out-26.mta1.migadu.com (out-26.mta1.migadu.com [95.215.58.26]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9638468C for ; Tue, 9 May 2023 09:57:22 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651441; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KkRPMSjK8YKASg8ZAP2GYnCrc/m4pRzmXY3p1j1RS/g=; b=e6zAL/14FO+1bslJPm07XIKa8FmztDcUWzG/+82TKmWU9JdQ37Lsk8HCZIAl/Ey4JWSA0n 3q9fReqLqSzv4BR/RUqz0QCO9NZZ9AIGDUd0kpAPw/7gjDB2t0CqgqTg8LW8BwDa7lZsi7 whDk5K1hE3oyJt+KNlYcAOFU9ZDzXmk= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet , Jens Axboe , linux-block@vger.kernel.org Subject: [PATCH 11/32] block: Bring back zero_fill_bio_iter Date: Tue, 9 May 2023 12:56:36 -0400 Message-Id: <20230509165657.1735798-12-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437199624777346?= X-GMAIL-MSGID: =?utf-8?q?1765437199624777346?= From: Kent Overstreet This reverts the commit that deleted it; it's used by bcachefs. Signed-off-by: Kent Overstreet Cc: Jens Axboe Cc: linux-block@vger.kernel.org --- block/bio.c | 6 +++--- include/linux/bio.h | 7 ++++++- 2 files changed, 9 insertions(+), 4 deletions(-) diff --git a/block/bio.c b/block/bio.c index e74a04ea14..70b5c987bc 100644 --- a/block/bio.c +++ b/block/bio.c @@ -606,15 +606,15 @@ struct bio *bio_kmalloc(unsigned short nr_vecs, gfp_t gfp_mask) } EXPORT_SYMBOL(bio_kmalloc); -void zero_fill_bio(struct bio *bio) +void zero_fill_bio_iter(struct bio *bio, struct bvec_iter start) { struct bio_vec bv; struct bvec_iter iter; - bio_for_each_segment(bv, bio, iter) + __bio_for_each_segment(bv, bio, iter, start) memzero_bvec(&bv); } -EXPORT_SYMBOL(zero_fill_bio); +EXPORT_SYMBOL(zero_fill_bio_iter); /** * bio_truncate - truncate the bio to small size of @new_size diff --git a/include/linux/bio.h b/include/linux/bio.h index d766be7152..3536f28c05 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -484,7 +484,12 @@ extern void bio_copy_data_iter(struct bio *dst, struct bvec_iter *dst_iter, extern void bio_copy_data(struct bio *dst, struct bio *src); extern void bio_free_pages(struct bio *bio); void guard_bio_eod(struct bio *bio); -void zero_fill_bio(struct bio *bio); +void zero_fill_bio_iter(struct bio *bio, struct bvec_iter iter); + +static inline void zero_fill_bio(struct bio *bio) +{ + zero_fill_bio_iter(bio, bio->bi_iter); +} static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { From patchwork Tue May 9 16:56:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91682 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3041622vqo; Tue, 9 May 2023 10:09:34 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5985d5IJkLYlZXJ5Gyh52sbuw+gk7hr/fNKOQalaAJUCx3fITG/lIG256zq5kxbDF8VvEs X-Received: by 2002:a17:90a:74c6:b0:249:8963:c72 with SMTP id p6-20020a17090a74c600b0024989630c72mr14790960pjl.17.1683652174555; Tue, 09 May 2023 10:09:34 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652174; cv=none; d=google.com; s=arc-20160816; b=EtESTXu/18XDv8EiYMLORfH8/8Qf2EYZAnh2AMSsIywHrRN5pwx8jIgAeknKjAKmWT lAFoVaZ4WbjJ3NPfZjTPNsb8rAwYBOwmWe4T0VoLNJkjRQsMw5kro+GzRpZtaySscMlQ 3jTmYcl28DVmBDchK+mc6k8ORsEL0vhfzmsydPKmSEkbYeKE9UbsWecSfXoiQRiWAfjb DIYh3sOCUAutSyH0VWh4bW08ey+oB/kcuzBiPspouGQ03gA+ikeRHc48/UWS0VkVkhHd et6kzKjJiN/Q8naBwiqzKSy6LvhJwiK0XYf+Dc+v3B2x1HZCHppwHJsuDRFdAqmA89+h 3POA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3/SbIkq+ijMraXia7ugf48JR6cDMRV7zGK3kOFrLB4E=; b=gOPISm7EjnkEA1nmURolHtkqcJ/HPcPBoEMXq1Li9gFfrFHU9ukWaUGDh+rHHZhMS8 BEk2IVrXAeXCrmbuUfRQzSmvKWxRLttf6om2mlVyCvVqxk4LXgxk2JRQkE2GhBd2S606 KoNKt3JkE1lElVpwVqfVitmPChSv8eAu0b/Zqu/iej2r1/I4HTQnOvreVRptLfa+3+aP rLyphBUChg8BfgITLfyjxbTOQ/JE32L9S8rEUWdBAJ9zBzpbFT4//NgWlr0+/kVsaihh zSqm6VGPWVzoK1w7nwW8KfteyqYjwy0J/DvS0GREZ7R65BnUH4GOqonAV7yeif8b52ZE bD6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=vJPR0Qn3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i9-20020a17090a718900b0024e1e15b1b0si15284409pjk.161.2023.05.09.10.09.21; Tue, 09 May 2023 10:09:34 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=vJPR0Qn3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233512AbjEIQ6s (ORCPT + 99 others); Tue, 9 May 2023 12:58:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41498 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234778AbjEIQ6E (ORCPT ); Tue, 9 May 2023 12:58:04 -0400 Received: from out-53.mta1.migadu.com (out-53.mta1.migadu.com [IPv6:2001:41d0:203:375::35]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4BC0B55BE for ; Tue, 9 May 2023 09:57:23 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651442; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3/SbIkq+ijMraXia7ugf48JR6cDMRV7zGK3kOFrLB4E=; b=vJPR0Qn3EC9bMAfq8VR6BdvcjQnruU4RBysk7bOJcEdPkHQQ93fxnhJ9ZWKcph9l3lY0I1 R5bXPGNCfAYY1ib2v6vX6JIE1rrAXAYEkkDYMnjj+M+FTt2MRnobhRwrHV9rcczDkI+sJV 2FXqWBiIrAt76SsKf+Gfu/OkIDLL8gk= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Jens Axboe , linux-block@vger.kernel.org, Ming Lei , Phillip Lougher Subject: [PATCH 12/32] block: Rework bio_for_each_segment_all() Date: Tue, 9 May 2023 12:56:37 -0400 Message-Id: <20230509165657.1735798-13-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437262830692396?= X-GMAIL-MSGID: =?utf-8?q?1765437262830692396?= This patch reworks bio_for_each_segment_all() to be more inline with how the other bio iterators work: - bio_iter_all_peek() now returns a synthesized bio_vec; we don't stash one in the iterator and pass a pointer to it - bad. This way makes it clearer what's a constructed value vs. a reference to something pre-existing, and it also will help with cleaning up and consolidating code with bio_for_each_folio_all(). - We now provide bio_for_each_segment_all_continue(), for squashfs: this makes their code clearer. Signed-off-by: Kent Overstreet Cc: Jens Axboe Cc: linux-block@vger.kernel.org Cc: Ming Lei Cc: Phillip Lougher --- block/bio.c | 38 ++++++++++++------------ block/blk-map.c | 38 ++++++++++++------------ block/bounce.c | 12 ++++---- drivers/md/bcache/btree.c | 8 ++--- drivers/md/dm-crypt.c | 10 +++---- drivers/md/raid1.c | 4 +-- fs/btrfs/disk-io.c | 4 +-- fs/btrfs/extent_io.c | 50 +++++++++++++++---------------- fs/btrfs/raid56.c | 14 ++++----- fs/erofs/zdata.c | 4 +-- fs/ext4/page-io.c | 8 ++--- fs/ext4/readpage.c | 4 +-- fs/f2fs/data.c | 20 ++++++------- fs/gfs2/lops.c | 10 +++---- fs/gfs2/meta_io.c | 8 ++--- fs/mpage.c | 4 +-- fs/squashfs/block.c | 48 +++++++++++++++++------------- fs/squashfs/lz4_wrapper.c | 17 ++++++----- fs/squashfs/lzo_wrapper.c | 17 ++++++----- fs/squashfs/xz_wrapper.c | 19 ++++++------ fs/squashfs/zlib_wrapper.c | 18 ++++++----- fs/squashfs/zstd_wrapper.c | 19 ++++++------ include/linux/bio.h | 34 ++++++++++++++++----- include/linux/bvec.h | 61 ++++++++++++++++++++++---------------- 24 files changed, 256 insertions(+), 213 deletions(-) diff --git a/block/bio.c b/block/bio.c index 70b5c987bc..f2845d4e47 100644 --- a/block/bio.c +++ b/block/bio.c @@ -1163,13 +1163,13 @@ EXPORT_SYMBOL(bio_add_folio); void __bio_release_pages(struct bio *bio, bool mark_dirty) { - struct bvec_iter_all iter_all; - struct bio_vec *bvec; + struct bvec_iter_all iter; + struct bio_vec bvec; - bio_for_each_segment_all(bvec, bio, iter_all) { - if (mark_dirty && !PageCompound(bvec->bv_page)) - set_page_dirty_lock(bvec->bv_page); - put_page(bvec->bv_page); + bio_for_each_segment_all(bvec, bio, iter) { + if (mark_dirty && !PageCompound(bvec.bv_page)) + set_page_dirty_lock(bvec.bv_page); + put_page(bvec.bv_page); } } EXPORT_SYMBOL_GPL(__bio_release_pages); @@ -1436,11 +1436,11 @@ EXPORT_SYMBOL(bio_copy_data); void bio_free_pages(struct bio *bio) { - struct bio_vec *bvec; - struct bvec_iter_all iter_all; + struct bvec_iter_all iter; + struct bio_vec bvec; - bio_for_each_segment_all(bvec, bio, iter_all) - __free_page(bvec->bv_page); + bio_for_each_segment_all(bvec, bio, iter) + __free_page(bvec.bv_page); } EXPORT_SYMBOL(bio_free_pages); @@ -1475,12 +1475,12 @@ EXPORT_SYMBOL(bio_free_pages); */ void bio_set_pages_dirty(struct bio *bio) { - struct bio_vec *bvec; - struct bvec_iter_all iter_all; + struct bvec_iter_all iter; + struct bio_vec bvec; - bio_for_each_segment_all(bvec, bio, iter_all) { - if (!PageCompound(bvec->bv_page)) - set_page_dirty_lock(bvec->bv_page); + bio_for_each_segment_all(bvec, bio, iter) { + if (!PageCompound(bvec.bv_page)) + set_page_dirty_lock(bvec.bv_page); } } EXPORT_SYMBOL_GPL(bio_set_pages_dirty); @@ -1524,12 +1524,12 @@ static void bio_dirty_fn(struct work_struct *work) void bio_check_pages_dirty(struct bio *bio) { - struct bio_vec *bvec; + struct bvec_iter_all iter; + struct bio_vec bvec; unsigned long flags; - struct bvec_iter_all iter_all; - bio_for_each_segment_all(bvec, bio, iter_all) { - if (!PageDirty(bvec->bv_page) && !PageCompound(bvec->bv_page)) + bio_for_each_segment_all(bvec, bio, iter) { + if (!PageDirty(bvec.bv_page) && !PageCompound(bvec.bv_page)) goto defer; } diff --git a/block/blk-map.c b/block/blk-map.c index 9137d16cec..5774a9e467 100644 --- a/block/blk-map.c +++ b/block/blk-map.c @@ -46,21 +46,21 @@ static struct bio_map_data *bio_alloc_map_data(struct iov_iter *data, */ static int bio_copy_from_iter(struct bio *bio, struct iov_iter *iter) { - struct bio_vec *bvec; - struct bvec_iter_all iter_all; + struct bvec_iter_all bv_iter; + struct bio_vec bvec; - bio_for_each_segment_all(bvec, bio, iter_all) { + bio_for_each_segment_all(bvec, bio, bv_iter) { ssize_t ret; - ret = copy_page_from_iter(bvec->bv_page, - bvec->bv_offset, - bvec->bv_len, + ret = copy_page_from_iter(bvec.bv_page, + bvec.bv_offset, + bvec.bv_len, iter); if (!iov_iter_count(iter)) break; - if (ret < bvec->bv_len) + if (ret < bvec.bv_len) return -EFAULT; } @@ -77,21 +77,21 @@ static int bio_copy_from_iter(struct bio *bio, struct iov_iter *iter) */ static int bio_copy_to_iter(struct bio *bio, struct iov_iter iter) { - struct bio_vec *bvec; - struct bvec_iter_all iter_all; + struct bvec_iter_all bv_iter; + struct bio_vec bvec; - bio_for_each_segment_all(bvec, bio, iter_all) { + bio_for_each_segment_all(bvec, bio, bv_iter) { ssize_t ret; - ret = copy_page_to_iter(bvec->bv_page, - bvec->bv_offset, - bvec->bv_len, + ret = copy_page_to_iter(bvec.bv_page, + bvec.bv_offset, + bvec.bv_len, &iter); if (!iov_iter_count(&iter)) break; - if (ret < bvec->bv_len) + if (ret < bvec.bv_len) return -EFAULT; } @@ -442,12 +442,12 @@ static void bio_copy_kern_endio(struct bio *bio) static void bio_copy_kern_endio_read(struct bio *bio) { char *p = bio->bi_private; - struct bio_vec *bvec; - struct bvec_iter_all iter_all; + struct bvec_iter_all iter; + struct bio_vec bvec; - bio_for_each_segment_all(bvec, bio, iter_all) { - memcpy_from_bvec(p, bvec); - p += bvec->bv_len; + bio_for_each_segment_all(bvec, bio, iter) { + memcpy_from_bvec(p, &bvec); + p += bvec.bv_len; } bio_copy_kern_endio(bio); diff --git a/block/bounce.c b/block/bounce.c index 7cfcb242f9..e701832d76 100644 --- a/block/bounce.c +++ b/block/bounce.c @@ -102,18 +102,18 @@ static void copy_to_high_bio_irq(struct bio *to, struct bio *from) static void bounce_end_io(struct bio *bio) { struct bio *bio_orig = bio->bi_private; - struct bio_vec *bvec, orig_vec; + struct bio_vec bvec, orig_vec; struct bvec_iter orig_iter = bio_orig->bi_iter; - struct bvec_iter_all iter_all; + struct bvec_iter_all iter; /* * free up bounce indirect pages used */ - bio_for_each_segment_all(bvec, bio, iter_all) { + bio_for_each_segment_all(bvec, bio, iter) { orig_vec = bio_iter_iovec(bio_orig, orig_iter); - if (bvec->bv_page != orig_vec.bv_page) { - dec_zone_page_state(bvec->bv_page, NR_BOUNCE); - mempool_free(bvec->bv_page, &page_pool); + if (bvec.bv_page != orig_vec.bv_page) { + dec_zone_page_state(bvec.bv_page, NR_BOUNCE); + mempool_free(bvec.bv_page, &page_pool); } bio_advance_iter(bio_orig, &orig_iter, orig_vec.bv_len); } diff --git a/drivers/md/bcache/btree.c b/drivers/md/bcache/btree.c index 147c493a98..98ce12b239 100644 --- a/drivers/md/bcache/btree.c +++ b/drivers/md/bcache/btree.c @@ -373,12 +373,12 @@ static void do_btree_node_write(struct btree *b) bset_sector_offset(&b->keys, i)); if (!bch_bio_alloc_pages(b->bio, __GFP_NOWARN|GFP_NOWAIT)) { - struct bio_vec *bv; + struct bio_vec bv; void *addr = (void *) ((unsigned long) i & ~(PAGE_SIZE - 1)); - struct bvec_iter_all iter_all; + struct bvec_iter_all iter; - bio_for_each_segment_all(bv, b->bio, iter_all) { - memcpy(page_address(bv->bv_page), addr, PAGE_SIZE); + bio_for_each_segment_all(bv, b->bio, iter) { + memcpy(page_address(bv.bv_page), addr, PAGE_SIZE); addr += PAGE_SIZE; } diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index 3ba53dc3cc..166bb4fdb4 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1713,12 +1713,12 @@ static struct bio *crypt_alloc_buffer(struct dm_crypt_io *io, unsigned int size) static void crypt_free_buffer_pages(struct crypt_config *cc, struct bio *clone) { - struct bio_vec *bv; - struct bvec_iter_all iter_all; + struct bvec_iter_all iter; + struct bio_vec bv; - bio_for_each_segment_all(bv, clone, iter_all) { - BUG_ON(!bv->bv_page); - mempool_free(bv->bv_page, &cc->page_pool); + bio_for_each_segment_all(bv, clone, iter) { + BUG_ON(!bv.bv_page); + mempool_free(bv.bv_page, &cc->page_pool); } } diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 68a9e2d998..4f58cae37e 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -2188,7 +2188,7 @@ static void process_checks(struct r1bio *r1_bio) blk_status_t status = sbio->bi_status; struct page **ppages = get_resync_pages(pbio)->pages; struct page **spages = get_resync_pages(sbio)->pages; - struct bio_vec *bi; + struct bio_vec bi; int page_len[RESYNC_PAGES] = { 0 }; struct bvec_iter_all iter_all; @@ -2198,7 +2198,7 @@ static void process_checks(struct r1bio *r1_bio) sbio->bi_status = 0; bio_for_each_segment_all(bi, sbio, iter_all) - page_len[j++] = bi->bv_len; + page_len[j++] = bi.bv_len; if (!status) { for (j = vcnt; j-- ; ) { diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index 9e1596bb20..92b3396c15 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -3804,12 +3804,12 @@ ALLOW_ERROR_INJECTION(open_ctree, ERRNO); static void btrfs_end_super_write(struct bio *bio) { struct btrfs_device *device = bio->bi_private; - struct bio_vec *bvec; + struct bio_vec bvec; struct bvec_iter_all iter_all; struct page *page; bio_for_each_segment_all(bvec, bio, iter_all) { - page = bvec->bv_page; + page = bvec.bv_page; if (bio->bi_status) { btrfs_warn_rl_in_rcu(device->fs_info, diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index 40300e8e5f..5796c99ea1 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -581,34 +581,34 @@ static void end_bio_extent_writepage(struct btrfs_bio *bbio) { struct bio *bio = &bbio->bio; int error = blk_status_to_errno(bio->bi_status); - struct bio_vec *bvec; + struct bio_vec bvec; u64 start; u64 end; struct bvec_iter_all iter_all; ASSERT(!bio_flagged(bio, BIO_CLONED)); bio_for_each_segment_all(bvec, bio, iter_all) { - struct page *page = bvec->bv_page; + struct page *page = bvec.bv_page; struct inode *inode = page->mapping->host; struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); const u32 sectorsize = fs_info->sectorsize; /* Our read/write should always be sector aligned. */ - if (!IS_ALIGNED(bvec->bv_offset, sectorsize)) + if (!IS_ALIGNED(bvec.bv_offset, sectorsize)) btrfs_err(fs_info, "partial page write in btrfs with offset %u and length %u", - bvec->bv_offset, bvec->bv_len); - else if (!IS_ALIGNED(bvec->bv_len, sectorsize)) + bvec.bv_offset, bvec.bv_len); + else if (!IS_ALIGNED(bvec.bv_len, sectorsize)) btrfs_info(fs_info, "incomplete page write with offset %u and length %u", - bvec->bv_offset, bvec->bv_len); + bvec.bv_offset, bvec.bv_len); - start = page_offset(page) + bvec->bv_offset; - end = start + bvec->bv_len - 1; + start = page_offset(page) + bvec.bv_offset; + end = start + bvec.bv_len - 1; end_extent_writepage(page, error, start, end); - btrfs_page_clear_writeback(fs_info, page, start, bvec->bv_len); + btrfs_page_clear_writeback(fs_info, page, start, bvec.bv_len); } bio_put(bio); @@ -736,7 +736,7 @@ static struct extent_buffer *find_extent_buffer_readpage( static void end_bio_extent_readpage(struct btrfs_bio *bbio) { struct bio *bio = &bbio->bio; - struct bio_vec *bvec; + struct bio_vec bvec; struct processed_extent processed = { 0 }; /* * The offset to the beginning of a bio, since one bio can never be @@ -749,7 +749,7 @@ static void end_bio_extent_readpage(struct btrfs_bio *bbio) ASSERT(!bio_flagged(bio, BIO_CLONED)); bio_for_each_segment_all(bvec, bio, iter_all) { bool uptodate = !bio->bi_status; - struct page *page = bvec->bv_page; + struct page *page = bvec.bv_page; struct inode *inode = page->mapping->host; struct btrfs_fs_info *fs_info = btrfs_sb(inode->i_sb); const u32 sectorsize = fs_info->sectorsize; @@ -769,19 +769,19 @@ static void end_bio_extent_readpage(struct btrfs_bio *bbio) * for unaligned offsets, and an error if they don't add up to * a full sector. */ - if (!IS_ALIGNED(bvec->bv_offset, sectorsize)) + if (!IS_ALIGNED(bvec.bv_offset, sectorsize)) btrfs_err(fs_info, "partial page read in btrfs with offset %u and length %u", - bvec->bv_offset, bvec->bv_len); - else if (!IS_ALIGNED(bvec->bv_offset + bvec->bv_len, + bvec.bv_offset, bvec.bv_len); + else if (!IS_ALIGNED(bvec.bv_offset + bvec.bv_len, sectorsize)) btrfs_info(fs_info, "incomplete page read with offset %u and length %u", - bvec->bv_offset, bvec->bv_len); + bvec.bv_offset, bvec.bv_len); - start = page_offset(page) + bvec->bv_offset; - end = start + bvec->bv_len - 1; - len = bvec->bv_len; + start = page_offset(page) + bvec.bv_offset; + end = start + bvec.bv_len - 1; + len = bvec.bv_len; mirror = bbio->mirror_num; if (uptodate && !is_data_inode(inode) && @@ -1993,7 +1993,7 @@ static void end_bio_subpage_eb_writepage(struct btrfs_bio *bbio) { struct bio *bio = &bbio->bio; struct btrfs_fs_info *fs_info; - struct bio_vec *bvec; + struct bio_vec bvec; struct bvec_iter_all iter_all; fs_info = btrfs_sb(bio_first_page_all(bio)->mapping->host->i_sb); @@ -2001,12 +2001,12 @@ static void end_bio_subpage_eb_writepage(struct btrfs_bio *bbio) ASSERT(!bio_flagged(bio, BIO_CLONED)); bio_for_each_segment_all(bvec, bio, iter_all) { - struct page *page = bvec->bv_page; - u64 bvec_start = page_offset(page) + bvec->bv_offset; - u64 bvec_end = bvec_start + bvec->bv_len - 1; + struct page *page = bvec.bv_page; + u64 bvec_start = page_offset(page) + bvec.bv_offset; + u64 bvec_end = bvec_start + bvec.bv_len - 1; u64 cur_bytenr = bvec_start; - ASSERT(IS_ALIGNED(bvec->bv_len, fs_info->nodesize)); + ASSERT(IS_ALIGNED(bvec.bv_len, fs_info->nodesize)); /* Iterate through all extent buffers in the range */ while (cur_bytenr <= bvec_end) { @@ -2050,14 +2050,14 @@ static void end_bio_subpage_eb_writepage(struct btrfs_bio *bbio) static void end_bio_extent_buffer_writepage(struct btrfs_bio *bbio) { struct bio *bio = &bbio->bio; - struct bio_vec *bvec; + struct bio_vec bvec; struct extent_buffer *eb; int done; struct bvec_iter_all iter_all; ASSERT(!bio_flagged(bio, BIO_CLONED)); bio_for_each_segment_all(bvec, bio, iter_all) { - struct page *page = bvec->bv_page; + struct page *page = bvec.bv_page; eb = (struct extent_buffer *)page->private; BUG_ON(!eb); diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c index 642828c1b2..39d8101541 100644 --- a/fs/btrfs/raid56.c +++ b/fs/btrfs/raid56.c @@ -1388,7 +1388,7 @@ static struct sector_ptr *find_stripe_sector(struct btrfs_raid_bio *rbio, static void set_bio_pages_uptodate(struct btrfs_raid_bio *rbio, struct bio *bio) { const u32 sectorsize = rbio->bioc->fs_info->sectorsize; - struct bio_vec *bvec; + struct bio_vec bvec; struct bvec_iter_all iter_all; ASSERT(!bio_flagged(bio, BIO_CLONED)); @@ -1397,9 +1397,9 @@ static void set_bio_pages_uptodate(struct btrfs_raid_bio *rbio, struct bio *bio) struct sector_ptr *sector; int pgoff; - for (pgoff = bvec->bv_offset; pgoff - bvec->bv_offset < bvec->bv_len; + for (pgoff = bvec.bv_offset; pgoff - bvec.bv_offset < bvec.bv_len; pgoff += sectorsize) { - sector = find_stripe_sector(rbio, bvec->bv_page, pgoff); + sector = find_stripe_sector(rbio, bvec.bv_page, pgoff); ASSERT(sector); if (sector) sector->uptodate = 1; @@ -1453,7 +1453,7 @@ static void verify_bio_data_sectors(struct btrfs_raid_bio *rbio, { struct btrfs_fs_info *fs_info = rbio->bioc->fs_info; int total_sector_nr = get_bio_sector_nr(rbio, bio); - struct bio_vec *bvec; + struct bio_vec bvec; struct bvec_iter_all iter_all; /* No data csum for the whole stripe, no need to verify. */ @@ -1467,8 +1467,8 @@ static void verify_bio_data_sectors(struct btrfs_raid_bio *rbio, bio_for_each_segment_all(bvec, bio, iter_all) { int bv_offset; - for (bv_offset = bvec->bv_offset; - bv_offset < bvec->bv_offset + bvec->bv_len; + for (bv_offset = bvec.bv_offset; + bv_offset < bvec.bv_offset + bvec.bv_len; bv_offset += fs_info->sectorsize, total_sector_nr++) { u8 csum_buf[BTRFS_CSUM_SIZE]; u8 *expected_csum = rbio->csum_buf + @@ -1479,7 +1479,7 @@ static void verify_bio_data_sectors(struct btrfs_raid_bio *rbio, if (!test_bit(total_sector_nr, rbio->csum_bitmap)) continue; - ret = btrfs_check_sector_csum(fs_info, bvec->bv_page, + ret = btrfs_check_sector_csum(fs_info, bvec.bv_page, bv_offset, csum_buf, expected_csum); if (ret < 0) set_bit(total_sector_nr, rbio->error_bitmap); diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index f1708c77a9..1fd0f01d11 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -1651,11 +1651,11 @@ static void z_erofs_decompressqueue_endio(struct bio *bio) { struct z_erofs_decompressqueue *q = bio->bi_private; blk_status_t err = bio->bi_status; - struct bio_vec *bvec; + struct bio_vec bvec; struct bvec_iter_all iter_all; bio_for_each_segment_all(bvec, bio, iter_all) { - struct page *page = bvec->bv_page; + struct page *page = bvec.bv_page; DBG_BUGON(PageUptodate(page)); DBG_BUGON(z_erofs_page_is_invalidated(page)); diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c index 1e4db96a04..81a1cc4518 100644 --- a/fs/ext4/page-io.c +++ b/fs/ext4/page-io.c @@ -99,15 +99,15 @@ static void buffer_io_error(struct buffer_head *bh) static void ext4_finish_bio(struct bio *bio) { - struct bio_vec *bvec; + struct bio_vec bvec; struct bvec_iter_all iter_all; bio_for_each_segment_all(bvec, bio, iter_all) { - struct page *page = bvec->bv_page; + struct page *page = bvec.bv_page; struct page *bounce_page = NULL; struct buffer_head *bh, *head; - unsigned bio_start = bvec->bv_offset; - unsigned bio_end = bio_start + bvec->bv_len; + unsigned bio_start = bvec.bv_offset; + unsigned bio_end = bio_start + bvec.bv_len; unsigned under_io = 0; unsigned long flags; diff --git a/fs/ext4/readpage.c b/fs/ext4/readpage.c index c61dc8a7c0..ce42b3d5c9 100644 --- a/fs/ext4/readpage.c +++ b/fs/ext4/readpage.c @@ -69,11 +69,11 @@ struct bio_post_read_ctx { static void __read_end_io(struct bio *bio) { struct page *page; - struct bio_vec *bv; + struct bio_vec bv; struct bvec_iter_all iter_all; bio_for_each_segment_all(bv, bio, iter_all) { - page = bv->bv_page; + page = bv.bv_page; if (bio->bi_status) ClearPageUptodate(page); diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index 06b552a0ab..e44bd8586f 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -139,12 +139,12 @@ struct bio_post_read_ctx { */ static void f2fs_finish_read_bio(struct bio *bio, bool in_task) { - struct bio_vec *bv; + struct bio_vec bv; struct bvec_iter_all iter_all; struct bio_post_read_ctx *ctx = bio->bi_private; bio_for_each_segment_all(bv, bio, iter_all) { - struct page *page = bv->bv_page; + struct page *page = bv.bv_page; if (f2fs_is_compressed_page(page)) { if (ctx && !ctx->decompression_attempted) @@ -189,11 +189,11 @@ static void f2fs_verify_bio(struct work_struct *work) * as those were handled separately by f2fs_end_read_compressed_page(). */ if (may_have_compressed_pages) { - struct bio_vec *bv; + struct bio_vec bv; struct bvec_iter_all iter_all; bio_for_each_segment_all(bv, bio, iter_all) { - struct page *page = bv->bv_page; + struct page *page = bv.bv_page; if (!f2fs_is_compressed_page(page) && !fsverity_verify_page(page)) { @@ -241,13 +241,13 @@ static void f2fs_verify_and_finish_bio(struct bio *bio, bool in_task) static void f2fs_handle_step_decompress(struct bio_post_read_ctx *ctx, bool in_task) { - struct bio_vec *bv; + struct bio_vec bv; struct bvec_iter_all iter_all; bool all_compressed = true; block_t blkaddr = ctx->fs_blkaddr; bio_for_each_segment_all(bv, ctx->bio, iter_all) { - struct page *page = bv->bv_page; + struct page *page = bv.bv_page; if (f2fs_is_compressed_page(page)) f2fs_end_read_compressed_page(page, false, blkaddr, @@ -327,7 +327,7 @@ static void f2fs_read_end_io(struct bio *bio) static void f2fs_write_end_io(struct bio *bio) { struct f2fs_sb_info *sbi; - struct bio_vec *bvec; + struct bio_vec bvec; struct bvec_iter_all iter_all; iostat_update_and_unbind_ctx(bio); @@ -337,7 +337,7 @@ static void f2fs_write_end_io(struct bio *bio) bio->bi_status = BLK_STS_IOERR; bio_for_each_segment_all(bvec, bio, iter_all) { - struct page *page = bvec->bv_page; + struct page *page = bvec.bv_page; enum count_type type = WB_DATA_TYPE(page); if (page_private_dummy(page)) { @@ -583,7 +583,7 @@ static void __submit_merged_bio(struct f2fs_bio_info *io) static bool __has_merged_page(struct bio *bio, struct inode *inode, struct page *page, nid_t ino) { - struct bio_vec *bvec; + struct bio_vec bvec; struct bvec_iter_all iter_all; if (!bio) @@ -593,7 +593,7 @@ static bool __has_merged_page(struct bio *bio, struct inode *inode, return true; bio_for_each_segment_all(bvec, bio, iter_all) { - struct page *target = bvec->bv_page; + struct page *target = bvec.bv_page; if (fscrypt_is_bounce_page(target)) { target = fscrypt_pagecache_page(target); diff --git a/fs/gfs2/lops.c b/fs/gfs2/lops.c index 1902413d5d..7f62fe8eb7 100644 --- a/fs/gfs2/lops.c +++ b/fs/gfs2/lops.c @@ -202,7 +202,7 @@ static void gfs2_end_log_write_bh(struct gfs2_sbd *sdp, static void gfs2_end_log_write(struct bio *bio) { struct gfs2_sbd *sdp = bio->bi_private; - struct bio_vec *bvec; + struct bio_vec bvec; struct page *page; struct bvec_iter_all iter_all; @@ -217,9 +217,9 @@ static void gfs2_end_log_write(struct bio *bio) } bio_for_each_segment_all(bvec, bio, iter_all) { - page = bvec->bv_page; + page = bvec.bv_page; if (page_has_buffers(page)) - gfs2_end_log_write_bh(sdp, bvec, bio->bi_status); + gfs2_end_log_write_bh(sdp, &bvec, bio->bi_status); else mempool_free(page, gfs2_page_pool); } @@ -395,11 +395,11 @@ static void gfs2_log_write_page(struct gfs2_sbd *sdp, struct page *page) static void gfs2_end_log_read(struct bio *bio) { struct page *page; - struct bio_vec *bvec; + struct bio_vec bvec; struct bvec_iter_all iter_all; bio_for_each_segment_all(bvec, bio, iter_all) { - page = bvec->bv_page; + page = bvec.bv_page; if (bio->bi_status) { int err = blk_status_to_errno(bio->bi_status); diff --git a/fs/gfs2/meta_io.c b/fs/gfs2/meta_io.c index 924361fa51..832572784e 100644 --- a/fs/gfs2/meta_io.c +++ b/fs/gfs2/meta_io.c @@ -193,15 +193,15 @@ struct buffer_head *gfs2_meta_new(struct gfs2_glock *gl, u64 blkno) static void gfs2_meta_read_endio(struct bio *bio) { - struct bio_vec *bvec; + struct bio_vec bvec; struct bvec_iter_all iter_all; bio_for_each_segment_all(bvec, bio, iter_all) { - struct page *page = bvec->bv_page; + struct page *page = bvec.bv_page; struct buffer_head *bh = page_buffers(page); - unsigned int len = bvec->bv_len; + unsigned int len = bvec.bv_len; - while (bh_offset(bh) < bvec->bv_offset) + while (bh_offset(bh) < bvec.bv_offset) bh = bh->b_this_page; do { struct buffer_head *next = bh->b_this_page; diff --git a/fs/mpage.c b/fs/mpage.c index 22b9de5ddd..49505456ba 100644 --- a/fs/mpage.c +++ b/fs/mpage.c @@ -45,11 +45,11 @@ */ static void mpage_end_io(struct bio *bio) { - struct bio_vec *bv; + struct bio_vec bv; struct bvec_iter_all iter_all; bio_for_each_segment_all(bv, bio, iter_all) { - struct page *page = bv->bv_page; + struct page *page = bv.bv_page; page_endio(page, bio_op(bio), blk_status_to_errno(bio->bi_status)); } diff --git a/fs/squashfs/block.c b/fs/squashfs/block.c index bed3bb8b27..83e8b44518 100644 --- a/fs/squashfs/block.c +++ b/fs/squashfs/block.c @@ -35,30 +35,33 @@ static int copy_bio_to_actor(struct bio *bio, int offset, int req_length) { void *actor_addr; - struct bvec_iter_all iter_all = {}; - struct bio_vec *bvec = bvec_init_iter_all(&iter_all); + struct bvec_iter_all iter; + struct bio_vec bvec; int copied_bytes = 0; int actor_offset = 0; + int bytes_to_copy; squashfs_actor_nobuff(actor); actor_addr = squashfs_first_page(actor); - if (WARN_ON_ONCE(!bio_next_segment(bio, &iter_all))) - return 0; + bvec_iter_all_init(&iter); + bio_iter_all_advance(bio, &iter, offset); - while (copied_bytes < req_length) { - int bytes_to_copy = min_t(int, bvec->bv_len - offset, + while (copied_bytes < req_length && + iter.idx < bio->bi_vcnt) { + bvec = bio_iter_all_peek(bio, &iter); + + bytes_to_copy = min_t(int, bvec.bv_len, PAGE_SIZE - actor_offset); bytes_to_copy = min_t(int, bytes_to_copy, req_length - copied_bytes); if (!IS_ERR(actor_addr)) - memcpy(actor_addr + actor_offset, bvec_virt(bvec) + - offset, bytes_to_copy); + memcpy(actor_addr + actor_offset, bvec_virt(&bvec), + bytes_to_copy); actor_offset += bytes_to_copy; copied_bytes += bytes_to_copy; - offset += bytes_to_copy; if (actor_offset >= PAGE_SIZE) { actor_addr = squashfs_next_page(actor); @@ -66,11 +69,8 @@ static int copy_bio_to_actor(struct bio *bio, break; actor_offset = 0; } - if (offset >= bvec->bv_len) { - if (!bio_next_segment(bio, &iter_all)) - break; - offset = 0; - } + + bio_iter_all_advance(bio, &iter, bytes_to_copy); } squashfs_finish_page(actor); return copied_bytes; @@ -159,8 +159,10 @@ int squashfs_read_data(struct super_block *sb, u64 index, int length, * Metadata block. */ const u8 *data; - struct bvec_iter_all iter_all = {}; - struct bio_vec *bvec = bvec_init_iter_all(&iter_all); + struct bvec_iter_all iter; + struct bio_vec bvec; + + bvec_iter_all_init(&iter); if (index + 2 > msblk->bytes_used) { res = -EIO; @@ -170,21 +172,25 @@ int squashfs_read_data(struct super_block *sb, u64 index, int length, if (res) goto out; - if (WARN_ON_ONCE(!bio_next_segment(bio, &iter_all))) { + bvec = bio_iter_all_peek(bio, &iter); + + if (WARN_ON_ONCE(!bvec.bv_len)) { res = -EIO; goto out_free_bio; } /* Extract the length of the metadata block */ - data = bvec_virt(bvec); + data = bvec_virt(&bvec); length = data[offset]; - if (offset < bvec->bv_len - 1) { + if (offset < bvec.bv_len - 1) { length |= data[offset + 1] << 8; } else { - if (WARN_ON_ONCE(!bio_next_segment(bio, &iter_all))) { + bio_iter_all_advance(bio, &iter, bvec.bv_len); + + if (WARN_ON_ONCE(!bvec.bv_len)) { res = -EIO; goto out_free_bio; } - data = bvec_virt(bvec); + data = bvec_virt(&bvec); length |= data[0] << 8; } bio_free_pages(bio); diff --git a/fs/squashfs/lz4_wrapper.c b/fs/squashfs/lz4_wrapper.c index 49797729f1..bd0dd787d2 100644 --- a/fs/squashfs/lz4_wrapper.c +++ b/fs/squashfs/lz4_wrapper.c @@ -92,20 +92,23 @@ static int lz4_uncompress(struct squashfs_sb_info *msblk, void *strm, struct bio *bio, int offset, int length, struct squashfs_page_actor *output) { - struct bvec_iter_all iter_all = {}; - struct bio_vec *bvec = bvec_init_iter_all(&iter_all); + struct bvec_iter_all iter; + struct bio_vec bvec; struct squashfs_lz4 *stream = strm; void *buff = stream->input, *data; int bytes = length, res; - while (bio_next_segment(bio, &iter_all)) { - int avail = min(bytes, ((int)bvec->bv_len) - offset); + bvec_iter_all_init(&iter); + bio_iter_all_advance(bio, &iter, offset); - data = bvec_virt(bvec); - memcpy(buff, data + offset, avail); + bio_for_each_segment_all_continue(bvec, bio, iter) { + unsigned avail = min_t(unsigned, bytes, bvec.bv_len); + + memcpy(buff, bvec_virt(&bvec), avail); buff += avail; bytes -= avail; - offset = 0; + if (!bytes) + break; } res = LZ4_decompress_safe(stream->input, stream->output, diff --git a/fs/squashfs/lzo_wrapper.c b/fs/squashfs/lzo_wrapper.c index d216aeefa8..bccfcfa12e 100644 --- a/fs/squashfs/lzo_wrapper.c +++ b/fs/squashfs/lzo_wrapper.c @@ -66,21 +66,24 @@ static int lzo_uncompress(struct squashfs_sb_info *msblk, void *strm, struct bio *bio, int offset, int length, struct squashfs_page_actor *output) { - struct bvec_iter_all iter_all = {}; - struct bio_vec *bvec = bvec_init_iter_all(&iter_all); + struct bvec_iter_all iter; + struct bio_vec bvec; struct squashfs_lzo *stream = strm; void *buff = stream->input, *data; int bytes = length, res; size_t out_len = output->length; - while (bio_next_segment(bio, &iter_all)) { - int avail = min(bytes, ((int)bvec->bv_len) - offset); + bvec_iter_all_init(&iter); + bio_iter_all_advance(bio, &iter, offset); - data = bvec_virt(bvec); - memcpy(buff, data + offset, avail); + bio_for_each_segment_all_continue(bvec, bio, iter) { + unsigned avail = min_t(unsigned, bytes, bvec.bv_len); + + memcpy(buff, bvec_virt(&bvec), avail); buff += avail; bytes -= avail; - offset = 0; + if (!bytes) + break; } res = lzo1x_decompress_safe(stream->input, (size_t)length, diff --git a/fs/squashfs/xz_wrapper.c b/fs/squashfs/xz_wrapper.c index 6c49481a2f..6cf0e11e3b 100644 --- a/fs/squashfs/xz_wrapper.c +++ b/fs/squashfs/xz_wrapper.c @@ -120,8 +120,7 @@ static int squashfs_xz_uncompress(struct squashfs_sb_info *msblk, void *strm, struct bio *bio, int offset, int length, struct squashfs_page_actor *output) { - struct bvec_iter_all iter_all = {}; - struct bio_vec *bvec = bvec_init_iter_all(&iter_all); + struct bvec_iter_all iter; int total = 0, error = 0; struct squashfs_xz *stream = strm; @@ -136,26 +135,28 @@ static int squashfs_xz_uncompress(struct squashfs_sb_info *msblk, void *strm, goto finish; } + bvec_iter_all_init(&iter); + bio_iter_all_advance(bio, &iter, offset); + for (;;) { enum xz_ret xz_err; if (stream->buf.in_pos == stream->buf.in_size) { - const void *data; - int avail; + struct bio_vec bvec = bio_iter_all_peek(bio, &iter); + unsigned avail = min_t(unsigned, length, bvec.bv_len); - if (!bio_next_segment(bio, &iter_all)) { + if (iter.idx >= bio->bi_vcnt) { /* XZ_STREAM_END must be reached. */ error = -EIO; break; } - avail = min(length, ((int)bvec->bv_len) - offset); - data = bvec_virt(bvec); length -= avail; - stream->buf.in = data + offset; + stream->buf.in = bvec_virt(&bvec); stream->buf.in_size = avail; stream->buf.in_pos = 0; - offset = 0; + + bio_iter_all_advance(bio, &iter, avail); } if (stream->buf.out_pos == stream->buf.out_size) { diff --git a/fs/squashfs/zlib_wrapper.c b/fs/squashfs/zlib_wrapper.c index cbb7afe7bc..981ca5e410 100644 --- a/fs/squashfs/zlib_wrapper.c +++ b/fs/squashfs/zlib_wrapper.c @@ -53,8 +53,7 @@ static int zlib_uncompress(struct squashfs_sb_info *msblk, void *strm, struct bio *bio, int offset, int length, struct squashfs_page_actor *output) { - struct bvec_iter_all iter_all = {}; - struct bio_vec *bvec = bvec_init_iter_all(&iter_all); + struct bvec_iter_all iter; int zlib_init = 0, error = 0; z_stream *stream = strm; @@ -67,25 +66,28 @@ static int zlib_uncompress(struct squashfs_sb_info *msblk, void *strm, goto finish; } + bvec_iter_all_init(&iter); + bio_iter_all_advance(bio, &iter, offset); + for (;;) { int zlib_err; if (stream->avail_in == 0) { - const void *data; + struct bio_vec bvec = bio_iter_all_peek(bio, &iter); int avail; - if (!bio_next_segment(bio, &iter_all)) { + if (iter.idx >= bio->bi_vcnt) { /* Z_STREAM_END must be reached. */ error = -EIO; break; } - avail = min(length, ((int)bvec->bv_len) - offset); - data = bvec_virt(bvec); + avail = min_t(unsigned, length, bvec.bv_len); length -= avail; - stream->next_in = data + offset; + stream->next_in = bvec_virt(&bvec); stream->avail_in = avail; - offset = 0; + + bio_iter_all_advance(bio, &iter, avail); } if (stream->avail_out == 0) { diff --git a/fs/squashfs/zstd_wrapper.c b/fs/squashfs/zstd_wrapper.c index 0e407c4d8b..658e5d462a 100644 --- a/fs/squashfs/zstd_wrapper.c +++ b/fs/squashfs/zstd_wrapper.c @@ -68,8 +68,7 @@ static int zstd_uncompress(struct squashfs_sb_info *msblk, void *strm, int error = 0; zstd_in_buffer in_buf = { NULL, 0, 0 }; zstd_out_buffer out_buf = { NULL, 0, 0 }; - struct bvec_iter_all iter_all = {}; - struct bio_vec *bvec = bvec_init_iter_all(&iter_all); + struct bvec_iter_all iter; stream = zstd_init_dstream(wksp->window_size, wksp->mem, wksp->mem_size); @@ -85,25 +84,27 @@ static int zstd_uncompress(struct squashfs_sb_info *msblk, void *strm, goto finish; } + bvec_iter_all_init(&iter); + bio_iter_all_advance(bio, &iter, offset); + for (;;) { size_t zstd_err; if (in_buf.pos == in_buf.size) { - const void *data; - int avail; + struct bio_vec bvec = bio_iter_all_peek(bio, &iter); + unsigned avail = min_t(unsigned, length, bvec.bv_len); - if (!bio_next_segment(bio, &iter_all)) { + if (iter.idx >= bio->bi_vcnt) { error = -EIO; break; } - avail = min(length, ((int)bvec->bv_len) - offset); - data = bvec_virt(bvec); length -= avail; - in_buf.src = data + offset; + in_buf.src = bvec_virt(&bvec); in_buf.size = avail; in_buf.pos = 0; - offset = 0; + + bio_iter_all_advance(bio, &iter, avail); } if (out_buf.pos == out_buf.size) { diff --git a/include/linux/bio.h b/include/linux/bio.h index 3536f28c05..f86c7190c3 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -78,22 +78,40 @@ static inline void *bio_data(struct bio *bio) return NULL; } -static inline bool bio_next_segment(const struct bio *bio, - struct bvec_iter_all *iter) +static inline struct bio_vec bio_iter_all_peek(const struct bio *bio, + struct bvec_iter_all *iter) { - if (iter->idx >= bio->bi_vcnt) - return false; + if (WARN_ON(iter->idx >= bio->bi_vcnt)) + return (struct bio_vec) { NULL }; - bvec_advance(&bio->bi_io_vec[iter->idx], iter); - return true; + return bvec_iter_all_peek(bio->bi_io_vec, iter); +} + +static inline void bio_iter_all_advance(const struct bio *bio, + struct bvec_iter_all *iter, + unsigned bytes) +{ + bvec_iter_all_advance(bio->bi_io_vec, iter, bytes); + + WARN_ON(iter->idx > bio->bi_vcnt || + (iter->idx == bio->bi_vcnt && iter->done)); } +#define bio_for_each_segment_all_continue(bvl, bio, iter) \ + for (; \ + iter.idx < bio->bi_vcnt && \ + ((bvl = bio_iter_all_peek(bio, &iter)), true); \ + bio_iter_all_advance((bio), &iter, bvl.bv_len)) + /* * drivers should _never_ use the all version - the bio may have been split * before it got to the driver and the driver won't own all of it */ -#define bio_for_each_segment_all(bvl, bio, iter) \ - for (bvl = bvec_init_iter_all(&iter); bio_next_segment((bio), &iter); ) +#define bio_for_each_segment_all(bvl, bio, iter) \ + for (bvec_iter_all_init(&iter); \ + iter.idx < (bio)->bi_vcnt && \ + ((bvl = bio_iter_all_peek((bio), &iter)), true); \ + bio_iter_all_advance((bio), &iter, bvl.bv_len)) static inline void bio_advance_iter(const struct bio *bio, struct bvec_iter *iter, unsigned int bytes) diff --git a/include/linux/bvec.h b/include/linux/bvec.h index 555aae5448..635fb54143 100644 --- a/include/linux/bvec.h +++ b/include/linux/bvec.h @@ -85,12 +85,6 @@ struct bvec_iter { current bvec */ } __packed; -struct bvec_iter_all { - struct bio_vec bv; - int idx; - unsigned done; -}; - /* * various member access, note that bio_data should of course not be used * on highmem page vectors @@ -184,7 +178,10 @@ static inline void bvec_iter_advance_single(const struct bio_vec *bv, ((bvl = bvec_iter_bvec((bio_vec), (iter))), 1); \ bvec_iter_advance_single((bio_vec), &(iter), (bvl).bv_len)) -/* for iterating one bio from start to end */ +/* + * bvec_iter_all: for advancing over a bio as it was originally created, but + * with the usual bio_for_each_segment interface - nonstandard, do not use: + */ #define BVEC_ITER_ALL_INIT (struct bvec_iter) \ { \ .bi_sector = 0, \ @@ -193,33 +190,45 @@ static inline void bvec_iter_advance_single(const struct bio_vec *bv, .bi_bvec_done = 0, \ } -static inline struct bio_vec *bvec_init_iter_all(struct bvec_iter_all *iter_all) +/* + * bvec_iter_all: for advancing over individual pages in a bio, as it was when + * it was first created: + */ +struct bvec_iter_all { + int idx; + unsigned done; +}; + +static inline void bvec_iter_all_init(struct bvec_iter_all *iter_all) { iter_all->done = 0; iter_all->idx = 0; +} - return &iter_all->bv; +static inline struct bio_vec bvec_iter_all_peek(const struct bio_vec *bvec, + struct bvec_iter_all *iter) +{ + struct bio_vec bv = bvec[iter->idx]; + + bv.bv_offset += iter->done; + bv.bv_len -= iter->done; + + bv.bv_page += bv.bv_offset >> PAGE_SHIFT; + bv.bv_offset &= ~PAGE_MASK; + bv.bv_len = min_t(unsigned, PAGE_SIZE - bv.bv_offset, bv.bv_len); + + return bv; } -static inline void bvec_advance(const struct bio_vec *bvec, - struct bvec_iter_all *iter_all) +static inline void bvec_iter_all_advance(const struct bio_vec *bvec, + struct bvec_iter_all *iter, + unsigned bytes) { - struct bio_vec *bv = &iter_all->bv; - - if (iter_all->done) { - bv->bv_page++; - bv->bv_offset = 0; - } else { - bv->bv_page = bvec->bv_page + (bvec->bv_offset >> PAGE_SHIFT); - bv->bv_offset = bvec->bv_offset & ~PAGE_MASK; - } - bv->bv_len = min_t(unsigned int, PAGE_SIZE - bv->bv_offset, - bvec->bv_len - iter_all->done); - iter_all->done += bv->bv_len; + iter->done += bytes; - if (iter_all->done == bvec->bv_len) { - iter_all->idx++; - iter_all->done = 0; + while (iter->done && iter->done >= bvec[iter->idx].bv_len) { + iter->done -= bvec[iter->idx].bv_len; + iter->idx++; } } From patchwork Tue May 9 16:56:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91669 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3037511vqo; Tue, 9 May 2023 10:04:36 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7yv6pvpDiY8+PJxr1B8RQJ55YBzSOyD9xf60kECrgKqKiPnw/T1YGafalXy6nVjwI4z67b X-Received: by 2002:a17:90a:c090:b0:24b:347b:2a0f with SMTP id o16-20020a17090ac09000b0024b347b2a0fmr14298157pjs.40.1683651875864; Tue, 09 May 2023 10:04:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683651875; cv=none; d=google.com; s=arc-20160816; b=SAsZfnSFMCCuSKDfkYWZkOfuLsrnUwgYCou/UMtaIsdzR674554YgbtBlMy7Y6lEVb 3JhlbJpPL9LQXGOMTfoUqiNiMMGotq5iB9sL6xgTPDmy6SL4eTBir3quW3RdGzoxnQtP VsOsWZK9yMopXBKdgI7rvtLqri2GBkeviRd78Puk04QTQgywgHTz6ArrW8TdjVUmJg6p FJKeR1fnSJDwApQE6qTZfKhMAW6xvH6aBtda/mi3SPwKAeKh10F8rT2w6jbUwCsLNgPS /JygEFAuCxPCQO4ZMC8OZ4+rwF4ZnB5uZva6XJCKlqxSCEoxUxnOAt3DX75yQ6qZLT0H Qfgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vmriaVgD/FyMiWTB+roCgu5nb9HM588jx+szztY1PGY=; b=oODTWmoadJ1AG4okpOXymR6Cr1y+RdGPep89vGhiq3YZuY2omUUXXFZvvyEHTiX0Y9 KSQWc9VUX55K19t5guSYJa5c5KMwQ4wIPFRtTZQHmMx7zl2lAcLeIe4KiwNGlibPh6jb azL0u3Pfcc47ZGib3hoe3ljgS6nmQwPzXnDwuLvPWDQrgJez73sXi99US/jsMqdPYcCx qQs+3zSKyyEck6bD1eOJqBzBz2lSUHFosFX0gdXIhFbO54TMvhvBDCJt5xwJYGmBFI+b tRK6tKZVqbZ7kBivxSIZv1zWc9EMzgBO+PnicBY17MMBgJUr4winiRVwoxju/vcZlcIA XfkA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Y8cJ4uv2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ay8-20020a1709028b8800b001ac482e25a0si1833486plb.374.2023.05.09.10.04.20; Tue, 09 May 2023 10:04:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Y8cJ4uv2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234929AbjEIQ6j (ORCPT + 99 others); Tue, 9 May 2023 12:58:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234798AbjEIQ6E (ORCPT ); Tue, 9 May 2023 12:58:04 -0400 Received: from out-57.mta1.migadu.com (out-57.mta1.migadu.com [IPv6:2001:41d0:203:375::39]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D75B423C for ; Tue, 9 May 2023 09:57:24 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651443; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vmriaVgD/FyMiWTB+roCgu5nb9HM588jx+szztY1PGY=; b=Y8cJ4uv28KMitwKkI+rMDqec7BQ2sqWlk0uW6vvAlG9jpKl1sie9uI9aDFHiFVb13u6wND clc46Jy46U5eEEXSGiLDH/Y7ZmJRRCvaF1kF5jdMQUo8DDKWHlDC3UIxIbGMs6goGBumcG BFSx62tvZjG0iuHHX2esbeVWEw0JrYU= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Matthew Wilcox , linux-block@vger.kernel.org Subject: [PATCH 13/32] block: Rework bio_for_each_folio_all() Date: Tue, 9 May 2023 12:56:38 -0400 Message-Id: <20230509165657.1735798-14-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765436949340917879?= X-GMAIL-MSGID: =?utf-8?q?1765436949340917879?= This reimplements bio_for_each_folio_all() on top of the newly-reworked bvec_iter_all, and since it's now trivial we also provide bio_for_each_folio. Signed-off-by: Kent Overstreet Cc: Matthew Wilcox Cc: linux-block@vger.kernel.org --- fs/crypto/bio.c | 9 +++-- fs/iomap/buffered-io.c | 14 ++++--- fs/verity/verify.c | 9 +++-- include/linux/bio.h | 91 +++++++++++++++++++++--------------------- include/linux/bvec.h | 15 +++++-- 5 files changed, 75 insertions(+), 63 deletions(-) diff --git a/fs/crypto/bio.c b/fs/crypto/bio.c index d57d0a020f..6469861add 100644 --- a/fs/crypto/bio.c +++ b/fs/crypto/bio.c @@ -30,11 +30,12 @@ */ bool fscrypt_decrypt_bio(struct bio *bio) { - struct folio_iter fi; + struct bvec_iter_all iter; + struct folio_vec fv; - bio_for_each_folio_all(fi, bio) { - int err = fscrypt_decrypt_pagecache_blocks(fi.folio, fi.length, - fi.offset); + bio_for_each_folio_all(fv, bio, iter) { + int err = fscrypt_decrypt_pagecache_blocks(fv.fv_folio, fv.fv_len, + fv.fv_offset); if (err) { bio->bi_status = errno_to_blk_status(err); diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index 6f4c97a6d7..60661c87d5 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -187,10 +187,11 @@ static void iomap_finish_folio_read(struct folio *folio, size_t offset, static void iomap_read_end_io(struct bio *bio) { int error = blk_status_to_errno(bio->bi_status); - struct folio_iter fi; + struct bvec_iter_all iter; + struct folio_vec fv; - bio_for_each_folio_all(fi, bio) - iomap_finish_folio_read(fi.folio, fi.offset, fi.length, error); + bio_for_each_folio_all(fv, bio, iter) + iomap_finish_folio_read(fv.fv_folio, fv.fv_offset, fv.fv_len, error); bio_put(bio); } @@ -1328,7 +1329,8 @@ iomap_finish_ioend(struct iomap_ioend *ioend, int error) u32 folio_count = 0; for (bio = &ioend->io_inline_bio; bio; bio = next) { - struct folio_iter fi; + struct bvec_iter_all iter; + struct folio_vec fv; /* * For the last bio, bi_private points to the ioend, so we @@ -1340,8 +1342,8 @@ iomap_finish_ioend(struct iomap_ioend *ioend, int error) next = bio->bi_private; /* walk all folios in bio, ending page IO on them */ - bio_for_each_folio_all(fi, bio) { - iomap_finish_folio_write(inode, fi.folio, fi.length, + bio_for_each_folio_all(fv, bio, iter) { + iomap_finish_folio_write(inode, fv.fv_folio, fv.fv_len, error); folio_count++; } diff --git a/fs/verity/verify.c b/fs/verity/verify.c index e250822275..b111ab0102 100644 --- a/fs/verity/verify.c +++ b/fs/verity/verify.c @@ -340,7 +340,8 @@ void fsverity_verify_bio(struct bio *bio) struct inode *inode = bio_first_page_all(bio)->mapping->host; struct fsverity_info *vi = inode->i_verity_info; struct ahash_request *req; - struct folio_iter fi; + struct bvec_iter_all iter; + struct folio_vec fv; unsigned long max_ra_pages = 0; /* This allocation never fails, since it's mempool-backed. */ @@ -359,9 +360,9 @@ void fsverity_verify_bio(struct bio *bio) max_ra_pages = bio->bi_iter.bi_size >> (PAGE_SHIFT + 2); } - bio_for_each_folio_all(fi, bio) { - if (!verify_data_blocks(inode, vi, req, fi.folio, fi.length, - fi.offset, max_ra_pages)) { + bio_for_each_folio_all(fv, bio, iter) { + if (!verify_data_blocks(inode, vi, req, fv.fv_folio, fv.fv_len, + fv.fv_offset, max_ra_pages)) { bio->bi_status = BLK_STS_IOERR; break; } diff --git a/include/linux/bio.h b/include/linux/bio.h index f86c7190c3..7ced281734 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -169,6 +169,42 @@ static inline void bio_advance(struct bio *bio, unsigned int nbytes) #define bio_for_each_segment(bvl, bio, iter) \ __bio_for_each_segment(bvl, bio, iter, (bio)->bi_iter) +struct folio_vec { + struct folio *fv_folio; + size_t fv_offset; + size_t fv_len; +}; + +static inline struct folio_vec biovec_to_foliovec(struct bio_vec bv) +{ + + struct folio *folio = page_folio(bv.bv_page); + size_t offset = (folio_page_idx(folio, bv.bv_page) << PAGE_SHIFT) + + bv.bv_offset; + size_t len = min_t(size_t, folio_size(folio) - offset, bv.bv_len); + + return (struct folio_vec) { + .fv_folio = folio, + .fv_offset = offset, + .fv_len = len, + }; +} + +static inline struct folio_vec bio_iter_iovec_folio(struct bio *bio, + struct bvec_iter iter) +{ + return biovec_to_foliovec(bio_iter_iovec(bio, iter)); +} + +#define __bio_for_each_folio(bvl, bio, iter, start) \ + for (iter = (start); \ + (iter).bi_size && \ + ((bvl = bio_iter_iovec_folio((bio), (iter))), 1); \ + bio_advance_iter_single((bio), &(iter), (bvl).fv_len)) + +#define bio_for_each_folio(bvl, bio, iter) \ + __bio_for_each_folio(bvl, bio, iter, (bio)->bi_iter) + #define __bio_for_each_bvec(bvl, bio, iter, start) \ for (iter = (start); \ (iter).bi_size && \ @@ -277,59 +313,22 @@ static inline struct bio_vec *bio_last_bvec_all(struct bio *bio) return &bio->bi_io_vec[bio->bi_vcnt - 1]; } -/** - * struct folio_iter - State for iterating all folios in a bio. - * @folio: The current folio we're iterating. NULL after the last folio. - * @offset: The byte offset within the current folio. - * @length: The number of bytes in this iteration (will not cross folio - * boundary). - */ -struct folio_iter { - struct folio *folio; - size_t offset; - size_t length; - /* private: for use by the iterator */ - struct folio *_next; - size_t _seg_count; - int _i; -}; - -static inline void bio_first_folio(struct folio_iter *fi, struct bio *bio, - int i) -{ - struct bio_vec *bvec = bio_first_bvec_all(bio) + i; - - fi->folio = page_folio(bvec->bv_page); - fi->offset = bvec->bv_offset + - PAGE_SIZE * (bvec->bv_page - &fi->folio->page); - fi->_seg_count = bvec->bv_len; - fi->length = min(folio_size(fi->folio) - fi->offset, fi->_seg_count); - fi->_next = folio_next(fi->folio); - fi->_i = i; -} - -static inline void bio_next_folio(struct folio_iter *fi, struct bio *bio) +static inline struct folio_vec bio_folio_iter_all_peek(const struct bio *bio, + const struct bvec_iter_all *iter) { - fi->_seg_count -= fi->length; - if (fi->_seg_count) { - fi->folio = fi->_next; - fi->offset = 0; - fi->length = min(folio_size(fi->folio), fi->_seg_count); - fi->_next = folio_next(fi->folio); - } else if (fi->_i + 1 < bio->bi_vcnt) { - bio_first_folio(fi, bio, fi->_i + 1); - } else { - fi->folio = NULL; - } + return biovec_to_foliovec(__bvec_iter_all_peek(bio->bi_io_vec, iter)); } /** * bio_for_each_folio_all - Iterate over each folio in a bio. - * @fi: struct folio_iter which is updated for each folio. + * @fi: struct bio_folio_iter_all which is updated for each folio. * @bio: struct bio to iterate over. */ -#define bio_for_each_folio_all(fi, bio) \ - for (bio_first_folio(&fi, bio, 0); fi.folio; bio_next_folio(&fi, bio)) +#define bio_for_each_folio_all(fv, bio, iter) \ + for (bvec_iter_all_init(&iter); \ + iter.idx < bio->bi_vcnt && \ + ((fv = bio_folio_iter_all_peek(bio, &iter)), true); \ + bio_iter_all_advance((bio), &iter, fv.fv_len)) enum bip_flags { BIP_BLOCK_INTEGRITY = 1 << 0, /* block layer owns integrity data */ diff --git a/include/linux/bvec.h b/include/linux/bvec.h index 635fb54143..d238f959e3 100644 --- a/include/linux/bvec.h +++ b/include/linux/bvec.h @@ -205,18 +205,27 @@ static inline void bvec_iter_all_init(struct bvec_iter_all *iter_all) iter_all->idx = 0; } -static inline struct bio_vec bvec_iter_all_peek(const struct bio_vec *bvec, - struct bvec_iter_all *iter) +static inline struct bio_vec __bvec_iter_all_peek(const struct bio_vec *bvec, + const struct bvec_iter_all *iter) { struct bio_vec bv = bvec[iter->idx]; + BUG_ON(iter->done >= bv.bv_len); + bv.bv_offset += iter->done; bv.bv_len -= iter->done; bv.bv_page += bv.bv_offset >> PAGE_SHIFT; bv.bv_offset &= ~PAGE_MASK; - bv.bv_len = min_t(unsigned, PAGE_SIZE - bv.bv_offset, bv.bv_len); + return bv; +} + +static inline struct bio_vec bvec_iter_all_peek(const struct bio_vec *bvec, + const struct bvec_iter_all *iter) +{ + struct bio_vec bv = __bvec_iter_all_peek(bvec, iter); + bv.bv_len = min_t(unsigned, PAGE_SIZE - bv.bv_offset, bv.bv_len); return bv; } From patchwork Tue May 9 16:56:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91670 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3039849vqo; Tue, 9 May 2023 10:07:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5eOxGOAlo5noX7XjmwoQNVSWk4hYGOPFk2QV00DybDCM4/EO6Cq2GGVoAqGAfKOEhS0+3e X-Received: by 2002:a05:6a20:72a6:b0:100:57cd:cdf5 with SMTP id o38-20020a056a2072a600b0010057cdcdf5mr10159462pzk.7.1683652034359; Tue, 09 May 2023 10:07:14 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652034; cv=none; d=google.com; s=arc-20160816; b=kqOhPF/xNjECpeLH6wXLFUgMvCopfZcoUGBe5qniWGR4hkI0WjK1zkcBUEkWBj+lho guuqzNo7tNhe6f8J1XROfFHLzgS2WD0+4flKrzpCIbgRvSqXJQCtXd0Q0khBKW1WtawO RfGL9l6fnkWVDm+pqkLHbFIKfVqUZO/HYu02ofEl5jczFKVysL6o5Qb3fYJH46dSzehQ TKxLYlwgplF1dQ30j90ciom3GinLqGMIrJujYXjQJK741cp4q3SNscvFHpNAO2V4K/4z 3hqSaRPOSFFXSNoTJ7LqrZs4hLKHSbud3h1ei+EELBPRJEhEe9ZsXWsVY854Y8SbVvmf n6FA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Bsc+1yI/guH+T2KE4bJnNuVNfNr1pSTLs0rK/pjjSAI=; b=FHz9gO6esjIPa+rKDbupFdZKQQly9GqVFK2ZqRxBrUq0qWxjEchbIvkT+rKs5cD/dU qSoG0sVoPZ878YOpdCbzfsKwCZdiUSccSIlIr6pgHshAmZQ3V3dneNTASeoU4lamWaUr u+EOm2wfH07cF1e3ZFY3dRIGt7AvxW08d53WWGzEVuMIPziGXx6pJvm51fdlWXd1zPP4 0jhrz4XW/snl+VQMy6ErA3nmhY0rEtbUsAnBJnho8PAAjuE0+rwrGVuuX7qm1F4u+s0v KmwOEFCiVuDcc0dlRwJ8nR3Wj3AL76QTT9P40Ph3inYhcfldn6nw3AVGn9FpS/2ijQNs 48sA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Reg17E4h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q21-20020a637515000000b005194496616bsi1968194pgc.348.2023.05.09.10.07.01; Tue, 09 May 2023 10:07:14 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Reg17E4h; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234941AbjEIQ6p (ORCPT + 99 others); Tue, 9 May 2023 12:58:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43178 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234872AbjEIQ6E (ORCPT ); Tue, 9 May 2023 12:58:04 -0400 Received: from out-4.mta1.migadu.com (out-4.mta1.migadu.com [95.215.58.4]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 848EB59DC for ; Tue, 9 May 2023 09:57:25 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651444; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bsc+1yI/guH+T2KE4bJnNuVNfNr1pSTLs0rK/pjjSAI=; b=Reg17E4hxC/htV66caDP4mIg8lk7TjjhYySdYeFxLdwU4L7ySzXsxSI3IE/jxUtwJLdQ6e geU1GuZFaJRSE+vUsNzOEQAHx8GKD9SxSVp/CscftZ+vi6/DaJin7NUQrUWOWUGTu7OsO7 iSaIQhOKsdg5do0xv0BevZNBx62CTAM= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 14/32] block: Don't block on s_umount from __invalidate_super() Date: Tue, 9 May 2023 12:56:39 -0400 Message-Id: <20230509165657.1735798-15-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437115553586097?= X-GMAIL-MSGID: =?utf-8?q?1765437115553586097?= __invalidate_super() is used to flush any filesystem mounted on a device, generally on some sort of media change event. However, when unmounting a filesystem and closing the underlying block devices, we can deadlock if the block driver then calls __invalidate_device() (e.g. because the block device goes away when it is no longer in use). This happens with bcachefs on top of loopback, and can be triggered by fstests generic/042: put_super -> blkdev_put -> lo_release -> disk_force_media_change -> __invalidate_device -> get_super This isn't inherently specific to bcachefs - it hasn't shown up with other filesystems before because most other filesystems use the sget() mechanism for opening/closing block devices (and enforcing exclusion), however sget() has its own downsides and weird/sketchy behaviour w.r.t. block device open lifetime - if that ever gets fixed more code will run into this issue. The __invalidate_device() call here is really a best effort "I just yanked the device for a mounted filesystem, please try not to lose my data" - if it's ever actually needed the user has already done something crazy, and we probably shouldn't make things worse by deadlocking. Switching to a trylock seems in keeping with what the code is trying to do. If we ever get revoke() at the block layer, perhaps we would look at rearchitecting to use that instead. Signed-off-by: Kent Overstreet --- block/bdev.c | 2 +- fs/super.c | 40 +++++++++++++++++++++++++++++++--------- include/linux/fs.h | 1 + 3 files changed, 33 insertions(+), 10 deletions(-) diff --git a/block/bdev.c b/block/bdev.c index 1795c7d4b9..743e969b7b 100644 --- a/block/bdev.c +++ b/block/bdev.c @@ -922,7 +922,7 @@ EXPORT_SYMBOL(lookup_bdev); int __invalidate_device(struct block_device *bdev, bool kill_dirty) { - struct super_block *sb = get_super(bdev); + struct super_block *sb = try_get_super(bdev); int res = 0; if (sb) { diff --git a/fs/super.c b/fs/super.c index 04bc62ab7d..a2decce02f 100644 --- a/fs/super.c +++ b/fs/super.c @@ -791,14 +791,7 @@ void iterate_supers_type(struct file_system_type *type, EXPORT_SYMBOL(iterate_supers_type); -/** - * get_super - get the superblock of a device - * @bdev: device to get the superblock for - * - * Scans the superblock list and finds the superblock of the file system - * mounted on the device given. %NULL is returned if no match is found. - */ -struct super_block *get_super(struct block_device *bdev) +static struct super_block *__get_super(struct block_device *bdev, bool try) { struct super_block *sb; @@ -813,7 +806,12 @@ struct super_block *get_super(struct block_device *bdev) if (sb->s_bdev == bdev) { sb->s_count++; spin_unlock(&sb_lock); - down_read(&sb->s_umount); + + if (!try) + down_read(&sb->s_umount); + else if (!down_read_trylock(&sb->s_umount)) + return NULL; + /* still alive? */ if (sb->s_root && (sb->s_flags & SB_BORN)) return sb; @@ -828,6 +826,30 @@ struct super_block *get_super(struct block_device *bdev) return NULL; } +/** + * get_super - get the superblock of a device + * @bdev: device to get the superblock for + * + * Scans the superblock list and finds the superblock of the file system + * mounted on the device given. %NULL is returned if no match is found. + */ +struct super_block *get_super(struct block_device *bdev) +{ + return __get_super(bdev, false); +} + +/** + * try_get_super - get the superblock of a device, using trylock on sb->s_umount + * @bdev: device to get the superblock for + * + * Scans the superblock list and finds the superblock of the file system + * mounted on the device given. %NULL is returned if no match is found. + */ +struct super_block *try_get_super(struct block_device *bdev) +{ + return __get_super(bdev, true); +} + /** * get_active_super - get an active reference to the superblock of a device * @bdev: device to get the superblock for diff --git a/include/linux/fs.h b/include/linux/fs.h index c85916e9f7..1a6f951942 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2878,6 +2878,7 @@ extern struct file_system_type *get_filesystem(struct file_system_type *fs); extern void put_filesystem(struct file_system_type *fs); extern struct file_system_type *get_fs_type(const char *name); extern struct super_block *get_super(struct block_device *); +extern struct super_block *try_get_super(struct block_device *); extern struct super_block *get_active_super(struct block_device *bdev); extern void drop_super(struct super_block *sb); extern void drop_super_exclusive(struct super_block *sb); From patchwork Tue May 9 16:56:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91681 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3041380vqo; Tue, 9 May 2023 10:09:13 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5izRJNw7Au7eJkU6/SZgfI9tppk/+JFlT+Q14UnH+8Vdnf7OFC4wuakXbSVEh39CeUZcJS X-Received: by 2002:a05:6a20:7484:b0:f8:a493:5290 with SMTP id p4-20020a056a20748400b000f8a4935290mr19062935pzd.20.1683652152800; Tue, 09 May 2023 10:09:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652152; cv=none; d=google.com; s=arc-20160816; b=MX0QjpYCiqzwZS49nn1rP9Q/s0iFiDy5DoG+zTQgexcP6qhQq5tgwi4+MLkxDpLEi8 DvWeUg6fS8VHUjtRp/p+tkXa/ac3UqDdjBHJK//434TJrUGfgEHiyFmCmIWlu40tJOih 9fUhCyw+N58ORDGzL/8sXz69PIdKtinNucawnYsf4sxz6IUpUJiFnoUINKkOAfU4fRZC Vf48bjqIdWlUA+enIllXdA54I4JWc8N61oeuCuPLjar5YxyLaZFLGLYMR9IovT/IUHzX gqvePR9wHVrAfbZQWNbcrDY7wEj223fEj9cWGTcQiMY7/BvZaGhsTrQzIJiNl7OGEuiR j3EA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=aFpAUdd6YceND8Z+9Ovo7o0lDo/i7abO9dGwiynsQMo=; b=i6+YOqTukUoEfl3VbKkmrx7Mloaoe/iQq9KvY4BXO932g2uheELs5b4NZbZKyly/pa Fzm2mfZcpjcvVNoDbTP2e3HVVshchFMoW8GGheFQjFvHq1YodzxhyKFpXgTfxd78StoE z8r9ezCucaYJ7L1S1qlCdoW58daSKc2/qiGVOeZ5by5TFtl1hDJq82DvOJLjoNEY5759 DodBQV0cYZq7BsjkQ194TZ+8u0cSOxmbdJ7oLKadhRXpZnuqcrttedQRwaOu632tYjLd do/dXdYVq3VNVktPiw8xlSDkTAdpKX5oBjXGbsOPvhhtY+VUkC0UrgoY2Okh5SjsdEzH VtYw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=YD9COKA+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m12-20020a633f0c000000b0050fad17282dsi1861660pga.217.2023.05.09.10.08.59; Tue, 09 May 2023 10:09:12 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=YD9COKA+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235126AbjEIQ6v (ORCPT + 99 others); Tue, 9 May 2023 12:58:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234945AbjEIQ6G (ORCPT ); Tue, 9 May 2023 12:58:06 -0400 Received: from out-13.mta1.migadu.com (out-13.mta1.migadu.com [IPv6:2001:41d0:203:375::d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC1215276 for ; Tue, 9 May 2023 09:57:26 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651444; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=aFpAUdd6YceND8Z+9Ovo7o0lDo/i7abO9dGwiynsQMo=; b=YD9COKA+IAd9H4LdPcSWdpB2IIbw6IWd437dPvp6Yad9eeznBqXnIuyLzzF7tD4D6rKEPL A6OHTjd4KHrQioFYDSpvjl82KOg3gF57iatX0Pn5jOxXjU/4KlG72uUyHlEboZvxcasXkZ Euc9DchkVx5xrUMNFtVxbPMdnRiXQEA= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet , Coly Li Subject: [PATCH 15/32] bcache: move closures to lib/ Date: Tue, 9 May 2023 12:56:40 -0400 Message-Id: <20230509165657.1735798-16-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437239720946965?= X-GMAIL-MSGID: =?utf-8?q?1765437239720946965?= From: Kent Overstreet Prep work for bcachefs - being a fork of bcache it also uses closures Signed-off-by: Kent Overstreet Acked-by: Coly Li --- drivers/md/bcache/Kconfig | 10 +----- drivers/md/bcache/Makefile | 4 +-- drivers/md/bcache/bcache.h | 2 +- drivers/md/bcache/super.c | 1 - drivers/md/bcache/util.h | 3 +- .../md/bcache => include/linux}/closure.h | 17 +++++---- lib/Kconfig | 3 ++ lib/Kconfig.debug | 9 +++++ lib/Makefile | 2 ++ {drivers/md/bcache => lib}/closure.c | 35 +++++++++---------- 10 files changed, 43 insertions(+), 43 deletions(-) rename {drivers/md/bcache => include/linux}/closure.h (97%) rename {drivers/md/bcache => lib}/closure.c (88%) diff --git a/drivers/md/bcache/Kconfig b/drivers/md/bcache/Kconfig index 529c9d04e9..b2d10063d3 100644 --- a/drivers/md/bcache/Kconfig +++ b/drivers/md/bcache/Kconfig @@ -4,6 +4,7 @@ config BCACHE tristate "Block device as cache" select BLOCK_HOLDER_DEPRECATED if SYSFS select CRC64 + select CLOSURES help Allows a block device to be used as cache for other devices; uses a btree for indexing and the layout is optimized for SSDs. @@ -19,15 +20,6 @@ config BCACHE_DEBUG Enables extra debugging tools, allows expensive runtime checks to be turned on. -config BCACHE_CLOSURES_DEBUG - bool "Debug closures" - depends on BCACHE - select DEBUG_FS - help - Keeps all active closures in a linked list and provides a debugfs - interface to list them, which makes it possible to see asynchronous - operations that get stuck. - config BCACHE_ASYNC_REGISTRATION bool "Asynchronous device registration" depends on BCACHE diff --git a/drivers/md/bcache/Makefile b/drivers/md/bcache/Makefile index 5b87e59676..054e8a33a7 100644 --- a/drivers/md/bcache/Makefile +++ b/drivers/md/bcache/Makefile @@ -2,6 +2,6 @@ obj-$(CONFIG_BCACHE) += bcache.o -bcache-y := alloc.o bset.o btree.o closure.o debug.o extents.o\ - io.o journal.o movinggc.o request.o stats.o super.o sysfs.o trace.o\ +bcache-y := alloc.o bset.o btree.o debug.o extents.o io.o\ + journal.o movinggc.o request.o stats.o super.o sysfs.o trace.o\ util.o writeback.o features.o diff --git a/drivers/md/bcache/bcache.h b/drivers/md/bcache/bcache.h index aebb7ef10e..c8b4914ad8 100644 --- a/drivers/md/bcache/bcache.h +++ b/drivers/md/bcache/bcache.h @@ -179,6 +179,7 @@ #define pr_fmt(fmt) "bcache: %s() " fmt, __func__ #include +#include #include #include #include @@ -192,7 +193,6 @@ #include "bcache_ondisk.h" #include "bset.h" #include "util.h" -#include "closure.h" struct bucket { atomic_t pin; diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c index ba3909bb6b..31b68a1b87 100644 --- a/drivers/md/bcache/super.c +++ b/drivers/md/bcache/super.c @@ -2912,7 +2912,6 @@ static int __init bcache_init(void) goto err; bch_debug_init(); - closure_debug_init(); bcache_is_reboot = false; diff --git a/drivers/md/bcache/util.h b/drivers/md/bcache/util.h index 6f3cb7c921..f61ab1bada 100644 --- a/drivers/md/bcache/util.h +++ b/drivers/md/bcache/util.h @@ -4,6 +4,7 @@ #define _BCACHE_UTIL_H #include +#include #include #include #include @@ -13,8 +14,6 @@ #include #include -#include "closure.h" - struct closure; #ifdef CONFIG_BCACHE_DEBUG diff --git a/drivers/md/bcache/closure.h b/include/linux/closure.h similarity index 97% rename from drivers/md/bcache/closure.h rename to include/linux/closure.h index c88cdc4ae4..0ec9e7bc8d 100644 --- a/drivers/md/bcache/closure.h +++ b/include/linux/closure.h @@ -155,7 +155,7 @@ struct closure { atomic_t remaining; -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES #define CLOSURE_MAGIC_DEAD 0xc054dead #define CLOSURE_MAGIC_ALIVE 0xc054a11e @@ -184,15 +184,13 @@ static inline void closure_sync(struct closure *cl) __closure_sync(cl); } -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES -void closure_debug_init(void); void closure_debug_create(struct closure *cl); void closure_debug_destroy(struct closure *cl); #else -static inline void closure_debug_init(void) {} static inline void closure_debug_create(struct closure *cl) {} static inline void closure_debug_destroy(struct closure *cl) {} @@ -200,21 +198,21 @@ static inline void closure_debug_destroy(struct closure *cl) {} static inline void closure_set_ip(struct closure *cl) { -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES cl->ip = _THIS_IP_; #endif } static inline void closure_set_ret_ip(struct closure *cl) { -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES cl->ip = _RET_IP_; #endif } static inline void closure_set_waiting(struct closure *cl, unsigned long f) { -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES cl->waiting_on = f; #endif } @@ -243,6 +241,7 @@ static inline void closure_queue(struct closure *cl) */ BUILD_BUG_ON(offsetof(struct closure, fn) != offsetof(struct work_struct, func)); + if (wq) { INIT_WORK(&cl->work, cl->work.func); BUG_ON(!queue_work(wq, &cl->work)); @@ -255,7 +254,7 @@ static inline void closure_queue(struct closure *cl) */ static inline void closure_get(struct closure *cl) { -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES BUG_ON((atomic_inc_return(&cl->remaining) & CLOSURE_REMAINING_MASK) <= 1); #else @@ -271,7 +270,7 @@ static inline void closure_get(struct closure *cl) */ static inline void closure_init(struct closure *cl, struct closure *parent) { - memset(cl, 0, sizeof(struct closure)); + cl->fn = NULL; cl->parent = parent; if (parent) closure_get(parent); diff --git a/lib/Kconfig b/lib/Kconfig index ce2abffb9e..1aa1c15a83 100644 --- a/lib/Kconfig +++ b/lib/Kconfig @@ -504,6 +504,9 @@ config ASSOCIATIVE_ARRAY for more information. +config CLOSURES + bool + config HAS_IOMEM bool depends on !NO_IOMEM diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 39d1d93164..3dba7a9aff 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -1618,6 +1618,15 @@ config DEBUG_NOTIFIERS This is a relatively cheap check but if you care about maximum performance, say N. +config DEBUG_CLOSURES + bool "Debug closures (bcache async widgits)" + depends on CLOSURES + select DEBUG_FS + help + Keeps all active closures in a linked list and provides a debugfs + interface to list them, which makes it possible to see asynchronous + operations that get stuck. + config BUG_ON_DATA_CORRUPTION bool "Trigger a BUG when data corruption is detected" select DEBUG_LIST diff --git a/lib/Makefile b/lib/Makefile index baf2821f7a..fd13ca6e0e 100644 --- a/lib/Makefile +++ b/lib/Makefile @@ -245,6 +245,8 @@ obj-$(CONFIG_ATOMIC64_SELFTEST) += atomic64_test.o obj-$(CONFIG_CPU_RMAP) += cpu_rmap.o +obj-$(CONFIG_CLOSURES) += closure.o + obj-$(CONFIG_DQL) += dynamic_queue_limits.o obj-$(CONFIG_GLOB) += glob.o diff --git a/drivers/md/bcache/closure.c b/lib/closure.c similarity index 88% rename from drivers/md/bcache/closure.c rename to lib/closure.c index d8d9394a6b..b38ded00b9 100644 --- a/drivers/md/bcache/closure.c +++ b/lib/closure.c @@ -6,13 +6,12 @@ * Copyright 2012 Google, Inc. */ +#include #include -#include +#include #include #include -#include "closure.h" - static inline void closure_put_after_sub(struct closure *cl, int flags) { int r = flags & CLOSURE_REMAINING_MASK; @@ -45,6 +44,7 @@ void closure_sub(struct closure *cl, int v) { closure_put_after_sub(cl, atomic_sub_return(v, &cl->remaining)); } +EXPORT_SYMBOL(closure_sub); /* * closure_put - decrement a closure's refcount @@ -53,6 +53,7 @@ void closure_put(struct closure *cl) { closure_put_after_sub(cl, atomic_dec_return(&cl->remaining)); } +EXPORT_SYMBOL(closure_put); /* * closure_wake_up - wake up all closures on a wait list, without memory barrier @@ -74,6 +75,7 @@ void __closure_wake_up(struct closure_waitlist *wait_list) closure_sub(cl, CLOSURE_WAITING + 1); } } +EXPORT_SYMBOL(__closure_wake_up); /** * closure_wait - add a closure to a waitlist @@ -93,6 +95,7 @@ bool closure_wait(struct closure_waitlist *waitlist, struct closure *cl) return true; } +EXPORT_SYMBOL(closure_wait); struct closure_syncer { struct task_struct *task; @@ -127,8 +130,9 @@ void __sched __closure_sync(struct closure *cl) __set_current_state(TASK_RUNNING); } +EXPORT_SYMBOL(__closure_sync); -#ifdef CONFIG_BCACHE_CLOSURES_DEBUG +#ifdef CONFIG_DEBUG_CLOSURES static LIST_HEAD(closure_list); static DEFINE_SPINLOCK(closure_list_lock); @@ -144,6 +148,7 @@ void closure_debug_create(struct closure *cl) list_add(&cl->all, &closure_list); spin_unlock_irqrestore(&closure_list_lock, flags); } +EXPORT_SYMBOL(closure_debug_create); void closure_debug_destroy(struct closure *cl) { @@ -156,8 +161,7 @@ void closure_debug_destroy(struct closure *cl) list_del(&cl->all); spin_unlock_irqrestore(&closure_list_lock, flags); } - -static struct dentry *closure_debug; +EXPORT_SYMBOL(closure_debug_destroy); static int debug_show(struct seq_file *f, void *data) { @@ -181,7 +185,7 @@ static int debug_show(struct seq_file *f, void *data) seq_printf(f, " W %pS\n", (void *) cl->waiting_on); - seq_printf(f, "\n"); + seq_puts(f, "\n"); } spin_unlock_irq(&closure_list_lock); @@ -190,18 +194,11 @@ static int debug_show(struct seq_file *f, void *data) DEFINE_SHOW_ATTRIBUTE(debug); -void __init closure_debug_init(void) +static int __init closure_debug_init(void) { - if (!IS_ERR_OR_NULL(bcache_debug)) - /* - * it is unnecessary to check return value of - * debugfs_create_file(), we should not care - * about this. - */ - closure_debug = debugfs_create_file( - "closures", 0400, bcache_debug, NULL, &debug_fops); + debugfs_create_file("closures", 0400, NULL, NULL, &debug_fops); + return 0; } -#endif +late_initcall(closure_debug_init) -MODULE_AUTHOR("Kent Overstreet "); -MODULE_LICENSE("GPL"); +#endif From patchwork Tue May 9 16:56:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91688 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3045264vqo; Tue, 9 May 2023 10:15:16 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7zsngUP1xfrL3pODiaNySkywV94X5IhCLmoYDcqYUO+/vOF7v378Hi4GlbvDh97beNoTC4 X-Received: by 2002:a17:90b:3142:b0:244:a412:47d7 with SMTP id ip2-20020a17090b314200b00244a41247d7mr14933771pjb.20.1683652515756; Tue, 09 May 2023 10:15:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652515; cv=none; d=google.com; s=arc-20160816; b=OKAX2JZZsvWjgFgXf2v+oT/M+Ctxh4hYf8iD7XCXsoozwJukDyUA7cMq+OrCzuoFCd NeDQTEmO1/doXfwLd2IOwwa98sfWjS0PrS+D0bdo8czQoyCxkW1jMzc8Quj/Y0416V/C WtseFUbdDl5Us8/7NqSF3Q0lLVa2QwiIH3kvrBPNLHzFA1gpwmnyNaXcB14ofKDZvCUg 0HKhobAMQRdO0iirFYYnJsAofocEMqqzB2fJGYnaej1XXNfMXzE1zy8Ejzi441WVBHHX 1hqKRMCDyUkaB9NtQNT8ptkMyrVLXhPquvfKW0DDiOqPF70T1wYJnTkFduQzMGOp9Pkd AIwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=rlmUvmP3OR+Hibyanmij5y/VCV+BtfYfFPsujlXOgZk=; b=HyWWVx6b6xOho3MfHfnnmf22WXXb5PsyDXkPjY8ZWct1YROX+xjwsmmMS0Oa+Mv/Yx 2Vrmfe3FcPXFaAVrXj1qMocEA4jCixJXTOgegfHVpdQXEcArOjlTAul/WZlEtOB8Y6LB 5Ti5d06xegvlMHLQ2rQkEfYxPNMjS6PDCVIwCF3zTrw8ZCSWZJ2f4LQclRVOm6Hy+rZ1 Qay0JE8mWFUMCerhVuD7gtqnuhuB7IaYN8zRCCFEv13hDjDyyyGv0nm2hdqfDkJBOmIG mMo/lM7gCBxW8weam3AQYeA79TM91JYy4Y9YBknbIs2LZch1zDb0AH5nkIc3F4GZp1e2 YiqA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=HPQouljO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h70-20020a638349000000b0051ba2478f52si1908097pge.524.2023.05.09.10.15.02; Tue, 09 May 2023 10:15:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=HPQouljO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235136AbjEIQ6y (ORCPT + 99 others); Tue, 9 May 2023 12:58:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41894 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234887AbjEIQ6H (ORCPT ); Tue, 9 May 2023 12:58:07 -0400 Received: from out-31.mta1.migadu.com (out-31.mta1.migadu.com [95.215.58.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C92FA4EEC for ; Tue, 9 May 2023 09:57:27 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651445; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rlmUvmP3OR+Hibyanmij5y/VCV+BtfYfFPsujlXOgZk=; b=HPQouljORsYWMK0E4MhzXx/8vB1JXRSqLRVKT23bU2e/im6dY2x9v6Z1F83/XFwP17gfS+ ETcOT56ZndtSR3AY+GLFyRQh2yGDW28ezO/09Xyg/OJePy6LCUe4npizReqKJsSHtW0R8P N0EM5wRPR17QAcgYwdjw5L9ZsBUVlqU= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Coly Li Subject: [PATCH 16/32] MAINTAINERS: Add entry for closures Date: Tue, 9 May 2023 12:56:41 -0400 Message-Id: <20230509165657.1735798-17-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437620303082314?= X-GMAIL-MSGID: =?utf-8?q?1765437620303082314?= closures, from bcache, are async widgets with a variety of uses. bcachefs also uses them, so they're being moved to lib/; mark them as maintained. Signed-off-by: Kent Overstreet Cc: Coly Li Acked-by: Coly Li --- MAINTAINERS | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 3fc37de3d6..5d76169140 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5044,6 +5044,14 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git timers/core F: Documentation/devicetree/bindings/timer/ F: drivers/clocksource/ +CLOSURES: +M: Kent Overstreet +L: linux-bcachefs@vger.kernel.org +S: Supported +C: irc://irc.oftc.net/bcache +F: include/linux/closure.h +F: lib/closure.c + CMPC ACPI DRIVER M: Thadeu Lima de Souza Cascardo M: Daniel Oliveira Nascimento From patchwork Tue May 9 16:56:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91672 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3040308vqo; Tue, 9 May 2023 10:07:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4USn49EhQFjiunuvQLBCMoRdE+Kj0Le3+1Oa6YcIPSOYZPYgCBmAL8ItmG9fGabcXL3Zfx X-Received: by 2002:a05:6a20:8e19:b0:fc:8e7e:92ce with SMTP id y25-20020a056a208e1900b000fc8e7e92cemr16230867pzj.47.1683652067512; Tue, 09 May 2023 10:07:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652067; cv=none; d=google.com; s=arc-20160816; b=UzFfNc3IhZCkS3PyczUA3Txw3AGz/1smybP2OGCKIB7XAXh9hsUud4E5mO9doKJja5 N37o61E2KXMw0Vf22gz1w9ObEuG/bofIMOJEFg0z+G2mxcMQzCWFY6EXeM21juBUtm2Q BHkBAQcc2pCFk+uigXjrzSR1eaJaa3JDiJlEvmTI6HbKeVBgYY1eaJxyisEmo++va19S imr//Q9vNS3TDlGxUgQ0qpkDToYvz30jeIp4lv9GCF4Li1/cBFJSGioHEL+GxxgcnDfh CsJiPDUBUlgLlT+49w+vVQxh1zqFSOBD+jXwHJLTqa3zBeqGO2wadKSQrg4tW2D9Noiy 3lmQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Ry7U36vaXDdBTb0o6BnFM/rdvkZja/IMfv3muX44WIU=; b=LYqt8GxqiJ6AMh7whxJE/vvLSu7LzYFD1DapNoGM3isQb1udXP9S1wRZeQesE6dqhj iWSmsdyQxwWSCHT+BJ+UTEwIGRzZVTGw83uHkOcK9nblG45ffZWGEayRoiBpRrZVfi/O WQkCNXxqgeaosQ8qbCk/auPm/0qUthWUQJfiEyJhfygVYGtZ5ulWIwnhWLeess4OW43C FKff0jQMid+8uyLVLrQAN7rf+jgW8pnp3oQtG5Gy8xH+oDu94o7cxQplEkonRjL4fq38 2bhZkpqEGgRXpTb5JLJ5W2jugJB+l1dxODEqY7kTrYkoszTAWYjFzNhYfzALcuv2usys kp6w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=hmk6zZus; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e21-20020a637455000000b0052856cec949si1787306pgn.875.2023.05.09.10.07.33; Tue, 09 May 2023 10:07:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=hmk6zZus; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235180AbjEIQ7T (ORCPT + 99 others); Tue, 9 May 2023 12:59:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41950 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234828AbjEIQ6U (ORCPT ); Tue, 9 May 2023 12:58:20 -0400 Received: from out-27.mta1.migadu.com (out-27.mta1.migadu.com [95.215.58.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D25B344AB for ; Tue, 9 May 2023 09:57:28 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651446; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Ry7U36vaXDdBTb0o6BnFM/rdvkZja/IMfv3muX44WIU=; b=hmk6zZus9aPNjnuKozz487uUnEFHglhCyPnG192psE7jxpj0jtRd3lgfUFRLrnylBVNfYj jRDbGMhvjUKYwThlzVlIPGFJ+UaBu3N9ewJDUGEdB9RFZEMhxIAqI/VtVc98O4HdMGd3fq 7kdWwbvh5n+XJihcYdZVsTA2++XS67g= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Coly Li , Kent Overstreet Subject: [PATCH 17/32] closures: closure_wait_event() Date: Tue, 9 May 2023 12:56:42 -0400 Message-Id: <20230509165657.1735798-18-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437150434751266?= X-GMAIL-MSGID: =?utf-8?q?1765437150434751266?= From: Kent Overstreet Like wait_event() - except, because it uses closures and closure waitlists it doesn't have the restriction on modifying task state inside the condition check, like wait_event() does. Signed-off-by: Kent Overstreet Acked-by: Coly Li Signed-off-by: Kent Overstreet --- include/linux/closure.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/include/linux/closure.h b/include/linux/closure.h index 0ec9e7bc8d..36b4a83f9b 100644 --- a/include/linux/closure.h +++ b/include/linux/closure.h @@ -374,4 +374,26 @@ static inline void closure_call(struct closure *cl, closure_fn fn, continue_at_nobarrier(cl, fn, wq); } +#define __closure_wait_event(waitlist, _cond) \ +do { \ + struct closure cl; \ + \ + closure_init_stack(&cl); \ + \ + while (1) { \ + closure_wait(waitlist, &cl); \ + if (_cond) \ + break; \ + closure_sync(&cl); \ + } \ + closure_wake_up(waitlist); \ + closure_sync(&cl); \ +} while (0) + +#define closure_wait_event(waitlist, _cond) \ +do { \ + if (!(_cond)) \ + __closure_wait_event(waitlist, _cond); \ +} while (0) + #endif /* _LINUX_CLOSURE_H */ From patchwork Tue May 9 16:56:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91689 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3045550vqo; Tue, 9 May 2023 10:15:39 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7fyfg8ArLm0QbemMc6RfNw5qD4YHOeCiMYlW+Xv+9TRGYtuknMiz91Hm+mCRohgpTVM5oy X-Received: by 2002:a05:6a20:4312:b0:fd:e806:9c95 with SMTP id h18-20020a056a20431200b000fde8069c95mr18556016pzk.13.1683652539316; Tue, 09 May 2023 10:15:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652539; cv=none; d=google.com; s=arc-20160816; b=y9hVU1OyrwlJHKX80pKRocdlSRTXsZIGMQmthVF1b+6FUjvWWljDwdPyu9PvDY4rsq AUzqmVMXZucNRx/Rb+zoOtT6vVhFHOPdF9vuuWRQS0vkVvOVmcp7AX5db77A0h/A96B0 s8otZRnHtwOMd1hKhTzlTzE+8pafYp/RjOfUR+DDcaZ5V80Pt8q1tRjDqS3YNEx/ly8Q XRcRaSuqrSGj361EyTLeGkNJiqoJ7JNpeh/WPttXikRKv7wBy/VWfou4GXuvVUNRAqhe lnEY+BIuQcRZJNcTG9xtmgK23IqPzUqxUtjgTlY7IOLt4GY7rrFw6jBSvDSvSUOrceNV GbZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=5eXxZII/AY+4X/BxaYnmkLZzUciXf9EGOnTBvq8dyQQ=; b=Lp/ig8HVea4HVuW3xvNZcYvvA4DHtQdLJX9N6MPLZ+JH882D7XUEHETqArR4AR2odg Yhim76MpyRJHkuJIYhdCn1sJjkGsiitBsJT0gMWEh4yLHRPYKPZzEZlX8IZEDUMgeF7H XlaApDhLJyt2okgsjfdsG3JDtBKbuZEgOQTuHrHRAZDBGwEQECR+zyCxGdpeuDnoUFYN A7Qzxx88Ykbjvarhph4q96YgGYifvtIFuR3GANHhfZuAfqgqV3CTVPpUTxFfV3t5N0V0 hsG/Ymrlsy5E//nHaqnePJpk5wCbo8258Pr4YB0/kmaHupHXoY376JLcJs/XHJQg9gws Ho5w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=P3WJrYDS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b23-20020a637157000000b0050beec93e77si1750339pgn.609.2023.05.09.10.15.24; Tue, 09 May 2023 10:15:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=P3WJrYDS; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235006AbjEIQ7W (ORCPT + 99 others); Tue, 9 May 2023 12:59:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234987AbjEIQ6V (ORCPT ); Tue, 9 May 2023 12:58:21 -0400 Received: from out-50.mta1.migadu.com (out-50.mta1.migadu.com [IPv6:2001:41d0:203:375::32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 60BF85FC9 for ; Tue, 9 May 2023 09:57:29 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5eXxZII/AY+4X/BxaYnmkLZzUciXf9EGOnTBvq8dyQQ=; b=P3WJrYDS9WYJIkDCpbnvLdDXfB2wMF192jc7g914mAZvMadSvCzAcfAvnGQgHG4ob3rALb IYQ299pn7imaicd+BdX8TMdqCyemGSuhAF+WYKm2U6bErM0UIDAwKPTKAVzlxPC7beBL3F C78AGlXiDX7tV3NvYLrZwMwpodbPG0Y= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 18/32] closures: closure_nr_remaining() Date: Tue, 9 May 2023 12:56:43 -0400 Message-Id: <20230509165657.1735798-19-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437644947637052?= X-GMAIL-MSGID: =?utf-8?q?1765437644947637052?= Factor out a new helper, which returns the number of events outstanding. Signed-off-by: Kent Overstreet --- include/linux/closure.h | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/include/linux/closure.h b/include/linux/closure.h index 36b4a83f9b..722a586bb2 100644 --- a/include/linux/closure.h +++ b/include/linux/closure.h @@ -172,6 +172,11 @@ void __closure_wake_up(struct closure_waitlist *list); bool closure_wait(struct closure_waitlist *list, struct closure *cl); void __closure_sync(struct closure *cl); +static inline unsigned closure_nr_remaining(struct closure *cl) +{ + return atomic_read(&cl->remaining) & CLOSURE_REMAINING_MASK; +} + /** * closure_sync - sleep until a closure a closure has nothing left to wait on * @@ -180,7 +185,7 @@ void __closure_sync(struct closure *cl); */ static inline void closure_sync(struct closure *cl) { - if ((atomic_read(&cl->remaining) & CLOSURE_REMAINING_MASK) != 1) + if (closure_nr_remaining(cl) != 1) __closure_sync(cl); } From patchwork Tue May 9 16:56:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91671 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3039893vqo; Tue, 9 May 2023 10:07:17 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6yne2lFkH+AUDThmMkHwkiHUgHNlZw9B0ks1oAKpKk8eCf6yaZcV0kPR3bXWUu0XYndBHK X-Received: by 2002:a05:6a20:7354:b0:101:1d1d:43a4 with SMTP id v20-20020a056a20735400b001011d1d43a4mr4678303pzc.15.1683652036515; Tue, 09 May 2023 10:07:16 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652036; cv=none; d=google.com; s=arc-20160816; b=QMNWFwliwfMCYltQf3uoEQP3rN7oNkNt0js2ynR4ylVfdEU9Ajeg9bG50w2rImdV0n VDEPsBDYgU0pE+fJr0bA+NE6UwzrjQvR/ZKYngbSsKCfcHu/MalMZSe7ECbZhiCxW+Av JDelnjNWEAc5GX17JW6EYjvEKUZIoM9g3jcY3Y3l2vb4LbS0k29gG6j3zUACogbYEeIh uSZLQKO+cGJPmCCdOszCaY6IEzrdWcDeRkoJ9dhxcp0w43on1uvL0Lq+8p+Dvlaj3YjC /MeQ/UDuDXUukTYVfOrwDQ3aRaj2FMnsjmvz3LONNYGpIAHjTq78E2/8DBQmPbeo9mDh IL6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=3bOiJuZA7MMn1dUAKvZtsb+DsEd9kjQVDcRKBJgcodE=; b=M8G6B8rACVFFAT0wR7jy75L80dZ+rnfGbEjn8vsWzJ4txfRclWqmbxjT7YH+BTq+hg Bo2ZwktQRnjtCUFS+6yjBUkhwsPOBeh389Ki4AxyTLN1N/IrDLJVOl1xWY+k7RUjQ7zD e6tGMYqAqqBoMm3xDYBgNyTWEZ9UIGIDPBLBBa3yGrwcseVCQg680zPhmOctmpCgefO0 UVgPH1sOjZyorM8ayOSYrpQmRDFijj22GcfaFzgwSdWl2Ibx4QF7CD7xwDFYKS3SstC4 uaaqQ/QNFDb/dhK0wce7Ffqvb5dmykAsXY+lZY/AtJ1l99WmoZSFVXAgekCXEeZ6VlmE j/2g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=P3hZFno7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z7-20020a633307000000b0052c73483bbfsi1800962pgz.830.2023.05.09.10.07.03; Tue, 09 May 2023 10:07:16 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=P3hZFno7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235001AbjEIQ7Z (ORCPT + 99 others); Tue, 9 May 2023 12:59:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234996AbjEIQ6V (ORCPT ); Tue, 9 May 2023 12:58:21 -0400 Received: from out-41.mta1.migadu.com (out-41.mta1.migadu.com [IPv6:2001:41d0:203:375::29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D448A4ED4 for ; Tue, 9 May 2023 09:57:29 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651448; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3bOiJuZA7MMn1dUAKvZtsb+DsEd9kjQVDcRKBJgcodE=; b=P3hZFno74bG0D5MVwOiAtytjPyoMXAxWkMWmWGCfGM3DquFD0ALELeBdVg3CPwnj+gUHgK NJJNgmjsEoEC/xiuoLdUK+Ku0A3kOpGxgm/PyfDlH1n2Qe/3QX9XUJTlO8rHac+1xpTczX 3qQ/rx1r/qiBVGrivA0TW/KTk46S0t0= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 19/32] closures: Add a missing include Date: Tue, 9 May 2023 12:56:44 -0400 Message-Id: <20230509165657.1735798-20-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437117608380195?= X-GMAIL-MSGID: =?utf-8?q?1765437117608380195?= Fixes building in userspace. Signed-off-by: Kent Overstreet --- lib/closure.c | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/closure.c b/lib/closure.c index b38ded00b9..0855e698ce 100644 --- a/lib/closure.c +++ b/lib/closure.c @@ -9,6 +9,7 @@ #include #include #include +#include #include #include From patchwork Tue May 9 16:56:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91675 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3040584vqo; Tue, 9 May 2023 10:08:08 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ41akrRGnP3RgIDfFEumULuf9Hbdqn9jF6+dsfoAd3i6Y3g5Sbt4jTgqPWqLsoWLeL76LQf X-Received: by 2002:a17:903:1209:b0:1a9:7912:850e with SMTP id l9-20020a170903120900b001a97912850emr18010400plh.10.1683652087622; Tue, 09 May 2023 10:08:07 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652087; cv=none; d=google.com; s=arc-20160816; b=KrMnNUKu+k3Gtu2r0gUfoJg12HN/vDtOoDnnYa6rV8dPcz4eORa5WdlJp4rbQfhvAm E1v6JRXnv1OAb0u/evPAna7Ul1IKw64qHK8EvFx/ggNmzBXYFc9tB5GFLleS57/qJVJc 3emc+bKR+SMM5KmAgxs3RWp6FFePUQfsDZ5R9Gxt9/2Qg2EGFSjcRaxHuujdqDm2/Rt/ 3zEoD5y7bIQIAmGhTIzXysF/Ve1480wGrGWUf/5wgb4w8QQZUIR4uWamuBmoN8tQdm0d etQ860Gs36GoPmwHulf16tl6vczwoc7knVRlYKrRh1V3Bd776w602ABPNT2ALUQMWvrh PG0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=KXhu0IMdhm7TtmY2sdq2+bF9aYr9Sm4xuSQ9rgukQdg=; b=ZO4LEksu0F6opm4tYeiQ94NtqbTLJVVhoEXavY9CNp9eFX1HBElxrRmJxPpiTlw6ko TXeUFvWkGF9wjS7bLZpm4z4NDGwWfQbGMe4k82vFvfHyrrM4nPBkOZ17sYglgQUMXQjK mjkxfzwBHkN5BnDhTx3ym14zMBq6qMFgrh6MSf52URKfnYtKcRT+6mQmYw9hyrMZDINp qwkTJV6RNKMIF962I4A+taSKPVD+z011mGf2tYmy5u+DGu4z8Hl7CqUNZuUHC6wWz/gx Nj0Lt/G27OV5pJBRZWs9Pq2uLNiZuKiW8bSrSNj1JUWuikoShFrj+jghn3FrgmsgRbIz PJpg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Gra0g7cw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id j4-20020a170903028400b001aaff9c8dd1si1841066plr.427.2023.05.09.10.07.55; Tue, 09 May 2023 10:08:07 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Gra0g7cw; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235016AbjEIQ70 (ORCPT + 99 others); Tue, 9 May 2023 12:59:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235013AbjEIQ6W (ORCPT ); Tue, 9 May 2023 12:58:22 -0400 Received: from out-20.mta1.migadu.com (out-20.mta1.migadu.com [95.215.58.20]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4908D525A for ; Tue, 9 May 2023 09:57:31 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651449; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KXhu0IMdhm7TtmY2sdq2+bF9aYr9Sm4xuSQ9rgukQdg=; b=Gra0g7cwhAqneFL51O788e/pMmLPbI05SL6bok25hYVdGn18I9cy8a3/KaT30uAxdqDTg2 SwDp+NtsmF5P0jw2xQkDhJ119T+9qju9K5I1q2D1uNjkIsQLuqoYRp9CqKmmQkS3AeA8xk Xg784HuVcBpPfrJnLnMCeN2m26D7MRs= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Dave Chinner , Alexander Viro , Christian Brauner , Kent Overstreet Subject: [PATCH 20/32] vfs: factor out inode hash head calculation Date: Tue, 9 May 2023 12:56:45 -0400 Message-Id: <20230509165657.1735798-21-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437171491248958?= X-GMAIL-MSGID: =?utf-8?q?1765437171491248958?= From: Dave Chinner In preparation for changing the inode hash table implementation. Signed-off-by: Dave Chinner Cc: Alexander Viro Cc: Christian Brauner Cc: linux-fsdevel@vger.kernel.org Signed-off-by: Kent Overstreet --- fs/inode.c | 44 +++++++++++++++++++++++++------------------- 1 file changed, 25 insertions(+), 19 deletions(-) diff --git a/fs/inode.c b/fs/inode.c index 4558dc2f13..41a10bcda1 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -60,6 +60,22 @@ static unsigned int i_hash_shift __read_mostly; static struct hlist_head *inode_hashtable __read_mostly; static __cacheline_aligned_in_smp DEFINE_SPINLOCK(inode_hash_lock); +static unsigned long hash(struct super_block *sb, unsigned long hashval) +{ + unsigned long tmp; + + tmp = (hashval * (unsigned long)sb) ^ (GOLDEN_RATIO_PRIME + hashval) / + L1_CACHE_BYTES; + tmp = tmp ^ ((tmp ^ GOLDEN_RATIO_PRIME) >> i_hash_shift); + return tmp & i_hash_mask; +} + +static inline struct hlist_head *i_hash_head(struct super_block *sb, + unsigned int hashval) +{ + return inode_hashtable + hash(sb, hashval); +} + /* * Empty aops. Can be used for the cases where the user does not * define any of the address_space operations. @@ -506,16 +522,6 @@ static inline void inode_sb_list_del(struct inode *inode) } } -static unsigned long hash(struct super_block *sb, unsigned long hashval) -{ - unsigned long tmp; - - tmp = (hashval * (unsigned long)sb) ^ (GOLDEN_RATIO_PRIME + hashval) / - L1_CACHE_BYTES; - tmp = tmp ^ ((tmp ^ GOLDEN_RATIO_PRIME) >> i_hash_shift); - return tmp & i_hash_mask; -} - /** * __insert_inode_hash - hash an inode * @inode: unhashed inode @@ -1163,7 +1169,7 @@ struct inode *inode_insert5(struct inode *inode, unsigned long hashval, int (*test)(struct inode *, void *), int (*set)(struct inode *, void *), void *data) { - struct hlist_head *head = inode_hashtable + hash(inode->i_sb, hashval); + struct hlist_head *head = i_hash_head(inode->i_sb, hashval); struct inode *old; again: @@ -1267,7 +1273,7 @@ EXPORT_SYMBOL(iget5_locked); */ struct inode *iget_locked(struct super_block *sb, unsigned long ino) { - struct hlist_head *head = inode_hashtable + hash(sb, ino); + struct hlist_head *head = i_hash_head(sb, ino); struct inode *inode; again: spin_lock(&inode_hash_lock); @@ -1335,7 +1341,7 @@ EXPORT_SYMBOL(iget_locked); */ static int test_inode_iunique(struct super_block *sb, unsigned long ino) { - struct hlist_head *b = inode_hashtable + hash(sb, ino); + struct hlist_head *b = i_hash_head(sb, ino); struct inode *inode; hlist_for_each_entry_rcu(inode, b, i_hash) { @@ -1422,7 +1428,7 @@ EXPORT_SYMBOL(igrab); struct inode *ilookup5_nowait(struct super_block *sb, unsigned long hashval, int (*test)(struct inode *, void *), void *data) { - struct hlist_head *head = inode_hashtable + hash(sb, hashval); + struct hlist_head *head = i_hash_head(sb, hashval); struct inode *inode; spin_lock(&inode_hash_lock); @@ -1477,7 +1483,7 @@ EXPORT_SYMBOL(ilookup5); */ struct inode *ilookup(struct super_block *sb, unsigned long ino) { - struct hlist_head *head = inode_hashtable + hash(sb, ino); + struct hlist_head *head = i_hash_head(sb, ino); struct inode *inode; again: spin_lock(&inode_hash_lock); @@ -1526,7 +1532,7 @@ struct inode *find_inode_nowait(struct super_block *sb, void *), void *data) { - struct hlist_head *head = inode_hashtable + hash(sb, hashval); + struct hlist_head *head = i_hash_head(sb, hashval); struct inode *inode, *ret_inode = NULL; int mval; @@ -1571,7 +1577,7 @@ EXPORT_SYMBOL(find_inode_nowait); struct inode *find_inode_rcu(struct super_block *sb, unsigned long hashval, int (*test)(struct inode *, void *), void *data) { - struct hlist_head *head = inode_hashtable + hash(sb, hashval); + struct hlist_head *head = i_hash_head(sb, hashval); struct inode *inode; RCU_LOCKDEP_WARN(!rcu_read_lock_held(), @@ -1609,7 +1615,7 @@ EXPORT_SYMBOL(find_inode_rcu); struct inode *find_inode_by_ino_rcu(struct super_block *sb, unsigned long ino) { - struct hlist_head *head = inode_hashtable + hash(sb, ino); + struct hlist_head *head = i_hash_head(sb, ino); struct inode *inode; RCU_LOCKDEP_WARN(!rcu_read_lock_held(), @@ -1629,7 +1635,7 @@ int insert_inode_locked(struct inode *inode) { struct super_block *sb = inode->i_sb; ino_t ino = inode->i_ino; - struct hlist_head *head = inode_hashtable + hash(sb, ino); + struct hlist_head *head = i_hash_head(sb, ino); while (1) { struct inode *old = NULL; From patchwork Tue May 9 16:56:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91694 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3046524vqo; Tue, 9 May 2023 10:17:07 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6Is7QNXcPiDaWdJkYYzOZZkNckRc5sPU9ZRZ6yVU1DhfaExG53tRoh1tm2X77jcl85aQRp X-Received: by 2002:a05:6a20:394f:b0:ef:c5b6:b6b9 with SMTP id r15-20020a056a20394f00b000efc5b6b6b9mr18951624pzg.23.1683652626706; Tue, 09 May 2023 10:17:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652626; cv=none; d=google.com; s=arc-20160816; b=FP5kqM+Hggf+y13tiIp+CtXgfRUsK5Ua3RD+QrrqNM6SwHDEhpH5GfxYIsZ2y288GI 6QAPYpEAKZCRYd4bOD9x92Z72ucFKv0dlGW1YRHU5wlGh9nGnKtCCOLoo4C6u0pwevQU Y9Y4ruO8XgyXu6Ct2j7VAiu3LHh1i85YZiNJkYc2qQQWvFUYPkrOdagdB3lD3TBQ/BY3 nFyYXpWtF70AVcu3kuC5BHIj2vP7ZeDY8PH4VMkidy+TkSrcCtdYE8i91bsUkmulRzhb tJhYX/YFM5Q66TztZTgLgaMNBq25KiEcbgDXQSUvI53Os/4n/XpDo2MkJd1GyHyl8tTu aLLA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4+qUD9R+DcetQOvg5f3GpMxzPZzmlaA8sue3bNTAfVo=; b=HoO2Zn3y5DzsClWxHJSTc47kpi0ywcvmZ69RP9tMumHxRV2t4vHKnLTq2rWLVSN+4N ebnKvJdwiWum7Ydf/wl7KH75JphbdL0W5+im3STZ1JFBf98tPfInhkyxnGYD2w2vz6o5 uR0Pb1B//6uEEVlu5RlKVeer5swoqCmN1r7U7CLeep4AOBHP9XWXTxKR2ZYYnd7KMQZE +468Z6o90pIA8+J4EnfHoxnrfSo7BgHYnvNcKsT5aWKawxUrzUIj5Izl/zZ/VlUVVZWC EvMxhjjlKNfT7pfktdpqDKLHMGtBoOxLkvXNRC29awXMZn93MmNJ1c2JTTaHq4iV0QsD sImw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=ENkhviM7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r134-20020a632b8c000000b005288509b88asi1884914pgr.681.2023.05.09.10.16.52; Tue, 09 May 2023 10:17:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=ENkhviM7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234248AbjEIQ7k (ORCPT + 99 others); Tue, 9 May 2023 12:59:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235034AbjEIQ6W (ORCPT ); Tue, 9 May 2023 12:58:22 -0400 Received: from out-6.mta1.migadu.com (out-6.mta1.migadu.com [95.215.58.6]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52D6B5FDD for ; Tue, 9 May 2023 09:57:32 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651450; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4+qUD9R+DcetQOvg5f3GpMxzPZzmlaA8sue3bNTAfVo=; b=ENkhviM7etnE84ZkD+4vtkRiiNspqxA/lNrO4RXNZ8yeN8c2YKqtbDxJAr9NJIhBYV8UsZ z6ktImL6+6VKYmFBJ10zEUcmqyFXw/Yt2IsylmWIqyAO9H8c6nRJDKPTxx9R8kf0Vj47zL WpWU+qb0j2vNs8POnr1oLPMdpMcv0Nc= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Dave Chinner , Kent Overstreet Subject: [PATCH 21/32] hlist-bl: add hlist_bl_fake() Date: Tue, 9 May 2023 12:56:46 -0400 Message-Id: <20230509165657.1735798-22-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437736894837106?= X-GMAIL-MSGID: =?utf-8?q?1765437736894837106?= From: Dave Chinner in preparation for switching the VFS inode cache over the hlist_bl lists, we nee dto be able to fake a list node that looks like it is hased for correct operation of filesystems that don't directly use the VFS indoe cache. Signed-off-by: Dave Chinner Signed-off-by: Kent Overstreet --- include/linux/list_bl.h | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/include/linux/list_bl.h b/include/linux/list_bl.h index ae1b541446..8ee2bf5af1 100644 --- a/include/linux/list_bl.h +++ b/include/linux/list_bl.h @@ -143,6 +143,28 @@ static inline void hlist_bl_del_init(struct hlist_bl_node *n) } } +/** + * hlist_bl_add_fake - create a fake list consisting of a single headless node + * @n: Node to make a fake list out of + * + * This makes @n appear to be its own predecessor on a headless hlist. + * The point of this is to allow things like hlist_bl_del() to work correctly + * in cases where there is no list. + */ +static inline void hlist_bl_add_fake(struct hlist_bl_node *n) +{ + n->pprev = &n->next; +} + +/** + * hlist_fake: Is this node a fake hlist_bl? + * @h: Node to check for being a self-referential fake hlist. + */ +static inline bool hlist_bl_fake(struct hlist_bl_node *n) +{ + return n->pprev == &n->next; +} + static inline void hlist_bl_lock(struct hlist_bl_head *b) { bit_spin_lock(0, (unsigned long *)b); From patchwork Tue May 9 16:56:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91687 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3044676vqo; Tue, 9 May 2023 10:14:09 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5kQwSWj9Iye7H25BLGBr3ulsl41etcA0AFyIQCky/kJM9KtfXC4kWynakweQ6fV5X837W6 X-Received: by 2002:a05:6a20:8409:b0:101:c6ee:4996 with SMTP id c9-20020a056a20840900b00101c6ee4996mr611966pzd.12.1683652449389; Tue, 09 May 2023 10:14:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652449; cv=none; d=google.com; s=arc-20160816; b=yDniR1M2DYQP0U++2jg05FX+gZzfFLPn8KJDkadWlPbZO9c2H4N8OjXexKVgh5ziIG JD3SDkYBrHOw5llsiiSw20z9IN7Qr5b1PFtii9JRWgDVQDaxbh+x/KI+vITquhtl6ddn BAkufIxsUFQSiz35YoYzOegWTCbgOA2thXIvG5p+s45jsvKoABJIWuxIWiIGfzCWFEJZ FUfsrdpWtkcajhiRwUfzUvlcWeolhT6+f/IkQ+AbcijdghRuVtbGhX3QdqU4NWOeSofY XWEfCgv0cZAt5R3EFTx48/DTtOAG85i5Q3fNOaOKoMX1QCduSUqMkncFDmuA1LTLgHde DaRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dk9fEV/nEtfTj8dEq6WD3l9lbR6xxkWjrj5WCgE+oUs=; b=l4pI++5yzhsxEj5whMySv7VkFBLICWW/NyQ/F7BMDG2Toy0Vj7V0t9B8U1ed8PlRrp uid7SbbcDLgOSJwsoWHyHRVL4CvPbKCELsbasG6gMAVffi4XDel70lDFkv+PkDtq/ENH y6ECmNBHQmxBH07u0A8mckj39Yu9RheZC7u3CXpL4lx35/8W5aqTn/4cARplIePcqUeJ WWmWsNKsGvQnrGeid0DuG4n/MHRYBNMBrbre11EBMt6Gsyn2e5iTd44iTAFAWRqiZGN/ KzWwsgXi+sNUoYJBUyXhYXyvSfpIKIFDhfymfZW3IE6TcKLp2T8qj6Nvj0UdMgrG/svZ CM6A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=xlpORcUQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h70-20020a638349000000b0051ba2478f52si1908097pge.524.2023.05.09.10.13.56; Tue, 09 May 2023 10:14:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=xlpORcUQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234609AbjEIRAH (ORCPT + 99 others); Tue, 9 May 2023 13:00:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235041AbjEIQ6g (ORCPT ); Tue, 9 May 2023 12:58:36 -0400 Received: from out-29.mta1.migadu.com (out-29.mta1.migadu.com [95.215.58.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36BCA5590 for ; Tue, 9 May 2023 09:57:34 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651451; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dk9fEV/nEtfTj8dEq6WD3l9lbR6xxkWjrj5WCgE+oUs=; b=xlpORcUQ0MniQ5ub5QSgCiBRk7D8kXP0OSY9kdkEWJmBXZfkdq/yckpdmoPfmndBpOqpyJ FSWmrEG8NQtokDyLL7rpwYkcl9hdRKPU4D0oBPfneQWyqg8hEOUKUgs3+Hd4gtgqZQ3k3a OmgHesQyP4GuRmhc+w7A9vUMfQt7tDQ= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Dave Chinner , Alexander Viro , Christian Brauner , Kent Overstreet Subject: [PATCH 22/32] vfs: inode cache conversion to hash-bl Date: Tue, 9 May 2023 12:56:47 -0400 Message-Id: <20230509165657.1735798-23-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437551097335787?= X-GMAIL-MSGID: =?utf-8?q?1765437551097335787?= From: Dave Chinner Because scalability of the global inode_hash_lock really, really sucks. 32-way concurrent create on a couple of different filesystems before: - 52.13% 0.04% [kernel] [k] ext4_create - 52.09% ext4_create - 41.03% __ext4_new_inode - 29.92% insert_inode_locked - 25.35% _raw_spin_lock - do_raw_spin_lock - 24.97% __pv_queued_spin_lock_slowpath - 72.33% 0.02% [kernel] [k] do_filp_open - 72.31% do_filp_open - 72.28% path_openat - 57.03% bch2_create - 56.46% __bch2_create - 40.43% inode_insert5 - 36.07% _raw_spin_lock - do_raw_spin_lock 35.86% __pv_queued_spin_lock_slowpath 4.02% find_inode Convert the inode hash table to a RCU-aware hash-bl table just like the dentry cache. Note that we need to store a pointer to the hlist_bl_head the inode has been added to in the inode so that when it comes to unhash the inode we know what list to lock. We need to do this because the hash value that is used to hash the inode is generated from the inode itself - filesystems can provide this themselves so we have to either store the hash or the head pointer in the inode to be able to find the right list head for removal... Same workload after: Signed-off-by: Dave Chinner Cc: Alexander Viro Cc: Christian Brauner Cc: linux-fsdevel@vger.kernel.org Signed-off-by: Kent Overstreet Signed-off-by: Dave Chinner --- fs/inode.c | 200 ++++++++++++++++++++++++++++----------------- include/linux/fs.h | 9 +- 2 files changed, 132 insertions(+), 77 deletions(-) diff --git a/fs/inode.c b/fs/inode.c index 41a10bcda1..d446b054ec 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -57,8 +57,7 @@ static unsigned int i_hash_mask __read_mostly; static unsigned int i_hash_shift __read_mostly; -static struct hlist_head *inode_hashtable __read_mostly; -static __cacheline_aligned_in_smp DEFINE_SPINLOCK(inode_hash_lock); +static struct hlist_bl_head *inode_hashtable __read_mostly; static unsigned long hash(struct super_block *sb, unsigned long hashval) { @@ -70,7 +69,7 @@ static unsigned long hash(struct super_block *sb, unsigned long hashval) return tmp & i_hash_mask; } -static inline struct hlist_head *i_hash_head(struct super_block *sb, +static inline struct hlist_bl_head *i_hash_head(struct super_block *sb, unsigned int hashval) { return inode_hashtable + hash(sb, hashval); @@ -433,7 +432,7 @@ EXPORT_SYMBOL(address_space_init_once); void inode_init_once(struct inode *inode) { memset(inode, 0, sizeof(*inode)); - INIT_HLIST_NODE(&inode->i_hash); + INIT_HLIST_BL_NODE(&inode->i_hash); INIT_LIST_HEAD(&inode->i_devices); INIT_LIST_HEAD(&inode->i_io_list); INIT_LIST_HEAD(&inode->i_wb_list); @@ -522,6 +521,17 @@ static inline void inode_sb_list_del(struct inode *inode) } } +/* + * Ensure that we store the hash head in the inode when we insert the inode into + * the hlist_bl_head... + */ +static inline void +__insert_inode_hash_head(struct inode *inode, struct hlist_bl_head *b) +{ + hlist_bl_add_head_rcu(&inode->i_hash, b); + inode->i_hash_head = b; +} + /** * __insert_inode_hash - hash an inode * @inode: unhashed inode @@ -532,13 +542,13 @@ static inline void inode_sb_list_del(struct inode *inode) */ void __insert_inode_hash(struct inode *inode, unsigned long hashval) { - struct hlist_head *b = inode_hashtable + hash(inode->i_sb, hashval); + struct hlist_bl_head *b = i_hash_head(inode->i_sb, hashval); - spin_lock(&inode_hash_lock); + hlist_bl_lock(b); spin_lock(&inode->i_lock); - hlist_add_head_rcu(&inode->i_hash, b); + __insert_inode_hash_head(inode, b); spin_unlock(&inode->i_lock); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); } EXPORT_SYMBOL(__insert_inode_hash); @@ -550,11 +560,44 @@ EXPORT_SYMBOL(__insert_inode_hash); */ void __remove_inode_hash(struct inode *inode) { - spin_lock(&inode_hash_lock); - spin_lock(&inode->i_lock); - hlist_del_init_rcu(&inode->i_hash); - spin_unlock(&inode->i_lock); - spin_unlock(&inode_hash_lock); + struct hlist_bl_head *b = inode->i_hash_head; + + /* + * There are some callers that come through here without synchronisation + * and potentially with multiple references to the inode. Hence we have + * to handle the case that we might race with a remove and insert to a + * different list. Coda, in particular, seems to have a userspace API + * that can directly trigger "unhash/rehash to different list" behaviour + * without any serialisation at all. + * + * Hence we have to handle the situation where the inode->i_hash_head + * might point to a different list than what we expect, indicating that + * we raced with another unhash and potentially a new insertion. This + * means we have to retest the head once we have everything locked up + * and loop again if it doesn't match. + */ + while (b) { + hlist_bl_lock(b); + spin_lock(&inode->i_lock); + if (b != inode->i_hash_head) { + hlist_bl_unlock(b); + b = inode->i_hash_head; + spin_unlock(&inode->i_lock); + continue; + } + /* + * Need to set the pprev pointer to NULL after list removal so + * that both RCU traversals and hlist_bl_unhashed() work + * correctly at this point. + */ + hlist_bl_del_rcu(&inode->i_hash); + inode->i_hash.pprev = NULL; + inode->i_hash_head = NULL; + spin_unlock(&inode->i_lock); + hlist_bl_unlock(b); + break; + } + } EXPORT_SYMBOL(__remove_inode_hash); @@ -904,26 +947,28 @@ long prune_icache_sb(struct super_block *sb, struct shrink_control *sc) return freed; } -static void __wait_on_freeing_inode(struct inode *inode); +static void __wait_on_freeing_inode(struct hlist_bl_head *b, + struct inode *inode); /* * Called with the inode lock held. */ static struct inode *find_inode(struct super_block *sb, - struct hlist_head *head, + struct hlist_bl_head *b, int (*test)(struct inode *, void *), void *data) { + struct hlist_bl_node *node; struct inode *inode = NULL; repeat: - hlist_for_each_entry(inode, head, i_hash) { + hlist_bl_for_each_entry(inode, node, b, i_hash) { if (inode->i_sb != sb) continue; if (!test(inode, data)) continue; spin_lock(&inode->i_lock); if (inode->i_state & (I_FREEING|I_WILL_FREE)) { - __wait_on_freeing_inode(inode); + __wait_on_freeing_inode(b, inode); goto repeat; } if (unlikely(inode->i_state & I_CREATING)) { @@ -942,19 +987,20 @@ static struct inode *find_inode(struct super_block *sb, * iget_locked for details. */ static struct inode *find_inode_fast(struct super_block *sb, - struct hlist_head *head, unsigned long ino) + struct hlist_bl_head *b, unsigned long ino) { + struct hlist_bl_node *node; struct inode *inode = NULL; repeat: - hlist_for_each_entry(inode, head, i_hash) { + hlist_bl_for_each_entry(inode, node, b, i_hash) { if (inode->i_ino != ino) continue; if (inode->i_sb != sb) continue; spin_lock(&inode->i_lock); if (inode->i_state & (I_FREEING|I_WILL_FREE)) { - __wait_on_freeing_inode(inode); + __wait_on_freeing_inode(b, inode); goto repeat; } if (unlikely(inode->i_state & I_CREATING)) { @@ -1162,25 +1208,25 @@ EXPORT_SYMBOL(unlock_two_nondirectories); * return it locked, hashed, and with the I_NEW flag set. The file system gets * to fill it in before unlocking it via unlock_new_inode(). * - * Note both @test and @set are called with the inode_hash_lock held, so can't - * sleep. + * Note both @test and @set are called with the inode hash chain lock held, + * so can't sleep. */ struct inode *inode_insert5(struct inode *inode, unsigned long hashval, int (*test)(struct inode *, void *), int (*set)(struct inode *, void *), void *data) { - struct hlist_head *head = i_hash_head(inode->i_sb, hashval); + struct hlist_bl_head *b = i_hash_head(inode->i_sb, hashval); struct inode *old; again: - spin_lock(&inode_hash_lock); - old = find_inode(inode->i_sb, head, test, data); + hlist_bl_lock(b); + old = find_inode(inode->i_sb, b, test, data); if (unlikely(old)) { /* * Uhhuh, somebody else created the same inode under us. * Use the old inode instead of the preallocated one. */ - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); if (IS_ERR(old)) return NULL; wait_on_inode(old); @@ -1202,7 +1248,7 @@ struct inode *inode_insert5(struct inode *inode, unsigned long hashval, */ spin_lock(&inode->i_lock); inode->i_state |= I_NEW; - hlist_add_head_rcu(&inode->i_hash, head); + __insert_inode_hash_head(inode, b); spin_unlock(&inode->i_lock); /* @@ -1212,7 +1258,7 @@ struct inode *inode_insert5(struct inode *inode, unsigned long hashval, if (list_empty(&inode->i_sb_list)) inode_sb_list_add(inode); unlock: - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); return inode; } @@ -1273,12 +1319,12 @@ EXPORT_SYMBOL(iget5_locked); */ struct inode *iget_locked(struct super_block *sb, unsigned long ino) { - struct hlist_head *head = i_hash_head(sb, ino); + struct hlist_bl_head *b = i_hash_head(sb, ino); struct inode *inode; again: - spin_lock(&inode_hash_lock); - inode = find_inode_fast(sb, head, ino); - spin_unlock(&inode_hash_lock); + hlist_bl_lock(b); + inode = find_inode_fast(sb, b, ino); + hlist_bl_unlock(b); if (inode) { if (IS_ERR(inode)) return NULL; @@ -1294,17 +1340,17 @@ struct inode *iget_locked(struct super_block *sb, unsigned long ino) if (inode) { struct inode *old; - spin_lock(&inode_hash_lock); + hlist_bl_lock(b); /* We released the lock, so.. */ - old = find_inode_fast(sb, head, ino); + old = find_inode_fast(sb, b, ino); if (!old) { inode->i_ino = ino; spin_lock(&inode->i_lock); inode->i_state = I_NEW; - hlist_add_head_rcu(&inode->i_hash, head); + __insert_inode_hash_head(inode, b); spin_unlock(&inode->i_lock); inode_sb_list_add(inode); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); /* Return the locked inode with I_NEW set, the * caller is responsible for filling in the contents @@ -1317,7 +1363,7 @@ struct inode *iget_locked(struct super_block *sb, unsigned long ino) * us. Use the old inode instead of the one we just * allocated. */ - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); destroy_inode(inode); if (IS_ERR(old)) return NULL; @@ -1341,10 +1387,11 @@ EXPORT_SYMBOL(iget_locked); */ static int test_inode_iunique(struct super_block *sb, unsigned long ino) { - struct hlist_head *b = i_hash_head(sb, ino); + struct hlist_bl_head *b = i_hash_head(sb, ino); + struct hlist_bl_node *node; struct inode *inode; - hlist_for_each_entry_rcu(inode, b, i_hash) { + hlist_bl_for_each_entry_rcu(inode, node, b, i_hash) { if (inode->i_ino == ino && inode->i_sb == sb) return 0; } @@ -1428,12 +1475,12 @@ EXPORT_SYMBOL(igrab); struct inode *ilookup5_nowait(struct super_block *sb, unsigned long hashval, int (*test)(struct inode *, void *), void *data) { - struct hlist_head *head = i_hash_head(sb, hashval); + struct hlist_bl_head *b = i_hash_head(sb, hashval); struct inode *inode; - spin_lock(&inode_hash_lock); - inode = find_inode(sb, head, test, data); - spin_unlock(&inode_hash_lock); + hlist_bl_lock(b); + inode = find_inode(sb, b, test, data); + hlist_bl_unlock(b); return IS_ERR(inode) ? NULL : inode; } @@ -1483,12 +1530,12 @@ EXPORT_SYMBOL(ilookup5); */ struct inode *ilookup(struct super_block *sb, unsigned long ino) { - struct hlist_head *head = i_hash_head(sb, ino); + struct hlist_bl_head *b = i_hash_head(sb, ino); struct inode *inode; again: - spin_lock(&inode_hash_lock); - inode = find_inode_fast(sb, head, ino); - spin_unlock(&inode_hash_lock); + hlist_bl_lock(b); + inode = find_inode_fast(sb, b, ino); + hlist_bl_unlock(b); if (inode) { if (IS_ERR(inode)) @@ -1532,12 +1579,13 @@ struct inode *find_inode_nowait(struct super_block *sb, void *), void *data) { - struct hlist_head *head = i_hash_head(sb, hashval); + struct hlist_bl_head *b = i_hash_head(sb, hashval); + struct hlist_bl_node *node; struct inode *inode, *ret_inode = NULL; int mval; - spin_lock(&inode_hash_lock); - hlist_for_each_entry(inode, head, i_hash) { + hlist_bl_lock(b); + hlist_bl_for_each_entry(inode, node, b, i_hash) { if (inode->i_sb != sb) continue; mval = match(inode, hashval, data); @@ -1548,7 +1596,7 @@ struct inode *find_inode_nowait(struct super_block *sb, goto out; } out: - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); return ret_inode; } EXPORT_SYMBOL(find_inode_nowait); @@ -1577,13 +1625,14 @@ EXPORT_SYMBOL(find_inode_nowait); struct inode *find_inode_rcu(struct super_block *sb, unsigned long hashval, int (*test)(struct inode *, void *), void *data) { - struct hlist_head *head = i_hash_head(sb, hashval); + struct hlist_bl_head *b = i_hash_head(sb, hashval); + struct hlist_bl_node *node; struct inode *inode; RCU_LOCKDEP_WARN(!rcu_read_lock_held(), "suspicious find_inode_rcu() usage"); - hlist_for_each_entry_rcu(inode, head, i_hash) { + hlist_bl_for_each_entry_rcu(inode, node, b, i_hash) { if (inode->i_sb == sb && !(READ_ONCE(inode->i_state) & (I_FREEING | I_WILL_FREE)) && test(inode, data)) @@ -1615,13 +1664,14 @@ EXPORT_SYMBOL(find_inode_rcu); struct inode *find_inode_by_ino_rcu(struct super_block *sb, unsigned long ino) { - struct hlist_head *head = i_hash_head(sb, ino); + struct hlist_bl_head *b = i_hash_head(sb, ino); + struct hlist_bl_node *node; struct inode *inode; RCU_LOCKDEP_WARN(!rcu_read_lock_held(), "suspicious find_inode_by_ino_rcu() usage"); - hlist_for_each_entry_rcu(inode, head, i_hash) { + hlist_bl_for_each_entry_rcu(inode, node, b, i_hash) { if (inode->i_ino == ino && inode->i_sb == sb && !(READ_ONCE(inode->i_state) & (I_FREEING | I_WILL_FREE))) @@ -1635,39 +1685,42 @@ int insert_inode_locked(struct inode *inode) { struct super_block *sb = inode->i_sb; ino_t ino = inode->i_ino; - struct hlist_head *head = i_hash_head(sb, ino); + struct hlist_bl_head *b = i_hash_head(sb, ino); while (1) { - struct inode *old = NULL; - spin_lock(&inode_hash_lock); - hlist_for_each_entry(old, head, i_hash) { - if (old->i_ino != ino) + struct hlist_bl_node *node; + struct inode *old = NULL, *t; + + hlist_bl_lock(b); + hlist_bl_for_each_entry(t, node, b, i_hash) { + if (t->i_ino != ino) continue; - if (old->i_sb != sb) + if (t->i_sb != sb) continue; - spin_lock(&old->i_lock); - if (old->i_state & (I_FREEING|I_WILL_FREE)) { - spin_unlock(&old->i_lock); + spin_lock(&t->i_lock); + if (t->i_state & (I_FREEING|I_WILL_FREE)) { + spin_unlock(&t->i_lock); continue; } + old = t; break; } if (likely(!old)) { spin_lock(&inode->i_lock); inode->i_state |= I_NEW | I_CREATING; - hlist_add_head_rcu(&inode->i_hash, head); + __insert_inode_hash_head(inode, b); spin_unlock(&inode->i_lock); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); return 0; } if (unlikely(old->i_state & I_CREATING)) { spin_unlock(&old->i_lock); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); return -EBUSY; } __iget(old); spin_unlock(&old->i_lock); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); wait_on_inode(old); if (unlikely(!inode_unhashed(old))) { iput(old); @@ -2192,17 +2245,18 @@ EXPORT_SYMBOL(inode_needs_sync); * wake_up_bit(&inode->i_state, __I_NEW) after removing from the hash list * will DTRT. */ -static void __wait_on_freeing_inode(struct inode *inode) +static void __wait_on_freeing_inode(struct hlist_bl_head *b, + struct inode *inode) { wait_queue_head_t *wq; DEFINE_WAIT_BIT(wait, &inode->i_state, __I_NEW); wq = bit_waitqueue(&inode->i_state, __I_NEW); prepare_to_wait(wq, &wait.wq_entry, TASK_UNINTERRUPTIBLE); spin_unlock(&inode->i_lock); - spin_unlock(&inode_hash_lock); + hlist_bl_unlock(b); schedule(); finish_wait(wq, &wait.wq_entry); - spin_lock(&inode_hash_lock); + hlist_bl_lock(b); } static __initdata unsigned long ihash_entries; @@ -2228,7 +2282,7 @@ void __init inode_init_early(void) inode_hashtable = alloc_large_system_hash("Inode-cache", - sizeof(struct hlist_head), + sizeof(struct hlist_bl_head), ihash_entries, 14, HASH_EARLY | HASH_ZERO, @@ -2254,7 +2308,7 @@ void __init inode_init(void) inode_hashtable = alloc_large_system_hash("Inode-cache", - sizeof(struct hlist_head), + sizeof(struct hlist_bl_head), ihash_entries, 14, HASH_ZERO, diff --git a/include/linux/fs.h b/include/linux/fs.h index 1a6f951942..db8d49cbf7 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -647,7 +647,8 @@ struct inode { unsigned long dirtied_when; /* jiffies of first dirtying */ unsigned long dirtied_time_when; - struct hlist_node i_hash; + struct hlist_bl_node i_hash; + struct hlist_bl_head *i_hash_head; struct list_head i_io_list; /* backing dev IO list */ #ifdef CONFIG_CGROUP_WRITEBACK struct bdi_writeback *i_wb; /* the associated cgroup wb */ @@ -713,7 +714,7 @@ static inline unsigned int i_blocksize(const struct inode *node) static inline int inode_unhashed(struct inode *inode) { - return hlist_unhashed(&inode->i_hash); + return hlist_bl_unhashed(&inode->i_hash); } /* @@ -724,7 +725,7 @@ static inline int inode_unhashed(struct inode *inode) */ static inline void inode_fake_hash(struct inode *inode) { - hlist_add_fake(&inode->i_hash); + hlist_bl_add_fake(&inode->i_hash); } /* @@ -2695,7 +2696,7 @@ static inline void insert_inode_hash(struct inode *inode) extern void __remove_inode_hash(struct inode *); static inline void remove_inode_hash(struct inode *inode) { - if (!inode_unhashed(inode) && !hlist_fake(&inode->i_hash)) + if (!inode_unhashed(inode) && !hlist_bl_fake(&inode->i_hash)) __remove_inode_hash(inode); } From patchwork Tue May 9 16:56:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91678 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3041164vqo; Tue, 9 May 2023 10:08:56 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7wMQtZ6KLL5nn9jnaR4upThyaolXro63TjVHNiLY0uUCQ4DMF8tyxMf8TWoFwPQxXG64pa X-Received: by 2002:a05:6a00:2443:b0:624:7c9a:c832 with SMTP id d3-20020a056a00244300b006247c9ac832mr18826728pfj.8.1683652135876; Tue, 09 May 2023 10:08:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652135; cv=none; d=google.com; s=arc-20160816; b=YXzyp0tmcazxDj9yscRXwgBtwZiCx9pS4UR6sy5VcIfJqshDkBEoRwwvDVnFxdrDNM 4UNH+SvMNzoz37ic2ZI3qxcZJtnGK9ayYSCg8x5vM5HKNVvuBzS+Oke9DBD7EyvJo/e7 QrETVLQ1xV6DxL0MhKwGl8wOYhTB5KWJHf5+81syKBJwUXEsntpY6FL/5xK11/6jtFab eHIhyDzQoCrEDEezSSNoZl6tkGLvaqky+jqHoB13YAWtMVo2B3CXSu7i2Xq+cA+z6RyQ uDpWW28lwd6kSUnU1SvuYyTSxDoAw7nDGQBoV858Gzv/qMnXqulWSZZY3yoNWVCuvv1K yKAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=oQzCDYvJ0lsDeSo/s7SzXt92I5sJ/6Jd0MY/kK+yD7U=; b=i1uTibG/wyc7gRJZDRizDYWT/IS1grecKd64FKskaxgNkmyTc0GBQ2CQZABfICsb3X UDnoFEUcECPpIX2gXcyk2/zYHY1ti0F1B1P/ctjaY8wXnuzP2/qkLfjPM6eLP0pX8+bo lVtwWtqWUNpeB0zPuDeEcYpFWleZI0cIePDeCdt6PBgUKEsBPnUjk8tkBq3nHgKrjxX+ uvTHicbN9epbpvcz4mbYeWczmzJeBZ4eeS2LZ6htWnfa7BEZPAx2pfh6sySs1Os7DnG+ PiUwerNcy2axtTZXYZyj8DC1P/DgKZy0Gbat0MuKvYcamglkXvRKbTesUvZx9JQbE1sV HA/Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=PHvXBmFr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id p16-20020aa79e90000000b00643e06f6c17si2910992pfq.142.2023.05.09.10.08.40; Tue, 09 May 2023 10:08:55 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=PHvXBmFr; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234984AbjEIRAL (ORCPT + 99 others); Tue, 9 May 2023 13:00:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234932AbjEIQ6j (ORCPT ); Tue, 9 May 2023 12:58:39 -0400 Received: from out-51.mta1.migadu.com (out-51.mta1.migadu.com [IPv6:2001:41d0:203:375::33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54F5F55BB for ; Tue, 9 May 2023 09:57:36 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651452; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oQzCDYvJ0lsDeSo/s7SzXt92I5sJ/6Jd0MY/kK+yD7U=; b=PHvXBmFrG1haHZH++ByGkbPhrVLFCqbuynFUtixhqKw8mL0m5MIf5c7OUEjz7eapdhaScW hCJHsByetQerY+RW9TsdnWKMQdY24yvdwZbxmirMlKSiBi5Di6g55JhMWL5pXOi8+suAQa Dg3gJDde1WSrFwgXqlCxQnFt2d4bGZY= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Alexander Viro , Matthew Wilcox Subject: [PATCH 23/32] iov_iter: copy_folio_from_iter_atomic() Date: Tue, 9 May 2023 12:56:48 -0400 Message-Id: <20230509165657.1735798-24-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437222047274417?= X-GMAIL-MSGID: =?utf-8?q?1765437222047274417?= Add a foliated version of copy_page_from_iter_atomic() Signed-off-by: Kent Overstreet Cc: Alexander Viro Cc: Matthew Wilcox --- include/linux/uio.h | 2 ++ lib/iov_iter.c | 53 ++++++++++++++++++++++++++++++++++++--------- 2 files changed, 45 insertions(+), 10 deletions(-) diff --git a/include/linux/uio.h b/include/linux/uio.h index 27e3fd9429..b2c281cb10 100644 --- a/include/linux/uio.h +++ b/include/linux/uio.h @@ -154,6 +154,8 @@ static inline struct iovec iov_iter_iovec(const struct iov_iter *iter) size_t copy_page_from_iter_atomic(struct page *page, unsigned offset, size_t bytes, struct iov_iter *i); +size_t copy_folio_from_iter_atomic(struct folio *folio, size_t offset, + size_t bytes, struct iov_iter *i); void iov_iter_advance(struct iov_iter *i, size_t bytes); void iov_iter_revert(struct iov_iter *i, size_t bytes); size_t fault_in_iov_iter_readable(const struct iov_iter *i, size_t bytes); diff --git a/lib/iov_iter.c b/lib/iov_iter.c index 274014e4ea..27ba7e9f9e 100644 --- a/lib/iov_iter.c +++ b/lib/iov_iter.c @@ -800,18 +800,10 @@ size_t iov_iter_zero(size_t bytes, struct iov_iter *i) } EXPORT_SYMBOL(iov_iter_zero); -size_t copy_page_from_iter_atomic(struct page *page, unsigned offset, size_t bytes, - struct iov_iter *i) +static inline size_t __copy_page_from_iter_atomic(struct page *page, unsigned offset, + size_t bytes, struct iov_iter *i) { char *kaddr = kmap_atomic(page), *p = kaddr + offset; - if (!page_copy_sane(page, offset, bytes)) { - kunmap_atomic(kaddr); - return 0; - } - if (WARN_ON_ONCE(!i->data_source)) { - kunmap_atomic(kaddr); - return 0; - } iterate_and_advance(i, bytes, base, len, off, copyin(p + off, base, len), memcpy(p + off, base, len) @@ -819,8 +811,49 @@ size_t copy_page_from_iter_atomic(struct page *page, unsigned offset, size_t byt kunmap_atomic(kaddr); return bytes; } + +size_t copy_page_from_iter_atomic(struct page *page, unsigned offset, size_t bytes, + struct iov_iter *i) +{ + if (!page_copy_sane(page, offset, bytes)) + return 0; + if (WARN_ON_ONCE(!i->data_source)) + return 0; + return __copy_page_from_iter_atomic(page, offset, bytes, i); +} EXPORT_SYMBOL(copy_page_from_iter_atomic); +size_t copy_folio_from_iter_atomic(struct folio *folio, size_t offset, + size_t bytes, struct iov_iter *i) +{ + size_t ret = 0; + + if (WARN_ON(offset + bytes > folio_size(folio))) + return 0; + if (WARN_ON_ONCE(!i->data_source)) + return 0; + +#ifdef CONFIG_HIGHMEM + while (bytes) { + struct page *page = folio_page(folio, offset >> PAGE_SHIFT); + unsigned b = min(bytes, PAGE_SIZE - (offset & PAGE_MASK)); + unsigned r = __copy_page_from_iter_atomic(page, offset, b, i); + + offset += r; + bytes -= r; + ret += r; + + if (r != b) + break; + } +#else + ret = __copy_page_from_iter_atomic(&folio->page, offset, bytes, i); +#endif + + return ret; +} +EXPORT_SYMBOL(copy_folio_from_iter_atomic); + static void pipe_advance(struct iov_iter *i, size_t size) { struct pipe_inode_info *pipe = i->pipe; From patchwork Tue May 9 16:56:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91673 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3040311vqo; Tue, 9 May 2023 10:07:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5RiNi9hdLyTHajUcx6g0jDP7p+rXHtDoiawdMPz9bnfE8ThPLg6oHtbV9ChC5o5kr6DBg4 X-Received: by 2002:a05:6a20:6a0a:b0:100:31b5:50c9 with SMTP id p10-20020a056a206a0a00b0010031b550c9mr10910400pzk.60.1683652067938; Tue, 09 May 2023 10:07:47 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652067; cv=none; d=google.com; s=arc-20160816; b=g2OE2f7xX2XwSQWCOKqa9kVhIgd2GpgTYhVZpUrMN9DDNVcnSJqT6IxvkbzWaVDYAL cCOEiu5cmLm1a6S6lXibYe6ePo4R/0w2z7UTuxmgJoT3iWXm1ASk5SutdgfcDVY8oxes khYb13YlUqPxw6DuMkHMzBb1hHlEcJqJ9wXfGkEWmJDhjTEoBTvSIY1wBJJOoDCizDdi yf/VDmEpMi67uBTCTF9KMsuhsIBLZoLGvgidTKF0D3tiL09dyQJpZQnEGJWbKSPohpaw eQwX2Otr1Mdk89sKlNX4H1+EQUSLkwmGhnBZDllV84s/CyIzb4Z+QA1pVt75pdDYwG3L fQ2g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4evz1kbIC6hOLTFQMkHcxrXWiRo0lENP+eBhEpndpY8=; b=s/mAQVOX3eRUm+Dabqo4VqQ1866BDZvDbE2CuFJxkIBI2TbCh4r+Z2O9K8+B8r3q7L qMJNBkT/TNw/BgrxyKw8jqKLt1k66l4mq3eu8ZWenPMu9Cb+UbRXzHsYp1GCADzWNkEs 3HThtn5R4s/+lYSm0TuBnEmVHCvxjKADYhXLefaRebepElSDix2Evbepgwml4/Vi1zSw Ii55uzwuVJ5Dm6xGp+zz98WC7gJIFe1pAm6K9YLr9Tbt6WYMc9xucB8igjXN16T0c9Za PzYHwxhFla68Nfvde15/LMiRHpphudwTbG2nXdawa2kdBDbRxAjj3ZVjTWPkczl63M1e usoQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=CBbntMb8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h23-20020a633857000000b0051374678f95si1692609pgn.808.2023.05.09.10.07.35; Tue, 09 May 2023 10:07:47 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=CBbntMb8; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235093AbjEIRAR (ORCPT + 99 others); Tue, 9 May 2023 13:00:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43206 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235095AbjEIQ6j (ORCPT ); Tue, 9 May 2023 12:58:39 -0400 Received: from out-10.mta1.migadu.com (out-10.mta1.migadu.com [IPv6:2001:41d0:203:375::a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D1285FF9 for ; Tue, 9 May 2023 09:57:36 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651454; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4evz1kbIC6hOLTFQMkHcxrXWiRo0lENP+eBhEpndpY8=; b=CBbntMb8JDhq2x3mCt4kaM+tJCcmDi4cB1kDlSpHxpH4RtVg8j9fjfe//o4jNosLeoF/eg s9Et5lUNFROInJTonZjVn6nHQVp+osd7shzlDUCaeEGaRpXw+XicD8gVRAkZd6DLr6LXhH lspVH1eipCUD5E57d10siAXsjDd7OtQ= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 24/32] MAINTAINERS: Add entry for generic-radix-tree Date: Tue, 9 May 2023 12:56:49 -0400 Message-Id: <20230509165657.1735798-25-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437151179597969?= X-GMAIL-MSGID: =?utf-8?q?1765437151179597969?= lib/generic-radix-tree.c is a simple radix tree that supports storing arbitrary types. Add a maintainers entry for it. Signed-off-by: Kent Overstreet --- MAINTAINERS | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 5d76169140..c550f5909e 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -8615,6 +8615,13 @@ F: Documentation/devicetree/bindings/power/power?domain* F: drivers/base/power/domain*.c F: include/linux/pm_domain.h +GENERIC RADIX TREE: +M: Kent Overstreet +S: Supported +C: irc://irc.oftc.net/bcache +F: include/linux/generic-radix-tree.h +F: lib/generic-radix-tree.c + GENERIC RESISTIVE TOUCHSCREEN ADC DRIVER M: Eugen Hristev L: linux-input@vger.kernel.org From patchwork Tue May 9 16:56:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91684 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3043689vqo; Tue, 9 May 2023 10:12:36 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4c7Dm3ta3599uSApd7sZJ+XdmpzVdlOQeienU91TCM+vubn+5a07RFzQDNm6tIVT+nNUUV X-Received: by 2002:a17:902:d48d:b0:1ac:8835:b89b with SMTP id c13-20020a170902d48d00b001ac8835b89bmr6504302plg.5.1683652356411; Tue, 09 May 2023 10:12:36 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652356; cv=none; d=google.com; s=arc-20160816; b=YfukimmQiq6rpAfaguhzIOtfXmK3jCbfPXojt9y4LdN9hkRqAs6HlcEGF9yGBzTajY XegEv6CRNAt3a6Kk3mW6FGgSzrxfdqorr1MuVkMSqZHtEl/YJamw4N3DQPLeGxgzM7JQ PM+ulqjvwwti3HODJNn+9bH6CQPnT17yNUSuZUQPo25JWnKfmRyI0xgrL/6UNJVKgBCr xcc7LVGn/trLsHswfUndKGHC0WTAmefovVVPz6YFmgyFFcd7iXeBanS+3kX38qHRC3mn JUalD3Cm349SR0oETeKxRGkG9AjL0fttbcRzWfSNMWthagnugnCHKnodqHCthwtWTIXB BLTQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Xn5bEL0Ei0eSLZ8JURgH14o2MvwdOSnzFDfE6+pvE7w=; b=gzxJtUQ04QIJZbw0HrCNhYdOTT0/IDy8Cc5ojz/UEujOPzjZyMLCdNMSEm5kflU2J2 JjyhI93FlXEFu+K2t0MXC7psmvSHGEj393Wq6O/gMXSEFZI/57YlePN+VCsWjLbUyoFm KDat3Qo8YXf5uux9YNpS4qtX8eih239NCDtCZTMK1hPpGV2xCeTZc/yG4ZSzs0qgmb0B mShRgGsIC/nmyL2QhNXMKNO9c79483fjPMGJGhb1g4rPVAIu6ZI3K+qGQSx53o402u2e T+CvD2s6trUloae5BspTHIozVhWSTyWDhYdG/YFseBdreU+mSAtHY8H1P1K8Hvcltcbe wo8g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="V93/4Ixd"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m12-20020a633f0c000000b0050fad17282dsi1861660pga.217.2023.05.09.10.12.23; Tue, 09 May 2023 10:12:36 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="V93/4Ixd"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235095AbjEIRAU (ORCPT + 99 others); Tue, 9 May 2023 13:00:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235128AbjEIQ6v (ORCPT ); Tue, 9 May 2023 12:58:51 -0400 Received: from out-56.mta1.migadu.com (out-56.mta1.migadu.com [IPv6:2001:41d0:203:375::38]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 435C04C10 for ; Tue, 9 May 2023 09:57:38 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651455; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Xn5bEL0Ei0eSLZ8JURgH14o2MvwdOSnzFDfE6+pvE7w=; b=V93/4IxdB6R1oDyNyXn+vk/3B37TkkxMELqTGRTHIdvE7v0ldmbzKSzJngOfYgV6W6nktq 6gemnOZwcnJwZ+kRszXRAdPyc6IAAKQfnfywGr1CjONwF7cgLuph94sKOvHMVxnGQn4zEB 2zXtaZe0mueWaZ7xrmXdVfPRpV15W24= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet Subject: [PATCH 25/32] lib/generic-radix-tree.c: Don't overflow in peek() Date: Tue, 9 May 2023 12:56:50 -0400 Message-Id: <20230509165657.1735798-26-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437453321511732?= X-GMAIL-MSGID: =?utf-8?q?1765437453321511732?= From: Kent Overstreet When we started spreading new inode numbers throughout most of the 64 bit inode space, that triggered some corner case bugs, in particular some integer overflows related to the radix tree code. Oops. Signed-off-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/generic-radix-tree.h | 6 ++++++ lib/generic-radix-tree.c | 17 ++++++++++++++--- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/include/linux/generic-radix-tree.h b/include/linux/generic-radix-tree.h index 107613f7d7..63080822dc 100644 --- a/include/linux/generic-radix-tree.h +++ b/include/linux/generic-radix-tree.h @@ -184,6 +184,12 @@ void *__genradix_iter_peek(struct genradix_iter *, struct __genradix *, size_t); static inline void __genradix_iter_advance(struct genradix_iter *iter, size_t obj_size) { + if (iter->offset + obj_size < iter->offset) { + iter->offset = SIZE_MAX; + iter->pos = SIZE_MAX; + return; + } + iter->offset += obj_size; if (!is_power_of_2(obj_size) && diff --git a/lib/generic-radix-tree.c b/lib/generic-radix-tree.c index f25eb111c0..7dfa88282b 100644 --- a/lib/generic-radix-tree.c +++ b/lib/generic-radix-tree.c @@ -166,6 +166,10 @@ void *__genradix_iter_peek(struct genradix_iter *iter, struct genradix_root *r; struct genradix_node *n; unsigned level, i; + + if (iter->offset == SIZE_MAX) + return NULL; + restart: r = READ_ONCE(radix->root); if (!r) @@ -184,10 +188,17 @@ void *__genradix_iter_peek(struct genradix_iter *iter, (GENRADIX_ARY - 1); while (!n->children[i]) { + size_t objs_per_ptr = genradix_depth_size(level); + + if (iter->offset + objs_per_ptr < iter->offset) { + iter->offset = SIZE_MAX; + iter->pos = SIZE_MAX; + return NULL; + } + i++; - iter->offset = round_down(iter->offset + - genradix_depth_size(level), - genradix_depth_size(level)); + iter->offset = round_down(iter->offset + objs_per_ptr, + objs_per_ptr); iter->pos = (iter->offset >> PAGE_SHIFT) * objs_per_page; if (i == GENRADIX_ARY) From patchwork Tue May 9 16:56:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91692 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3046272vqo; Tue, 9 May 2023 10:16:43 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7LAFu2iXmCMiglkXGRzNjJo1ID2wyGPktVxHBfsnExR+gDCm+j87ZVvV5h6WDrKdV9PI9t X-Received: by 2002:a05:6a21:6317:b0:eb:6e43:8e1b with SMTP id wu23-20020a056a21631700b000eb6e438e1bmr15238675pzb.30.1683652603034; Tue, 09 May 2023 10:16:43 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652603; cv=none; d=google.com; s=arc-20160816; b=Xcb9TGTeUOJUYcFKhZnfyWJPcCMD0gXZqcCKsxGvhicgvzldesgaRRFzgy/xlFGslG 9BVMNZyB8fT7FZEVlWP5smPCxnj04gQQANtkdvNLV7DWIjghYOH7ah56K2eVV2Pbcgmd TJsb2NrFlwdWbC+RpRm5/ssvF6CyeRhQMCb4kjHI1NN8Vq0hCtHQXjOt2ImBWbcDH4L5 s7w8ht7O16QJzf+6dtjoKJPlwdgdSWtwyVdxOxivcQiL9mqhMXbdo6sGTlrIrMHtTrY2 37cCCogDk0VIiU0GZTl4xx8qG33EQnt/bXrtmCFxwuY+ekTL+VRAqgBPkkeVn3fzM30t zzfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=W3IMW7979m4jEebRkm8ameWmv+95K3nEqS903wmmHh0=; b=Y3pGkWLZa4ViDkn9jzGW3roK0valndO61apQ+9wZ5CMJf/ejwkUvWtKClgrG2gWFqn oAy1xPAXO5P5uKNB0vnRqhUCLaufk/IZZ2i2BMoFU65R7spaE1s5REKoKX5jev8AL1+j aqHIAI4GCRjd/XWZ1Ufyo9veSN9AeSFcnDsG2hynLmWB/K7GWWzoLVe6eODg8HVMbKcb Gs269Zh6NUpac3v8PdDyagKCXjkaXRBh9aDYWcE2pJSxDBRdvyEDohcZpQEaWMuqYeFu 58E+eh9Bvi03GKmpVmR0/Tcpxpl7bDqUQ0N/FcgGWBXgEH2jmPq3CZJt+mBMd0Oj/I1w jwlQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=rePqwIEQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b188-20020a6334c5000000b00524fcd54553si1843728pga.485.2023.05.09.10.16.30; Tue, 09 May 2023 10:16:43 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=rePqwIEQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235416AbjEIRA5 (ORCPT + 99 others); Tue, 9 May 2023 13:00:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42244 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235388AbjEIRAC (ORCPT ); Tue, 9 May 2023 13:00:02 -0400 Received: from out-9.mta1.migadu.com (out-9.mta1.migadu.com [95.215.58.9]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3E120729D for ; Tue, 9 May 2023 09:57:55 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651456; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W3IMW7979m4jEebRkm8ameWmv+95K3nEqS903wmmHh0=; b=rePqwIEQtfDZ7Lr1PZGsbFUi63Nh5oKibQRVDY7ErwYGzoTrCqeYvKqOCoZlO888YVeiOy A8lw/mTSAfWfN8ZSaPlBg2oZ8cefaiEUcoyOCcUyxFtzdFyxYu2f2khlg2LltB/a6SIRiV Kb36uSeA1vkHxSqaSroCgGgH2A7sRek= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet Subject: [PATCH 26/32] lib/generic-radix-tree.c: Add a missing include Date: Tue, 9 May 2023 12:56:51 -0400 Message-Id: <20230509165657.1735798-27-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437712120403011?= X-GMAIL-MSGID: =?utf-8?q?1765437712120403011?= From: Kent Overstreet We now need linux/limits.h for SIZE_MAX. Signed-off-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/generic-radix-tree.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/generic-radix-tree.h b/include/linux/generic-radix-tree.h index 63080822dc..f6cd0f909d 100644 --- a/include/linux/generic-radix-tree.h +++ b/include/linux/generic-radix-tree.h @@ -38,6 +38,7 @@ #include #include +#include #include #include #include From patchwork Tue May 9 16:56:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91679 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3041198vqo; Tue, 9 May 2023 10:08:59 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4URh9jKzhmPrs5z0CmHbgis6vxlWyxUwYvJ4Xtrsu6obzeSvA6bzL4qOw50wH3DVRbIVc6 X-Received: by 2002:a17:90a:94cc:b0:24e:edd:4d63 with SMTP id j12-20020a17090a94cc00b0024e0edd4d63mr14506805pjw.5.1683652138989; Tue, 09 May 2023 10:08:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683652138; cv=none; d=google.com; s=arc-20160816; b=dvajwyIiIfK8PZQmlqCicZw4cN1MHFwQeou1ZkqKxn7ROLUp8lX7sJhQkrkvUGstbt 0NC7IHqdK0OWUUBbQ9AQkCqc0Datx2vKtq5eXwEqqplbyP6DjQlUWgZVidouUKrO6KwV lrJbZbClcDvGLWzV4l3nso9ypZ+79gf0z3XtyrqAItcFLz121kOAemx3F7OjExd3pkwq snV7b15rK+g0tw5f+8cRcOHJI2qkHYlVuzFuVs3CzngLphEklW9EuFpWI/V/Rc4zopZc ge/h4UUSMn8LwcT5qVtQgGLBeUWyh79MTfQwjHZ57/GRZnrFGZjY5U/tTh2wGN8Tttou rCRg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=+ZhmCl+n4yHEwTa1THztHgqHnTTx5bTorYdBdx9eWUk=; b=onfgMOsa71FxXqyh+I0UNBObwVBMFn81HIk1reZTZJgvosINqJoA92DOTrGMyElPKH PAbSH3qtfpo9ySrMHu8A1x9pq+uYexb1lpKCZ3U/7olj/R7hNX8Ajy9FRzmitLCFijR4 xHm3yOuKRU/+/3Ry1b0VFaAxECzovMW/gwTCn9J1yvgU1xfrDlinP1PEjSIQVvytIhy4 tIF8H4YP0qSyNDyd9yryVgsV8VevzCVb75mIFmTrlkvJW8IZYSUi5uYxWaM5z7J4USs/ fP5lLImcT4z1aYXILjE9rEahUuUFBnLgcoaor1gVcsjhPfjsP+rC7aNFOcA3a7a6ck+4 Vrmg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=AnOcSonl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i9-20020a17090a718900b0024e1e15b1b0si15284409pjk.161.2023.05.09.10.08.22; Tue, 09 May 2023 10:08:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=AnOcSonl; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235209AbjEIRAY (ORCPT + 99 others); Tue, 9 May 2023 13:00:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234954AbjEIQ7Q (ORCPT ); Tue, 9 May 2023 12:59:16 -0400 Received: from out-22.mta1.migadu.com (out-22.mta1.migadu.com [IPv6:2001:41d0:203:375::16]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95687FA for ; Tue, 9 May 2023 09:57:40 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651457; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=+ZhmCl+n4yHEwTa1THztHgqHnTTx5bTorYdBdx9eWUk=; b=AnOcSonlgfJwVL1TfpYCrbzb3btH35FbKemYRBb7Tu6SNotf+y3JjRuDfJ26u2TA5Ga1Xu WMjtnAl0/RYF0F/+W96E3Zb9Pwc42jua50CeTq8jkOPIADAjkuqMarQVzkNNGhQyHCJQWM qzXJsyauJIJIQ/Y3wqhmN1555VfHPNs= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet Subject: [PATCH 27/32] lib/generic-radix-tree.c: Add peek_prev() Date: Tue, 9 May 2023 12:56:52 -0400 Message-Id: <20230509165657.1735798-28-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765437225025551773?= X-GMAIL-MSGID: =?utf-8?q?1765437225025551773?= From: Kent Overstreet This patch adds genradix_peek_prev(), genradix_iter_rewind(), and genradix_for_each_reverse(), for iterating backwards over a generic radix tree. Signed-off-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/generic-radix-tree.h | 61 +++++++++++++++++++++++++++++- lib/generic-radix-tree.c | 59 +++++++++++++++++++++++++++++ 2 files changed, 119 insertions(+), 1 deletion(-) diff --git a/include/linux/generic-radix-tree.h b/include/linux/generic-radix-tree.h index f6cd0f909d..c74b737699 100644 --- a/include/linux/generic-radix-tree.h +++ b/include/linux/generic-radix-tree.h @@ -117,6 +117,11 @@ static inline size_t __idx_to_offset(size_t idx, size_t obj_size) #define __genradix_cast(_radix) (typeof((_radix)->type[0]) *) #define __genradix_obj_size(_radix) sizeof((_radix)->type[0]) +#define __genradix_objs_per_page(_radix) \ + (PAGE_SIZE / sizeof((_radix)->type[0])) +#define __genradix_page_remainder(_radix) \ + (PAGE_SIZE % sizeof((_radix)->type[0])) + #define __genradix_idx_to_offset(_radix, _idx) \ __idx_to_offset(_idx, __genradix_obj_size(_radix)) @@ -180,7 +185,25 @@ void *__genradix_iter_peek(struct genradix_iter *, struct __genradix *, size_t); #define genradix_iter_peek(_iter, _radix) \ (__genradix_cast(_radix) \ __genradix_iter_peek(_iter, &(_radix)->tree, \ - PAGE_SIZE / __genradix_obj_size(_radix))) + __genradix_objs_per_page(_radix))) + +void *__genradix_iter_peek_prev(struct genradix_iter *, struct __genradix *, + size_t, size_t); + +/** + * genradix_iter_peek - get first entry at or below iterator's current + * position + * @_iter: a genradix_iter + * @_radix: genradix being iterated over + * + * If no more entries exist at or below @_iter's current position, returns NULL + */ +#define genradix_iter_peek_prev(_iter, _radix) \ + (__genradix_cast(_radix) \ + __genradix_iter_peek_prev(_iter, &(_radix)->tree, \ + __genradix_objs_per_page(_radix), \ + __genradix_obj_size(_radix) + \ + __genradix_page_remainder(_radix))) static inline void __genradix_iter_advance(struct genradix_iter *iter, size_t obj_size) @@ -203,6 +226,25 @@ static inline void __genradix_iter_advance(struct genradix_iter *iter, #define genradix_iter_advance(_iter, _radix) \ __genradix_iter_advance(_iter, __genradix_obj_size(_radix)) +static inline void __genradix_iter_rewind(struct genradix_iter *iter, + size_t obj_size) +{ + if (iter->offset == 0 || + iter->offset == SIZE_MAX) { + iter->offset = SIZE_MAX; + return; + } + + if ((iter->offset & (PAGE_SIZE - 1)) == 0) + iter->offset -= PAGE_SIZE % obj_size; + + iter->offset -= obj_size; + iter->pos--; +} + +#define genradix_iter_rewind(_iter, _radix) \ + __genradix_iter_rewind(_iter, __genradix_obj_size(_radix)) + #define genradix_for_each_from(_radix, _iter, _p, _start) \ for (_iter = genradix_iter_init(_radix, _start); \ (_p = genradix_iter_peek(&_iter, _radix)) != NULL; \ @@ -220,6 +262,23 @@ static inline void __genradix_iter_advance(struct genradix_iter *iter, #define genradix_for_each(_radix, _iter, _p) \ genradix_for_each_from(_radix, _iter, _p, 0) +#define genradix_last_pos(_radix) \ + (SIZE_MAX / PAGE_SIZE * __genradix_objs_per_page(_radix) - 1) + +/** + * genradix_for_each_reverse - iterate over entry in a genradix, reverse order + * @_radix: genradix to iterate over + * @_iter: a genradix_iter to track current position + * @_p: pointer to genradix entry type + * + * On every iteration, @_p will point to the current entry, and @_iter.pos + * will be the current entry's index. + */ +#define genradix_for_each_reverse(_radix, _iter, _p) \ + for (_iter = genradix_iter_init(_radix, genradix_last_pos(_radix));\ + (_p = genradix_iter_peek_prev(&_iter, _radix)) != NULL;\ + genradix_iter_rewind(&_iter, _radix)) + int __genradix_prealloc(struct __genradix *, size_t, gfp_t); /** diff --git a/lib/generic-radix-tree.c b/lib/generic-radix-tree.c index 7dfa88282b..41f1bcdc44 100644 --- a/lib/generic-radix-tree.c +++ b/lib/generic-radix-tree.c @@ -1,4 +1,5 @@ +#include #include #include #include @@ -212,6 +213,64 @@ void *__genradix_iter_peek(struct genradix_iter *iter, } EXPORT_SYMBOL(__genradix_iter_peek); +void *__genradix_iter_peek_prev(struct genradix_iter *iter, + struct __genradix *radix, + size_t objs_per_page, + size_t obj_size_plus_page_remainder) +{ + struct genradix_root *r; + struct genradix_node *n; + unsigned level, i; + + if (iter->offset == SIZE_MAX) + return NULL; + +restart: + r = READ_ONCE(radix->root); + if (!r) + return NULL; + + n = genradix_root_to_node(r); + level = genradix_root_to_depth(r); + + if (ilog2(iter->offset) >= genradix_depth_shift(level)) { + iter->offset = genradix_depth_size(level); + iter->pos = (iter->offset >> PAGE_SHIFT) * objs_per_page; + + iter->offset -= obj_size_plus_page_remainder; + iter->pos--; + } + + while (level) { + level--; + + i = (iter->offset >> genradix_depth_shift(level)) & + (GENRADIX_ARY - 1); + + while (!n->children[i]) { + size_t objs_per_ptr = genradix_depth_size(level); + + iter->offset = round_down(iter->offset, objs_per_ptr); + iter->pos = (iter->offset >> PAGE_SHIFT) * objs_per_page; + + if (!iter->offset) + return NULL; + + iter->offset -= obj_size_plus_page_remainder; + iter->pos--; + + if (!i) + goto restart; + --i; + } + + n = n->children[i]; + } + + return &n->data[iter->offset & (PAGE_SIZE - 1)]; +} +EXPORT_SYMBOL(__genradix_iter_peek_prev); + static void genradix_free_recurse(struct genradix_node *n, unsigned level) { if (level) { From patchwork Tue May 9 16:56:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91700 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3058796vqo; Tue, 9 May 2023 10:39:13 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4hn/Dnf0x1RN3mMKyZQu85wOpenkMyArzEIbzH3V8CUbn/hp2Sj9zBFbn8iDQ/BdFBTKjH X-Received: by 2002:a17:903:41ca:b0:1a6:abac:9cc with SMTP id u10-20020a17090341ca00b001a6abac09ccmr17896173ple.66.1683653953227; Tue, 09 May 2023 10:39:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683653953; cv=none; d=google.com; s=arc-20160816; b=fmAj9+0WeapJxwQ4ewsg1iQ6HOKdkgWEEe5xPzi3EZ8P8urudHu5QU96+kzC/lVCZo E/AV2NRmkIwBmWUyf1kZDTTaGKkB3eveF1iDG8K9rtdKKbc+/g4dnI0eX51IcUoMnlVL Bz0QgriKwXFR5lgQF41j+TK2taQy0DrNINTb9lPItsiWaBs9+6DqWdMUm5cbScuFHzZ5 rpWGNmHvvyvyUEISkk6sO7plcyoy+3cUgnB3ZCf0BQMApaTGMSvNKsQfal3ya9AknnkW xDtgPsdjKK9OBX7jZ01tpGc88j3nwikT78RIkVSM6ApvaDT7eKiyFd2NkYJY3mkXvBAC 2stw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=R1CKn2DNdt+nUP3OEYLc0FEKerxTzeneNiYGijAjw4w=; b=xeouBhRKXscWN9cZ0jG32B+tH1GgraeJ2REg2ccr8s6gM8MIi04pMwWaK5cFr+tQGa ypPvvMj1PMf/pMzh/uh9THuq1InSGwD9FB9B+6QcUIb2qG5E58HwnMQb+nu8KaRxJZWJ G8mAk50M0zEYrxnUVf34yyNdoc7wlMdEy0pQklc/ESo4gVbX8SDL5PoHxAxfK5cpph4O QuuGgwWuAtyqh2klnX5xK5aWbW4vjBXXNSyJ0FZCahXam2NsYnCa7DUySzz2y151sTJw m/xi9qbb4carldkOyyYEx0zYGVTs9ByFkrF2zbOWfZ4sGdyARLBiCAxvlqp9Wig2uCRI L2eQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=MWyrh9lT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id m17-20020a170902bb9100b001a8d80c7bb1si1856213pls.128.2023.05.09.10.39.00; Tue, 09 May 2023 10:39:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=MWyrh9lT; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235525AbjEIRBA (ORCPT + 99 others); Tue, 9 May 2023 13:01:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235450AbjEIRAE (ORCPT ); Tue, 9 May 2023 13:00:04 -0400 Received: from out-2.mta1.migadu.com (out-2.mta1.migadu.com [95.215.58.2]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 395C06A5A for ; Tue, 9 May 2023 09:57:57 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651458; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=R1CKn2DNdt+nUP3OEYLc0FEKerxTzeneNiYGijAjw4w=; b=MWyrh9lTZEKvAjK54wQLwMmHmVWpNM37wFWrK8Q5auXGHq22dsN5xqqiOumHjYP49lY2bB o8u0C/aFn62GOYNPBbIfgOahC2Fj+hR7+2lBPSyPpU+gL7KBAllKr1kecg10NBdQmKTRqa dOIjSLmj5brVSuAmKbeJPBDHpevngMs= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Christopher James Halse Rogers , Kent Overstreet Subject: [PATCH 28/32] stacktrace: Export stack_trace_save_tsk Date: Tue, 9 May 2023 12:56:53 -0400 Message-Id: <20230509165657.1735798-29-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765439127636470577?= X-GMAIL-MSGID: =?utf-8?q?1765439127636470577?= From: Christopher James Halse Rogers The bcachefs module wants it, and there doesn't seem to be any reason it shouldn't be exported like the other functions. Signed-off-by: Christopher James Halse Rogers Signed-off-by: Kent Overstreet --- kernel/stacktrace.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/stacktrace.c b/kernel/stacktrace.c index 9ed5ce9894..4f65824879 100644 --- a/kernel/stacktrace.c +++ b/kernel/stacktrace.c @@ -151,6 +151,7 @@ unsigned int stack_trace_save_tsk(struct task_struct *tsk, unsigned long *store, put_task_stack(tsk); return c.len; } +EXPORT_SYMBOL_GPL(stack_trace_save_tsk); /** * stack_trace_save_regs - Save a stack trace based on pt_regs into a storage array @@ -301,6 +302,7 @@ unsigned int stack_trace_save_tsk(struct task_struct *task, save_stack_trace_tsk(task, &trace); return trace.nr_entries; } +EXPORT_SYMBOL_GPL(stack_trace_save_tsk); /** * stack_trace_save_regs - Save a stack trace based on pt_regs into a storage array From patchwork Tue May 9 16:56:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91667 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3037044vqo; Tue, 9 May 2023 10:03:59 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ51Vlu7iBmkWaGpL6bdLDI9ON/CS5jCAPzB0dnTyQWllnoqGAZkLFk4j54IN+4Uw65calN6 X-Received: by 2002:a05:6a21:788c:b0:100:5c04:4903 with SMTP id bf12-20020a056a21788c00b001005c044903mr10376183pzc.55.1683651838938; Tue, 09 May 2023 10:03:58 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683651838; cv=none; d=google.com; s=arc-20160816; b=mbgleMN8ofy1D2O45XG/piQRqkho4TXchxPs0YE+kpx/SN97JvET65pFOIY0m8QQfu SY1gnfEnA65z46d1sDiRHC3cg2rTIiNBR+B+1KXAyG200OINo58XPghay7uCLG0laJID z4JPzqhR8l0YRtu9FLjP1QLq7RbFsxwU1I+BrW8xR87QLtZdSfJkVbb47Ez5X5yp2mua uhjlTxJ5yQK6vjt8S85BRh3dw+tRtSCNi5szEkvplNWaEVXKHz1L/1DsvJazqsMmJg5p d+4xNQfLLfKKOaEdsqePqQuk2QeFZNUhrm8q+QZDVKTqIg+1AMwnqqIwtcs1Fx5TDyoF ligg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=LbLEE4jIbrZpJwXBO7eCGOTCJ14bvYB1GBwSVBBfWTo=; b=w+I56aid/HK6cQWoqNHjSZO8e3eDvGKjLtdh2c71kw0ck0XGv69msJozqHmx9roeAo ok+GkhApDDpzqPWqX4ctPGk59sJYioGb347NFl82ga74N04ba7o2u4emeL9cDP9vtQBd 5xliWtADSFfBp91+kj80RpTGLRl8dF3cQ7gQTvlekNCNC+T+kg651XIGsWLpGxA6idNU 4kqj4HWoO5N86zYWEYTd/b0MmeWBuUysKvxMLEDwMQqC7JE1KFtdFrJjeFeQeQ1/Keal Jc+rt2ycG4vBGDfw6vqwyisxqwyQD7GXjjLxEgCIepiNb+WMrVp/YwnUhOkQFks1GJCz di7g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Ema++DBy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i69-20020a639d48000000b0051948f1993fsi1923557pgd.40.2023.05.09.10.03.37; Tue, 09 May 2023 10:03:58 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Ema++DBy; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235240AbjEIRA3 (ORCPT + 99 others); Tue, 9 May 2023 13:00:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234983AbjEIQ7T (ORCPT ); Tue, 9 May 2023 12:59:19 -0400 Received: from out-33.mta1.migadu.com (out-33.mta1.migadu.com [IPv6:2001:41d0:203:375::21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6CCBB65A4 for ; Tue, 9 May 2023 09:57:44 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651459; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LbLEE4jIbrZpJwXBO7eCGOTCJ14bvYB1GBwSVBBfWTo=; b=Ema++DByNMG9DiB+n947beAEq7g/Q71m9KoPfjHSpli1UraJyM8iCLpH2dVD7dGmlPZham POXN8Vj6rrwc/2I6l2HTF1fjTiqvONuKx/tIEtEaCgCG3vT3KEZtJaEGlgxgJ8gxdR7Ezw AkgHuGvv7tkdEsQ6O1IeMS4LPLXMVs0= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Kent Overstreet Subject: [PATCH 29/32] lib/string_helpers: string_get_size() now returns characters wrote Date: Tue, 9 May 2023 12:56:54 -0400 Message-Id: <20230509165657.1735798-30-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765436910393074224?= X-GMAIL-MSGID: =?utf-8?q?1765436910393074224?= From: Kent Overstreet printbuf now needs to know the number of characters that would have been written if the buffer was too small, like snprintf(); this changes string_get_size() to return the the return value of snprintf(). Signed-off-by: Kent Overstreet Signed-off-by: Kent Overstreet --- include/linux/string_helpers.h | 4 ++-- lib/string_helpers.c | 8 ++++---- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/include/linux/string_helpers.h b/include/linux/string_helpers.h index fae6beaaa2..44148f8feb 100644 --- a/include/linux/string_helpers.h +++ b/include/linux/string_helpers.h @@ -23,8 +23,8 @@ enum string_size_units { STRING_UNITS_2, /* use binary powers of 2^10 */ }; -void string_get_size(u64 size, u64 blk_size, enum string_size_units units, - char *buf, int len); +int string_get_size(u64 size, u64 blk_size, enum string_size_units units, + char *buf, int len); int parse_int_array_user(const char __user *from, size_t count, int **array); diff --git a/lib/string_helpers.c b/lib/string_helpers.c index 230020a2e0..ca36ceba0e 100644 --- a/lib/string_helpers.c +++ b/lib/string_helpers.c @@ -32,8 +32,8 @@ * at least 9 bytes and will always be zero terminated. * */ -void string_get_size(u64 size, u64 blk_size, const enum string_size_units units, - char *buf, int len) +int string_get_size(u64 size, u64 blk_size, const enum string_size_units units, + char *buf, int len) { static const char *const units_10[] = { "B", "kB", "MB", "GB", "TB", "PB", "EB", "ZB", "YB" @@ -126,8 +126,8 @@ void string_get_size(u64 size, u64 blk_size, const enum string_size_units units, else unit = units_str[units][i]; - snprintf(buf, len, "%u%s %s", (u32)size, - tmp, unit); + return snprintf(buf, len, "%u%s %s", (u32)size, + tmp, unit); } EXPORT_SYMBOL(string_get_size); From patchwork Tue May 9 16:56:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91699 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3058505vqo; Tue, 9 May 2023 10:38:40 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ40b5AdLsfvhIDgNsOuhY+bIx6r9tfXjeOuoWuaylLpYTPBxfWBqKqYxpbBFQtR4sjnIUEB X-Received: by 2002:a05:6a20:7f83:b0:d5:73ad:87c2 with SMTP id d3-20020a056a207f8300b000d573ad87c2mr17162747pzj.56.1683653920643; Tue, 09 May 2023 10:38:40 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683653920; cv=none; d=google.com; s=arc-20160816; b=Nm6bSYbI3F5vwW+H7fdahiBoBOjR2nhL7xf9W1YvHopZq0QOZh01uUSTUw6HY9pPDd 37W4JP1svqKezPAQNZubou9BdVs1KUWxSk1fvt4gLIQcCH6xyoFygdjTcXCzz9MQkrCs QAmCGYgafXyZbwUFEifPR+VNdC8ZMhrWCiyVYfvayj1zHeHO2x3dNVh7LM+8FtPdC0Nd +i24tZMyZlV6f+w3zJezX/PjHCLc03CcjfohJTA2DlsDpW7ZWr3cydAza630zxLRKMET 3Fa4JYwNQ7/8aKDHSg433NDhfNVdeR8RDa7oE7qSYmsJgY89zWIxVaCBPLW9UVOTsHXU Wd1Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NRkDg8/QDL78Ozn75JlJivS7ptmCxyI7V8VfiBBetMw=; b=T7FR1ISybfX8yadRcLJAHDR0ftH3OakTUvTpG1lbsis1PfmF18tPqptL2GyuOQ5LAV VCCXOL75jsk7Tq3SzOxi7KDUgaqB1m+CvWp2zqdHQInVK0Nq8wurCW4pDC8T9VZ/dadJ JjEx/4w2Xv++YnxExptQLhdduJ/5e14dnUFNUQXd50xP0OtZExQ1+PJ8VvegLFkIqrCF rgFLG6YGHrZFO1DFbkIT3AwfvAngPvU6mtiZcxRoUz5czq4KMDey+6J3GZfLAq+b43wx E6zNUgOM40uiE0h1oNPkyGB2LHm3WFDAFMZkO4e0TuSYlWl4gRNXcXhXGfYKM27JZZo0 +t9A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=v9vcpKP2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t17-20020a639551000000b004fb171df693si1932528pgn.352.2023.05.09.10.38.28; Tue, 09 May 2023 10:38:40 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=v9vcpKP2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235540AbjEIRBJ (ORCPT + 99 others); Tue, 9 May 2023 13:01:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41556 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234562AbjEIRAH (ORCPT ); Tue, 9 May 2023 13:00:07 -0400 Received: from out-19.mta1.migadu.com (out-19.mta1.migadu.com [95.215.58.19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5962768B for ; Tue, 9 May 2023 09:57:59 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651460; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NRkDg8/QDL78Ozn75JlJivS7ptmCxyI7V8VfiBBetMw=; b=v9vcpKP2I5oVsGiASi9Z0yhNlg6/OYU386b+4kL3mJUwu9isTUH1JIw2lUNkRwA5UwTzmh dP5UkDVzix9yn6X3BvFL/2jYJQDuPcJHnH652qQP87dseSaZP2loDaloabniLSAIIVOiqr mJsb3JdfLSw+HpSncpkM+6GoKL/zt0E= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet , Christopher James Halse Rogers Subject: [PATCH 30/32] lib: Export errname Date: Tue, 9 May 2023 12:56:55 -0400 Message-Id: <20230509165657.1735798-31-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765439093609832288?= X-GMAIL-MSGID: =?utf-8?q?1765439093609832288?= The bcachefs module now wants this and it seems sensible. Signed-off-by: Christopher James Halse Rogers Signed-off-by: Kent Overstreet --- lib/errname.c | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/errname.c b/lib/errname.c index 67739b174a..dd1b998552 100644 --- a/lib/errname.c +++ b/lib/errname.c @@ -228,3 +228,4 @@ const char *errname(int err) return err > 0 ? name + 1 : name; } +EXPORT_SYMBOL(errname); From patchwork Tue May 9 16:56:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91702 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3065404vqo; Tue, 9 May 2023 10:51:28 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ778/fKYphKkmD7ywUaOUWqdzUDvTjoWrsoB9pP4mOU2BSKEZvr/lnD5Qr/CwI6zlkBN19B X-Received: by 2002:a05:6a20:3d94:b0:d9:6660:8746 with SMTP id s20-20020a056a203d9400b000d966608746mr19175913pzi.18.1683654688216; Tue, 09 May 2023 10:51:28 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683654688; cv=none; d=google.com; s=arc-20160816; b=Oir+ah0rEYyjPBJsKjxp2dhaKQ0IBudG401x837/Nre1iAb0VtM/MOjA0OaZssexl8 zAvp9g6F/RU0tkcYV2Uxc+k66M+7S3oewMYqxiwObvkKN9HOTppJshiuswLOKqDeR3/o VlVPEOV8TejiBNvzJAPGmRxCoxJU1aPl4rk2Qvr4h59smJkMAauXkkXCi9dHyIwJMdWA ikz835msgRG622Qub3mU3lz0GI7/Z1veYw1rkfTgsKdEgyWR7I5LopTpRS0aqLNxnYcV 1DitGwYLpzz0u9ayTkKpXgS7JUB5WNSNsWGTZvHu4Zs6SFcgt4V5qHezbtf9eunxfNBP mFxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=I3IAVGyB/8A3XrcmgOov/HZ/oVTBD4Irg0eSJUzhLg0=; b=skih0k9rxf6dwKg5yCI55ZQm7hFKRt0WM2pYOYNUsrp4bCmR7k5dThqJdq9MFm/LNv q9NnnAjNHyP8zF5622kRIJJ+bDXW/eG4bWd5UjqAyKD6GE+Boha24BlEp7lBhuqdmbKq 0r2LCm9xnb+CsCyh+jP5edFLwtZ1oohCMIqkWgcOhEAJAQqaLlKojVt1poAP0ce4sTEX 8xENKOps2fAawav3Ywui1jKNiSIb5tMSgYS+YiyitqIIAitEZ7Me5igAxyJYChOvzz0O 28yQUHcQXb0UbzpQ/3ngJShNcoVLrHo8oUrE6w9Fve9oD/NovyFluKfpzBuscYX6oRSU Fy7Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Hk+jz+6u; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 14-20020a63184e000000b00519f8ebcc5esi2022412pgy.121.2023.05.09.10.51.13; Tue, 09 May 2023 10:51:28 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b=Hk+jz+6u; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235040AbjEIRBN (ORCPT + 99 others); Tue, 9 May 2023 13:01:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234826AbjEIRAJ (ORCPT ); Tue, 9 May 2023 13:00:09 -0400 Received: from out-27.mta1.migadu.com (out-27.mta1.migadu.com [95.215.58.27]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 06C2A6A6E for ; Tue, 9 May 2023 09:58:01 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651460; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I3IAVGyB/8A3XrcmgOov/HZ/oVTBD4Irg0eSJUzhLg0=; b=Hk+jz+6u/vONyMe0EiexHJsx9U/I3cL9tyjBDqO+gXS9IdFRbCNXBolMrC5tfKj+zoQzGF Jnb0bwuPUOc4kTRGI33xo/pfeKYkK8QKHshuWvDs1jt2rEdNnnJAwocuqTkcdrREcX+0Dl WpTR0b54NqG/KaC1kLnIEnZn6C6GJNI= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Daniel Hill , Kent Overstreet Subject: [PATCH 31/32] lib: add mean and variance module. Date: Tue, 9 May 2023 12:56:56 -0400 Message-Id: <20230509165657.1735798-32-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765439898275233885?= X-GMAIL-MSGID: =?utf-8?q?1765439898275233885?= From: Daniel Hill This module provides a fast 64bit implementation of basic statistics functions, including mean, variance and standard deviation in both weighted and unweighted variants, the unweighted variant has a 32bit limitation per sample to prevent overflow when squaring. Signed-off-by: Daniel Hill Signed-off-by: Kent Overstreet --- MAINTAINERS | 9 ++ include/linux/mean_and_variance.h | 219 ++++++++++++++++++++++++++++++ lib/Kconfig.debug | 9 ++ lib/math/Kconfig | 3 + lib/math/Makefile | 2 + lib/math/mean_and_variance.c | 136 +++++++++++++++++++ lib/math/mean_and_variance_test.c | 155 +++++++++++++++++++++ 7 files changed, 533 insertions(+) create mode 100644 include/linux/mean_and_variance.h create mode 100644 lib/math/mean_and_variance.c create mode 100644 lib/math/mean_and_variance_test.c diff --git a/MAINTAINERS b/MAINTAINERS index c550f5909e..dbf3c33c31 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -12767,6 +12767,15 @@ F: Documentation/devicetree/bindings/net/ieee802154/mcr20a.txt F: drivers/net/ieee802154/mcr20a.c F: drivers/net/ieee802154/mcr20a.h +MEAN AND VARIANCE LIBRARY +M: Daniel B. Hill +M: Kent Overstreet +S: Maintained +T: git https://github.com/YellowOnion/linux/ +F: include/linux/mean_and_variance.h +F: lib/math/mean_and_variance.c +F: lib/math/mean_and_variance_test.c + MEASUREMENT COMPUTING CIO-DAC IIO DRIVER M: William Breathitt Gray L: linux-iio@vger.kernel.org diff --git a/include/linux/mean_and_variance.h b/include/linux/mean_and_variance.h new file mode 100644 index 0000000000..89540628e8 --- /dev/null +++ b/include/linux/mean_and_variance.h @@ -0,0 +1,219 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef MEAN_AND_VARIANCE_H_ +#define MEAN_AND_VARIANCE_H_ + +#include +#include +#include + +#define SQRT_U64_MAX 4294967295ULL + + +#if defined(CONFIG_ARCH_SUPPORTS_INT128) && defined(__SIZEOF_INT128__) + +typedef unsigned __int128 u128; + +static inline u128 u64_to_u128(u64 a) +{ + return (u128)a; +} + +static inline u64 u128_to_u64(u128 a) +{ + return (u64)a; +} + +static inline u64 u128_shr64_to_u64(u128 a) +{ + return (u64)(a >> 64); +} + +static inline u128 u128_add(u128 a, u128 b) +{ + return a + b; +} + +static inline u128 u128_sub(u128 a, u128 b) +{ + return a - b; +} + +static inline u128 u128_shl(u128 i, s8 shift) +{ + return i << shift; +} + +static inline u128 u128_shl64_add(u64 a, u64 b) +{ + return ((u128)a << 64) + b; +} + +static inline u128 u128_square(u64 i) +{ + return i*i; +} + +#else + +typedef struct { + u64 hi, lo; +} u128; + +static inline u128 u64_to_u128(u64 a) +{ + return (u128){ .lo = a }; +} + +static inline u64 u128_to_u64(u128 a) +{ + return a.lo; +} + +static inline u64 u128_shr64_to_u64(u128 a) +{ + return a.hi; +} + +static inline u128 u128_add(u128 a, u128 b) +{ + u128 c; + + c.lo = a.lo + b.lo; + c.hi = a.hi + b.hi + (c.lo < a.lo); + return c; +} + +static inline u128 u128_sub(u128 a, u128 b) +{ + u128 c; + + c.lo = a.lo - b.lo; + c.hi = a.hi - b.hi - (c.lo > a.lo); + return c; +} + +static inline u128 u128_shl(u128 i, s8 shift) +{ + u128 r; + + r.lo = i.lo << shift; + if (shift < 64) + r.hi = (i.hi << shift) | (i.lo >> (64 - shift)); + else { + r.hi = i.lo << (shift - 64); + r.lo = 0; + } + return r; +} + +static inline u128 u128_shl64_add(u64 a, u64 b) +{ + return u128_add(u128_shl(u64_to_u128(a), 64), u64_to_u128(b)); +} + +static inline u128 u128_square(u64 i) +{ + u128 r; + u64 h = i >> 32, l = i & (u64)U32_MAX; + + r = u128_shl(u64_to_u128(h*h), 64); + r = u128_add(r, u128_shl(u64_to_u128(h*l), 32)); + r = u128_add(r, u128_shl(u64_to_u128(l*h), 32)); + r = u128_add(r, u64_to_u128(l*l)); + return r; +} + +#endif + +static inline u128 u128_div(u128 n, u64 d) +{ + u128 r; + u64 rem; + u64 hi = u128_shr64_to_u64(n); + u64 lo = u128_to_u64(n); + u64 h = hi & ((u64)U32_MAX << 32); + u64 l = (hi & (u64)U32_MAX) << 32; + + r = u128_shl(u64_to_u128(div64_u64_rem(h, d, &rem)), 64); + r = u128_add(r, u128_shl(u64_to_u128(div64_u64_rem(l + (rem << 32), d, &rem)), 32)); + r = u128_add(r, u64_to_u128(div64_u64_rem(lo + (rem << 32), d, &rem))); + return r; +} + +struct mean_and_variance { + s64 n; + s64 sum; + u128 sum_squares; +}; + +/* expontentially weighted variant */ +struct mean_and_variance_weighted { + bool init; + u8 w; + s64 mean; + u64 variance; +}; + +/** + * fast_divpow2() - fast approximation for n / (1 << d) + * @n: numerator + * @d: the power of 2 denominator. + * + * note: this rounds towards 0. + */ +static inline s64 fast_divpow2(s64 n, u8 d) +{ + return (n + ((n < 0) ? ((1 << d) - 1) : 0)) >> d; +} + +static inline struct mean_and_variance +mean_and_variance_update_inlined(struct mean_and_variance s1, s64 v1) +{ + struct mean_and_variance s2; + u64 v2 = abs(v1); + + s2.n = s1.n + 1; + s2.sum = s1.sum + v1; + s2.sum_squares = u128_add(s1.sum_squares, u128_square(v2)); + return s2; +} + +static inline struct mean_and_variance_weighted +mean_and_variance_weighted_update_inlined(struct mean_and_variance_weighted s1, s64 x) +{ + struct mean_and_variance_weighted s2; + // previous weighted variance. + u64 var_w0 = s1.variance; + u8 w = s2.w = s1.w; + // new value weighted. + s64 x_w = x << w; + s64 diff_w = x_w - s1.mean; + s64 diff = fast_divpow2(diff_w, w); + // new mean weighted. + s64 u_w1 = s1.mean + diff; + + BUG_ON(w % 2 != 0); + + if (!s1.init) { + s2.mean = x_w; + s2.variance = 0; + } else { + s2.mean = u_w1; + s2.variance = ((var_w0 << w) - var_w0 + ((diff_w * (x_w - u_w1)) >> w)) >> w; + } + s2.init = true; + + return s2; +} + +struct mean_and_variance mean_and_variance_update(struct mean_and_variance s1, s64 v1); + s64 mean_and_variance_get_mean(struct mean_and_variance s); + u64 mean_and_variance_get_variance(struct mean_and_variance s1); + u32 mean_and_variance_get_stddev(struct mean_and_variance s); + +struct mean_and_variance_weighted mean_and_variance_weighted_update(struct mean_and_variance_weighted s1, s64 v1); + s64 mean_and_variance_weighted_get_mean(struct mean_and_variance_weighted s); + u64 mean_and_variance_weighted_get_variance(struct mean_and_variance_weighted s); + u32 mean_and_variance_weighted_get_stddev(struct mean_and_variance_weighted s); + +#endif // MEAN_AND_VAIRANCE_H_ diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug index 3dba7a9aff..9ca88e0027 100644 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -2101,6 +2101,15 @@ config CPUMASK_KUNIT_TEST If unsure, say N. +config MEAN_AND_VARIANCE_UNIT_TEST + tristate "mean_and_variance unit tests" if !KUNIT_ALL_TESTS + depends on KUNIT + select MEAN_AND_VARIANCE + default KUNIT_ALL_TESTS + help + This option enables the kunit tests for mean_and_variance module. + If unsure, say N. + config TEST_LIST_SORT tristate "Linked list sorting test" if !KUNIT_ALL_TESTS depends on KUNIT diff --git a/lib/math/Kconfig b/lib/math/Kconfig index 0634b428d0..7530ae9a35 100644 --- a/lib/math/Kconfig +++ b/lib/math/Kconfig @@ -15,3 +15,6 @@ config PRIME_NUMBERS config RATIONAL tristate + +config MEAN_AND_VARIANCE + tristate diff --git a/lib/math/Makefile b/lib/math/Makefile index bfac26ddfc..2ef1487e01 100644 --- a/lib/math/Makefile +++ b/lib/math/Makefile @@ -4,6 +4,8 @@ obj-y += div64.o gcd.o lcm.o int_pow.o int_sqrt.o reciprocal_div.o obj-$(CONFIG_CORDIC) += cordic.o obj-$(CONFIG_PRIME_NUMBERS) += prime_numbers.o obj-$(CONFIG_RATIONAL) += rational.o +obj-$(CONFIG_MEAN_AND_VARIANCE) += mean_and_variance.o obj-$(CONFIG_TEST_DIV64) += test_div64.o obj-$(CONFIG_RATIONAL_KUNIT_TEST) += rational-test.o +obj-$(CONFIG_MEAN_AND_VARIANCE_UNIT_TEST) += mean_and_variance_test.o diff --git a/lib/math/mean_and_variance.c b/lib/math/mean_and_variance.c new file mode 100644 index 0000000000..6e315d3a13 --- /dev/null +++ b/lib/math/mean_and_variance.c @@ -0,0 +1,136 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Functions for incremental mean and variance. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms of the GNU General Public License version 2 as published by + * the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for + * more details. + * + * Copyright © 2022 Daniel B. Hill + * + * Author: Daniel B. Hill + * + * Description: + * + * This is includes some incremental algorithms for mean and variance calculation + * + * Derived from the paper: https://fanf2.user.srcf.net/hermes/doc/antiforgery/stats.pdf + * + * Create a struct and if it's the weighted variant set the w field (weight = 2^k). + * + * Use mean_and_variance[_weighted]_update() on the struct to update it's state. + * + * Use the mean_and_variance[_weighted]_get_* functions to calculate the mean and variance, some computation + * is deferred to these functions for performance reasons. + * + * see lib/math/mean_and_variance_test.c for examples of usage. + * + * DO NOT access the mean and variance fields of the weighted variants directly. + * DO NOT change the weight after calling update. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +/** + * mean_and_variance_update() - update a mean_and_variance struct @s1 with a new sample @v1 + * and return it. + * @s1: the mean_and_variance to update. + * @v1: the new sample. + * + * see linked pdf equation 12. + */ +struct mean_and_variance mean_and_variance_update(struct mean_and_variance s1, s64 v1) +{ + return mean_and_variance_update_inlined(s1, v1); +} +EXPORT_SYMBOL_GPL(mean_and_variance_update); + +/** + * mean_and_variance_get_mean() - get mean from @s + */ +s64 mean_and_variance_get_mean(struct mean_and_variance s) +{ + return div64_u64(s.sum, s.n); +} +EXPORT_SYMBOL_GPL(mean_and_variance_get_mean); + +/** + * mean_and_variance_get_variance() - get variance from @s1 + * + * see linked pdf equation 12. + */ +u64 mean_and_variance_get_variance(struct mean_and_variance s1) +{ + u128 s2 = u128_div(s1.sum_squares, s1.n); + u64 s3 = abs(mean_and_variance_get_mean(s1)); + + return u128_to_u64(u128_sub(s2, u128_square(s3))); +} +EXPORT_SYMBOL_GPL(mean_and_variance_get_variance); + +/** + * mean_and_variance_get_stddev() - get standard deviation from @s + */ +u32 mean_and_variance_get_stddev(struct mean_and_variance s) +{ + return int_sqrt64(mean_and_variance_get_variance(s)); +} +EXPORT_SYMBOL_GPL(mean_and_variance_get_stddev); + +/** + * mean_and_variance_weighted_update() - exponentially weighted variant of mean_and_variance_update() + * @s1: .. + * @s2: .. + * + * see linked pdf: function derived from equations 140-143 where alpha = 2^w. + * values are stored bitshifted for performance and added precision. + */ +struct mean_and_variance_weighted mean_and_variance_weighted_update(struct mean_and_variance_weighted s1, + s64 x) +{ + return mean_and_variance_weighted_update_inlined(s1, x); +} +EXPORT_SYMBOL_GPL(mean_and_variance_weighted_update); + +/** + * mean_and_variance_weighted_get_mean() - get mean from @s + */ +s64 mean_and_variance_weighted_get_mean(struct mean_and_variance_weighted s) +{ + return fast_divpow2(s.mean, s.w); +} +EXPORT_SYMBOL_GPL(mean_and_variance_weighted_get_mean); + +/** + * mean_and_variance_weighted_get_variance() -- get variance from @s + */ +u64 mean_and_variance_weighted_get_variance(struct mean_and_variance_weighted s) +{ + // always positive don't need fast divpow2 + return s.variance >> s.w; +} +EXPORT_SYMBOL_GPL(mean_and_variance_weighted_get_variance); + +/** + * mean_and_variance_weighted_get_stddev() - get standard deviation from @s + */ +u32 mean_and_variance_weighted_get_stddev(struct mean_and_variance_weighted s) +{ + return int_sqrt64(mean_and_variance_weighted_get_variance(s)); +} +EXPORT_SYMBOL_GPL(mean_and_variance_weighted_get_stddev); + +MODULE_AUTHOR("Daniel B. Hill"); +MODULE_LICENSE("GPL"); diff --git a/lib/math/mean_and_variance_test.c b/lib/math/mean_and_variance_test.c new file mode 100644 index 0000000000..79a96d7307 --- /dev/null +++ b/lib/math/mean_and_variance_test.c @@ -0,0 +1,155 @@ +// SPDX-License-Identifier: GPL-2.0 +#include +#include + +#define MAX_SQR (SQRT_U64_MAX*SQRT_U64_MAX) + +static void mean_and_variance_basic_test(struct kunit *test) +{ + struct mean_and_variance s = {}; + + s = mean_and_variance_update(s, 2); + s = mean_and_variance_update(s, 2); + + KUNIT_EXPECT_EQ(test, mean_and_variance_get_mean(s), 2); + KUNIT_EXPECT_EQ(test, mean_and_variance_get_variance(s), 0); + KUNIT_EXPECT_EQ(test, s.n, 2); + + s = mean_and_variance_update(s, 4); + s = mean_and_variance_update(s, 4); + + KUNIT_EXPECT_EQ(test, mean_and_variance_get_mean(s), 3); + KUNIT_EXPECT_EQ(test, mean_and_variance_get_variance(s), 1); + KUNIT_EXPECT_EQ(test, s.n, 4); +} + +/* + * Test values computed using a spreadsheet from the psuedocode at the bottom: + * https://fanf2.user.srcf.net/hermes/doc/antiforgery/stats.pdf + */ + +static void mean_and_variance_weighted_test(struct kunit *test) +{ + struct mean_and_variance_weighted s = {}; + + s.w = 2; + + s = mean_and_variance_weighted_update(s, 10); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_mean(s), 10); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_variance(s), 0); + + s = mean_and_variance_weighted_update(s, 20); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_mean(s), 12); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_variance(s), 18); + + s = mean_and_variance_weighted_update(s, 30); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_mean(s), 16); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_variance(s), 72); + + s = (struct mean_and_variance_weighted){}; + s.w = 2; + + s = mean_and_variance_weighted_update(s, -10); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_mean(s), -10); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_variance(s), 0); + + s = mean_and_variance_weighted_update(s, -20); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_mean(s), -12); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_variance(s), 18); + + s = mean_and_variance_weighted_update(s, -30); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_mean(s), -16); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_variance(s), 72); + +} + +static void mean_and_variance_weighted_advanced_test(struct kunit *test) +{ + struct mean_and_variance_weighted s = {}; + s64 i; + + s.w = 8; + for (i = 10; i <= 100; i += 10) + s = mean_and_variance_weighted_update(s, i); + + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_mean(s), 11); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_variance(s), 107); + + s = (struct mean_and_variance_weighted){}; + + s.w = 8; + for (i = -10; i >= -100; i -= 10) + s = mean_and_variance_weighted_update(s, i); + + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_mean(s), -11); + KUNIT_EXPECT_EQ(test, mean_and_variance_weighted_get_variance(s), 107); + +} + +static void mean_and_variance_fast_divpow2(struct kunit *test) +{ + s64 i; + u8 d; + + for (i = 0; i < 100; i++) { + d = 0; + KUNIT_EXPECT_EQ(test, fast_divpow2(i, d), div_u64(i, 1LLU << d)); + KUNIT_EXPECT_EQ(test, abs(fast_divpow2(-i, d)), div_u64(i, 1LLU << d)); + for (d = 1; d < 32; d++) { + KUNIT_EXPECT_EQ_MSG(test, abs(fast_divpow2(i, d)), + div_u64(i, 1 << d), "%lld %u", i, d); + KUNIT_EXPECT_EQ_MSG(test, abs(fast_divpow2(-i, d)), + div_u64(i, 1 << d), "%lld %u", -i, d); + } + } +} + +static void mean_and_variance_u128_basic_test(struct kunit *test) +{ + u128 a = u128_shl64_add(0, U64_MAX); + u128 a1 = u128_shl64_add(0, 1); + u128 b = u128_shl64_add(1, 0); + u128 c = u128_shl64_add(0, 1LLU << 63); + u128 c2 = u128_shl64_add(U64_MAX, U64_MAX); + + KUNIT_EXPECT_EQ(test, u128_shr64_to_u64(u128_add(a, a1)), 1); + KUNIT_EXPECT_EQ(test, u128_to_u64(u128_add(a, a1)), 0); + KUNIT_EXPECT_EQ(test, u128_shr64_to_u64(u128_add(a1, a)), 1); + KUNIT_EXPECT_EQ(test, u128_to_u64(u128_add(a1, a)), 0); + + KUNIT_EXPECT_EQ(test, u128_to_u64(u128_sub(b, a1)), U64_MAX); + KUNIT_EXPECT_EQ(test, u128_shr64_to_u64(u128_sub(b, a1)), 0); + + KUNIT_EXPECT_EQ(test, u128_shr64_to_u64(u128_shl(c, 1)), 1); + KUNIT_EXPECT_EQ(test, u128_to_u64(u128_shl(c, 1)), 0); + + KUNIT_EXPECT_EQ(test, u128_shr64_to_u64(u128_square(U64_MAX)), U64_MAX - 1); + KUNIT_EXPECT_EQ(test, u128_to_u64(u128_square(U64_MAX)), 1); + + KUNIT_EXPECT_EQ(test, u128_to_u64(u128_div(b, 2)), 1LLU << 63); + + KUNIT_EXPECT_EQ(test, u128_shr64_to_u64(u128_div(c2, 2)), U64_MAX >> 1); + KUNIT_EXPECT_EQ(test, u128_to_u64(u128_div(c2, 2)), U64_MAX); + + KUNIT_EXPECT_EQ(test, u128_shr64_to_u64(u128_div(u128_shl(u64_to_u128(U64_MAX), 32), 2)), U32_MAX >> 1); + KUNIT_EXPECT_EQ(test, u128_to_u64(u128_div(u128_shl(u64_to_u128(U64_MAX), 32), 2)), U64_MAX << 31); +} + +static struct kunit_case mean_and_variance_test_cases[] = { + KUNIT_CASE(mean_and_variance_fast_divpow2), + KUNIT_CASE(mean_and_variance_u128_basic_test), + KUNIT_CASE(mean_and_variance_basic_test), + KUNIT_CASE(mean_and_variance_weighted_test), + KUNIT_CASE(mean_and_variance_weighted_advanced_test), + {} +}; + +static struct kunit_suite mean_and_variance_test_suite = { +.name = "mean and variance tests", +.test_cases = mean_and_variance_test_cases +}; + +kunit_test_suite(mean_and_variance_test_suite); + +MODULE_AUTHOR("Daniel B. Hill"); +MODULE_LICENSE("GPL"); From patchwork Tue May 9 16:56:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kent Overstreet X-Patchwork-Id: 91701 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3059206vqo; Tue, 9 May 2023 10:39:56 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4vDt0ZTjnaihFXoOgb4dK+LVOfwcjN2B6n38JHpV+SHjlHntlUAMbS/Wx8LznjmfRPM/OS X-Received: by 2002:a17:902:ec83:b0:1ac:4d01:dff8 with SMTP id x3-20020a170902ec8300b001ac4d01dff8mr17894981plg.45.1683653996640; Tue, 09 May 2023 10:39:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683653996; cv=none; d=google.com; s=arc-20160816; b=j8ZvNZ/j7uNUxolNf0X/8fjikeB+ZsBr+rkVjLQeSqtCEBPEfpZG0/vMPu99pnHm6I Jb6odrqUeIioA9iLyMXXJmb05JfisE0Fu5uN1cfqmeFV4b02FOrYdiTQh8NT+gmdbjgd T0ZL27oTILgUvYF+K1cgwESkOKgqqdrOw3OjYu85s2IkvZdPbgWnbYmplU1XJ3XccHLi CKHWVMKK60QkToxPqfgBHKAK7nS1CNAwVR1zkqNJjyuWhh/2U2NwbtZ1g9AqBhv9lQlW Ks7g/Lit5QswRdbNjx2vUr5HI/f68kXdR2AyX9fBPMaYfOXTp1NL4B1OOU6GPBkZ7zJS e42A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xcHhLx9X66ZiP+hkbacjwT2I7KoTF7hsRShtmv0sqzA=; b=qRoB1nRAJuAjNWgZBZHFuapLchIStcT3U5faoAozjmqxwBdf+wbtuv21gpgp4pdk4L q8DFUqJA2S0YaJ7ik/PXusacPX+Z6U+O85/ZUVqo+YFTBa0x/iD80FFmZ1q5ozdmgsdo uhpvQZHKPnz5o4ErAp77n12OGYw0LRCRZTPdCLeS4RwesQxm85wumlr1mLG8QagV2ONQ jfogk4gnnKzsbdbZV6asH9narkbyL4mduywk1BI9zxG41LAukwiyQkRZ5zRVW4e9uZmQ UiWq/Q1WuMhB1Swlddaziqp+sAy7xJkx8qH8xiQl5MokLijKAnEC7pZRQkTbZLnQsV0H 4Igg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="biwaHuM/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o16-20020a170902779000b001ac82e60516si1857177pll.323.2023.05.09.10.39.44; Tue, 09 May 2023 10:39:56 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@linux.dev header.s=key1 header.b="biwaHuM/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=linux.dev Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235314AbjEIRAs (ORCPT + 99 others); Tue, 9 May 2023 13:00:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235303AbjEIQ75 (ORCPT ); Tue, 9 May 2023 12:59:57 -0400 Received: from out-25.mta1.migadu.com (out-25.mta1.migadu.com [IPv6:2001:41d0:203:375::19]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A84366E99 for ; Tue, 9 May 2023 09:57:51 -0700 (PDT) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1683651463; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xcHhLx9X66ZiP+hkbacjwT2I7KoTF7hsRShtmv0sqzA=; b=biwaHuM/eMP7HCr+Axj3/+gSQknHQmfD/m+5P9FSCxSV1Qo7DgDdKxnIfUnw7Z52iep6WD 5RtlZ21/HtbKg2eI2eaHqsksKozLugNUlmXy/eh7CPd/+wwd2fDjXTK5MrowYJjuWCoxcf A3j0RF4EpAMOrAqGuKGPzDWm6rjV1QI= From: Kent Overstreet To: linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-bcachefs@vger.kernel.org Cc: Kent Overstreet Subject: [PATCH 32/32] MAINTAINERS: Add entry for bcachefs Date: Tue, 9 May 2023 12:56:57 -0400 Message-Id: <20230509165657.1735798-33-kent.overstreet@linux.dev> In-Reply-To: <20230509165657.1735798-1-kent.overstreet@linux.dev> References: <20230509165657.1735798-1-kent.overstreet@linux.dev> MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765439173186904090?= X-GMAIL-MSGID: =?utf-8?q?1765439173186904090?= bcachefs is a new copy-on-write filesystem; add a MAINTAINERS entry for it. Signed-off-by: Kent Overstreet --- MAINTAINERS | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index dbf3c33c31..0ac2b432f0 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3509,6 +3509,13 @@ W: http://bcache.evilpiepirate.org C: irc://irc.oftc.net/bcache F: drivers/md/bcache/ +BCACHEFS: +M: Kent Overstreet +L: linux-bcachefs@vger.kernel.org +S: Supported +C: irc://irc.oftc.net/bcache +F: fs/bcachefs/ + BDISP ST MEDIA DRIVER M: Fabien Dessenne L: linux-media@vger.kernel.org