From patchwork Thu Feb 23 16:43:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Breno Leitao X-Patchwork-Id: 61034 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp418781wrd; Thu, 23 Feb 2023 09:02:07 -0800 (PST) X-Google-Smtp-Source: AK7set9z6+V28HykaxbUcw4u5yjaINJ8+vk4iuytNov87n94/Ajy+UcdoR23nEMZ0ZshecGOyye9 X-Received: by 2002:a05:6a20:12cd:b0:cc:416f:2dd0 with SMTP id v13-20020a056a2012cd00b000cc416f2dd0mr1184917pzg.3.1677171726871; Thu, 23 Feb 2023 09:02:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677171726; cv=none; d=google.com; s=arc-20160816; b=vPQjo6LJlToJWx9KzJcZrJu/0JzEgI1cP+B2JRaHfnyCCNLNE0JgLttJr4JpMiNLFk Ej/wAYdLSRnK+NeoSeBiBA8A/ec1cSA6+K7Yqjwp9vL9w+XPnfU1HCNgAiaJhj2F3NK6 dcN5muL9rwtZ3J94MUukF91HszMgAX6mDfrdGWLnDB32Mj42shDqtU9VUkYE9cnRD76Q hHMi/K0A1iBc1OFT0z+Y/YHIFLQueED3KcpIOS2F/Cn/0n6l4F+unr3Q8quHFyKtF9al YNbS8X5X85lSc/0VD/BUMoqSzNBiAHlSqq3qL7JKXgq1u12o6EhLaMQAjseKQidhP+FD m1+Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=+k0d3ITubH2glaeJAp2cfaEZXtQloEAVR31lcQ0bITk=; b=IxUXmkyJj6TYjnWfgcD2kAmpV17No+UV5lHIS6KmQQEUp/WNWMbESKxTUCRz4lrDeJ h2QjT2YInY6EtlugfTw4V7iBzXxs1qENDdujwcMoPN0QMJ2IxTkSat23f5ANXL+w1azA q4CklWgMz24EZlb8T6KFVX+4GvNUhLCEmF932VdZrQ9xYMc4Rk/WenvmHQlY8Pn0Je3O YxYdrVL75GVaBh0ViABEncHEoacT+fGwx7kXjTJcwDwSMvomvnv8YBPubLTjkYfA6Nnm BPNWJ7cuKqiTukIsjebTtlxoQiTtm/43uNk1oZxhlmfcnMmzagDOJddce8BwFMqeklG4 Jdfg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u9-20020a63df09000000b004fc1d8eaf07si5894520pgg.406.2023.02.23.09.01.41; Thu, 23 Feb 2023 09:02:06 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234681AbjBWQoZ (ORCPT + 99 others); Thu, 23 Feb 2023 11:44:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234026AbjBWQoX (ORCPT ); Thu, 23 Feb 2023 11:44:23 -0500 Received: from mail-wr1-f42.google.com (mail-wr1-f42.google.com [209.85.221.42]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EBFC639CD4; Thu, 23 Feb 2023 08:44:21 -0800 (PST) Received: by mail-wr1-f42.google.com with SMTP id j2so10990011wrh.9; Thu, 23 Feb 2023 08:44:21 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+k0d3ITubH2glaeJAp2cfaEZXtQloEAVR31lcQ0bITk=; b=QlBPgV3u59OIN4TN6/CCgtQNRFPZhpiFaAhJ2/VczU9uZlblr1x7kJncMuRYoMWAUz e6dadVz/ncEOn/QbV0oQRQN7pTjaEoX7h2I5p3gM6OQfMC8iVQlpElqA4l4q39xrqtEi QLDBt6/6N+741jKJCe+Svf69qt6ilGbWp2BuYGdl3GKUSs5bl50SyEk/HsyIDNXV7kcD 4csb1Kagp60HLc29eWJ/CqEARf+6q+tVdecRt9hoM0rxz3fbuonNFOKqXQDwUga6izN8 HhVOGrKLbPB+JuP78CaHE7pWvOWT0f8j1rsdBt5GZ5j4tjk0yQ45K90J5tZvNfML4nge vqXg== X-Gm-Message-State: AO0yUKUVwyhXgcrWy1zvMTW2Mc1Q6MN6gcrebfiJmbk8gWoWuxRXdKbZ cJEbx/Uq2eFyX9i2jn38jbI= X-Received: by 2002:a5d:51ca:0:b0:2c7:1159:ea43 with SMTP id n10-20020a5d51ca000000b002c71159ea43mr2492370wrv.51.1677170660333; Thu, 23 Feb 2023 08:44:20 -0800 (PST) Received: from localhost (fwdproxy-cln-008.fbsv.net. [2a03:2880:31ff:8::face:b00c]) by smtp.gmail.com with ESMTPSA id v26-20020a5d591a000000b002c573cff730sm7700970wrd.68.2023.02.23.08.44.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 08:44:19 -0800 (PST) From: Breno Leitao To: axboe@kernel.dk, asml.silence@gmail.com, io-uring@vger.kernel.org Cc: linux-kernel@vger.kernel.org, gustavold@meta.com, leit@meta.com, kasan-dev@googlegroups.com Subject: [PATCH v3 1/2] io_uring: Move from hlist to io_wq_work_node Date: Thu, 23 Feb 2023 08:43:52 -0800 Message-Id: <20230223164353.2839177-2-leitao@debian.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230223164353.2839177-1-leitao@debian.org> References: <20230223164353.2839177-1-leitao@debian.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758642020785591678?= X-GMAIL-MSGID: =?utf-8?q?1758642020785591678?= Having cache entries linked using the hlist format brings no benefit, and also requires an unnecessary extra pointer address per cache entry. Use the internal io_wq_work_node single-linked list for the internal alloc caches (async_msghdr and async_poll) This is required to be able to use KASAN on cache entries, since we do not need to touch unused (and poisoned) cache entries when adding more entries to the list. Suggested-by: Pavel Begunkov Signed-off-by: Breno Leitao --- include/linux/io_uring_types.h | 2 +- io_uring/alloc_cache.h | 24 +++++++++++++----------- 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 0efe4d784358..efa66b6c32c9 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -188,7 +188,7 @@ struct io_ev_fd { }; struct io_alloc_cache { - struct hlist_head list; + struct io_wq_work_node list; unsigned int nr_cached; }; diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index 729793ae9712..301855e94309 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -7,7 +7,7 @@ #define IO_ALLOC_CACHE_MAX 512 struct io_cache_entry { - struct hlist_node node; + struct io_wq_work_node node; }; static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, @@ -15,7 +15,7 @@ static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, { if (cache->nr_cached < IO_ALLOC_CACHE_MAX) { cache->nr_cached++; - hlist_add_head(&entry->node, &cache->list); + wq_stack_add_head(&entry->node, &cache->list); return true; } return false; @@ -23,11 +23,12 @@ static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, static inline struct io_cache_entry *io_alloc_cache_get(struct io_alloc_cache *cache) { - if (!hlist_empty(&cache->list)) { - struct hlist_node *node = cache->list.first; + if (cache->list.next) { + struct io_cache_entry *entry; - hlist_del(node); - return container_of(node, struct io_cache_entry, node); + entry = container_of(cache->list.next, struct io_cache_entry, node); + cache->list.next = cache->list.next->next; + return entry; } return NULL; @@ -35,18 +36,19 @@ static inline struct io_cache_entry *io_alloc_cache_get(struct io_alloc_cache *c static inline void io_alloc_cache_init(struct io_alloc_cache *cache) { - INIT_HLIST_HEAD(&cache->list); + cache->list.next = NULL; cache->nr_cached = 0; } static inline void io_alloc_cache_free(struct io_alloc_cache *cache, void (*free)(struct io_cache_entry *)) { - while (!hlist_empty(&cache->list)) { - struct hlist_node *node = cache->list.first; + while (1) { + struct io_cache_entry *entry = io_alloc_cache_get(cache); - hlist_del(node); - free(container_of(node, struct io_cache_entry, node)); + if (!entry) + break; + free(entry); } cache->nr_cached = 0; } From patchwork Thu Feb 23 16:43:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Breno Leitao X-Patchwork-Id: 61035 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp419126wrd; Thu, 23 Feb 2023 09:02:32 -0800 (PST) X-Google-Smtp-Source: AK7set+V9ogHvJ0BVOWRX6hgNt9jgzKfDf5WlRz9BjDitRf8rLMob27jvvTQC3tfwQ5sueEr5143 X-Received: by 2002:a17:906:196:b0:8c5:3625:ffc5 with SMTP id 22-20020a170906019600b008c53625ffc5mr17308545ejb.7.1677171752049; Thu, 23 Feb 2023 09:02:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1677171752; cv=none; d=google.com; s=arc-20160816; b=P1j6AYL48H/01TzCaW/eUKr+yCOKTZV0/gSuKtnHbsnQQh+9Yuk801DRN0jPvL8hrU s9FjjOwyIL6GuX8K0w0cxzSawlpQRZyUjVdzq13Q3JItEIq/F9vOwoESiZr2SzJ0M3Wv enC31aG/RhTxNgqwjTlLAWtHhtXBD/UoTHLBEqST7i28RKHGSSwK5Im9UKciPVEAdOP+ VENFsYWucbb97mERogGelbdTZc6EMz3r/Zw91vFFEV+jW8nO/MHxZVD5cxefhtPSIkgX ac1ffe9Ep5wx8Q70llIKlWG1V6+wcf0qfIUB0msw2gS+OAY9oTknJ57LZL9n6/e9dFEy RT3A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=EXP5WLQ9Rub89Gkwnqh+Yqr8BCe99md1ePNpoibd3cQ=; b=jF6hIlZFU2OhnaUBeDk3Sib+5MWG9BddzrJIXNtk3HT5pnKM4uXkb7Ge0xIaKb/ld3 PZwnWMpuiVrgl3ayG+sXI4sMC+/oAGcKY7EPlCAxmO7bHNegbjMNGddTMHLtl2R62oi6 QOcBfA3s++P8mQpDl11yTeqL8Ba1flOr2bph+Wn7062+vTe5dX/4FtUeNCwYsByoy+pM n8gz54mJVmMmeam59RzABcYIY9Is5pJJ9yj9ud12lfF+I6oRsQPI4WRNG01BZpCbH4t0 mp+lTNZpAk17Whh5GmlLhuVPbVP5NZ2gVYUNkG6Uij8DyMYkDtLFKK9bpyqim4YgcOpy F/fw== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vf4-20020a170907238400b008e66d041e32si4905925ejb.129.2023.02.23.09.02.01; Thu, 23 Feb 2023 09:02:32 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234723AbjBWQo1 (ORCPT + 99 others); Thu, 23 Feb 2023 11:44:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44654 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234589AbjBWQoX (ORCPT ); Thu, 23 Feb 2023 11:44:23 -0500 Received: from mail-wr1-f46.google.com (mail-wr1-f46.google.com [209.85.221.46]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D48783A0BA; Thu, 23 Feb 2023 08:44:22 -0800 (PST) Received: by mail-wr1-f46.google.com with SMTP id j2so10990037wrh.9; Thu, 23 Feb 2023 08:44:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EXP5WLQ9Rub89Gkwnqh+Yqr8BCe99md1ePNpoibd3cQ=; b=LYMm2oP4lSIzZSlbBMC560ETr2rafqjaUYA4Kj/BExQR2jbO8T9OCsIIu7MZXUrZeA I9izJvZIKLJpFOBQyoHFhFU9EKcVsDiwDSbV4wkKaapjsEPbn/1T46YhlhzGAAK4flOn GBXXUV5bksugkPeNltn04noWBn2F91A4azJEFFlk/8pkN0jWK1P5+MkSKStvsJnosSf/ h65gY0FfE66CfYFvx9/azTOL0JS8FrRSJyuALd71U7WOiJO6OesD+RmpVJA/d7zP2A/Y yZPQhgo2xQJ9MYeiByJW6perIdLBLBL1IQv+FokTHkul/ybvKg9ARppN4314kht2oBz+ WDuw== X-Gm-Message-State: AO0yUKUwdDDKwbc3bVSVKP56/VVkkf9mTlHfyyuSDqfTCoErkpNax2la 8Cu/mxp1tT3VaoFMZWKx8cw= X-Received: by 2002:a5d:410b:0:b0:2c5:8c56:42d3 with SMTP id l11-20020a5d410b000000b002c58c5642d3mr9678543wrp.23.1677170661338; Thu, 23 Feb 2023 08:44:21 -0800 (PST) Received: from localhost (fwdproxy-cln-026.fbsv.net. [2a03:2880:31ff:1a::face:b00c]) by smtp.gmail.com with ESMTPSA id x4-20020adfdd84000000b002c556a4f1casm10274330wrl.42.2023.02.23.08.44.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Feb 2023 08:44:21 -0800 (PST) From: Breno Leitao To: axboe@kernel.dk, asml.silence@gmail.com, io-uring@vger.kernel.org Cc: linux-kernel@vger.kernel.org, gustavold@meta.com, leit@meta.com, kasan-dev@googlegroups.com Subject: [PATCH v3 2/2] io_uring: Add KASAN support for alloc_caches Date: Thu, 23 Feb 2023 08:43:53 -0800 Message-Id: <20230223164353.2839177-3-leitao@debian.org> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230223164353.2839177-1-leitao@debian.org> References: <20230223164353.2839177-1-leitao@debian.org> MIME-Version: 1.0 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1758642047133160455?= X-GMAIL-MSGID: =?utf-8?q?1758642047133160455?= Add support for KASAN in the alloc_caches (apoll and netmsg_cache). Thus, if something touches the unused caches, it will raise a KASAN warning/exception. It poisons the object when the object is put to the cache, and unpoisons it when the object is gotten or freed. Signed-off-by: Breno Leitao Reviewed-by: Gabriel Krisman Bertazi --- include/linux/io_uring_types.h | 1 + io_uring/alloc_cache.h | 6 +++++- io_uring/io_uring.c | 4 ++-- io_uring/net.h | 5 ++++- 4 files changed, 12 insertions(+), 4 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index efa66b6c32c9..35ebcfb46047 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -190,6 +190,7 @@ struct io_ev_fd { struct io_alloc_cache { struct io_wq_work_node list; unsigned int nr_cached; + size_t elem_size; }; struct io_ring_ctx { diff --git a/io_uring/alloc_cache.h b/io_uring/alloc_cache.h index 301855e94309..3aba7b356320 100644 --- a/io_uring/alloc_cache.h +++ b/io_uring/alloc_cache.h @@ -16,6 +16,8 @@ static inline bool io_alloc_cache_put(struct io_alloc_cache *cache, if (cache->nr_cached < IO_ALLOC_CACHE_MAX) { cache->nr_cached++; wq_stack_add_head(&entry->node, &cache->list); + /* KASAN poisons object */ + kasan_slab_free_mempool(entry); return true; } return false; @@ -27,6 +29,7 @@ static inline struct io_cache_entry *io_alloc_cache_get(struct io_alloc_cache *c struct io_cache_entry *entry; entry = container_of(cache->list.next, struct io_cache_entry, node); + kasan_unpoison_range(entry, cache->elem_size); cache->list.next = cache->list.next->next; return entry; } @@ -34,10 +37,11 @@ static inline struct io_cache_entry *io_alloc_cache_get(struct io_alloc_cache *c return NULL; } -static inline void io_alloc_cache_init(struct io_alloc_cache *cache) +static inline void io_alloc_cache_init(struct io_alloc_cache *cache, size_t size) { cache->list.next = NULL; cache->nr_cached = 0; + cache->elem_size = size; } static inline void io_alloc_cache_free(struct io_alloc_cache *cache, diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 80b6204769e8..7a30a3e72fcc 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -309,8 +309,8 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) INIT_LIST_HEAD(&ctx->sqd_list); INIT_LIST_HEAD(&ctx->cq_overflow_list); INIT_LIST_HEAD(&ctx->io_buffers_cache); - io_alloc_cache_init(&ctx->apoll_cache); - io_alloc_cache_init(&ctx->netmsg_cache); + io_alloc_cache_init(&ctx->apoll_cache, sizeof(struct async_poll)); + io_alloc_cache_init(&ctx->netmsg_cache, sizeof(struct io_async_msghdr)); init_completion(&ctx->ref_comp); xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1); mutex_init(&ctx->uring_lock); diff --git a/io_uring/net.h b/io_uring/net.h index 5ffa11bf5d2e..191009979bcb 100644 --- a/io_uring/net.h +++ b/io_uring/net.h @@ -5,8 +5,8 @@ #include "alloc_cache.h" -#if defined(CONFIG_NET) struct io_async_msghdr { +#if defined(CONFIG_NET) union { struct iovec fast_iov[UIO_FASTIOV]; struct { @@ -22,8 +22,11 @@ struct io_async_msghdr { struct sockaddr __user *uaddr; struct msghdr msg; struct sockaddr_storage addr; +#endif }; +#if defined(CONFIG_NET) + struct io_async_connect { struct sockaddr_storage address; };