From patchwork Tue Nov 21 02:00:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shifeng Li X-Patchwork-Id: 167472 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2b07:b0:403:3b70:6f57 with SMTP id io7csp346647vqb; Mon, 20 Nov 2023 18:04:19 -0800 (PST) X-Google-Smtp-Source: AGHT+IGILo5ttNQkI/rjwKwQD5K4EHf2tRaXh0fHUSK6C3hs0NFgpU3j9wa4HVG38zl6MECUWJQ8 X-Received: by 2002:a17:902:dad1:b0:1c9:d948:33ea with SMTP id q17-20020a170902dad100b001c9d94833eamr9090863plx.21.1700532258979; Mon, 20 Nov 2023 18:04:18 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1700532258; cv=none; d=google.com; s=arc-20160816; b=wHCQp2eXl15lS3a+p1o92SQb2OUf3Xk26yTWrxxIuNc9z4x3Pq7d+6rnMqf41Lvz3J ydqAy+kz7o0H4WJiRwMkX8rSCt7wEGMU2qn/tE+x4VZJXfI303WSrderodG1/gzqCDch M93qZ/ozgWDv1/FSjBxlEZ28JDyFWufAyWOJITADlxAaIq2HUXUVCM2GAhcaiGXufFMZ 0M3NovaPaRZJeQ6OAPBOjEX18N9S9rnyocz/fQS+yD8Gic9K+VLzjuIxUFAxCqtTSSMg Ggra8pueNJZZCUbHhhnFsBxued834JqUJ9NNSUuvqzNYHEcxn8vdIA/lJxCotEeKXSWk TqVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=JGU6KFKruZOeFPVjR6IpK0hLwoBGgsH3T/tqFT/aOhU=; fh=3SStDU4fFdviccOo4SpYO5bKLhzU2qUTpT6bMS13JAQ=; b=BCWR0akd0eI5uLtFZ1/NOKhGVm9DxQQbwM8oiXenHxy2o78Mjg81jk8xYiBvvMt9bU ENnlZa94QBF6RC+Jtsci67anivFm5bHi9npYkx7+8km1MB02mdTYdJd5MxlU5NwUgjBB pkDOkFuL1/OrM4OwQ9G0fXZ8IOuwHBV+yM0TNQytxUWYiZh+VtABRJobU6LJwnKsjNwb I1DGMNCkwKiJp1dRgQwMhabwWw3EvOvaYlx+ZS+bqFo8813jzjF+xxL6x/lkw+dvNjW9 1iozGacmnJfWe+v9X36bvkXr5GqAlpK4a+k26/c9GaYfts6KQgIx4vinolml+OK/T/xA Gbew== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@126.com header.s=s110527 header.b=LGktvv6A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=126.com Received: from snail.vger.email (snail.vger.email. [23.128.96.37]) by mx.google.com with ESMTPS id j7-20020a170903024700b001c60ee79b6esi9714640plh.0.2023.11.20.18.04.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Nov 2023 18:04:18 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) client-ip=23.128.96.37; Authentication-Results: mx.google.com; dkim=pass header.i=@126.com header.s=s110527 header.b=LGktvv6A; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.37 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=126.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by snail.vger.email (Postfix) with ESMTP id 93EA3801C087; Mon, 20 Nov 2023 18:03:12 -0800 (PST) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.11 at snail.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233168AbjKUCDD (ORCPT + 27 others); Mon, 20 Nov 2023 21:03:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50932 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229679AbjKUCCw (ORCPT ); Mon, 20 Nov 2023 21:02:52 -0500 Received: from m15.mail.126.com (m15.mail.126.com [45.254.50.224]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 2923ECF; Mon, 20 Nov 2023 18:02:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=126.com; s=s110527; h=From:Subject:Date:Message-Id:MIME-Version; bh=JGU6K FKruZOeFPVjR6IpK0hLwoBGgsH3T/tqFT/aOhU=; b=LGktvv6AGKieT73AWCqGb dT1GP1D4TZ3yWgg/ZgH18DRqGn9NWEhQ4fLsK9w28fFPRRbQwJfB+WGrB3DNuIPT jAIrj0hJJRf155ZpRLbYvNQCpjiabFPPsKjuKioKzKrGrud7jql0MqTIvovFYPq/ L6UI47JWP840pROXz5hIro= Received: from ubuntu.localdomain (unknown [111.222.250.119]) by zwqz-smtp-mta-g0-0 (Coremail) with SMTP id _____wB3_9UnD1xltn6HAw--.26642S2; Tue, 21 Nov 2023 10:00:16 +0800 (CST) From: Shifeng Li To: saeedm@nvidia.com, leon@kernel.org, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, eli@mellanox.com, ogerlitz@mellanox.com, jackm@dev.mellanox.co.il, roland@purestorage.com Cc: netdev@vger.kernel.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, dinghui@sangfor.com.cn, Shifeng Li Subject: [PATCH] net/mlx5e: Fix a race in command alloc flow Date: Mon, 20 Nov 2023 18:00:04 -0800 Message-Id: <20231121020004.115815-1-lishifeng1992@126.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-CM-TRANSID: _____wB3_9UnD1xltn6HAw--.26642S2 X-Coremail-Antispam: 1Uf129KBjvJXoWxWrWkKw1kKFy5GF1UZryDAwb_yoWrtFWrpF W7W343AF4kGa1q9r40vF40v3W8A39Fg3srGF1I93Z3W3Z8A34kAa4DJFyjgryUuFW8tFy7 JFWDt3W8Ars3XF7anT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07U-zVbUUUUU= X-Originating-IP: [111.222.250.119] X-CM-SenderInfo: xolvxx5ihqwiqzzsqiyswou0bp/1tbi1xsur153c1R7WAABsh X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_ENVFROM_END_DIGIT, FREEMAIL_FROM,RCVD_IN_DNSWL_BLOCKED,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (snail.vger.email [0.0.0.0]); Mon, 20 Nov 2023 18:03:12 -0800 (PST) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1783137314042850733 X-GMAIL-MSGID: 1783137314042850733 Fix a cmd->ent use after free due to a race on command entry. Such race occurs when one of the commands releases its last refcount and frees its index and entry while another process running command flush flow takes refcount to this command entry. The process which handles commands flush may see this command as needed to be flushed if the other process allocated a ent->idx but didn't set ent to cmd->ent_arr in cmd_work_handler(). Fix it by moving the assignment of cmd->ent_arr into the spin lock. [70013.081955] BUG: KASAN: use-after-free in mlx5_cmd_trigger_completions+0x1e2/0x4c0 [mlx5_core] [70013.081967] Write of size 4 at addr ffff88880b1510b4 by task kworker/26:1/1433361 [70013.081968] [70013.081989] CPU: 26 PID: 1433361 Comm: kworker/26:1 Kdump: loaded Tainted: G OE 4.19.90-25.17.v2101.osc.sfc.6.10.0.0030.ky10.x86_64+debug #1 [70013.082001] Hardware name: SANGFOR 65N32-US/ASERVER-G-2605, BIOS SSSS5203 08/19/2020 [70013.082028] Workqueue: events aer_isr [70013.082053] Call Trace: [70013.082067] dump_stack+0x8b/0xbb [70013.082086] print_address_description+0x6a/0x270 [70013.082102] kasan_report+0x179/0x2c0 [70013.082133] ? mlx5_cmd_trigger_completions+0x1e2/0x4c0 [mlx5_core] [70013.082173] mlx5_cmd_trigger_completions+0x1e2/0x4c0 [mlx5_core] [70013.082213] ? mlx5_cmd_use_polling+0x20/0x20 [mlx5_core] [70013.082223] ? kmem_cache_free+0x1ad/0x1e0 [70013.082267] mlx5_cmd_flush+0x80/0x180 [mlx5_core] [70013.082304] mlx5_enter_error_state+0x106/0x1d0 [mlx5_core] [70013.082338] mlx5_try_fast_unload+0x2ea/0x4d0 [mlx5_core] [70013.082377] remove_one+0x200/0x2b0 [mlx5_core] [70013.082390] ? __pm_runtime_resume+0x58/0x70 [70013.082409] pci_device_remove+0xf3/0x280 [70013.082426] ? pcibios_free_irq+0x10/0x10 [70013.082439] device_release_driver_internal+0x1c3/0x470 [70013.082453] pci_stop_bus_device+0x109/0x160 [70013.082468] pci_stop_and_remove_bus_device+0xe/0x20 [70013.082485] pcie_do_fatal_recovery+0x167/0x550 [70013.082493] aer_isr+0x7d2/0x960 [70013.082510] ? aer_get_device_error_info+0x420/0x420 [70013.082526] ? __schedule+0x821/0x2040 [70013.082536] ? strscpy+0x85/0x180 [70013.082543] process_one_work+0x65f/0x12d0 [70013.082556] worker_thread+0x87/0xb50 [70013.082563] ? __kthread_parkme+0x82/0xf0 [70013.082569] ? process_one_work+0x12d0/0x12d0 [70013.082571] kthread+0x2e9/0x3a0 [70013.082579] ? kthread_create_worker_on_cpu+0xc0/0xc0 [70013.082592] ret_from_fork+0x1f/0x40 Fixes: e126ba97dba9 ("mlx5: Add driver for Mellanox Connect-IB adapters") Signed-off-by: Shifeng Li --- drivers/net/ethernet/mellanox/mlx5/core/cmd.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c index d3ca745d107d..1f9c09065249 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c @@ -115,15 +115,18 @@ static u8 alloc_token(struct mlx5_cmd *cmd) return token; } -static int cmd_alloc_index(struct mlx5_cmd *cmd) +static int cmd_alloc_index(struct mlx5_cmd *cmd, struct mlx5_cmd_work_ent *ent) { unsigned long flags; int ret; spin_lock_irqsave(&cmd->alloc_lock, flags); ret = find_first_bit(&cmd->bitmask, cmd->max_reg_cmds); - if (ret < cmd->max_reg_cmds) + if (ret < cmd->max_reg_cmds) { clear_bit(ret, &cmd->bitmask); + ent->idx = ret; + cmd->ent_arr[ent->idx] = ent; + } spin_unlock_irqrestore(&cmd->alloc_lock, flags); return ret < cmd->max_reg_cmds ? ret : -ENOMEM; @@ -957,7 +960,7 @@ static void cmd_work_handler(struct work_struct *work) sem = ent->page_queue ? &cmd->pages_sem : &cmd->sem; down(sem); if (!ent->page_queue) { - alloc_ret = cmd_alloc_index(cmd); + alloc_ret = cmd_alloc_index(cmd, ent); if (alloc_ret < 0) { mlx5_core_err_rl(dev, "failed to allocate command entry\n"); if (ent->callback) { @@ -972,15 +975,14 @@ static void cmd_work_handler(struct work_struct *work) up(sem); return; } - ent->idx = alloc_ret; } else { ent->idx = cmd->max_reg_cmds; spin_lock_irqsave(&cmd->alloc_lock, flags); clear_bit(ent->idx, &cmd->bitmask); + cmd->ent_arr[ent->idx] = ent; spin_unlock_irqrestore(&cmd->alloc_lock, flags); } - cmd->ent_arr[ent->idx] = ent; lay = get_inst(cmd, ent->idx); ent->lay = lay; memset(lay, 0, sizeof(*lay));