Message ID | 20221019083708.27138-1-nstange@suse.de |
---|---|
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:4ac7:0:0:0:0:0 with SMTP id y7csp201461wrs; Wed, 19 Oct 2022 01:38:05 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5Rq8f8chkvT6rw5FsPS1FqdQmbAZWswiePb6Fw9cNszN2rR5mgzBejQpgKVcrqyG4Y5kDQ X-Received: by 2002:a17:902:f789:b0:17f:8cb6:7da3 with SMTP id q9-20020a170902f78900b0017f8cb67da3mr7196848pln.167.1666168685736; Wed, 19 Oct 2022 01:38:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1666168685; cv=none; d=google.com; s=arc-20160816; b=DRrVc13wRUTVAVX+7/Om4jsHDCXZ8G2QKAz7PhSv7mHil9lRiQsgSYwJhBDUow2UfM XrAHFSFaJV9w2sVVXIs7DLfrf1Ab2obKK8DVApeHBP4LdhsQFCfHt0WL9SCGJTd/BYWs lQFW6/iYFOTZyldmn+yaKwx0/xMO1E6BEUqk9oh0jf8c6jl3y4EbVAx6eYZgNjdyz3Ng erPis4w6e9GkvPKmQtQZdrWnPEkUT/jUpsOY1DpEa0pfvpEsooKI5mVfiLIEXul4SgFm stg977Iz+6nOwPfNFmEtcuxrXZIM3A43YCUpkDNG0joN3GEikyLwpFeeTMkAe96/wM8Q jOog== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature:dkim-signature; bh=OnRTNV6BJLlEJNjkgcYAe4uit2FAkqV3YDM031DnAjE=; b=jAh8n61nOtlc5urxW1e4udRZHx3TZVhLaj8eoLYtHN17+Cy7rNlI8CmtuQBNKH06kN xID0F3Kc99X1xFwoa+xzGjQvXXQVGEuzCHPAt8JcKtjGWBFzWSH70Sd0nbVVscczZGCs q8duaOXRZjNNUU++BIHYJ/J9F+leBRi20wpUMC2fWy/zuS0RxenJppCbGNm2se+1h4F3 Ymu4uBx0Ecg3ix9rhgGowX0mf/vF/uobS3TZeIHPPU7dZGJMAihSZAXxcKn7DD98ZaNI gkAMlppafoWW2hLTajsEUBlqKXuUcjWZXh6BUu4eCPcymBaO7qXEwja+XJqp5exyCOzu HRLw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=qdw21VjJ; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=xKJYrZFc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id br10-20020a17090b0f0a00b0020d4f2e056csi154863pjb.151.2022.10.19.01.37.53; Wed, 19 Oct 2022 01:38:05 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@suse.de header.s=susede2_rsa header.b=qdw21VjJ; dkim=neutral (no key) header.i=@suse.de header.s=susede2_ed25519 header.b=xKJYrZFc; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=suse.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230062AbiJSIh0 (ORCPT <rfc822;samuel.l.nystrom@gmail.com> + 99 others); Wed, 19 Oct 2022 04:37:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53856 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229592AbiJSIhW (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Wed, 19 Oct 2022 04:37:22 -0400 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D8975FF70; Wed, 19 Oct 2022 01:37:20 -0700 (PDT) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 51CC72036B; Wed, 19 Oct 2022 08:37:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_rsa; t=1666168638; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=OnRTNV6BJLlEJNjkgcYAe4uit2FAkqV3YDM031DnAjE=; b=qdw21VjJrnUFHHve/t0gJpZJrl62EO2IZSnjlOAoITfWFEkT0wpCEuSbRCqVZ9vmZXb3y0 x56RQLq+J70YeXXpA2v3tuQvylQx6txaPSegsN6SboM3fwSfttzOd3H3X09O/LZoBJ58oN fr2xkdlVhofeo9oW/ZbtnBd/fRUkTzo= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.de; s=susede2_ed25519; t=1666168638; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=OnRTNV6BJLlEJNjkgcYAe4uit2FAkqV3YDM031DnAjE=; b=xKJYrZFcwpB00oIPj6Yk7HYJL59Obnxgk8Jw2A58hltPHw4q8T0IssWxG+RVIqTyR21Eho IFV7pgcjnpXexVCg== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 408DD13345; Wed, 19 Oct 2022 08:37:18 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id Api1Dz63T2M/ZQAAMHmgww (envelope-from <nstange@suse.de>); Wed, 19 Oct 2022 08:37:18 +0000 From: Nicolai Stange <nstange@suse.de> To: Steffen Klassert <steffen.klassert@secunet.com>, Daniel Jordan <daniel.m.jordan@oracle.com> Cc: Herbert Xu <herbert@gondor.apana.org.au>, Martin Doucha <mdoucha@suse.cz>, linux-crypto@vger.kernel.org, linux-kernel@vger.kernel.org, Nicolai Stange <nstange@suse.de> Subject: [PATCH 0/5] padata: fix liftime issues after ->serial() has completed Date: Wed, 19 Oct 2022 10:37:03 +0200 Message-Id: <20221019083708.27138-1-nstange@suse.de> X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-4.4 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1747104495856101306?= X-GMAIL-MSGID: =?utf-8?q?1747104495856101306?= |
Series |
padata: fix liftime issues after ->serial() has completed
|
|
Message
Nicolai Stange
Oct. 19, 2022, 8:37 a.m. UTC
Hi all, this series is supposed to fix some lifetime issues all related to the fact that once the last ->serial() has been invoked, the padata user (i.e. pcrypt) is well with its right to tear down the associated padata_shell or parallel_data instance respectively. Only the first one, addressed by patch [2/5], has actually been observed, namely on a (downstream) RT kernel under a very specific workload involving LTP's pcrypt_aead01. On non-RT, I've been unable to reproduce. The remainder of this series, 3-5/5, fixes two more, somewhat related, but purely theoretical issues I spotted when scratching my head about possible reasons for the original Oops. Thanks! Nicolai Nicolai Stange (5): padata: introduce internal padata_get/put_pd() helpers padata: make padata_free_shell() to respect pd's ->refcnt padata: grab parallel_data refcnt for reorder padata: split out dequeue operation from padata_find_next() padata: avoid potential UAFs to the padata_shell from padata_reorder() kernel/padata.c | 129 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 96 insertions(+), 33 deletions(-)
Comments
Hi Nicolai, On Wed, Oct 19, 2022 at 10:37:03AM +0200, Nicolai Stange wrote: > Hi all, > > this series is supposed to fix some lifetime issues all related to the fact that > once the last ->serial() has been invoked, the padata user (i.e. pcrypt) is well > with its right to tear down the associated padata_shell or parallel_data > instance respectively. > > Only the first one, addressed by patch [2/5], has actually been observed, namely > on a (downstream) RT kernel under a very specific workload involving LTP's > pcrypt_aead01. On non-RT, I've been unable to reproduce. I haven't been able to hit the issue in 2/5 on RT on a v6.0 kernel in an x86 vm. Were there any other things running on the system besides pcrypt_aead01? More details about your environment and your kernel config would be helpful. The first two patches seem ok but I want to think about the second more next week. I'll look over the rest of the series then too.
Hi Daniel, Daniel Jordan <daniel.m.jordan@oracle.com> writes: > On Wed, Oct 19, 2022 at 10:37:03AM +0200, Nicolai Stange wrote: >> this series is supposed to fix some lifetime issues all related to the fact that >> once the last ->serial() has been invoked, the padata user (i.e. pcrypt) is well >> with its right to tear down the associated padata_shell or parallel_data >> instance respectively. >> >> Only the first one, addressed by patch [2/5], has actually been observed, namely >> on a (downstream) RT kernel under a very specific workload involving LTP's >> pcrypt_aead01. On non-RT, I've been unable to reproduce. > > I haven't been able to hit the issue in 2/5 on RT on a v6.0 kernel in an > x86 vm. Were there any other things running on the system besides > pcrypt_aead01? More details about your environment and your kernel > config would be helpful. Right, the issue is indeed hard to reproduce, unfortunately. It has originally been reported internally by our QA Maintenance team, which -- for unknown reason -- suddenly started to hit the issue once every while in their testing environment. I did manage to reproduce it once or twice myself, but it took me several days running pcrypt_aead01 in a loop each time. AFAIR, I allocated a single cpu to the VM only and increased the priority of pcrypt_aead01 a bit, with the intent to make preemption of the ->serial() worker by DELALG more likely. But really, I cannot tell if that did in fact contribute to the likelihood of triggering the race or whether I've just been lucky. Also, as mentioned in the cover letter, the RT kernel this has been observed on is a downstream one, based on 5.3.18 (source tree at [1], config at [2]), but with quite some additional patches on top. A backport of this patch series here had been subject to testing in the same environment the issue originally showed up in on a fairly regular basis and no new crashes have been observed since. Let me know if I could provide you with any more details. Thanks, Nicolai [1] https://github.com/SUSE/kernel/tree/SLE15-SP3-RT [2] https://github.com/SUSE/kernel-source/blob/SLE15-SP3-RT/config/x86_64/rt