From patchwork Thu Nov 24 00:11:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rishabh Bhatnagar X-Patchwork-Id: 2150 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp3106428wrr; Wed, 23 Nov 2022 16:19:52 -0800 (PST) X-Google-Smtp-Source: AA0mqf55xkqNmGjbWHkIq8NVuuo5JGNwept29pFkD2pQ/T4OjJ+/1W+Ojs4wv4SuAmNasIo6S3hC X-Received: by 2002:a63:da10:0:b0:477:ccac:6eb5 with SMTP id c16-20020a63da10000000b00477ccac6eb5mr1034066pgh.41.1669249192042; Wed, 23 Nov 2022 16:19:52 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669249192; cv=none; d=google.com; s=arc-20160816; b=cGqZ65LSIV11E8IKveGhOlIvdWtHTG2MdnW+a60lCJRiREntaLV2xBhnyNAIxvFHdY VAlLfI0mh4idGfC2l/uBG4iyNBzeylpYRP16wQEZni+ZoASPom1His3E93bpJQ1X1lta LZyAanRJoVqlz5dwzCifDBu0yBxnZS3EPnyuez3Np/reyh7GfAzAFTEEBcgxpl1yj3Zb uSDfxIwabXqiXBAhHoGPqiXaFBPjgjumXEh1e1qkDpHt0k2rk4iHYQdERrG+6ZTRCIiL 6f9L4xQLVRx+AMoI8SechkKUgHRwhYyQJN8T7xK9eNgatKL8HvvRx5RBro2y6t/CRRXh jxNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=WEHHMHmAbC3y7/sIYj2mPueeyoULLmYszFzqAgbKTL4=; b=Z51wkkK2QqcahtzFXnGZI1KByUF26/lKqUXqhNFQsWanLWPVA8Mpyqwmfe18jnNecz LX4pc/kQX/Nqw+qTNGgeKdZnF8rXmEEYhxfn0SDRwdO/GxcK7VzkMEcjhfmMNS0/PDjD bggt6Hq5808445/YmNOcUovgBP3/iUiGtCQ9RT3ATx7IrFx7bl06cdpFpdHkDscOQNjB qTY4ASZ873dR+ybIxXY6YOYQJqXChjhaZeDU65xV3DmU19ruNo1oDltRoX4JMDOBRS09 VB+Q6ZnqZhDoTBNNVSKd5smP8/U/4/XzlEkJ9eQCm9HiAw+e00+blHDylWG9XehGaXDs Os4Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b="SyPHf/aP"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g3-20020a056a0023c300b00574694c70e1si2260720pfc.22.2022.11.23.16.19.38; Wed, 23 Nov 2022 16:19:52 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@amazon.com header.s=amazon201209 header.b="SyPHf/aP"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amazon.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229541AbiKXALb (ORCPT + 99 others); Wed, 23 Nov 2022 19:11:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229453AbiKXAL2 (ORCPT ); Wed, 23 Nov 2022 19:11:28 -0500 Received: from smtp-fw-80007.amazon.com (smtp-fw-80007.amazon.com [99.78.197.218]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E33D2D740; Wed, 23 Nov 2022 16:11:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1669248687; x=1700784687; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=WEHHMHmAbC3y7/sIYj2mPueeyoULLmYszFzqAgbKTL4=; b=SyPHf/aPDryPRp2m5FtxXerfX69uaHrhYFL93nM4kcENS6MgPM4F9jk9 IEeRi/1DdjxhOwVgExvGqMGXfYKAbwJTkk2ayUAzlWWb42KHcE0dPm1bq exqNn+lvpCtLktvnClXCHT7F5LjDJwhwYuL8D0tZXrZIne4FHAOGHqn03 Q=; X-IronPort-AV: E=Sophos;i="5.96,187,1665446400"; d="scan'208";a="154218801" Received: from pdx4-co-svc-p1-lb2-vlan3.amazon.com (HELO email-inbound-relay-pdx-2c-m6i4x-94edd59b.us-west-2.amazon.com) ([10.25.36.214]) by smtp-border-fw-80007.pdx80.corp.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 24 Nov 2022 00:11:27 +0000 Received: from EX13MTAUWC001.ant.amazon.com (pdx1-ws-svc-p6-lb9-vlan3.pdx.amazon.com [10.236.137.198]) by email-inbound-relay-pdx-2c-m6i4x-94edd59b.us-west-2.amazon.com (Postfix) with ESMTPS id 5CE1D41A0F; Thu, 24 Nov 2022 00:11:27 +0000 (UTC) Received: from EX19D002UWA003.ant.amazon.com (10.13.138.235) by EX13MTAUWC001.ant.amazon.com (10.43.162.135) with Microsoft SMTP Server (TLS) id 15.0.1497.42; Thu, 24 Nov 2022 00:11:25 +0000 Received: from EX13MTAUWA001.ant.amazon.com (10.43.160.58) by EX19D002UWA003.ant.amazon.com (10.13.138.235) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1118.20; Thu, 24 Nov 2022 00:11:25 +0000 Received: from dev-dsk-risbhat-2b-8bdc64cd.us-west-2.amazon.com (10.189.73.169) by mail-relay.amazon.com (10.43.160.118) with Microsoft SMTP Server id 15.0.1497.42 via Frontend Transport; Thu, 24 Nov 2022 00:11:25 +0000 Received: by dev-dsk-risbhat-2b-8bdc64cd.us-west-2.amazon.com (Postfix, from userid 22673075) id 74476ABFD; Thu, 24 Nov 2022 00:11:25 +0000 (UTC) From: Rishabh Bhatnagar To: , , , CC: , , , Rishabh Bhatnagar Subject: [PATCH 5.4 0/2] Fix epoll issue in 5.4 kernels Date: Thu, 24 Nov 2022 00:11:21 +0000 Message-ID: <20221124001123.3248571-1-risbhat@amazon.com> X-Mailer: git-send-email 2.37.1 MIME-Version: 1.0 X-Spam-Status: No, score=-11.9 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_MED, RCVD_IN_MSPIKE_H2,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750334640926152658?= X-GMAIL-MSGID: =?utf-8?q?1750334640926152658?= Hi Greg After upgrading to 5.4.211 we were started seeing some nodes getting stuck in our Kubernetes cluster. All nodes are running this kernel version. After taking a closer look it seems that runc was command getting stuck. Looking at the stack it appears the thread is stuck in epoll wait for sometime. [<0>] do_syscall_64+0x48/0xf0 [<0>] entry_SYSCALL_64_after_hwframe+0x5c/0xc1 [<0>] ep_poll+0x48d/0x4e0 [<0>] do_epoll_wait+0xab/0xc0 [<0>] __x64_sys_epoll_pwait+0x4d/0xa0 [<0>] do_syscall_64+0x48/0xf0 [<0>] entry_SYSCALL_64_after_hwframe+0x5c/0xc1 [<0>] futex_wait_queue_me+0xb6/0x110 [<0>] futex_wait+0xe2/0x260 [<0>] do_futex+0x372/0x4f0 [<0>] __x64_sys_futex+0x134/0x180 [<0>] do_syscall_64+0x48/0xf0 [<0>] entry_SYSCALL_64_after_hwframe+0x5c/0xc1 I noticed there are other discussions going on as well regarding this. https://lore.kernel.org/all/Y1pY2n6E1Xa58MXv@kroah.com/ Reverting the below patch does fix the issue: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?h=linux-5.4.y&id=cf2db24ec4b8e9d399005ececd6f6336916ab6fc We don't see this issue in latest upstream kernel or even latest 5.10 stable tree. Looking at the patches that went in for 5.10 stable there's one that stands out that seems to be missing in 5.4. 289caf5d8f6c61c6d2b7fd752a7f483cd153f182 (epoll: check for events when removing a timed out thread from the wait queue) Backporting this patch to 5.4 we don't see the hangups anymore. Looks like this patch fixes time out scenarios which might cause missed wake ups. The other patch in the patch series also fixes a race and is needed for the second patch to apply. Roman Penyaev (1): epoll: call final ep_events_available() check under the lock Soheil Hassas Yeganeh (1): epoll: check for events when removing a timed out thread from the wait queue fs/eventpoll.c | 68 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 27 deletions(-)