From patchwork Tue May 30 17:47:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lino Sanfilippo X-Patchwork-Id: 101053 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2366881vqr; Tue, 30 May 2023 11:07:04 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4GI1anUUckQjToeG5qVTS5wh2zT5mwy+tBlAn4SZIdjmkdkZ9DYHeZ0xQsma2RK5GIrgsE X-Received: by 2002:a05:6a00:1308:b0:645:fc7b:63db with SMTP id j8-20020a056a00130800b00645fc7b63dbmr3198460pfu.20.1685470023955; Tue, 30 May 2023 11:07:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1685470023; cv=none; d=google.com; s=arc-20160816; b=Azz5eH3cIV47cBTAVy/lVJR1pM2kBFa/h0ageJFra7WXr1vEFmTnwP2Tp1fKei/ucE AFd/VX+U5f5F09fuVD3qx7Bi+7klKLzk6GfxG7cRsYmGbaTkwl537C0Wq510iiTvTCAL DNd1MLNQ3gFOc7IcjLKRqYeZAmzydShQHQ7HYaLpYao1WKeWn1yuUHdP9tv2WVLSff8r NlKxghJe0FE96n/Tkm30VAd8Ttsr9tPaYnS3vrtY/3hkIcz8MeGJaqR0g3FCbvGITevE pfhKuv2ye3Bu6AWrBCpoPmk1msKGHIP1QqNv0lYMNQCcZaS05Twv/ayAc/02mYlAGWbj G+dw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:ui-outboundreport:content-transfer-encoding :mime-version:message-id:date:subject:cc:to:from:dkim-signature; bh=++AWfGg+huXGq6oS7nTH1/C1B8ZsQI4CHiv2Tr/rOQg=; b=hhT0Xo7CImH4qFMfsCrO63oPb4KsGCyAB4dpoYJlGGM56pjNqIkMl0jIklS56vQ+1p Goo6QOZjhPPqmfeqWMykBsAaSFleKO03DGuJORETgHiotcclM0p6LmLPQ/7oAad47gPM 0sTSPPYlKGD6Vm/t0PFRU8DkVhSsYrWl+dLVimCyZjXJFCYf+yyiI3WA0ltVxy2ci2Kt HFFNpJIbXrAFR23OZWemd7iN/TzWA6SQPlQsy6CKDUk0XmHd1DYL6IUFPtefgmhQwPRf D++EQrUvm3YNLfMgCfw0GVRG5TiZ0uEMaF0fArUgJOjfeG+Pr0fH6DUhge9+Ia2J7xA0 zWHQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmx.de header.s=s31663417 header.b=MGxErC8K; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gmx.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f28-20020a63381c000000b0052868a865d4si4486962pga.553.2023.05.30.11.06.48; Tue, 30 May 2023 11:07:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmx.de header.s=s31663417 header.b=MGxErC8K; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gmx.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232634AbjE3RsQ (ORCPT + 99 others); Tue, 30 May 2023 13:48:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230096AbjE3RsP (ORCPT ); Tue, 30 May 2023 13:48:15 -0400 Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D9FEB2; Tue, 30 May 2023 10:48:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1685468859; i=linosanfilippo@gmx.de; bh=++AWfGg+huXGq6oS7nTH1/C1B8ZsQI4CHiv2Tr/rOQg=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date; b=MGxErC8KrqcS+ussukmTT8GRSHHm8QIlE8GxT/J8r9/rKabfKPqivojQEbULZ4W7a NCcOQoRDEszWdQK8oD732maLD80BRFbzBm0Cx9ASqMZFV9ShV3KNFFoWNsgFHAY0Ll Q8ZNdnlIUd6u2z7Yub2nxZzvwZJOXnk6C3x/fOMXo/vRdsoqzrIjP2wZPhIPnDYT3B LIjSC8KDFWov2wHm+uT7SIDFtRgJx1akgokJw/j0ZEf053S//reF8AkI6L0FXAlkeR dv67AQ2PQUKgvBDiIOh62Qv3pE10FDIYLeoADl64J+J5y9wWbuWZqaU1MwtQ1lnKeo svULC9Zhi10rQ== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from Venus.speedport.ip ([84.162.2.106]) by mail.gmx.net (mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id 1MTzf6-1pd8en0aeY-00QyBO; Tue, 30 May 2023 19:47:39 +0200 From: Lino Sanfilippo To: peterhuewe@gmx.de, jarkko@kernel.org, jgg@ziepe.ca Cc: jsnitsel@redhat.com, hdegoede@redhat.com, oe-lkp@lists.linux.dev, lkp@intel.com, peter.ujfalusi@linux.intel.com, peterz@infradead.org, linux@mniewoehner.de, linux-integrity@vger.kernel.org, linux-kernel@vger.kernel.org, l.sanfilippo@kunbus.com, LinoSanfilippo@gmx.de, lukas@wunner.de, p.rosenberger@kunbus.com, kernel test robot Subject: [PATCH v2] tpm,tpm_tis: Handle interrupt storm Date: Tue, 30 May 2023 19:47:12 +0200 Message-Id: <20230530174712.6989-1-LinoSanfilippo@gmx.de> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-Provags-ID: V03:K1:6I6uOiEXoQ914p4DH3/mKhxPboZliuI1/kHOD8HviEPdYFnOAIF UImaGD1j11GryfRWdZ9+Cz5WKBy6G74kFm0cavF+jNcx/UKjELl+kJcgv03hbaNN3z/LiJu lQRUxHLN9htBKIoOfaMlWHsqV12ESz++5btTXWEad1aD1J7eiEBpIh2aOyrGtieOpRV3eig 8XfHAvahfI04oX6muDz6w== UI-OutboundReport: notjunk:1;M01:P0:5MoqWh4wBzk=;I1siaBsEOSZY6/rpiK4svW0k1sK VjKtY0yzx/hI22+Xjt26Wq1ggeYJr0rt0P+fU15X40YbBWydESaU8Y4Vp6F/20DlNR99VM0vO SOgu5opqh6gYBioTPCSU8ANYdveQHFhHWdsekq9ZhSuTu/EfD0N5sK/bfy1Enhqt22PhOteY0 bE/frup9hqkuHW17eSsF4xygm34phxf1AttLzjk9Sy1vAewOy7kkBqkoPY4hzeQ2LE1Tix5S+ itS0lhGk0fiMoZWt/JrsQnmcD+dItbcBUYj5P2yGsTii1Yo3xfERFpkrVQWKVKZVvlLMMVReX RsWVFdyiWl621A3ZqCp8jNb/eU69Ntg4drNUFK25kMzhkiVx+lnv+od+/3uGHUV4OzxXqOwEk xxbXI6M4pmCQo094CKSATqQ5Y7OD2lbxOB2xVPKWxp30M0O3SbckvvT2rDjnR4cUoOyHkOAwG sDgqLIPUxTrlciVZaBErPSpxYdGqxW4PRWsle3XE1UhYZ5MAm7Ujx5eEaZ8/QIUXoj0AGSIbO LVnyqWXW03cf+QMRSJInB/MuJ9JOtV57CpDo000IpbeLXH4k7unwY0Ik7YTLi7lf6H/eZxFTN jTk3DVviiVLHNykkPacQLPTm7iZGUUZ0ZBWAy+43dAwklxDGbGRrEG5U//mmkRswXv2ULkCUw t5mIieBwlw/+Xq435js0C6sU5kpQK42Y0P/AzVOHyZ+ZCBYARhdFZzlKOOJSfoxK7gAib/gQq ripoOLSTT0QhvnqTb3U8f+h8Y5KpsWODXi5yDcFWVFlSsh2jRQW8SL5JDO69WFDHc936exGGU aZMIi6xlA2blzEMDFEQnAMTy/icdD/RaMQYh1EY16KIIDFCi+tRDwfW1KsKfWg7dmmFXQwTBz VNNDrEFUA0RSCfpXFvWcNTsyRjXxCuu0AfYQ1aoUJ9+d6utj5xhojnOkwSsmuE20eSPBeR235 1S4nsX9F2zQU2TYcoFYKYhzBqbw= X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,MIME_BASE64_TEXT, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H5,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766606338435968010?= X-GMAIL-MSGID: =?utf-8?q?1767343415721373493?= From: Lino Sanfilippo After activation of interrupts for TPM TIS drivers 0-day reports an interrupt storm on an Inspur NF5180M6/NF5180M6 server. Fix this by detecting the storm and falling back to polling: Count the number of unhandled interrupts within a 10 ms time interval. In case that more than 1000 were unhandled deactivate interrupts entirely, deregister the handler and use polling instead. The storm detection logic equals the implementation in note_interrupt() which uses timestamps and counters stored in struct irq_desc. Since this structure is private to the generic interrupt core the TPM TIS core uses its own timestamps and counters. Furthermore the TPM interrupt handler always returns IRQ_HANDLED to prevent the generic interrupt core from processing the interrupt storm. Since the interrupt deregistration function devm_free_irq() waits for all interrupt handlers to finish, only trigger a worker in the interrupt handler and do the unregistration in the worker to avoid a deadlock. Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305041325.ae8b0c43-yujie.liu@intel.com/ Suggested-by: Lukas Wunner Signed-off-by: Lino Sanfilippo --- drivers/char/tpm/tpm_tis_core.c | 93 ++++++++++++++++++++++++++++----- drivers/char/tpm/tpm_tis_core.h | 4 ++ 2 files changed, 85 insertions(+), 12 deletions(-) base-commit: 7877cb91f1081754a1487c144d85dc0d2e2e7fc4 diff --git a/drivers/char/tpm/tpm_tis_core.c b/drivers/char/tpm/tpm_tis_core.c index 558144fa707a..7ae8228e803f 100644 --- a/drivers/char/tpm/tpm_tis_core.c +++ b/drivers/char/tpm/tpm_tis_core.c @@ -468,25 +468,32 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len) return rc; } +static void __tpm_tis_disable_interrupts(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + u32 intmask = 0; + + tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask); + intmask &= ~TPM_GLOBAL_INT_ENABLE; + + tpm_tis_request_locality(chip, 0); + tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask); + tpm_tis_relinquish_locality(chip, 0); + + chip->flags &= ~TPM_CHIP_FLAG_IRQ; +} + static void disable_interrupts(struct tpm_chip *chip) { struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); - u32 intmask; - int rc; if (priv->irq == 0) return; - rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask); - if (rc < 0) - intmask = 0; - - intmask &= ~TPM_GLOBAL_INT_ENABLE; - rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask); + __tpm_tis_disable_interrupts(chip); devm_free_irq(chip->dev.parent, priv->irq, chip); priv->irq = 0; - chip->flags &= ~TPM_CHIP_FLAG_IRQ; } /* @@ -752,6 +759,53 @@ static bool tpm_tis_req_canceled(struct tpm_chip *chip, u8 status) return status == TPM_STS_COMMAND_READY; } +static void tpm_tis_reenable_polling(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + + dev_warn(&chip->dev, FW_BUG + "TPM interrupt storm detected, polling instead\n"); + + __tpm_tis_disable_interrupts(chip); + + /* + * devm_free_irq() must not be called from within the interrupt handler, + * since this function waits for running handlers to finish and thus it + * would deadlock. Instead trigger a worker that takes care of the + * unregistration. + */ + schedule_work(&priv->free_irq_work); +} + +static irqreturn_t tpm_tis_check_for_interrupt_storm(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + const unsigned int MAX_UNHANDLED_IRQS = 1000; + + /* + * The worker to free the TPM interrupt (free_irq_work) may already + * be scheduled, so make sure it is not scheduled again. + */ + if (!(chip->flags & TPM_CHIP_FLAG_IRQ)) + return IRQ_HANDLED; + + if (time_after(jiffies, priv->last_unhandled_irq + HZ/10)) + priv->unhandled_irqs = 1; + else + priv->unhandled_irqs++; + + priv->last_unhandled_irq = jiffies; + + if (priv->unhandled_irqs > MAX_UNHANDLED_IRQS) + tpm_tis_reenable_polling(chip); + + /* + * Prevent the genirq code from starting its own interrupt storm + * handling by always reporting that the interrupt was handled. + */ + return IRQ_HANDLED; +} + static irqreturn_t tis_int_handler(int dummy, void *dev_id) { struct tpm_chip *chip = dev_id; @@ -761,10 +815,10 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id) rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt); if (rc < 0) - return IRQ_NONE; + goto unhandled; if (interrupt == 0) - return IRQ_NONE; + goto unhandled; set_bit(TPM_TIS_IRQ_TESTED, &priv->flags); if (interrupt & TPM_INTF_DATA_AVAIL_INT) @@ -780,10 +834,13 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id) rc = tpm_tis_write32(priv, TPM_INT_STATUS(priv->locality), interrupt); tpm_tis_relinquish_locality(chip, 0); if (rc < 0) - return IRQ_NONE; + goto unhandled; tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt); return IRQ_HANDLED; + +unhandled: + return tpm_tis_check_for_interrupt_storm(chip); } static void tpm_tis_gen_interrupt(struct tpm_chip *chip) @@ -804,6 +861,15 @@ static void tpm_tis_gen_interrupt(struct tpm_chip *chip) chip->flags &= ~TPM_CHIP_FLAG_IRQ; } +static void tpm_tis_free_irq_func(struct work_struct *work) +{ + struct tpm_tis_data *priv = container_of(work, typeof(*priv), free_irq_work); + struct tpm_chip *chip = priv->chip; + + devm_free_irq(chip->dev.parent, priv->irq, chip); + priv->irq = 0; +} + /* Register the IRQ and issue a command that will cause an interrupt. If an * irq is seen then leave the chip setup for IRQ operation, otherwise reverse * everything and leave in polling mode. Returns 0 on success. @@ -816,6 +882,7 @@ static int tpm_tis_probe_irq_single(struct tpm_chip *chip, u32 intmask, int rc; u32 int_status; + INIT_WORK(&priv->free_irq_work, tpm_tis_free_irq_func); rc = devm_request_threaded_irq(chip->dev.parent, irq, NULL, tis_int_handler, IRQF_ONESHOT | flags, @@ -918,6 +985,7 @@ void tpm_tis_remove(struct tpm_chip *chip) interrupt = 0; tpm_tis_write32(priv, reg, ~TPM_GLOBAL_INT_ENABLE & interrupt); + flush_work(&priv->free_irq_work); tpm_tis_clkrun_enable(chip, false); @@ -1021,6 +1089,7 @@ int tpm_tis_core_init(struct device *dev, struct tpm_tis_data *priv, int irq, chip->timeout_b = msecs_to_jiffies(TIS_TIMEOUT_B_MAX); chip->timeout_c = msecs_to_jiffies(TIS_TIMEOUT_C_MAX); chip->timeout_d = msecs_to_jiffies(TIS_TIMEOUT_D_MAX); + priv->chip = chip; priv->timeout_min = TPM_TIMEOUT_USECS_MIN; priv->timeout_max = TPM_TIMEOUT_USECS_MAX; priv->phy_ops = phy_ops; diff --git a/drivers/char/tpm/tpm_tis_core.h b/drivers/char/tpm/tpm_tis_core.h index e978f457fd4d..b1fa42367052 100644 --- a/drivers/char/tpm/tpm_tis_core.h +++ b/drivers/char/tpm/tpm_tis_core.h @@ -91,11 +91,15 @@ enum tpm_tis_flags { }; struct tpm_tis_data { + struct tpm_chip *chip; u16 manufacturer_id; struct mutex locality_count_mutex; unsigned int locality_count; int locality; int irq; + struct work_struct free_irq_work; + unsigned long last_unhandled_irq; + unsigned int unhandled_irqs; unsigned int int_mask; unsigned long flags; void __iomem *ilb_base_addr;