From patchwork Mon Jun 19 09:22:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Lino Sanfilippo X-Patchwork-Id: 109853 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp2873016vqr; Mon, 19 Jun 2023 02:32:36 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ48NAREBSez++A4f2Lo5wM1d1Cp7vFYcs8t67KFqh+n+Du4fKqSPahw91kBqF4uRpGEwiyX X-Received: by 2002:a17:90a:ad8f:b0:25e:85fd:5749 with SMTP id s15-20020a17090aad8f00b0025e85fd5749mr9774086pjq.8.1687167155988; Mon, 19 Jun 2023 02:32:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687167155; cv=none; d=google.com; s=arc-20160816; b=iJjrsYpowGxM4hbM3rgtxHGjUV3RuvPnZZFsS7n12XYoRVWJavuHX6Ud0QQ9Cex5Af 2TtoBkW/47604AksxgOqm1xVxYcv1/YxEZOKZd9Q+Lh+6tCbt6XX8qRvriaGiWYdMFhq 4xEs2bTiwhkfdEE5EvL/qQ3ZE2Vf+T2VoyqnAQfMxnay4zXzhyU6u+iMqvr0tUztSizs Z0uTkAmn0w3oDYbmnCIUVYRNAZCQgynjiyKVExMyDI8Am1S6IM/7qwXdZe1u0tQwWls9 C5G2DkFIzuC03+JLGlzfhoi0mfUwIPOcPul7zf3Qn8oz7Qg+3EZqAl23Gm3UnXpCvarg KOow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:ui-outboundreport:content-transfer-encoding :mime-version:message-id:date:subject:cc:to:from:dkim-signature; bh=YZh32p7Z0wnP31W9pdXwjnNl16gtCtu5z5U3zFyYmD0=; b=t3yCumyjenduMPO9+I+52/4CseGb8DA3cAtX5j3zqDoUZkZkvoO+7jhD2FWNWLGokJ o4nL4VDmyIEpq8J+Q8bAF85K7bPFG5bzCmn7L357IvHBG2YaQd7DGuIKHdE51QgbSKqx nPQ8SUXj2ar6jzNtlbc8GQyBlVVhoHv4xTdrIqCeR3+ON3DSpcqwuKZheK+HPlZiOBUy XXOodb3F2ZrIAwsIE80Ig2IwcOfPuZv5ZFovLGjUuiwi1s3mlmln7K1HWZ39CngaJEH7 283D736WizAuXmOo8eoqOBZ0eXASKU5JvBBoJJGB/Iz7CZTzWkLv5kbsZY2dkWlNjOPt swxQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmx.de header.s=s31663417 header.b=A2Ey5YQF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gmx.de Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s32-20020a17090a69a300b00250a2c9a793si7001354pjj.152.2023.06.19.02.32.23; Mon, 19 Jun 2023 02:32:35 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmx.de header.s=s31663417 header.b=A2Ey5YQF; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=gmx.de Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231397AbjFSJXg (ORCPT + 99 others); Mon, 19 Jun 2023 05:23:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231386AbjFSJXT (ORCPT ); Mon, 19 Jun 2023 05:23:19 -0400 Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63242130; Mon, 19 Jun 2023 02:23:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=gmx.de; s=s31663417; t=1687166558; x=1687771358; i=linosanfilippo@gmx.de; bh=YZh32p7Z0wnP31W9pdXwjnNl16gtCtu5z5U3zFyYmD0=; h=X-UI-Sender-Class:From:To:Cc:Subject:Date; b=A2Ey5YQF1enTJU7sN7EMKgATH91rxZfwyOzE7vWkks0lERnePVGACZCA738C0/CkBs5lbBU q3Q10wB5kOsYd/qrHS0dckVo1t7vqMzmkP/ufjMJU6n8QOtpc2dQtEi7ZPSQIGUmU1rH9p19q hLQKgocE7ACzaAh+W3b1x6oEHTzs7zPL0Le8SxuJ01RkKzqBB0G95pMmHgjHxPElWPWTyiqNp o1qVSrya6hbbelphu7s332DBTn4uesNselmUbG0ohSrBUEQBKeck2zERQ0RuVlKQUqCbH6TQj Js1Q5Vb/e6WwHO3PtIabupQZxpodz2WVzIKOLCvJzEi4O6UoI8Mg== X-UI-Sender-Class: 724b4f7f-cbec-4199-ad4e-598c01a50d3a Received: from Venus.speedport.ip ([84.162.2.106]) by mail.gmx.net (mrgmx104 [212.227.17.168]) with ESMTPSA (Nemesis) id 1MLR1f-1qTfY50WmZ-00IVzb; Mon, 19 Jun 2023 11:22:38 +0200 From: Lino Sanfilippo To: peterhuewe@gmx.de, jarkko@kernel.org, jgg@ziepe.ca Cc: jsnitsel@redhat.com, hdegoede@redhat.com, oe-lkp@lists.linux.dev, lkp@intel.com, peter.ujfalusi@linux.intel.com, peterz@infradead.org, linux@mniewoehner.de, linux-integrity@vger.kernel.org, linux-kernel@vger.kernel.org, l.sanfilippo@kunbus.com, LinoSanfilippo@gmx.de, lukas@wunner.de, p.rosenberger@kunbus.com, kernel test robot Subject: [PATCH 3] tpm,tpm_tis: Disable interrupts after 1000 unhandled IRQs Date: Mon, 19 Jun 2023 11:22:19 +0200 Message-Id: <20230619092219.2600-1-LinoSanfilippo@gmx.de> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-Provags-ID: V03:K1:taDtb8rWEaK48/aZ8g6C1/tF9k7JOfj7aFkWy7a6ZgDeQU/b5YC kidobZabla0nZJUvzmho1oKNMX4TDfrywG+qHTTmpKG1ZDC2EMkJsgA8faQ/9FTZ1+/W1W1 NvFh2Ug5Rzxwy26Vt6HgD76Z4Yp1HQdTC5dmCYVTIMwLfHdQYutaVBbtAl0yhm+npE1/sYR jVHVRAgd4zksWWoM92ikw== UI-OutboundReport: notjunk:1;M01:P0:qnDdBP6gpZM=;8RKMauwupN5cVf5Utd1kkg1WIx9 M9kkr1INUUcgiKO06kMNoPksemB+i58S+8GfRG2D8dbIIIlfumcbABeIVFSCHYQ+4fy6q0c9B MmLBJtnrZXPvG02xXGhrI7D6rNyimqovXNVT0G2QfB1yevLEnDqtgCTE6/bkozktdwtSnBAVj osK1OBWDRlrt87E/YZuRlS5hi0tVfj9eIbkv9eeSp2QYrzpHM5EyZeAoHjZ8e0rD4aA9qy93+ w+nZqBTSWb7AX5J2jTGELZVfdtQSgMr5kcbIWGESlZjuMJKSn67D2ry9AppFPLxwXcdmasRRV GVH1bTfsyLBKHW+DXEE3RiuhmJjwsNpEub96uQyPxwEf2N4GBxQp138Tr8temUCSuecksQaPw iueLG8LfCOTh6AnMdO0w7X9z4RkQEo3sKQMjqs4kAYwDBkWrABKlwaZDUttkazmkfEaYW27Hx TjinTENX0Coo6nzZ3cTTY9TQsIPCR3iJ1k7C2P/AoRB2ENZU2Oz32TZHkIZ3AmkUKM4UbphWb EtrF/ngcpiR1+B+VAJXf0eXBcMZfBTTaY24x4xRqq0nCjDnvxJyA042xMqg5j0JvkjFqxEd9I MEo1G6YC6bVt5C4Holo9UWhY4gWsWemKQsV/Vx6CiRXB66KLrPuJJJ35W8wgbuY5izxtD5Rhe vvQC1WmFz7EBy9ik7EXnu0jOW5jQAf73dOzwbAtN9iDCxC4FC1quw1VN7UhQk7yLOWI0XvJ+m bk55t3fNBtqjE6ZCcibsdm7hV1wxJeNj4T1Qhr5w68VCyu8gILlAZnFll42H4QwEqXzgVfNC+ tmo0V1UM2h+s/iACjWgIN7xMtGpWu+fi9/ylqfv4xopQew92kaMhu2pyVMYPvDdSXa4Ap33Cs SU6g5NnciTlWxFsXide2YQ08H/bIwK6R+rMWbqPbd3c7a+gVD2Gjkec7e0fx1C0E0/8E9YWGA FAXKcemle6p6oa4nsRzCGyMYNbc= X-Spam-Status: No, score=-1.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM,MIME_BASE64_TEXT, RCVD_IN_DNSWL_LOW,RCVD_IN_MSPIKE_H4,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769122987896509310?= X-GMAIL-MSGID: =?utf-8?q?1769122987896509310?= From: Lino Sanfilippo After activation of interrupts for TPM TIS drivers 0-day reports an interrupt storm on an Inspur NF5180M6 server. Fix this by detecting the storm and falling back to polling: Count the number of unhandled interrupts within a 10 ms time interval. In case that more than 1000 were unhandled deactivate interrupts entirely, deregister the handler and use polling instead. Also print a note to point to the tpm_tis_dmi_table. Since the interrupt deregistration function devm_free_irq() waits for all interrupt handlers to finish, only trigger a worker in the interrupt handler and do the unregistration in the worker to avoid a deadlock. Note: the storm detection logic equals the implementation in note_interrupt() which uses timestamps and counters stored in struct irq_desc. Since this structure is private to the generic interrupt core the TPM TIS core uses its own timestamps and counters. Furthermore the TPM interrupt handler always returns IRQ_HANDLED to prevent the generic interrupt core from processing the interrupt storm. Reported-by: kernel test robot Closes: https://lore.kernel.org/oe-lkp/202305041325.ae8b0c43-yujie.liu@intel.com/ Suggested-by: Lukas Wunner Signed-off-by: Lino Sanfilippo --- drivers/char/tpm/tpm_tis_core.c | 117 ++++++++++++++++++++++++++++---- drivers/char/tpm/tpm_tis_core.h | 4 ++ 2 files changed, 106 insertions(+), 15 deletions(-) Changes to v2: - use define for max number of unhandles irqs(requested by Jarko) - rename intmask to int_mask (requested by Jarko) - rephrased short summary (requested by Jarko) - rename disable_interrupts to tpm_tis_disable_interrupts (requested by Jarko) - print info message concerning adding an entry to tpm_tis_dmi_table (suggested by Jerry) - amended commit message - handle failure of locality request by returning IRQ_NONE - dont take and release locality in __tpm_tis_disable_interrupts but in its caller base-commit: 45a3e24f65e90a047bef86f927ebdc4c710edaa1 diff --git a/drivers/char/tpm/tpm_tis_core.c b/drivers/char/tpm/tpm_tis_core.c index 558144fa707a..d42537b985c5 100644 --- a/drivers/char/tpm/tpm_tis_core.c +++ b/drivers/char/tpm/tpm_tis_core.c @@ -24,9 +24,12 @@ #include #include #include +#include #include "tpm.h" #include "tpm_tis_core.h" +#define TPM_TIS_MAX_UNHANDLED_IRQS 1000 + static void tpm_tis_clkrun_enable(struct tpm_chip *chip, bool value); static bool wait_for_tpm_stat_cond(struct tpm_chip *chip, u8 mask, @@ -468,25 +471,29 @@ static int tpm_tis_send_data(struct tpm_chip *chip, const u8 *buf, size_t len) return rc; } -static void disable_interrupts(struct tpm_chip *chip) +static void __tpm_tis_disable_interrupts(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + u32 int_mask = 0; + + tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &int_mask); + int_mask &= ~TPM_GLOBAL_INT_ENABLE; + tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), int_mask); + + chip->flags &= ~TPM_CHIP_FLAG_IRQ; +} + +static void tpm_tis_disable_interrupts(struct tpm_chip *chip) { struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); - u32 intmask; - int rc; if (priv->irq == 0) return; - rc = tpm_tis_read32(priv, TPM_INT_ENABLE(priv->locality), &intmask); - if (rc < 0) - intmask = 0; - - intmask &= ~TPM_GLOBAL_INT_ENABLE; - rc = tpm_tis_write32(priv, TPM_INT_ENABLE(priv->locality), intmask); + __tpm_tis_disable_interrupts(chip); devm_free_irq(chip->dev.parent, priv->irq, chip); priv->irq = 0; - chip->flags &= ~TPM_CHIP_FLAG_IRQ; } /* @@ -552,7 +559,7 @@ static int tpm_tis_send(struct tpm_chip *chip, u8 *buf, size_t len) if (!test_bit(TPM_TIS_IRQ_TESTED, &priv->flags)) tpm_msleep(1); if (!test_bit(TPM_TIS_IRQ_TESTED, &priv->flags)) - disable_interrupts(chip); + tpm_tis_disable_interrupts(chip); set_bit(TPM_TIS_IRQ_TESTED, &priv->flags); return rc; } @@ -752,6 +759,71 @@ static bool tpm_tis_req_canceled(struct tpm_chip *chip, u8 status) return status == TPM_STS_COMMAND_READY; } +static irqreturn_t tpm_tis_reenable_polling(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + const char *product; + const char *vendor; + + dev_warn(&chip->dev, FW_BUG + "TPM interrupt storm detected, polling instead\n"); + + vendor = dmi_get_system_info(DMI_SYS_VENDOR); + product = dmi_get_system_info(DMI_PRODUCT_VERSION); + + if (vendor && product) { + dev_info(&chip->dev, + "Consider adding the following entry to tpm_tis_dmi_table:\n"); + dev_info(&chip->dev, "\tDMI_SYS_VENDOR: %s\n", vendor); + dev_info(&chip->dev, "\tDMI_PRODUCT_VERSION: %s\n", product); + } + + if (tpm_tis_request_locality(chip, 0) != 0) + return IRQ_NONE; + + __tpm_tis_disable_interrupts(chip); + tpm_tis_relinquish_locality(chip, 0); + + /* + * devm_free_irq() must not be called from within the interrupt handler, + * since this function waits for running handlers to finish and thus it + * would deadlock. Instead trigger a worker that takes care of the + * unregistration. + */ + schedule_work(&priv->free_irq_work); + + return IRQ_HANDLED; +} + +static irqreturn_t tpm_tis_check_for_interrupt_storm(struct tpm_chip *chip) +{ + struct tpm_tis_data *priv = dev_get_drvdata(&chip->dev); + irqreturn_t irqret = IRQ_HANDLED; + + /* + * The worker to free the TPM interrupt (free_irq_work) may already + * be scheduled, so make sure it is not scheduled again. + */ + if (!(chip->flags & TPM_CHIP_FLAG_IRQ)) + return IRQ_HANDLED; + + if (time_after(jiffies, priv->last_unhandled_irq + HZ/10)) + priv->unhandled_irqs = 1; + else + priv->unhandled_irqs++; + + priv->last_unhandled_irq = jiffies; + + if (priv->unhandled_irqs > TPM_TIS_MAX_UNHANDLED_IRQS) + irqret = tpm_tis_reenable_polling(chip); + + /* + * Prevent the genirq code from starting its own interrupt storm + * handling by always reporting that the interrupt was handled. + */ + return irqret; +} + static irqreturn_t tis_int_handler(int dummy, void *dev_id) { struct tpm_chip *chip = dev_id; @@ -761,10 +833,10 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id) rc = tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt); if (rc < 0) - return IRQ_NONE; + goto unhandled; if (interrupt == 0) - return IRQ_NONE; + goto unhandled; set_bit(TPM_TIS_IRQ_TESTED, &priv->flags); if (interrupt & TPM_INTF_DATA_AVAIL_INT) @@ -780,10 +852,13 @@ static irqreturn_t tis_int_handler(int dummy, void *dev_id) rc = tpm_tis_write32(priv, TPM_INT_STATUS(priv->locality), interrupt); tpm_tis_relinquish_locality(chip, 0); if (rc < 0) - return IRQ_NONE; + goto unhandled; tpm_tis_read32(priv, TPM_INT_STATUS(priv->locality), &interrupt); return IRQ_HANDLED; + +unhandled: + return tpm_tis_check_for_interrupt_storm(chip); } static void tpm_tis_gen_interrupt(struct tpm_chip *chip) @@ -804,6 +879,15 @@ static void tpm_tis_gen_interrupt(struct tpm_chip *chip) chip->flags &= ~TPM_CHIP_FLAG_IRQ; } +static void tpm_tis_free_irq_func(struct work_struct *work) +{ + struct tpm_tis_data *priv = container_of(work, typeof(*priv), free_irq_work); + struct tpm_chip *chip = priv->chip; + + devm_free_irq(chip->dev.parent, priv->irq, chip); + priv->irq = 0; +} + /* Register the IRQ and issue a command that will cause an interrupt. If an * irq is seen then leave the chip setup for IRQ operation, otherwise reverse * everything and leave in polling mode. Returns 0 on success. @@ -816,6 +900,7 @@ static int tpm_tis_probe_irq_single(struct tpm_chip *chip, u32 intmask, int rc; u32 int_status; + INIT_WORK(&priv->free_irq_work, tpm_tis_free_irq_func); rc = devm_request_threaded_irq(chip->dev.parent, irq, NULL, tis_int_handler, IRQF_ONESHOT | flags, @@ -918,6 +1003,7 @@ void tpm_tis_remove(struct tpm_chip *chip) interrupt = 0; tpm_tis_write32(priv, reg, ~TPM_GLOBAL_INT_ENABLE & interrupt); + flush_work(&priv->free_irq_work); tpm_tis_clkrun_enable(chip, false); @@ -1021,6 +1107,7 @@ int tpm_tis_core_init(struct device *dev, struct tpm_tis_data *priv, int irq, chip->timeout_b = msecs_to_jiffies(TIS_TIMEOUT_B_MAX); chip->timeout_c = msecs_to_jiffies(TIS_TIMEOUT_C_MAX); chip->timeout_d = msecs_to_jiffies(TIS_TIMEOUT_D_MAX); + priv->chip = chip; priv->timeout_min = TPM_TIMEOUT_USECS_MIN; priv->timeout_max = TPM_TIMEOUT_USECS_MAX; priv->phy_ops = phy_ops; @@ -1179,7 +1266,7 @@ int tpm_tis_core_init(struct device *dev, struct tpm_tis_data *priv, int irq, rc = tpm_tis_request_locality(chip, 0); if (rc < 0) goto out_err; - disable_interrupts(chip); + tpm_tis_disable_interrupts(chip); tpm_tis_relinquish_locality(chip, 0); } } diff --git a/drivers/char/tpm/tpm_tis_core.h b/drivers/char/tpm/tpm_tis_core.h index 610bfadb6acf..b1a169d7d1ca 100644 --- a/drivers/char/tpm/tpm_tis_core.h +++ b/drivers/char/tpm/tpm_tis_core.h @@ -91,11 +91,15 @@ enum tpm_tis_flags { }; struct tpm_tis_data { + struct tpm_chip *chip; u16 manufacturer_id; struct mutex locality_count_mutex; unsigned int locality_count; int locality; int irq; + struct work_struct free_irq_work; + unsigned long last_unhandled_irq; + unsigned int unhandled_irqs; unsigned int int_mask; unsigned long flags; void __iomem *ilb_base_addr;