From patchwork Tue Mar 28 15:35:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 76152 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2315509vqo; Tue, 28 Mar 2023 08:43:24 -0700 (PDT) X-Google-Smtp-Source: AKy350a9xrSwnk4VlVnsURLL5avRpSULhnYDLjgxl59FHjZ+GI+x9Fj5y0u7RdjXKadTbIaruS+o X-Received: by 2002:a17:907:9c08:b0:93f:9b4a:12f3 with SMTP id ld8-20020a1709079c0800b0093f9b4a12f3mr12771467ejc.10.1680018204414; Tue, 28 Mar 2023 08:43:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1680018204; cv=none; d=google.com; s=arc-20160816; b=dUyC9Txhu9jGoS284HFE5H4mK1nudqgSCB26cpqm0RKs8jT8coMrHa7XajzpRdy8nh 6c0oPbdzh62Z4N7MbHcYB9eADMxxjljzvRqezKg0OeMm7dGOPu2ljdIN8/L5Inb21RGy cVazeECGCCK6eubn5wVEGSb2bEVxIj6qmj3bq/HHGLbACCeyqWOAu4wsz6aJVC9yyJ4U Jup2t4kpDw77dS9u264zLIQWUvaoUkw3XGWfbVlc0L+/6jhEiUV4uu4TGbRNmZeRxUPB cMDivCJW/UgGraIDNiA7PdUxkBAPALJcUN4kueh9pbv/40iPniMRGXsUAJggaUhyQNn8 lj3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VU6Ogi7E8AOzHKJ5AYuATmbfqIKJbUJ0ELwsUb1FAww=; b=MxJnNWzIvLVfV1T+ou2a9bIpmh8eyNEt27YY/+Schrh+F/O83j/soPPLZ8lGYSakWJ hPUBCErrqanfNOCPHcoA49nnpwg9Q4gvGhvGzgQjmNVsFVDW6BB6cxVDY763QOxNVpho Ri7H5Bk0Q2FKR952hshvNTqVT8zc3BXrQaSejEh+J1LXH4RtGK853VCaKbkAJI8IhvDz rlSYKBVPjFv/wY6SeYXi76SSOmDkuvNszfdE42Xp8PEZFRXBzSwUiXVHJI3MeSRlg9iI i4l9HNRUr0u1/GUjVL3gvCXWCoIYuoxqdmAPksCb6H2nVltHjJ8D357nHRB1vcOKCSB0 gc0g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=BCnwbE8D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v7-20020aa7d807000000b00501d4810b84si21305577edq.376.2023.03.28.08.43.00; Tue, 28 Mar 2023 08:43:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=BCnwbE8D; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234098AbjC1Pik (ORCPT + 99 others); Tue, 28 Mar 2023 11:38:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233192AbjC1Pht (ORCPT ); Tue, 28 Mar 2023 11:37:49 -0400 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34ACC11162; Tue, 28 Mar 2023 08:36:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680017809; x=1711553809; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ySf76nZG5BSVEABaWShXg0z5KKr8JGvJzU73vkpnzAg=; b=BCnwbE8DoJ5/T1pnLnKy6qHasRAjRNBgpXLFbY/nMDviT1Zc1EvDFfV3 OdUil8HaOV8BWJE57KD4Nvi4MAse4NuT0F8ZVplzgKdyFmWB4X6EaXXYS Hqz+Nlz9ELSN8E9FXju8k1D1thtZun7cyvYXEuNRrcrhgbVdoDF6IW47J n0mzahGQwirk8jEYSWj2jh3G/TjcGfWyMkhNN364ehCG/NN9Mk3vMtc4N 2R3iBAIMfY+kIcs8BkNfadurlVdR3YXIkV+conWuj94/W/tf0NgPnZqH7 PUq+hE81JjXXdl59trHLs78WU8SHpuvh7Whr+TjxtmlzMmtLesFmpOag8 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="342192250" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="342192250" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:36:19 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10663"; a="683948523" X-IronPort-AV: E=Sophos;i="5.98,297,1673942400"; d="scan'208";a="683948523" Received: from sdwarak1-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.intel.com) ([10.212.127.200]) by orsmga002-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Mar 2023 08:36:17 -0700 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, giovanni.cabiddu@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH v2 14/15] crypto: iaa - Add irq support for the crypto async interface Date: Tue, 28 Mar 2023 10:35:34 -0500 Message-Id: <20230328153535.126223-15-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230328153535.126223-1-tom.zanussi@linux.intel.com> References: <20230328153535.126223-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.4 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,RCVD_IN_MSPIKE_H3, RCVD_IN_MSPIKE_WL,SPF_HELO_NONE,SPF_NONE autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1761626768727359310?= X-GMAIL-MSGID: =?utf-8?q?1761626768727359310?= The existing iaa crypto async support provides an implementation that satisfies the interface but does so in a synchronous manner - it fills and submits the IDXD descriptor and then waits for it to complete before returning. This isn't a problem at the moment, since all existing callers (e.g. zswap) wrap any asynchronous callees in a synchronous wrapper anyway. This change makes the iaa crypto async implementation truly asynchronous: it fills and submits the IDXD descriptor, then returns immediately with -EINPROGRESS. It also sets the descriptor's 'request completion irq' bit and sets up a callback with the IDXD driver which is called when the operation completes and the irq fires. The existing callers such as zswap use synchronous wrappers to deal with -EINPROGRESS and so work as expected without any changes. This mode can be enabled by writing 'async_irq' to the sync_mode iaa_crypto driver attribute: echo async_irq > /sys/bus/dsa/drivers/crypto/sync_mode Async mode without interrupts (caller must poll) can be enabled by writing 'async' to it: echo async > /sys/bus/dsa/drivers/crypto/sync_mode The default sync mode can be enabled by writing 'sync' to it: echo sync > /sys/bus/dsa/drivers/crypto/sync_mode Signed-off-by: Tom Zanussi --- drivers/crypto/intel/iaa/iaa_crypto_main.c | 224 ++++++++++++++++++++- 1 file changed, 222 insertions(+), 2 deletions(-) diff --git a/drivers/crypto/intel/iaa/iaa_crypto_main.c b/drivers/crypto/intel/iaa/iaa_crypto_main.c index b6d41642f0dd..7a2c0ed80108 100644 --- a/drivers/crypto/intel/iaa/iaa_crypto_main.c +++ b/drivers/crypto/intel/iaa/iaa_crypto_main.c @@ -116,6 +116,102 @@ static ssize_t verify_compress_store(struct device_driver *driver, } static DRIVER_ATTR_RW(verify_compress); +/* + * The iaa crypto driver supports three 'sync' methods determining how + * compressions and decompressions are performed: + * + * - sync: the compression or decompression completes before + * returning. This is the mode used by the async crypto + * interface when the sync mode is set to 'sync' and by + * the sync crypto interface regardless of setting. + * + * - async: the compression or decompression is submitted and returns + * immediately. Completion interrupts are not used so + * the caller is responsible for polling the descriptor + * for completion. This mode is applicable to only the + * async crypto interface and is ignored for anything + * else. + * + * - async_irq: the compression or decompression is submitted and + * returns immediately. Completion interrupts are + * enabled so the caller can wait for the completion and + * yield to other threads. When the compression or + * decompression completes, the completion is signaled + * and the caller awakened. This mode is applicable to + * only the async crypto interface and is ignored for + * anything else. + * + * These modes can be set using the iaa_crypto sync_mode driver + * attribute. + */ + +/* Use async mode */ +static bool async_mode; +/* Use interrupts */ +static bool use_irq; + +/** + * set_iaa_sync_mode - Set IAA sync mode + * @name: The name of the sync mode + * + * Make the IAA sync mode named @name the current sync mode used by + * compression/decompression. + */ + +static int set_iaa_sync_mode(const char *name) +{ + int ret = 0; + + if (sysfs_streq(name, "sync")) { + async_mode = false; + use_irq = false; + } else if (sysfs_streq(name, "async")) { + async_mode = true; + use_irq = false; + } else if (sysfs_streq(name, "async_irq")) { + async_mode = true; + use_irq = true; + } else { + ret = -EINVAL; + } + + return ret; +} + +static ssize_t sync_mode_show(struct device_driver *driver, char *buf) +{ + int ret = 0; + + if (!async_mode && !use_irq) + ret = sprintf(buf, "%s\n", "sync"); + else if (async_mode && !use_irq) + ret = sprintf(buf, "%s\n", "async"); + else if (async_mode && use_irq) + ret = sprintf(buf, "%s\n", "async_irq"); + + return ret; +} + +static ssize_t sync_mode_store(struct device_driver *driver, + const char *buf, size_t count) +{ + int ret = -EBUSY; + + mutex_lock(&iaa_devices_lock); + + if (iaa_crypto_enabled) + goto out; + + ret = set_iaa_sync_mode(buf); + if (ret == 0) + ret = count; +out: + mutex_unlock(&iaa_devices_lock); + + return ret; +} +static DRIVER_ATTR_RW(sync_mode); + static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; static int active_compression_mode; @@ -1033,6 +1129,72 @@ static inline int check_completion(struct device *dev, return ret; } +static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, + struct idxd_wq *wq, + dma_addr_t src_addr, unsigned int slen, + dma_addr_t dst_addr, unsigned int *dlen, + u32 compression_crc); + +static void iaa_desc_complete(struct idxd_desc *idxd_desc, + enum idxd_complete_type comp_type, + bool free_desc, void *__ctx, + u32 *status) +{ + struct crypto_ctx *ctx = __ctx; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + int ret, err = 0; + + iaa_wq = idxd_desc->wq->private_data; + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + dev_dbg(dev, "%s: compression mode %s," + " ctx->src_addr %llx, ctx->dst_addr %llx\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + ctx->src_addr, ctx->dst_addr); + + ret = check_completion(dev, idxd_desc->iax_completion, + ctx->compress, false); + if (ret) { + dev_dbg(dev, "%s: check_completion failed ret=%d\n", __func__, ret); + err = -EIO; + goto err; + } + + ctx->req->dlen = idxd_desc->iax_completion->output_size; + + if (ctx->compress && iaa_verify_compress) { + u32 compression_crc; + + compression_crc = idxd_desc->iax_completion->crc; + dma_sync_sg_for_device(dev, ctx->req->dst, 1, DMA_FROM_DEVICE); + dma_sync_sg_for_device(dev, ctx->req->src, 1, DMA_TO_DEVICE); + ret = iaa_compress_verify(ctx->tfm, ctx->req, iaa_wq->wq, ctx->src_addr, + ctx->req->slen, ctx->dst_addr, &ctx->req->dlen, + compression_crc); + if (ret) { + dev_dbg(dev, "%s: compress verify failed ret=%d\n", __func__, ret); + err = -EIO; + } + } +err: + if (ctx->req->base.complete) + acomp_request_complete(ctx->req, err); + + dma_unmap_sg(dev, ctx->req->dst, sg_nents(ctx->req->dst), DMA_FROM_DEVICE); + dma_unmap_sg(dev, ctx->req->src, sg_nents(ctx->req->src), DMA_TO_DEVICE); + + if (ret != 0) + dev_dbg(dev, "asynchronous compress failed ret=%d\n", ret); + + if (free_desc) + idxd_free_desc(idxd_desc->wq, idxd_desc); +} + static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_wq *wq, dma_addr_t src_addr, unsigned int slen, @@ -1075,6 +1237,22 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, desc->src2_size = sizeof(struct aecs_comp_table_record); desc->completion_addr = idxd_desc->compl_dma; + if (use_irq) { + desc->flags |= IDXD_OP_FLAG_RCI; + + idxd_desc->crypto.req = req; + idxd_desc->crypto.tfm = tfm; + idxd_desc->crypto.src_addr = src_addr; + idxd_desc->crypto.dst_addr = dst_addr; + idxd_desc->crypto.compress = true; + + dev_dbg(dev, "%s use_async_irq: compression mode %s," + " src_addr %llx, dst_addr %llx\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + src_addr, dst_addr); + } else if (async_mode && !disable_async) + req->base.data = idxd_desc; + dev_dbg(dev, "%s: compression mode %s," " desc->src1_addr %llx, desc->src1_size %d," " desc->dst_addr %llx, desc->max_dst_size %d," @@ -1089,6 +1267,12 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, goto err; } + if (async_mode && !disable_async) { + ret = -EINPROGRESS; + dev_dbg(dev, "%s: returning -EINPROGRESS\n", __func__); + goto out; + } + ret = check_completion(dev, idxd_desc->iax_completion, true, false); if (ret) { dev_dbg(dev, "check_completion failed ret=%d\n", ret); @@ -1099,7 +1283,8 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, *compression_crc = idxd_desc->iax_completion->crc; - idxd_free_desc(wq, idxd_desc); + if (!async_mode) + idxd_free_desc(wq, idxd_desc); out: return ret; err: @@ -1246,6 +1431,22 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, desc->flags |= IDXD_OP_FLAG_RD_SRC2_AECS; } + if (use_irq && !disable_async) { + desc->flags |= IDXD_OP_FLAG_RCI; + + idxd_desc->crypto.req = req; + idxd_desc->crypto.tfm = tfm; + idxd_desc->crypto.src_addr = src_addr; + idxd_desc->crypto.dst_addr = dst_addr; + idxd_desc->crypto.compress = false; + + dev_dbg(dev, "%s: use_async_irq compression mode %s," + " src_addr %llx, dst_addr %llx\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + src_addr, dst_addr); + } else if (async_mode && !disable_async) + req->base.data = idxd_desc; + dev_dbg(dev, "%s: decompression mode %s," " desc->src1_addr %llx, desc->src1_size %d," " desc->dst_addr %llx, desc->max_dst_size %d," @@ -1260,6 +1461,12 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, goto err; } + if (async_mode && !disable_async) { + ret = -EINPROGRESS; + dev_dbg(dev, "%s: returning -EINPROGRESS\n", __func__); + goto out; + } + ret = check_completion(dev, idxd_desc->iax_completion, false, false); if (ret) { dev_dbg(dev, "check_completion failed ret=%d\n", ret); @@ -1268,7 +1475,8 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, *dlen = idxd_desc->iax_completion->output_size; - idxd_free_desc(wq, idxd_desc); + if (!async_mode) + idxd_free_desc(wq, idxd_desc); out: return ret; err: @@ -1675,6 +1883,7 @@ static struct idxd_device_driver iaa_crypto_driver = { .remove = iaa_crypto_remove, .name = IDXD_SUBDRIVER_NAME, .type = dev_types, + .desc_complete = iaa_desc_complete, }; static int __init iaa_crypto_init_module(void) @@ -1691,6 +1900,13 @@ static int __init iaa_crypto_init_module(void) goto out; } + ret = driver_create_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); + if (ret) { + pr_debug("IAA sync mode attr creation failed\n"); + goto err_attr_create; + } + ret = driver_create_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); if (ret) { @@ -1723,6 +1939,8 @@ static int __init iaa_crypto_init_module(void) return ret; err_compression_mode: + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); driver_remove_file(&iaa_crypto_driver.drv, @@ -1735,6 +1953,8 @@ static int __init iaa_crypto_init_module(void) static void __exit iaa_crypto_cleanup_module(void) { + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); driver_remove_file(&iaa_crypto_driver.drv,