From patchwork Mon Mar 6 18:52:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65058 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2015805wrd; Mon, 6 Mar 2023 10:56:35 -0800 (PST) X-Google-Smtp-Source: AK7set/69sFtrYcsvlZJ1Hbm6mdQTfkhFzyLfPZkp8NcNzrtq18Bh31iJXg0cnzoES1jAoYIXjBa X-Received: by 2002:a05:6402:6d2:b0:4af:51b6:fe49 with SMTP id n18-20020a05640206d200b004af51b6fe49mr13233444edy.13.1678128995443; Mon, 06 Mar 2023 10:56:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678128995; cv=none; d=google.com; s=arc-20160816; b=ix0b+1W1473PZ4UA6TumP/RfJqYa257Hb2oZ3NHRzWJkCTzvzd5WmL476wSlbKYYMh Zs6Z1OcH8EXEl2Rm77Z617aB/NSKEjg7nZQMrExbjb9DrZqPqShVSouP7V3MUuTky3CY HXQIYR2Wg2QDOSwgZ3l2Tjmt5eLrxchV41oNxEkNOmJhJDttV/GbVz99pHX/k5diezLa VLM+T2AcgG6CwJt5nZwH/qqSp/GbZZQI0szIl17YxX/Ersrz2XuFbFDCj4Q0/mVfsjDH PIoXhC7EhtUceKyhTWJkTMwV7XVOas2/Scv2zG7KTaVHYEClSj7cbyQyKYvs+4DBvyia rCMw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=VpWdogxCA/pbRFVofBYTOnVD5fVbfE2/JUysW7G4nf0=; b=1CR30OnBEmnQpzndet3SGZbvKaiRqgHaP9mR+vmi0EGrFCtSYojdjNMQiRINTQWWgI q2DB9A3xcCOqiNaMwdP6M/CRtpPkdUbDJVXtPzcEpHRM7QCS0sPA1rB/VSYDSyeXL4qz l7MZ+4fbmiEjQQ9tZZZTqDQd0ETy/J5CfBX0AqSrrfCSmmjTx9GDqdGktdp0kOeIPfuK iR2LOE4yVIjmktgEBLODZlxIcnTdGvvqLL6rrYnNB3NAkwhysVMCr1WVrwpy4dSjWcpu Y1Rnd0+I4yZQPDvQylo/J7ly86Ksi4Bsn07YO9DJB9nWL7QXCIXuDzO4Q9iS0SPwLjlY gsXw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=N10jfWfu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ji17-20020a170907981100b008dd83608273si7381032ejc.370.2023.03.06.10.56.11; Mon, 06 Mar 2023 10:56:35 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=N10jfWfu; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230164AbjCFSwv (ORCPT + 99 others); Mon, 6 Mar 2023 13:52:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59918 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230315AbjCFSwk (ORCPT ); Mon, 6 Mar 2023 13:52:40 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F11741B4E; Mon, 6 Mar 2023 10:52:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128757; x=1709664757; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oowziMwXwVGyoGa9lxmkLd9ABSwpuSTfsdgiXE790Ps=; b=N10jfWfuvdc1zFQza1b6LM2f0RikZCsF9tzAH/vhd6CKAiVvuzge0BD5 fKkSOsBKiSYYj5dNM4hTv/VHafJAwPoAPRWhB0TbEhvjeV5N7dO0EqplN LrzemKLIkRXRiYx2XEduaWAJn8SiUF2QA+00qMgVEMjkUovd1BLRYuI8M Z2VuGbhiBvmBqwx9uABjj1yi10gLANHEDpDSV8OcvdXtVwCbHsWZj6/+U uCttXGXbWC8SX4HdcZS59RCgeXvm8hyH9/HDlPoqytaRE0pQbDUnKBfFh XtBA66wAenTjPUM2xGXTw7SriG8WEvbW7qhpet46ttGk5TNhETM5ehWRO w==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227656" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227656" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255730" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255730" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:32 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 01/16] dmaengine: idxd: add wq driver name support for accel-config user tool Date: Mon, 6 Mar 2023 12:52:11 -0600 Message-Id: <20230306185226.26483-2-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645789492059578?= X-GMAIL-MSGID: =?utf-8?q?1759645789492059578?= From: Dave Jiang With the possibility of multiple wq drivers that can be bound to the wq, the user config tool accel-config needs a way to know which wq driver to bind to the wq. Introduce per wq driver_name sysfs attribute where the user can indicate the driver to be bound to the wq. This allows accel-config to just bind to the driver using wq->driver_name. Signed-off-by: Dave Jiang Signed-off-by: Tom Zanussi --- .../ABI/stable/sysfs-driver-dma-idxd | 6 ++++ drivers/dma/idxd/cdev.c | 8 ++++++ drivers/dma/idxd/dma.c | 6 ++++ drivers/dma/idxd/idxd.h | 7 +++++ drivers/dma/idxd/sysfs.c | 28 +++++++++++++++++++ include/uapi/linux/idxd.h | 1 + 6 files changed, 56 insertions(+) diff --git a/Documentation/ABI/stable/sysfs-driver-dma-idxd b/Documentation/ABI/stable/sysfs-driver-dma-idxd index 3becc9a82bdf..e9a37e064193 100644 --- a/Documentation/ABI/stable/sysfs-driver-dma-idxd +++ b/Documentation/ABI/stable/sysfs-driver-dma-idxd @@ -244,6 +244,12 @@ Description: Shows the operation capability bits displayed in bitmap format correlates to the operations allowed. It's visible only on platforms that support the capability. +What: /sys/bus/dsa/devices/wq./driver_name +Date: Feb 23, 2023 +KernelVersion: 6.3.0 +Contact: dmaengine@vger.kernel.org +Description: Name of driver to be bounded to the wq. + What: /sys/bus/dsa/devices/engine./group_id Date: Oct 25, 2019 KernelVersion: 5.6.0 diff --git a/drivers/dma/idxd/cdev.c b/drivers/dma/idxd/cdev.c index 674bfefca088..f3d1604e753d 100644 --- a/drivers/dma/idxd/cdev.c +++ b/drivers/dma/idxd/cdev.c @@ -304,6 +304,7 @@ void idxd_wq_del_cdev(struct idxd_wq *wq) static int idxd_user_drv_probe(struct idxd_dev *idxd_dev) { + struct device *dev = &idxd_dev->conf_dev; struct idxd_wq *wq = idxd_dev_to_wq(idxd_dev); struct idxd_device *idxd = wq->idxd; int rc; @@ -330,6 +331,13 @@ static int idxd_user_drv_probe(struct idxd_dev *idxd_dev) } mutex_lock(&wq->wq_lock); + + if (!idxd_wq_driver_name_match(wq, dev)) { + idxd->cmd_status = IDXD_SCMD_WQ_NO_DRV_NAME; + rc = -ENODEV; + goto err; + } + wq->type = IDXD_WQT_USER; rc = drv_enable_wq(wq); if (rc < 0) diff --git a/drivers/dma/idxd/dma.c b/drivers/dma/idxd/dma.c index eb35ca313684..8bb7e7ff8d6a 100644 --- a/drivers/dma/idxd/dma.c +++ b/drivers/dma/idxd/dma.c @@ -305,6 +305,12 @@ static int idxd_dmaengine_drv_probe(struct idxd_dev *idxd_dev) return -ENXIO; mutex_lock(&wq->wq_lock); + if (!idxd_wq_driver_name_match(wq, dev)) { + idxd->cmd_status = IDXD_SCMD_WQ_NO_DRV_NAME; + rc = -ENODEV; + goto err; + } + wq->type = IDXD_WQT_KERNEL; rc = drv_enable_wq(wq); diff --git a/drivers/dma/idxd/idxd.h b/drivers/dma/idxd/idxd.h index 7ced8d283d98..505118fc19de 100644 --- a/drivers/dma/idxd/idxd.h +++ b/drivers/dma/idxd/idxd.h @@ -214,6 +214,8 @@ struct idxd_wq { char name[WQ_NAME_SIZE + 1]; u64 max_xfer_bytes; u32 max_batch_size; + + char driver_name[WQ_NAME_SIZE + 1]; }; struct idxd_engine { @@ -580,6 +582,11 @@ static inline void idxd_wqcfg_set_max_batch_shift(int idxd_type, union wqcfg *wq wqcfg->max_batch_shift = max_batch_shift; } +static inline int idxd_wq_driver_name_match(struct idxd_wq *wq, struct device *dev) +{ + return (strncmp(wq->driver_name, dev->driver->name, strlen(dev->driver->name)) == 0); +} + int __must_check __idxd_driver_register(struct idxd_device_driver *idxd_drv, struct module *module, const char *mod_name); #define idxd_driver_register(driver) \ diff --git a/drivers/dma/idxd/sysfs.c b/drivers/dma/idxd/sysfs.c index 18cd8151dee0..cb5864c98d5a 100644 --- a/drivers/dma/idxd/sysfs.c +++ b/drivers/dma/idxd/sysfs.c @@ -1224,6 +1224,33 @@ static ssize_t wq_op_config_store(struct device *dev, struct device_attribute *a static struct device_attribute dev_attr_wq_op_config = __ATTR(op_config, 0644, wq_op_config_show, wq_op_config_store); +static ssize_t wq_driver_name_show(struct device *dev, struct device_attribute *attr, char *buf) +{ + struct idxd_wq *wq = confdev_to_wq(dev); + + return sysfs_emit(buf, "%s\n", wq->driver_name); +} + +static ssize_t wq_driver_name_store(struct device *dev, struct device_attribute *attr, + const char *buf, size_t count) +{ + struct idxd_wq *wq = confdev_to_wq(dev); + + if (wq->state != IDXD_WQ_DISABLED) + return -EPERM; + + if (strlen(buf) > WQ_NAME_SIZE || strlen(buf) == 0) + return -EINVAL; + + memset(wq->driver_name, 0, WQ_NAME_SIZE + 1); + strncpy(wq->driver_name, buf, WQ_NAME_SIZE); + strreplace(wq->name, '\n', '\0'); + return count; +} + +static struct device_attribute dev_attr_wq_driver_name = + __ATTR(driver_name, 0644, wq_driver_name_show, wq_driver_name_store); + static struct attribute *idxd_wq_attributes[] = { &dev_attr_wq_clients.attr, &dev_attr_wq_state.attr, @@ -1242,6 +1269,7 @@ static struct attribute *idxd_wq_attributes[] = { &dev_attr_wq_occupancy.attr, &dev_attr_wq_enqcmds_retries.attr, &dev_attr_wq_op_config.attr, + &dev_attr_wq_driver_name.attr, NULL, }; diff --git a/include/uapi/linux/idxd.h b/include/uapi/linux/idxd.h index 1d553bedbdb5..2f85c8f59eb5 100644 --- a/include/uapi/linux/idxd.h +++ b/include/uapi/linux/idxd.h @@ -30,6 +30,7 @@ enum idxd_scmd_stat { IDXD_SCMD_WQ_NO_PRIV = 0x800f0000, IDXD_SCMD_WQ_IRQ_ERR = 0x80100000, IDXD_SCMD_WQ_USER_NO_IOMMU = 0x80110000, + IDXD_SCMD_WQ_NO_DRV_NAME = 0x80200000, }; #define IDXD_SCMD_SOFTERR_MASK 0x80000000 From patchwork Mon Mar 6 18:52:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65076 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2026699wrd; Mon, 6 Mar 2023 11:18:05 -0800 (PST) X-Google-Smtp-Source: AK7set+R1vFW3jkun80J0phkW43tGBnki0Evo0ZTJn0niYJtByrxtps4tjo0xpAvCwmZHmMlSGD8 X-Received: by 2002:a17:906:5e0c:b0:872:82d3:4162 with SMTP id n12-20020a1709065e0c00b0087282d34162mr11095827eju.44.1678130285578; Mon, 06 Mar 2023 11:18:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678130285; cv=none; d=google.com; s=arc-20160816; b=Ebe5E3S3qPSh+ePotIarf70EEQcipbK7WI5hnIGkn9wczfr7R/XonqpFulfc+0+TfH lKAl0wObZ7Ex4RiKrSY4kXs6/UHEkNEJnFwz4+5vodslGBGpyva8hF7TrxRprTE32A+I ucYEyvI1diISHKjC63K0a0ELrbyfWp+xnd6egIPwpieOh9ey3NskaI/2v4zvJG/Uvif/ LF3warkJ32DqQerWDh1pt/+M00B9YU7VjR8qdIjL1u+2LlokhN+OFGV8TzUhPB8XR6Yp p8pyyG+i/gnYe6gK86z5MRQSZgnURO5+aAktTs/ngJQp47sRbKpJKuRBgKTmSJbcMZ9x WrJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=EIVP1ZmNYGjAKT/9hLWJcUFkj7zr1dSvcBY/B+ThRrU=; b=SXgq/bx88a9xsyLJ2UblX/XWV0L6CnAdTvSZk7DBJD4Ex2WZuB/Z/OtWDbSpiBa1oe GtsPRlsfLz7qs8Lz2qJlI47N78ZoEFHmLbLKPj0vSrmVdEOCWkR38yANJCTem2PrSBWv /11fcQ+w2qGzYfasXsm3iBN12qVubkiFw5dsfQLXl2S4K4Ikgc+z4w0V6ss1bTu0kTAG kBV+4K83dJ2nFh0niFgGf/L8Gn0AQ+HoW21IMRQzVM0MYRiVK5icjNgWDGY3saLy204Z M3zqPzh6z/cgGm4Rs/c85rpaJ/725vMvYhO4wFHU2rSENfFhKdSBs25hjxTpsK/9+NiK uD+Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=NU2WFnRb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f12-20020a1709064dcc00b008b71cb4a5bcsi10595074ejw.218.2023.03.06.11.17.41; Mon, 06 Mar 2023 11:18:05 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=NU2WFnRb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230385AbjCFSwt (ORCPT + 99 others); Mon, 6 Mar 2023 13:52:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229958AbjCFSwk (ORCPT ); Mon, 6 Mar 2023 13:52:40 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4EF9D65115; Mon, 6 Mar 2023 10:52:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128757; x=1709664757; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kO38bfPMogxI1tGFhan6UK8YLGWBnl59qm86GfdzocM=; b=NU2WFnRbxnoTE6pLYc9VlwFTzNxpmCl2L/yd0bpuoXbDCadkc+U1bbPf gYaMJN/0hL/wKnVDYG5WL7AWqztuKxelb9dMTGoQiWufdv5qpTWxG/GSF n7XCSwfgrR4u7kva1kgxFJsSh26+/V7GEOOKTWmQcKyBU3M4A3us8u4pX QTu8JVfxTdsv0ZwJTkju6Bo4EiT77rThpERtylQiVZ82rZX54VgNoEjD2 ElA6eJpAW8ouIB3Jpenmv28R7Wn+gDRSxT6clPc1UhqZfeG7RqYSCbONu rmhQea2Srln7Au8f0IcBFMtCaGWdpKdY5NdbeM7h7TeFzcF4o4MAAWxLF Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227674" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227674" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255749" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255749" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:34 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 02/16] dmaengine: idxd: add external module driver support for dsa_bus_type Date: Mon, 6 Mar 2023 12:52:12 -0600 Message-Id: <20230306185226.26483-3-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759647142073029489?= X-GMAIL-MSGID: =?utf-8?q?1759647142073029489?= From: Dave Jiang Add support to allow an external driver to be registered to the dsa_bus_type and also auto-loaded. Signed-off-by: Dave Jiang Signed-off-by: Tom Zanussi --- drivers/dma/idxd/bus.c | 6 ++++++ drivers/dma/idxd/idxd.h | 3 +++ 2 files changed, 9 insertions(+) diff --git a/drivers/dma/idxd/bus.c b/drivers/dma/idxd/bus.c index 6f84621053c6..0c9e689a2e77 100644 --- a/drivers/dma/idxd/bus.c +++ b/drivers/dma/idxd/bus.c @@ -67,11 +67,17 @@ static void idxd_config_bus_remove(struct device *dev) idxd_drv->remove(idxd_dev); } +static int idxd_bus_uevent(const struct device *dev, struct kobj_uevent_env *env) +{ + return add_uevent_var(env, "MODALIAS=" IDXD_DEVICES_MODALIAS_FMT, 0); +} + struct bus_type dsa_bus_type = { .name = "dsa", .match = idxd_config_bus_match, .probe = idxd_config_bus_probe, .remove = idxd_config_bus_remove, + .uevent = idxd_bus_uevent, }; EXPORT_SYMBOL_GPL(dsa_bus_type); diff --git a/drivers/dma/idxd/idxd.h b/drivers/dma/idxd/idxd.h index 505118fc19de..719f9f1662ad 100644 --- a/drivers/dma/idxd/idxd.h +++ b/drivers/dma/idxd/idxd.h @@ -587,6 +587,9 @@ static inline int idxd_wq_driver_name_match(struct idxd_wq *wq, struct device *d return (strncmp(wq->driver_name, dev->driver->name, strlen(dev->driver->name)) == 0); } +#define MODULE_ALIAS_IDXD_DEVICE(type) MODULE_ALIAS("idxd:t" __stringify(type) "*") +#define IDXD_DEVICES_MODALIAS_FMT "idxd:t%d" + int __must_check __idxd_driver_register(struct idxd_device_driver *idxd_drv, struct module *module, const char *mod_name); #define idxd_driver_register(driver) \ From patchwork Mon Mar 6 18:52:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65059 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2015818wrd; Mon, 6 Mar 2023 10:56:39 -0800 (PST) X-Google-Smtp-Source: AK7set9U8AJGuM+f4N/2NTctdKT+BfucvNki1FjZe3kxCi/G8NpwljFPc0JoVqWo+YoCa4CWUQTC X-Received: by 2002:a17:906:4f94:b0:8ed:e8d6:42c4 with SMTP id o20-20020a1709064f9400b008ede8d642c4mr10911421eju.12.1678128999236; Mon, 06 Mar 2023 10:56:39 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678128999; cv=none; d=google.com; s=arc-20160816; b=Sgx45dZbFi2F6Av/kExwRnYcwga5XIl2r4eYbh1skud+xVInF7wdFeLlX+ZyS9CZir Tcx1/XIrsrkutAvCoIgFhNxYXIW0yWEFN/xmL3aB9bRl4fROJ46q+u5V/mjxReR5TdiE H3P1wlrdryGaPkPbFarGDsavMagVWISB8EjbF9JmSRzdHWezGP2/3zLK/1YrevcrHSAk 13UrFdQKkWTf9Pr6mmhDx1/mgsKbouxr26hHNfOVw9SsN39AGXQ0yguJWjl1KeaMaC0s dOJ/d9A858MsMTF5wFUE0IRLw8BgeTztqtljRnhZZHvkIxemTmJJTj10LfDPr2XvRFTT YRIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fc+w/n3fblmet3EH8DmyyeKV6USg/mM8CKwczWQMDMw=; b=z4W67kK7Wf38E2Sv0oSQFKYLNRm1uLRM1DRi/U0fsdH0KXWq1XTzVzZ4tGj/MI2NdB sttOyAE9Wm/e8rmuhOPTvK/YQCPNCBSyuMFGWmjd/PXPb0j0EYCnGNG6IUnb0bO1v5lA hDXlu8tVJz/CmltzxuYzojtWQ6TvCPQ9HhzZei7TibhGzTpGO6663Fe0ws5NP9neSK+J 3AYJUh1JWzzKmTFcmYRz5Su3ZcfGB6giCUdyUA0RL2i2yoUNnVJE/Aq04A4RFRH6P3n/ iLRkIzD3qbDjh1+3tq7bVFbu6mfvce6u9JX/UuVSwmll+BKuVgC2g22xpFQ5USFdbZ4R KrMQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=W91YTFjn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id mj3-20020a170906af8300b008d278c51d55si9080974ejb.805.2023.03.06.10.56.15; Mon, 06 Mar 2023 10:56:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=W91YTFjn; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230428AbjCFSw4 (ORCPT + 99 others); Mon, 6 Mar 2023 13:52:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229895AbjCFSwl (ORCPT ); Mon, 6 Mar 2023 13:52:41 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E0F56A05C; Mon, 6 Mar 2023 10:52:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128759; x=1709664759; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=OtM3rn2qVNEodUF6b7WwLHWyrC+y34N0YmCPeW0BQzE=; b=W91YTFjnHYD/cTQduA+KXpXaq3CMFHxZkWjXym/eokwl2KPBcdeqzQd0 gR1JrG321whGXwr1FY9FXweufScAjRsFeQ/eSnv5nPnAzMtpA23HTvipR MgNREecmbUJX3/3eOY9sgceKu0JS3/annAseNT9uMCfp0xGEvYh36j7Jz VTodgPa4zk8aZVWaIFJ3pWEotNI/ogxAtgos+AIzUpules1LXUtYN/N/A +6eBqorQlM7JZy2Hso9TzUBNOD4+l6ljMBxUW2Wm5Xzxs7+NVYP4V92Qz ZTGWtCAQbqXZkWpPSBhH8waU/cK4Qezv8P60eofWF875x6aelJhnDhBqM w==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227689" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227689" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:38 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255759" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255759" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:37 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 03/16] dmaengine: idxd: Export drv_enable/disable and related functions Date: Mon, 6 Mar 2023 12:52:13 -0600 Message-Id: <20230306185226.26483-4-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645793619905769?= X-GMAIL-MSGID: =?utf-8?q?1759645793619905769?= To allow idxd sub-drivers to enable and disable wqs, export them. Signed-off-by: Tom Zanussi --- drivers/dma/idxd/device.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/dma/idxd/device.c b/drivers/dma/idxd/device.c index 5f321f3b4242..af1aa88a66c1 100644 --- a/drivers/dma/idxd/device.c +++ b/drivers/dma/idxd/device.c @@ -1406,6 +1406,7 @@ int drv_enable_wq(struct idxd_wq *wq) err: return rc; } +EXPORT_SYMBOL_NS_GPL(drv_enable_wq, IDXD); void drv_disable_wq(struct idxd_wq *wq) { @@ -1427,6 +1428,7 @@ void drv_disable_wq(struct idxd_wq *wq) wq->type = IDXD_WQT_NONE; wq->client_count = 0; } +EXPORT_SYMBOL_NS_GPL(drv_disable_wq, IDXD); int idxd_device_drv_probe(struct idxd_dev *idxd_dev) { From patchwork Mon Mar 6 18:52:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65060 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2015849wrd; Mon, 6 Mar 2023 10:56:42 -0800 (PST) X-Google-Smtp-Source: AK7set8MfCe/CQTiVfaVs7zwYRobB8RCHb60LCLKFhSrr56gxx7gNCSKoQ8h10XH2IhdWk8je8a/ X-Received: by 2002:a17:906:80c1:b0:87b:d62c:d87 with SMTP id a1-20020a17090680c100b0087bd62c0d87mr11912806ejx.1.1678129002379; Mon, 06 Mar 2023 10:56:42 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678129002; cv=none; d=google.com; s=arc-20160816; b=q2WoyhuJbCABbixgCAR+EaFvPBTNTWQjHvVStVHPSX5TVFU8bDKG6WOzhrUl/TTHpF nnM/jEaPzuKlO7nKF8QZ6LUB/CgUEaE+Pz3Vvd61Qy4A678C8D24evn0T5F5Yc2roh1g ZOssGyjBd48Cg7A2GLQ84X9B9MdMRvoR+NdFBfAjTAQIdWZq7LMuFm50oni2vQMXnkKn Ta6AN9ueIIJXhQpBbDeFTCD9uHa4CVrNLBYlZVXao0bxcueWOTP0gYx6TgqN8hXqjoWX X+9jyv+0q2I/eSZuISmMGBi6y949OhfZCPOTiRIxsGlY1DT/n1HnJuD4JSsZTqqYc2r3 P78g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=80e71FBn8S9bF27KzNvgPbp+w47gEL/d2teCfylXU24=; b=AWCRCGQeSrYCGCzc6vxpzgM3dg/tWP1XjYYZIqQSazFqB+3U+i9BIZfXeft60QpKtJ 54cVVIPhERGM6ZGgGTPiC55aXmk5cyPVYkbJIcSzhVsV8/ZFwRrBIz2o2cV2zN1WHqZB o67hLm1UksQGmr11rB0jNJYPri99dgO4aAHkNHKSswU5xoKHxo1DbjmDr28SUTPpYMzr nU6WU6JdtjDB6Fmb6+/vj0HbPjoBPOhgxeTt6/JLG+CbzxXsWA2EjZskeoP/WeU6HPLN quSvffIl4L7SfVo0LT9LLzXqL7TDi8i7/V8/YubpQuac+tIBLG1uIsMgODL2Bb7Q0Mre w4bA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Lx1JWg02; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w6-20020a50fa86000000b004c2822f85cfsi11645361edr.506.2023.03.06.10.56.18; Mon, 06 Mar 2023 10:56:42 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Lx1JWg02; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229968AbjCFSw6 (ORCPT + 99 others); Mon, 6 Mar 2023 13:52:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230366AbjCFSws (ORCPT ); Mon, 6 Mar 2023 13:52:48 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5AADE72B21; Mon, 6 Mar 2023 10:52:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128761; x=1709664761; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=7/ydwUXLdHl1bpDjApixDXbPYE6quD+r8XqadnXkUFI=; b=Lx1JWg02HcS6libptapPeHYUKrVggeQchxnpHYtePCKx+872OrcdkZp2 aU5R9d7BvOvjyWug7Uk/39hH/yjbJkjz2vOa4MBiCjsBF7f0xV6ljVg5R MvqOelZ91guo46pa2GopCKlAHQryNw9hPNuZY6hp99JvChMp6M+E2q7mb MVpS0nL3CYi6RcD0Y4f0J35fZPR7uhT9DEYxE6l9oIrglQdy3seFQrI9T RHBb/JNsmC8Yhf7JPmQxuN900wRdm+5gZjV/iIU2OCONkJc1vaxiX3ZBh jusrdRfhsag0nGs3alNJbQ6UGQawLdmHWQbclTaMNZh2XuhoRasq7pLqK A==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227706" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227706" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:41 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255775" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255775" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:39 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 04/16] dmaengine: idxd: Export descriptor management functions Date: Mon, 6 Mar 2023 12:52:14 -0600 Message-Id: <20230306185226.26483-5-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645796788700515?= X-GMAIL-MSGID: =?utf-8?q?1759645796788700515?= To allow idxd sub-drivers to access the descriptor management functions, export them. Signed-off-by: Tom Zanussi --- drivers/dma/idxd/submit.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/dma/idxd/submit.c b/drivers/dma/idxd/submit.c index c01db23e3333..9d9ec0b76ccd 100644 --- a/drivers/dma/idxd/submit.c +++ b/drivers/dma/idxd/submit.c @@ -61,6 +61,7 @@ struct idxd_desc *idxd_alloc_desc(struct idxd_wq *wq, enum idxd_op_type optype) return __get_desc(wq, idx, cpu); } +EXPORT_SYMBOL_NS_GPL(idxd_alloc_desc, IDXD); void idxd_free_desc(struct idxd_wq *wq, struct idxd_desc *desc) { @@ -69,6 +70,7 @@ void idxd_free_desc(struct idxd_wq *wq, struct idxd_desc *desc) desc->cpu = -1; sbitmap_queue_clear(&wq->sbq, desc->id, cpu); } +EXPORT_SYMBOL_NS_GPL(idxd_free_desc, IDXD); static struct idxd_desc *list_abort_desc(struct idxd_wq *wq, struct idxd_irq_entry *ie, struct idxd_desc *desc) @@ -215,3 +217,4 @@ int idxd_submit_desc(struct idxd_wq *wq, struct idxd_desc *desc) percpu_ref_put(&wq->wq_active); return 0; } +EXPORT_SYMBOL_GPL(idxd_submit_desc); From patchwork Mon Mar 6 18:52:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65073 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2025322wrd; Mon, 6 Mar 2023 11:15:22 -0800 (PST) X-Google-Smtp-Source: AK7set+sTQ54CcY1qlLJsSy8op/1jfNkJRbwXM7gfu35We0zhOOcrhFJojucJ/Ic7YlDhQdpn0yY X-Received: by 2002:aa7:d910:0:b0:4ad:7c44:35fc with SMTP id a16-20020aa7d910000000b004ad7c4435fcmr10152407edr.6.1678130121978; Mon, 06 Mar 2023 11:15:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678130121; cv=none; d=google.com; s=arc-20160816; b=g4RHCI8Oy0fjPWr2/AfzhwxlgTIF49XdjkbulJCy30HeOpV2vE4xXfGnrMrMvT8Vjv qHegcv0ycv0EzbUZRBAIM4Gv/sWzf+hgAAwELqZbmT9rGP8ugfWDXKyJwnCWt3Qw6X19 Zr74x0fRewzJhnk8PL4LTdyMiV7mu2JxUb73S0h9JOQNr8yvtdMxBTugrzp2vWwnZcXz zB9UJz5Cgf94cMy19s2XMt6DbATBCuQ/7ppGgdSGjEhdO4rGMN2AnWGXKOOX4vRsFk3c cpJh/OfYCWV+ArA2WSD1Qj4KWffajPsfaYnJajP+WHNSCvZIZzXrMB0VdqFpcCF7YmZh g3hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=chbnLG3pwvAuAO4obUP7nQbKnfcQUjUxF9mXlvPd6uw=; b=IdTYOX9Z/gnPQaqtbNUBprbxWVtCpojH3ICquj6vP/88KZFNTk2EPfDHgU2EB17Pvp dABBGwV1Gw2V92+A0a6fawnJnG5FTBbYGzaVqPfn1+ZmNQwq4LQQM3kT43QNrx7W2HDk vQl/lQUM8VWjk1I9d303jrr4JL9xZRAqlCgIvq6aTGKdLe5NpNuaIg3CiogO270IktPM h7V3Tu498Z29UtgtsUEl1ti27mr+AoqpGPXo11DbP8hH6UK/mUrJeI4NPApkG+UXo+E+ 58w853u0yALQgmR75aOteWnEndf6G94EuhPhvn3uAoAanFOj2bPBqH9dxCyJFvqtou5B iaEg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YXX0ri3R; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i9-20020a50fc09000000b004bf644fd789si10868348edr.345.2023.03.06.11.14.58; Mon, 06 Mar 2023 11:15:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=YXX0ri3R; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229784AbjCFSxB (ORCPT + 99 others); Mon, 6 Mar 2023 13:53:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230382AbjCFSws (ORCPT ); Mon, 6 Mar 2023 13:52:48 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3066E7040B; Mon, 6 Mar 2023 10:52:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128764; x=1709664764; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=UWiyeSaMJRXIRpsgST4Bs27UAB5ID/L9rOXwzjeNkgk=; b=YXX0ri3Rc5tESxrxW0gkf9MjZAgkfAdNtOOfjzsqf35F96W4ZX+8T3Yb TtTVQWGxqDhQ8f2GEmowEJ2k2cWKkgG5hD3BPzRtT6qEQmGSZiTrOtm+M P+cnB21088GoCEE3zMDHY/nV3t8qWC7veYRPxApv7rXYxOCeq27uWORyV dPPIO/75i1BiLX1ilBOhtDiR+V1Zkqi8drTozgi9TrIo4V4+kKVPgE+ta /UQ7FkMRYJY5HpBv/gblyxgpeAT+LniuWusPKg34X8riDBZwtgsaUvf+H XLKUbNf70Jk7PsQnIavfKWOvkONkd9a9Qvpw+381RIAkfHVs23+WpbTBz w==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227719" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227719" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:43 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255788" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255788" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:41 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 05/16] dmaengine: idxd: Export wq resource management functions Date: Mon, 6 Mar 2023 12:52:15 -0600 Message-Id: <20230306185226.26483-6-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759646970963200144?= X-GMAIL-MSGID: =?utf-8?q?1759646970963200144?= To allow idxd sub-drivers to access the wq resource management functions, export them. Signed-off-by: Tom Zanussi --- drivers/dma/idxd/device.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/drivers/dma/idxd/device.c b/drivers/dma/idxd/device.c index af1aa88a66c1..7837d0a56314 100644 --- a/drivers/dma/idxd/device.c +++ b/drivers/dma/idxd/device.c @@ -161,6 +161,7 @@ int idxd_wq_alloc_resources(struct idxd_wq *wq) free_hw_descs(wq); return rc; } +EXPORT_SYMBOL_NS_GPL(idxd_wq_alloc_resources, IDXD); void idxd_wq_free_resources(struct idxd_wq *wq) { @@ -174,6 +175,7 @@ void idxd_wq_free_resources(struct idxd_wq *wq) dma_free_coherent(dev, wq->compls_size, wq->compls, wq->compls_addr); sbitmap_queue_free(&wq->sbq); } +EXPORT_SYMBOL_NS_GPL(idxd_wq_free_resources, IDXD); int idxd_wq_enable(struct idxd_wq *wq) { @@ -422,6 +424,7 @@ int idxd_wq_init_percpu_ref(struct idxd_wq *wq) reinit_completion(&wq->wq_resurrect); return 0; } +EXPORT_SYMBOL_NS_GPL(idxd_wq_init_percpu_ref, IDXD); void __idxd_wq_quiesce(struct idxd_wq *wq) { @@ -431,6 +434,7 @@ void __idxd_wq_quiesce(struct idxd_wq *wq) complete_all(&wq->wq_resurrect); wait_for_completion(&wq->wq_dead); } +EXPORT_SYMBOL_NS_GPL(__idxd_wq_quiesce, IDXD); void idxd_wq_quiesce(struct idxd_wq *wq) { @@ -438,6 +442,7 @@ void idxd_wq_quiesce(struct idxd_wq *wq) __idxd_wq_quiesce(wq); mutex_unlock(&wq->wq_lock); } +EXPORT_SYMBOL_NS_GPL(idxd_wq_quiesce, IDXD); /* Device control bits */ static inline bool idxd_is_enabled(struct idxd_device *idxd) From patchwork Mon Mar 6 18:52:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65071 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2023943wrd; Mon, 6 Mar 2023 11:12:40 -0800 (PST) X-Google-Smtp-Source: AK7set/VcV4CjBZ1F9Nq9x4oTeGWpu/x+n3bJxgeuZKYHvMTJ2zNaslTSPoZih1vm+Bh+xBsCXON X-Received: by 2002:a17:906:13d7:b0:881:a3ec:2b43 with SMTP id g23-20020a17090613d700b00881a3ec2b43mr9949522ejc.56.1678129960798; Mon, 06 Mar 2023 11:12:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678129960; cv=none; d=google.com; s=arc-20160816; b=U0lGtbIikrb+t2Yzru606z/G31wQXaOX1tDJSck/6XfKrfOFxAf1EUgMdnCD69LCkE gT08BGnfHsFkR4zxFyY22BJUMpErGiMRfNkOoZrT4kZmi0HzTcNs6NYZLAHUfLU7lk4C I1SGc1AYDVAmQVqkEiji7FP1vUpG7RuyvsT95zSbN8lHrZzSzdZU/kVJ5l3PBfv9CSYy oGqLUcB1jmYhNVEWHseRZq5xHJ4gwYLvSf7Y9geo8QfImZRO2LIPLivm5vKFKc1Q4ZlD Tms1SGakMZxU6tXDrIQSMNZsoVH2MSLBgN1Cpfi3V/5te8kc1yMxsYf9hTeFUjztceQw NDLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WWbzRMXMJ8cxAN3UhR8xWWH29mL7D3PHrgc/GscIiyo=; b=ebiPbCMh/FAhy1JfsfE0ZLLJ2qGAbv+eKLoFd/22u+TY1MbOGq5ucmzoT6IFSg0VS/ d8Ay2OXAqdCpNPA/j1X7tiha4B4peJkRFE06kQ6/k91uC+Prz4u0VBh5l3mz7gIQggO0 QTSxqPWjRSw3ek85m3vE5MU5ygdMUHEm5TuvnALXQ+t3MNb3KAgDg85oLaqLkOdgO3Hx nt9LBJppPZwBT7JF8Aks4AYClkDkNGGj0yNcTN4KITVyq5AmrK1ndif0As89HrfBOqsq m868hXMt0V0QXLFPAgPGMEcPk2Gbc+i9OeN2VftSz4sJ/Q0DEmXgsi9COjWSAy/Vuq+p Wm3A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=IGzNEBzs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id hc39-20020a17090716a700b008ddf3c18314si13868312ejc.372.2023.03.06.11.12.17; Mon, 06 Mar 2023 11:12:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=IGzNEBzs; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230225AbjCFSxV (ORCPT + 99 others); Mon, 6 Mar 2023 13:53:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230413AbjCFSwt (ORCPT ); Mon, 6 Mar 2023 13:52:49 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CAD766BDFA; Mon, 6 Mar 2023 10:52:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128765; x=1709664765; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=T24PujgQYZHqfnkCbqb1TyiF+r+EUpp8fid20iaYFLk=; b=IGzNEBzsssHXOGdryWx89dYwbWQhS0wK2r6eNvLUF7FpVt6WM1Z4Kebb BYUNkNexLu9e3LpO/jEIweFzVurXpIAxdZ3W35aOWZNkkxWqQBcF2al+C b8gTmRpyGjfXqo4SNcHop6BbHrIaPH3q7mPVjnpvvsdc4E2rPwnbi15IP wGTGuz5H5rBetyir/dSgKGfVyXnzZJYXkWo1xlzlDJHBOSjqgXXnS3udL hI4gzqgfZ0ahmmu17Oz2FWGYAM8LE5VQ6AuN2WE25ULgGjqN2WA+anW+W qwgLC4sfOHWXyVq9mTqmOB2K12/A/6kSkaCzvwBY/gU5uyPn5vwkgYhnN Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227727" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227727" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:45 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255798" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255798" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:44 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 06/16] dmaengine: idxd: Add private_data to struct idxd_wq Date: Mon, 6 Mar 2023 12:52:16 -0600 Message-Id: <20230306185226.26483-7-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759646801550520236?= X-GMAIL-MSGID: =?utf-8?q?1759646801550520236?= Add a void * to idxd_wqs for user-defined context data. Signed-off-by: Tom Zanussi --- drivers/dma/idxd/idxd.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/dma/idxd/idxd.h b/drivers/dma/idxd/idxd.h index 719f9f1662ad..e55213b5153e 100644 --- a/drivers/dma/idxd/idxd.h +++ b/drivers/dma/idxd/idxd.h @@ -216,6 +216,8 @@ struct idxd_wq { u32 max_batch_size; char driver_name[WQ_NAME_SIZE + 1]; + + void *private_data; }; struct idxd_engine { From patchwork Mon Mar 6 18:52:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65069 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2016787wrd; Mon, 6 Mar 2023 10:58:56 -0800 (PST) X-Google-Smtp-Source: AK7set+uy1ytdw6Pygjj/0OEtMnwklXaRLFtZMazBlTtFr15gKSu5s2fxtB3AVzc5GJ2hv5XCEQu X-Received: by 2002:a17:907:888b:b0:877:573d:e919 with SMTP id rp11-20020a170907888b00b00877573de919mr11558626ejc.20.1678129135961; Mon, 06 Mar 2023 10:58:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678129135; cv=none; d=google.com; s=arc-20160816; b=mWpYKT3XUEzsUxiO9jAYiDykwLdMI1+7UKqWCmUT7DKmXqpZeQNCxWpbFXuTWIys8X 0IXwt3rjNRTdvb/ecMpUHEdih+KOPe9TQ6ma+K26tkGQx4Jr8+uk/Kp1n1urON5C1/lp n+P6qfwZfr4mx9UIjTMxfq5DwN6iWSLUN7fD06AI7aKyUVWL6urp+UqRIWLdNpTmQgIH D3sSIV+ydD23NkHWVjywZJGHK/HWAWEzp+7jlbscm4wS1FqHUeCQfNl763bDrznFP2Oq 6Pi/GhRr0Oc/GojfqXKmVfme+RYUPpl5AAukfo0or1aYUQiqnxz6SB1cvM0l54Q7p+hg TljA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=s11+mhtW1nouN7zTeMxURD12edwrYgg6KWwo18kIhrc=; b=U/Y4H+FRHAmcIcz8JGjQfmpMof3YrQAJRgEW8ZiY4CrePaNsISr2Rmdx/x1RtwrIYZ UycJAiJiGhFf3f9P7RaOyShinagbwfPdAIiBGe5USU2ioEWIBIj/4LmHmyedyauo8lf4 lREMrChAlnrIb8K4aKrUkalNTFCQk9sQEoWYKCHvfJ1A53egpVzaDwYZEPW1Fu4pn5ki FozYxEhEOEV92e/0kfTZHXf6Jh9Lkmfmdi5EUoTbHgD77CUdk734paDQHrgZ7OrDLlfw nLt+WmZY7ALqcB1nhYtyMJ6D1llsAooPpg9MyzqyekPjGZOtxKj5sBffGzwQqn1gxVWR 6mPw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=P9C4bzTX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id xc2-20020a170907074200b009140782d1fasi1169129ejb.410.2023.03.06.10.58.32; Mon, 06 Mar 2023 10:58:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=P9C4bzTX; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229922AbjCFSx3 (ORCPT + 99 others); Mon, 6 Mar 2023 13:53:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230270AbjCFSwx (ORCPT ); Mon, 6 Mar 2023 13:52:53 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1318E65115; Mon, 6 Mar 2023 10:52:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128768; x=1709664768; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KqSKXDDfqjmEx9XDqpAUTxuP4dSo8DvMvYWaPrJ7Sw0=; b=P9C4bzTXEPb1SuhiS4ReK1mBMLH71IvJJnN1pFsOFcdosIrEV4ATLikg JQoVqL9rnHcFc1tguxCt3Mh2sm1uT+kpwj59vAKBmdVeJACXl93O7s0+F OAA/+NTXv4gjFERJ0sC0D8/NbNK9oa9L3xRkzK/8DH1fZaYfN4sTTboA9 A5VJfnsWGViz43SOzJlXZJkaTrrff9u9mA6+kLQrK48IKZXb21qQGfxi5 sRoISVy3FSMRWWqIfA6kqoBAHGNZiWRWiGQ62zVYNy/uwcA3t+kBN7N4C 4wyhJD2O87Jic4xfOsd6ZpbYXTrm5EJ2uL+3rAK676bU4wbI4Y8r99S+o w==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227734" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227734" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:47 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255815" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255815" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:46 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 07/16] dmaengine: idxd: add callback support for iaa crypto Date: Mon, 6 Mar 2023 12:52:17 -0600 Message-Id: <20230306185226.26483-8-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645936557789457?= X-GMAIL-MSGID: =?utf-8?q?1759645936557789457?= Create a lightweight callback interface to allow idxd sub-drivers to be notified when work sent to idxd wqs has completed. For a sub-driver to be notified of work completion, it needs to: - Set the descriptor's 'Request Completion Interrupt' (IDXD_OP_FLAG_RCI) - Set the sub-driver desc_complete() callback when registering the sub-driver e.g.: struct idxd_device_driver my_drv = { .probe = my_probe, .desc_complete = my_complete, } - Set the sub-driver-specific context in the sub-driver's descriptor e.g: idxd_desc->crypto.req = req; idxd_desc->crypto.tfm = tfm; idxd_desc->crypto.src_addr = src_addr; idxd_desc->crypto.dst_addr = dst_addr; When the work completes and the completion irq fires, idxd will invoke the desc_complete() callback with pointers to the descriptor, context, and completion_type. Signed-off-by: Dave Jiang Signed-off-by: Tom Zanussi --- drivers/dma/idxd/device.c | 2 +- drivers/dma/idxd/dma.c | 3 +- drivers/dma/idxd/idxd.h | 62 ++++++++++++++++++++++++++++++++------- drivers/dma/idxd/irq.c | 12 ++++---- drivers/dma/idxd/submit.c | 6 ++-- 5 files changed, 65 insertions(+), 20 deletions(-) diff --git a/drivers/dma/idxd/device.c b/drivers/dma/idxd/device.c index 7837d0a56314..636451249046 100644 --- a/drivers/dma/idxd/device.c +++ b/drivers/dma/idxd/device.c @@ -1190,7 +1190,7 @@ static void idxd_flush_pending_descs(struct idxd_irq_entry *ie) tx = &desc->txd; tx->callback = NULL; tx->callback_result = NULL; - idxd_dma_complete_txd(desc, ctype, true); + idxd_dma_complete_txd(desc, ctype, true, NULL, NULL); } } diff --git a/drivers/dma/idxd/dma.c b/drivers/dma/idxd/dma.c index 8bb7e7ff8d6a..9a12b02b0549 100644 --- a/drivers/dma/idxd/dma.c +++ b/drivers/dma/idxd/dma.c @@ -22,7 +22,7 @@ static inline struct idxd_wq *to_idxd_wq(struct dma_chan *c) void idxd_dma_complete_txd(struct idxd_desc *desc, enum idxd_complete_type comp_type, - bool free_desc) + bool free_desc, void *ctx, u32 *status) { struct idxd_device *idxd = desc->wq->idxd; struct dma_async_tx_descriptor *tx; @@ -358,6 +358,7 @@ static enum idxd_dev_type dev_types[] = { struct idxd_device_driver idxd_dmaengine_drv = { .probe = idxd_dmaengine_drv_probe, .remove = idxd_dmaengine_drv_remove, + .desc_complete = idxd_dma_complete_txd, .name = "dmaengine", .type = dev_types, }; diff --git a/drivers/dma/idxd/idxd.h b/drivers/dma/idxd/idxd.h index e55213b5153e..9b8d7043e79c 100644 --- a/drivers/dma/idxd/idxd.h +++ b/drivers/dma/idxd/idxd.h @@ -13,6 +13,7 @@ #include #include #include +#include #include #include "registers.h" @@ -56,11 +57,23 @@ enum idxd_type { #define IDXD_ENQCMDS_RETRIES 32 #define IDXD_ENQCMDS_MAX_RETRIES 64 +enum idxd_complete_type { + IDXD_COMPLETE_NORMAL = 0, + IDXD_COMPLETE_ABORT, + IDXD_COMPLETE_DEV_FAIL, +}; + +struct idxd_desc; + struct idxd_device_driver { const char *name; enum idxd_dev_type *type; int (*probe)(struct idxd_dev *idxd_dev); void (*remove)(struct idxd_dev *idxd_dev); + void (*desc_complete)(struct idxd_desc *desc, + enum idxd_complete_type comp_type, + bool free_desc, + void *ctx, u32 *status); struct device_driver drv; }; @@ -164,12 +177,6 @@ enum idxd_op_type { IDXD_OP_NONBLOCK = 1, }; -enum idxd_complete_type { - IDXD_COMPLETE_NORMAL = 0, - IDXD_COMPLETE_ABORT, - IDXD_COMPLETE_DEV_FAIL, -}; - struct idxd_dma_chan { struct dma_chan chan; struct idxd_wq *wq; @@ -322,6 +329,14 @@ struct idxd_device { unsigned long *opcap_bmap; }; +struct crypto_ctx { + struct acomp_req *req; + struct crypto_tfm *tfm; + dma_addr_t src_addr; + dma_addr_t dst_addr; + bool compress; +}; + /* IDXD software descriptor */ struct idxd_desc { union { @@ -334,7 +349,10 @@ struct idxd_desc { struct iax_completion_record *iax_completion; }; dma_addr_t compl_dma; - struct dma_async_tx_descriptor txd; + union { + struct dma_async_tx_descriptor txd; + struct crypto_ctx crypto; + }; struct llist_node llnode; struct list_head list; int id; @@ -360,6 +378,15 @@ enum idxd_completion_status { #define idxd_dev_to_idxd(idxd_dev) container_of(idxd_dev, struct idxd_device, idxd_dev) #define idxd_dev_to_wq(idxd_dev) container_of(idxd_dev, struct idxd_wq, idxd_dev) +static inline struct idxd_device_driver *wq_to_idxd_drv(struct idxd_wq *wq) +{ + struct device *dev = wq_confdev(wq); + struct idxd_device_driver *idxd_drv = + container_of(dev->driver, struct idxd_device_driver, drv); + + return idxd_drv; +} + static inline struct idxd_device *confdev_to_idxd(struct device *dev) { struct idxd_dev *idxd_dev = confdev_to_idxd_dev(dev); @@ -602,6 +629,24 @@ void idxd_driver_unregister(struct idxd_device_driver *idxd_drv); #define module_idxd_driver(__idxd_driver) \ module_driver(__idxd_driver, idxd_driver_register, idxd_driver_unregister) +void idxd_free_desc(struct idxd_wq *wq, struct idxd_desc *desc); +void idxd_dma_complete_txd(struct idxd_desc *desc, + enum idxd_complete_type comp_type, + bool free_desc, void *ctx, u32 *status); + +static inline void idxd_desc_complete(struct idxd_desc *desc, + enum idxd_complete_type comp_type, + bool free_desc) +{ + struct idxd_device_driver *drv; + u32 status; + + drv = wq_to_idxd_drv(desc->wq); + if (drv->desc_complete) + drv->desc_complete(desc, comp_type, free_desc, + &desc->txd, &status); +} + int idxd_register_bus_type(void); void idxd_unregister_bus_type(void); int idxd_register_devices(struct idxd_device *idxd); @@ -658,15 +703,12 @@ int idxd_wq_request_irq(struct idxd_wq *wq); /* submission */ int idxd_submit_desc(struct idxd_wq *wq, struct idxd_desc *desc); struct idxd_desc *idxd_alloc_desc(struct idxd_wq *wq, enum idxd_op_type optype); -void idxd_free_desc(struct idxd_wq *wq, struct idxd_desc *desc); int idxd_enqcmds(struct idxd_wq *wq, void __iomem *portal, const void *desc); /* dmaengine */ int idxd_register_dma_device(struct idxd_device *idxd); void idxd_unregister_dma_device(struct idxd_device *idxd); void idxd_parse_completion_status(u8 status, enum dmaengine_tx_result *res); -void idxd_dma_complete_txd(struct idxd_desc *desc, - enum idxd_complete_type comp_type, bool free_desc); /* cdev */ int idxd_cdev_register(void); diff --git a/drivers/dma/idxd/irq.c b/drivers/dma/idxd/irq.c index aa314ebec587..3f89236c97c1 100644 --- a/drivers/dma/idxd/irq.c +++ b/drivers/dma/idxd/irq.c @@ -121,7 +121,7 @@ static void idxd_abort_invalid_int_handle_descs(struct idxd_irq_entry *ie) list_for_each_entry_safe(d, t, &flist, list) { list_del(&d->list); - idxd_dma_complete_txd(d, IDXD_COMPLETE_ABORT, true); + idxd_desc_complete(d, IDXD_COMPLETE_ABORT, true); } } @@ -376,7 +376,7 @@ static void idxd_int_handle_resubmit_work(struct work_struct *work) */ if (rc != -EAGAIN) { desc->completion->status = IDXD_COMP_DESC_ABORT; - idxd_dma_complete_txd(desc, IDXD_COMPLETE_ABORT, false); + idxd_desc_complete(desc, IDXD_COMPLETE_ABORT, false); } idxd_free_desc(wq, desc); } @@ -417,11 +417,11 @@ static void irq_process_pending_llist(struct idxd_irq_entry *irq_entry) * and 0xff, which DSA_COMP_STATUS_MASK can mask out. */ if (unlikely(desc->completion->status == IDXD_COMP_DESC_ABORT)) { - idxd_dma_complete_txd(desc, IDXD_COMPLETE_ABORT, true); + idxd_desc_complete(desc, IDXD_COMPLETE_ABORT, true); continue; } - idxd_dma_complete_txd(desc, IDXD_COMPLETE_NORMAL, true); + idxd_desc_complete(desc, IDXD_COMPLETE_NORMAL, true); } else { spin_lock(&irq_entry->list_lock); list_add_tail(&desc->list, @@ -460,11 +460,11 @@ static void irq_process_work_list(struct idxd_irq_entry *irq_entry) * and 0xff, which DSA_COMP_STATUS_MASK can mask out. */ if (unlikely(desc->completion->status == IDXD_COMP_DESC_ABORT)) { - idxd_dma_complete_txd(desc, IDXD_COMPLETE_ABORT, true); + idxd_desc_complete(desc, IDXD_COMPLETE_ABORT, true); continue; } - idxd_dma_complete_txd(desc, IDXD_COMPLETE_NORMAL, true); + idxd_desc_complete(desc, IDXD_COMPLETE_NORMAL, true); } } diff --git a/drivers/dma/idxd/submit.c b/drivers/dma/idxd/submit.c index 9d9ec0b76ccd..7ab37c349b2f 100644 --- a/drivers/dma/idxd/submit.c +++ b/drivers/dma/idxd/submit.c @@ -127,7 +127,8 @@ static void llist_abort_desc(struct idxd_wq *wq, struct idxd_irq_entry *ie, spin_unlock(&ie->list_lock); if (found) - idxd_dma_complete_txd(found, IDXD_COMPLETE_ABORT, false); + idxd_dma_complete_txd(found, IDXD_COMPLETE_ABORT, false, + NULL, NULL); /* * completing the descriptor will return desc to allocator and @@ -137,7 +138,8 @@ static void llist_abort_desc(struct idxd_wq *wq, struct idxd_irq_entry *ie, */ list_for_each_entry_safe(d, t, &flist, list) { list_del_init(&d->list); - idxd_dma_complete_txd(found, IDXD_COMPLETE_ABORT, true); + idxd_dma_complete_txd(found, IDXD_COMPLETE_ABORT, true, + NULL, NULL); } } From patchwork Mon Mar 6 18:52:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65074 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2026635wrd; Mon, 6 Mar 2023 11:17:55 -0800 (PST) X-Google-Smtp-Source: AK7set+rChIRIwSX52K4FDm2gwlpq5au1RRsHqJc5T/69XAFQcBURPFU+Pa5z+F4qcRw0jyn0r7B X-Received: by 2002:a17:907:204c:b0:8b1:7e1a:ef22 with SMTP id pg12-20020a170907204c00b008b17e1aef22mr10961620ejb.45.1678130275413; Mon, 06 Mar 2023 11:17:55 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678130275; cv=none; d=google.com; s=arc-20160816; b=qJidAQ4i43uf0SVkkEPjiK5gSbGRtZH5BXrCqbMPZWjLvQJGHz0xV8tvxXy5Ars5eC EZ8Vo92w2bHDO+uw8UH7a5Xnvi+sqCrO0uqgiZWBoREMhnA+llvdEWgeHgiVoe6NePZV HE+fLX8dwFW6A2aRkqzk9HD+UidwbIm6Dvfgu5t7eHgefdCS2zstTF+8iXVyaQNwPsd8 whSqLwVz/J3LbaN+khkDNcmHs3kyl9talsADiUiSYTdJIZRXKrJmgCVrUkey/2Soq9yY PHbPvfvp+XcQQzBvueogWtV3v7zMyyjX+FS+zS7SzyIW2Iz1WEZMSY1J55hdAjtA5evU IY9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=vXysz9SvojmVk7PZtIZk4GPtZxT2hDGUtvhUyC8rHAw=; b=qudSj0j0Qi246pySLDBk9BvBlyBPcLN+eBBW0omB1Ba3UfjWGaJ22LW4kRLWNU6GW9 TyhL8SeMwywRxCIbT2M5vxROuU4RqeUlllC9XWXLdRzqh4sEkdwaVYD9NGJCKsBfTiaZ TQr8PxTmxdccfoRHvKzA6QdzrTdIHrcz3yLXvgFhkGXDTziLGfn74DwoMKy66K8rt/qC q0uKeCc46e+X/wqTGQtIIWYjcs2MI49ealRC694NGkeisgfZqqkU9voKk/tYQIZRhoM7 OD5DmFq+A5xOsnqmFcYd2E37ikfEdPJbCKGY6RoRf918IVwEQD1AjZmSC6+luyMct+49 xlZw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=THmCv4w3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ga22-20020a1709070c1600b008caedbfc3b0si1161900ejc.755.2023.03.06.11.17.32; Mon, 06 Mar 2023 11:17:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=THmCv4w3; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230344AbjCFSxl (ORCPT + 99 others); Mon, 6 Mar 2023 13:53:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230118AbjCFSxT (ORCPT ); Mon, 6 Mar 2023 13:53:19 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E611572B29; Mon, 6 Mar 2023 10:52:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128770; x=1709664770; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qz/Y7pzaDWJSk/YJ+9MShHfc55gV8C2hlMea4CKzUnM=; b=THmCv4w3nMrlnJzpZrHGJ3/F2m4GMyLjfQQIpJMQiHHuNPVZJgD5cAvP CzOjWk/xK7tssgknUh6y9tHSzk+OZmopN7wOeoMLiO4a244FtbQsfQ2ma 2Ug7IZxyKCzt036yZjKBdwwkqb+47OYNm34dlO8zLnsQw1LHaAvW9f9ur h+v57XGl6A3mTEbY2In+4rZ8wNH+9VEm/h0NoRBMf6vGHIYWmkDUkggx7 5R/Uid2A0Xq4yQsptdcplyo1MiVbC+onfbmYBHycPvQ+lJP3lFmmWh70S sCWGZ+st1p/IRH2/7nbcjoWBLdtNJyn/SWrruBaMXsYuBhoZPR6fpAIyx g==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227740" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227740" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:50 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255823" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255823" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:48 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 08/16] crypto: iaa - Add IAA Compression Accelerator Documentation Date: Mon, 6 Mar 2023 12:52:18 -0600 Message-Id: <20230306185226.26483-9-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759647131851708325?= X-GMAIL-MSGID: =?utf-8?q?1759647131851708325?= Because the IAA Compression Accelerator requires significant user setup in order to be used properly, this adds documentation on the iaa_crypto driver including setup, usage, and examples. Signed-off-by: Tom Zanussi --- .../driver-api/crypto/iaa/iaa-crypto.rst | 743 ++++++++++++++++++ Documentation/driver-api/crypto/iaa/index.rst | 20 + Documentation/driver-api/crypto/index.rst | 20 + Documentation/driver-api/index.rst | 1 + 4 files changed, 784 insertions(+) create mode 100644 Documentation/driver-api/crypto/iaa/iaa-crypto.rst create mode 100644 Documentation/driver-api/crypto/iaa/index.rst create mode 100644 Documentation/driver-api/crypto/index.rst diff --git a/Documentation/driver-api/crypto/iaa/iaa-crypto.rst b/Documentation/driver-api/crypto/iaa/iaa-crypto.rst new file mode 100644 index 000000000000..13808700bf27 --- /dev/null +++ b/Documentation/driver-api/crypto/iaa/iaa-crypto.rst @@ -0,0 +1,743 @@ +.. SPDX-License-Identifier: GPL-2.0 + +========================================= +IAA Compression Accelerator Crypto Driver +========================================= + +Tom Zanussi + +The IAA crypto driver supports compression/decompression compatible +with the DEFLATE compression standard described in RFC 1951, which is +the compression/decompression algorithm exported by this module. + +The IAA hardware spec can be found here: + + https://cdrdv2.intel.com/v1/dl/getContent/721858 + +The iaa_crypto driver is designed to work as a layer underneath +higher-level compression devices such as zswap and zram. + +Users can select IAA compress/decompress acceleration by specifying +'iaa_crypto' as the compression algorithm to use by whatever facility +allows compression algorithms to be selected. + +For example, a zram drive can select iaa_crypto via:: + + # echo iaa_crypto > /sys/block/zram1/comp_algorithm + +Similarly for zswap:: + + # echo iaa_crypto > /sys/module/zswap/parameters/compressor + + +Config options and other setup +============================== + +The IAA crypto driver is available via menuconfig using the following +path:: + + Cryptographic API -> Hardware crypto devices -> Support for Intel(R) IAA Compression Accelerator + +In the configuration file the option called CONFIG_CRYPTO_DEV_IAA_CRYPTO. + +The IAA crypto driver also supports statistics, which are available +via menuconfig using the following path:: + + Cryptographic API -> Hardware crypto devices -> Support for Intel(R) IAA Compression -> Enable Intel(R) IAA Compression Accelerator Statistics + +In the configuration file the option called CONFIG_CRYPTO_DEV_IAA_CRYPTO_STATS. + +The following config options should also be enabled:: + + CONFIG_IRQ_REMAP=y + CONFIG_INTEL_IOMMU=y + CONFIG_INTEL_IOMMU_SVM=y + CONFIG_PCI_ATS=y + CONFIG_PCI_PRI=y + CONFIG_PCI_PASID=y + CONFIG_INTEL_IDXD=m + CONFIG_INTEL_IDXD_SVM=y + +IAA is one of the first Intel accelerator IPs that can work in +conjunction with the Intel IOMMU. There are multiple modes that exist +for testing. Based on IOMMU configuration, there are 3 modes:: + + - Scalable + - Legacy + - No IOMMU + + +Scalable mode +------------- + +Scalable mode supports Shared Virtual Memory (SVM or SVA). It is +entered when using the kernel boot commandline:: + + intel_iommu=on,sm_on + +with VT-d turned on in BIOS. + +With scalable mode, both shared and dedicated workqueues are available +for use. + +For scalable mode, the following BIOS settings should be enabled:: + + Socket Configuration > IIO Configuration > Intel VT for Directed I/O (VT-d) > Intel VT for Directed I/O + + Socket Configuration > IIO Configuration > PCIe ENQCMD > ENQCMDS + + +Legacy mode +----------- + +Legacy mode is entered when using the kernel boot commandline:: + + intel_iommu=off, intel_iommu=on + +or VT-d is not turned on in BIOS. + +If you have booted into Linux and not sure if VT-d is on, do a "dmesg +| grep -i dmar". If you don't see a number of DMAR devices enumerated, +most likely VT-d is not on. + +With legacy mode, only dedicated workqueues are available for use. + + +No IOMMU mode +------------- + +No IOMMU mode is entered when using the kernel boot commandline:: + + iommu=off. + +With no IOMMU mode, only dedicated workqueues are available for use. + + +Usage +===== + +accel-config +------------ + +Unlike typical drivers, the iaa_crypto driver does not enable the +device on driver load. Due to complexity and configurability of the +accelerator devices, it was a design decision to have the user +configure the device and manually enable the desired devices and +workqueues. + +The userspace tool to help doing that is called accel-config. Using +accel-config to configure device or loading a previously saved config +is highly recommended. The device can be controlled via sysfs +directly but comes with the warning that do this ONLY if you know +exactly what you are doing. This document will not cover the sysfs +interface but assumes you will be using accel-config. + +The accel-config tool along with instructions for building it can be +found here: + + https://github.com/intel/idxd-config/#readme + + +Typical usage +------------- + +In order for the iaa_crypto module to actually do any +compression/decompression work on behalf of a facility, one or more +IAA workqueues need to be bound to the iaa_crypto driver. + +For instance, here's an example of configuring an IAA workqueue and +binding it to the iaa_crypto driver (note that device names are +specified as 'iax' rather than 'iaa' - this is because upstream still +has the old 'iax' device naming in place) :: + + # configure wq1.0 + + accel-config config-wq --group-id=0 --mode=shared --type=kernel --name="iaa_crypto" --device_name="crypto" iax1/wq1.0 + + # enable IAA device iax1 + + accel-config enable-device iax1 + + # enable wq1.0 on IAX device iax1 + + accel-config enable-wq iax1/wq1.0 + +Whenever a new workqueue is bound to or unbound from the iaa_crypto +driver, the available workqueues are 'rebalanced' such that work +submitted from a particular CPU is given to the most appropriate +workqueue available. Current best practice is to configure and bind +at least one workqueue for each IAA device, but as long as there is at +least one workqueue configured and bound to any IAA device in the +system, the iaa_crypto driver will work, albeit most likely not as +efficiently. + +The IAA crypto algorigthms is operational and compression and +decompression operations are fully enabled following the successful +binding of the first IAA workqueue to the iaa_crypto driver, and +similarly, the IAA crypto algorithm is automatically unregistered when +there are no IAA workqueues bound to the driver, following their +removal. + +As a result, the iaa_crypto crypto algorithm and thus the IAA hardware +are only available when one or more workques are bound to the +iaa_crypto driver. + +Driver attributes +----------------- + +There are a few user-configurable driver attributes that can be used +to configure various modes of operation. They're listed below, along +with their default values. To set any of these attributes, echo the +appropriate values to the attribute file located under +/sys/bus/dsa/drivers/crypto/ + + - verify_compress + + Toggle compression verification. If set, each compress will be + internally decompressed and the contents verified, returning error + codes if unsuccessful. This can be toggled with 0/1: + + echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress + + The default setting is '1' - verify all compresses. + + - compression_mode + + Select compression mode to be used by all compresses and + decompresses. There are a number ofcompression modes available, + each identified by a unique string. These can be selected by + echoing the string to the 'compression_mode' driver attribute. + Currently, there are only two compression modes available, + 'canned' and 'fixed' modes. + + The 'fixed' compression mode implements the compression scheme + specified by RFC 1951. + + The 'canned' compression mode implements a good general-purpose + compression scheme whose tables were generated from statistics + derived from a wide variety of SPEC17 workloads. It provides much + better overall characteristics than the existing deflate-1951 + tables implemented by 'fixed'. + + Either 'fixed' or 'canned' modes can be chosen as the mode to be used + for compression/decompression via the iaa_crypto compression_mode + iaa_crypto driver attribute: + + echo "canned" > /sys/bus/dsa/drivers/crypto/compression_mode + + The default setting is 'fixed'. + + - sync_mode + + Select mode to be used to wait for completion of each compresses + and decompress operation. + + The crypto async interface support implemented by iaa_crypto + provides an implementation that satisfies the interface but does + so in a synchronous manner - it fills and submits the IDXD + descriptor and then loops around waiting for it to complete before + returning. This isn't a problem at the moment, since all existing + callers (e.g. zswap) wrap any asynchronous callees in a + synchronous wrapper anyway. + + The iaa_crypto driver does however provide true asynchronous + support for callers that can make use of it. In this mode, it + fills and submits the IDXD descriptor, then returns immediately + with -EINPROGRESS. The caller can then either poll for completion + itself, which requires specific code in the caller which currently + nothing in the upstream kernel implements, or go to sleep and wait + for an interrupt signaling completion. This latter mode is + supported by current users in the kernel such as zswap via + synchronous wrappers. Although it is supported this mode is + significantly slower than the synchronous mode that does the + polling in the iaa_crypto driver previously mentioned. + + This mode can be enabled by writing 'async_irq' to the sync_mode + iaa_crypto driver attribute: + + echo async_irq > /sys/bus/dsa/drivers/crypto/sync_mode + + Async mode without interrupts (caller must poll) can be enabled by + writing 'async' to it: + + echo async > /sys/bus/dsa/drivers/crypto/sync_mode + + The mode that does the polling in the iaa_crypto driver can be + enabled by writing 'sync' to it: + + echo sync > /sys/bus/dsa/drivers/crypto/sync_mode + + The default mode is 'sync'. + + +Statistics +========== + +If the optional debugfs statistics support is enabled, the IAA crypto +driver will generate statistics which can be accessed in debugfs at:: + + # ls -al /sys/kernel/debug/iaa-crypto/ + total 0 + drwxr-xr-x 2 root root 0 Mar 3 09:35 . + drwx------ 47 root root 0 Mar 3 09:35 .. + -rw-r--r-- 1 root root 0 Mar 3 09:35 max_acomp_delay_ns + -rw-r--r-- 1 root root 0 Mar 3 09:35 max_adecomp_delay_ns + -rw-r--r-- 1 root root 0 Mar 3 09:35 max_comp_delay_ns + -rw-r--r-- 1 root root 0 Mar 3 09:35 max_decomp_delay_ns + -rw-r--r-- 1 root root 0 Mar 3 09:35 stats_reset + -rw-r--r-- 1 root root 0 Mar 3 09:35 total_comp_bytes_out + -rw-r--r-- 1 root root 0 Mar 3 09:35 total_comp_calls + -rw-r--r-- 1 root root 0 Mar 3 09:35 total_decomp_bytes_in + -rw-r--r-- 1 root root 0 Mar 3 09:35 total_decomp_calls + -rw-r--r-- 1 root root 0 Mar 3 09:35 wq_stats + +Most of the above statisticss are self-explanatory. The wq_stats file +shows per-wq stats, a set for each iaa device and wq in addition to +some global stats:: + + # cat wq_stats + global stats: + total_comp_calls: 100 + total_decomp_calls: 100 + total_comp_bytes_out: 22800 + total_decomp_bytes_in: 22800 + total_completion_einval_errors: 0 + total_completion_timeout_errors: 0 + total_completion_comp_buf_overflow_errors: 0 + + iaa device: + id: 1 + n_wqs: 1 + comp_calls: 0 + comp_bytes: 0 + decomp_calls: 0 + decomp_bytes: 0 + wqs: + name: iaa_crypto + comp_calls: 0 + comp_bytes: 0 + decomp_calls: 0 + decomp_bytes: 0 + + iaa device: + id: 3 + n_wqs: 1 + comp_calls: 0 + comp_bytes: 0 + decomp_calls: 0 + decomp_bytes: 0 + wqs: + name: iaa_crypto + comp_calls: 0 + comp_bytes: 0 + decomp_calls: 0 + decomp_bytes: 0 + + iaa device: + id: 5 + n_wqs: 1 + comp_calls: 100 + comp_bytes: 22800 + decomp_calls: 100 + decomp_bytes: 22800 + wqs: + name: iaa_crypto + comp_calls: 100 + comp_bytes: 22800 + decomp_calls: 100 + decomp_bytes: 22800 + +Writing 0 to 'stats_reset' resets all the stats, including the +per-device and per-wq stats:: + + # echo 0 > stats_reset + # cat wq_stats + global stats: + total_comp_calls: 0 + total_decomp_calls: 0 + total_comp_bytes_out: 0 + total_decomp_bytes_in: 0 + total_completion_einval_errors: 0 + total_completion_timeout_errors: 0 + total_completion_comp_buf_overflow_errors: 0 + ... + + +Use cases +========= + +Simple zram test +---------------- + +For this example, the kernel should be configured according to the +scalable mode options described above, and zram should be enabled as +well:: + + CONFIG_ZRAM=m + CONFIG_ZSMALLOC=m + CONFIG_ZSMALLOC_STAT=y + CONFIG_ZRAM_WRITEBACK=y + +This is a simple test that uses iaa_compress as the compressor for a +zram filesystem. It simply copies a file to an iaa_compressed zram +disk and then copies it back, to demonstrate both compress and +decompress. + +The first step is to make sure the iaa_crypto module is loaded:: + + modprobe iaa_crypto + +Following that the IAA device(s) should be configured and enabled. + +The zram test expects the work queues for each IAA device on the +system to be configured properly as a kernel workqueue with a +workqueue driver_name of "crypto". + +The below script automatically does that:: + + #!/bin/bash + + echo "IAA devices:" + lspci -d:0cfe + echo "# IAA devices:" + lspci -d:0cfe | wc -l + + # + # count iaa instances + # + iaa_dev_id="0cfe" + num_iaa=$(lspci -d:${iaa_dev_id} | wc -l) + echo "Found ${num_iaa} IAA instances" + + # + # disable iaa wqs and devices + # + echo "Disable IAA" + + for ((i = 1; i < ${num_iaa} * 2; i += 2)); do + echo disable wq iax${i}/wq${i}.0 + accel-config disable-wq iax${i}/wq${i}.0 + echo disable iaa iax${i} + accel-config disable-device iax${i} + done + + echo "End Disable IAA" + + # + # configure iaa wqs and devices + # + echo "Configure IAA" + for ((i = 1; i < ${num_iaa} * 2; i += 2)); do + accel-config config-wq --group-id=0 --mode=shared --size=128 --priority=10 --type=kernel --name="iaa_crypto" --device_name="crypto" --threshold=10 iax${i}/wq${i} + done + + echo "End Configure IAA" + + # + # enable iaa wqs and devices + # + echo "Enable IAA" + + for ((i = 1; i < ${num_iaa} * 2; i += 2)); do + echo enable iaa iaa${i} + accel-config enable-device iaa${i} + echo enable wq iaa${i}/wq${i}.0 + accel-config enable-wq iaa${i}/wq${i}.0 + done + + echo "End Enable IAA" + +When the workqueues are bound to the iaa_crypto driver, you should +see something similar to the following in dmesg output if you've +enabled debug output (echo -n 'module iaa_crypto +p' > +/sys/kernel/debug/dynamic_debug/control):: + + [ 60.752344] idxd 0000:f6:02.0: add_iaa_wq: added wq 000000004068d14d to iaa 00000000c9585ba2, n_wq 1 + [ 60.752346] iaa_crypto: rebalance_wq_table: nr_nodes=2, nr_cpus 160, nr_iaa 8, cpus_per_iaa 20 + [ 60.752347] iaa_crypto: rebalance_wq_table: iaa=0 + [ 60.752349] idxd 0000:6a:02.0: request_iaa_wq: getting wq from iaa_device 0000000042d7bc52 (0) + [ 60.752350] idxd 0000:6a:02.0: request_iaa_wq: returning unused wq 00000000c8bb4452 (0) from iaa device 0000000042d7bc52 (0) + [ 60.752352] iaa_crypto: rebalance_wq_table: assigned wq for cpu=0, node=0 = wq 00000000c8bb4452 + [ 60.752354] iaa_crypto: rebalance_wq_table: iaa=0 + [ 60.752355] idxd 0000:6a:02.0: request_iaa_wq: getting wq from iaa_device 0000000042d7bc52 (0) + [ 60.752356] idxd 0000:6a:02.0: request_iaa_wq: returning unused wq 00000000c8bb4452 (0) from iaa device 0000000042d7bc52 (0) + [ 60.752358] iaa_crypto: rebalance_wq_table: assigned wq for cpu=1, node=0 = wq 00000000c8bb4452 + [ 60.752359] iaa_crypto: rebalance_wq_table: iaa=0 + [ 60.752360] idxd 0000:6a:02.0: request_iaa_wq: getting wq from iaa_device 0000000042d7bc52 (0) + [ 60.752361] idxd 0000:6a:02.0: request_iaa_wq: returning unused wq 00000000c8bb4452 (0) from iaa device 0000000042d7bc52 (0) + [ 60.752362] iaa_crypto: rebalance_wq_table: assigned wq for cpu=2, node=0 = wq 00000000c8bb4452 + [ 60.752364] iaa_crypto: rebalance_wq_table: iaa=0 + . + . + . + +Once the workqueues and devices have been enabled, the iaa_crypto +algorithm is enabled and available. When the iaa_crypto algorithm has +been successfully enabled, you should see the following dmesg output:: + + [ 64.893759] iaa_crypto: iaa_crypto_enable: iaa_crypto now ENABLED + +Now that the iaa_crypto module is ready to go, we can set up +zram. Start by setting up a few vm params:: + + echo 100 > /proc/sys/vm/swappiness + echo never > /sys/kernel/mm/transparent_hugepage/enabled + echo 1 > /proc/sys/vm/overcommit_memory + +Next, insert the zram module for 1 device and configure the new +/dev/zram1 to use iaa_crypto as the compression algorithm:: + + modprobe zram num_devices=1 + + echo iaa_crypto > /sys/block/zram1/comp_algorithm + echo 32G > /sys/block/zram1/disksize + echo 32G > /sys/block/zram1/mem_limit + +Now create an ext4 filesytem on /dev/zram1 and mount it:: + + mkfs.ext4 /dev/zram1 + mount /dev/zram1 /tmp + +Now that our /tmp filesystem is set up and using iaa_crypto for +compress/decompress, copy a big file to /tmp:: + + cp -f somebigfile.txt /tmp; sync + +You should see something like the following in the dmesg output if +you've enabled debug output (echo -n 'module iaa_crypto +p' > +/sys/kernel/debug/dynamic_debug/control):: + + [ 836.226331] idxd 0000:6f:02.0: iaa_comp_compress: dma_map_single, src_addr 2cdffb000, dev 0000000063891379, src 000000004d193d53, slen 4096 + [ 836.226333] idxd 0000:6f:02.0: iaa_comp_compress: dma_map_single, dst_addr 281182000, dev 0000000063891379, dst 0000000014a9f578, *dlen 8192 + [ 836.226335] idxd 0000:6f:02.0: iaa_comp_compress: src 000000004d193d53, src_addr 2cdffb000, slen 4096, dst 0000000014a9f578, dst_addr 281182000, dlen 8192 + [ 836.226337] idxd 0000:6f:02.0: iaa_compress: desc->src1_addr 2cdffb000, desc->src1_size 4096, desc->dst_addr 281182000, desc->max_dst_size 8192, desc->src2_addr 28270a000, desc->src2_size 1568 + [ 836.226344] idxd 0000:6f:02.0: iaa_compress_verify: (verify) desc->src1_addr 281182000, desc->src1_size 468, desc->dst_addr 2cdffb000, desc->max_dst_size 4096, desc->src2_addr 0, desc->src2_size 0 + [ 836.226381] idxd 0000:6f:02.0: iaa_comp_compress: dma_map_single, src_addr 280367000, dev 0000000063891379, src 00000000b72795f1, slen 4096 + [ 836.226383] idxd 0000:6f:02.0: iaa_comp_compress: dma_map_single, dst_addr 281182000, dev 0000000063891379, dst 0000000014a9f578, *dlen 8192 + [ 836.226385] idxd 0000:6f:02.0: iaa_comp_compress: src 00000000b72795f1, src_addr 280367000, slen 4096, dst 0000000014a9f578, dst_addr 281182000, dlen 8192 + [ 836.226388] idxd 0000:6f:02.0: iaa_compress: desc->src1_addr 280367000, desc->src1_size 4096, desc->dst_addr 281182000, desc->max_dst_size 8192, desc->src2_addr 28270a000, desc->src2_size 1568 + [ 836.226394] idxd 0000:6f:02.0: iaa_compress_verify: (verify) desc->src1_addr 281182000, desc->src1_size 285, desc->dst_addr 280367000, desc->max_dst_size 4096, desc->src2_addr 0, desc->src2_size 0 + +To see the file decompress (again, turn on debug output in the +kernel):: + + sync; echo 3 > /proc/sys/vm/drop_caches; sync + cat /tmp/somebigfile.txt + +You should see something like the following in the dmesg output if +you've enabled debug output (echo -n 'module iaa_crypto +p' > +/sys/kernel/debug/dynamic_debug/control):: + + [ 929.394416] idxd 0000:74:02.0: iaa_comp_decompress: using wq for cpu=81 = wq 000000007e629f8f + [ 929.394417] idxd 0000:74:02.0: iaa_comp_decompress: dma_map_single, src_addr 272a58008, dev 000000002bf7270d, src 000000007e580802, slen 1564 + [ 929.394418] idxd 0000:74:02.0: iaa_comp_decompress: dma_map_single, dst_addr 274bbe000, dev 000000002bf7270d, dst 0000000037de3d04, *dlen 4096 + [ 929.394420] idxd 0000:74:02.0: iaa_comp_decompress: src 000000007e580802, src_addr 272a58008, slen 1564, dst 0000000037de3d04, dst_addr 274bbe000, dlen 4096 + [ 929.394421] idxd 0000:74:02.0: iaa_decompress: desc->src1_addr 272a58008, desc->src1_size 1564, desc->dst_addr 274bbe000, desc->max_dst_size 4096, desc->src2_addr 0, desc->src2_size 0 + [ 929.394427] idxd 0000:74:02.0: iaa_comp_decompress: using wq for cpu=81 = wq 000000007e629f8f + [ 929.394428] idxd 0000:74:02.0: iaa_comp_decompress: dma_map_single, src_addr 2b8539008, dev 000000002bf7270d, src 0000000021939c1e, slen 849 + [ 929.394430] idxd 0000:74:02.0: iaa_comp_decompress: dma_map_single, dst_addr 2b89b7000, dev 000000002bf7270d, dst 00000000850a85fa, *dlen 4096 + [ 929.394431] idxd 0000:74:02.0: iaa_comp_decompress: src 0000000021939c1e, src_addr 2b8539008, slen 849, dst 00000000850a85fa, dst_addr 2b89b7000, dlen 4096 + [ 929.394433] idxd 0000:74:02.0: iaa_decompress: desc->src1_addr 2b8539008, desc->src1_size 849, desc->dst_addr 2b89b7000, desc->max_dst_size 4096, desc->src2_addr 0, desc->src2_size 0 + +When finished, disable the device(s), and remove the zram and +iaa_crypto modules:: + + umount -f /dev/zram1 + echo 0 > /sys/class/zram-control/hot_remove + + rmmod zram + rmmod iaa_crypto + + +Simple zswap test +----------------- + +For this example, the kernel should be configured according to the +scalable mode options described above, and zswap should be enabled as +well:: + + CONFIG_ZSWAP=y + +This is a simple test that uses iaa_compress as the compressor for a +swap (zswap) device. It sets up the zswap device and then uses the +memory_memadvice program listed below to forcibly swap out and in a +specified number of pages, demonstrating both compress and decompress. + +The zswap test expects the work queues for each IAA device on the +system to be configured properly as a kernel workqueue with a +workqueue driver_name of "crypto". + +The first step is to make sure the iaa_crypto module is loaded:: + + modprobe iaa_crypto + +Following that the IAA device(s) should be configured and enabled. + +Repeat the above steps but instead of the zram setup following the iaa +device/workqueue setup script, run the following zswap-specific setup +commands. For zswap, CONFIG_ZSWAP=y should be enabled as well: + +The below script automatically does that:: + + #!/bin/bash + + echo "IAA devices:" + lspci -d:0cfe + echo "# IAA devices:" + lspci -d:0cfe | wc -l + + # + # count iaa instances + # + iaa_dev_id="0cfe" + num_iaa=$(lspci -d:${iaa_dev_id} | wc -l) + echo "Found ${num_iaa} IAA instances" + + # + # disable iaa wqs and devices + # + echo "Disable IAA" + + for ((i = 1; i < ${num_iaa} * 2; i += 2)); do + echo disable wq iax${i}/wq${i}.0 + accel-config disable-wq iax${i}/wq${i}.0 + echo disable iaa iax${i} + accel-config disable-device iax${i} + done + + echo "End Disable IAA" + + # + # configure iaa wqs and devices + # + echo "Configure IAA" + for ((i = 1; i < ${num_iaa} * 2; i += 2)); do + accel-config config-wq --group-id=0 --mode=shared --size=128 --priority=10 --type=kernel --name="iaa_crypto" --device_name="crypto" --threshold=10 iax${i}/wq${i} + done + + echo "End Configure IAA" + + # + # enable iaa wqs and devices + # + echo "Enable IAA" + + for ((i = 1; i < ${num_iaa} * 2; i += 2)); do + echo enable iaa iaa${i} + accel-config enable-device iaa${i} + echo enable wq iaa${i}/wq${i}.0 + accel-config enable-wq iaa${i}/wq${i}.0 + done + + echo "End Enable IAA" + +When the iaa_crypto algorithm has been successfully enabled, you +should see the following dmesg output:: + + [ 196.377326] iaa_crypto: iaa_crypto_enable: iaa_crypto now ENABLED + +Now run the following zswap-specific setup commands:: + + echo 0 > /sys/module/zswap/parameters/enabled + echo 50 > /sys/module/zswap/parameters/max_pool_percent + echo iaa_crypto > /sys/module/zswap/parameters/compressor + echo zsmalloc > /sys/module/zswap/parameters/zpool + echo 1 > /sys/module/zswap/parameters/enabled + echo 0 > /sys/module/zswap/parameters/same_filled_pages_enabled + + echo 100 > /proc/sys/vm/swappiness + echo never > /sys/kernel/mm/transparent_hugepage/enabled + echo 1 > /proc/sys/vm/overcommit_memory + +Finally, you can now run the zswap workload you want to measure. For +example, using the code below, the following command will swap in and +out 100 pages:: + + ./memory_madvise 100 + + Allocating 100 pages to swap in/out + Swapping out 100 pages + Swapping in 100 pages + Swapped out and in 100 pages + +You should see something like the following in the dmesg output if +you've enabled debug output (echo -n 'module iaa_crypto +p' > +/sys/kernel/debug/dynamic_debug/control):: + + [ 404.202972] idxd 0000:e7:02.0: iaa_comp_acompress: dma_map_sg, src_addr 223925c000, nr_sgs 1, req->src 00000000ee7cb5e6, req->slen 4096, sg_dma_len(sg) 4096 + [ 404.202973] idxd 0000:e7:02.0: iaa_comp_acompress: dma_map_sg, dst_addr 21dadf8000, nr_sgs 1, req->dst 000000008d6acea8, req->dlen 4096, sg_dma_len(sg) 8192 + [ 404.202975] idxd 0000:e7:02.0: iaa_compress: desc->src1_addr 223925c000, desc->src1_size 4096, desc->dst_addr 21dadf8000, desc->max_dst_size 4096, desc->src2_addr 2203543000, desc->src2_size 1568 + [ 404.202981] idxd 0000:e7:02.0: iaa_compress_verify: (verify) desc->src1_addr 21dadf8000, desc->src1_size 228, desc->dst_addr 223925c000, desc->max_dst_size 4096, desc->src2_addr 0, desc->src2_size 0 + [ 409.203227] idxd 0000:e7:02.0: iaa_comp_adecompress: dma_map_sg, src_addr 21ddd8b100, nr_sgs 1, req->src 0000000084adab64, req->slen 228, sg_dma_len(sg) 228 + [ 409.203235] idxd 0000:e7:02.0: iaa_comp_adecompress: dma_map_sg, dst_addr 21ee3dc000, nr_sgs 1, req->dst 000000004e2990d0, req->dlen 4096, sg_dma_len(sg) 4096 + [ 409.203239] idxd 0000:e7:02.0: iaa_decompress: desc->src1_addr 21ddd8b100, desc->src1_size 228, desc->dst_addr 21ee3dc000, desc->max_dst_size 4096, desc->src2_addr 0, desc->src2_size 0 + [ 409.203254] idxd 0000:e7:02.0: iaa_comp_adecompress: dma_map_sg, src_addr 21ddd8b100, nr_sgs 1, req->src 0000000084adab64, req->slen 228, sg_dma_len(sg) 228 + [ 409.203256] idxd 0000:e7:02.0: iaa_comp_adecompress: dma_map_sg, dst_addr 21f1551000, nr_sgs 1, req->dst 000000004e2990d0, req->dlen 4096, sg_dma_len(sg) 4096 + [ 409.203257] idxd 0000:e7:02.0: iaa_decompress: desc->src1_addr 21ddd8b100, desc->src1_size 228, desc->dst_addr 21f1551000, desc->max_dst_size 4096, desc->src2_addr 0, desc->src2_size 0 + +memory_madvise.c (gcc -o memory_memadvise memory_madvise.c):: + + #include + #include + #include + #include + #include + #include + + #ifndef MADV_PAGEOUT + #define MADV_PAGEOUT 21 /* force pages out immediately */ + #endif + + #define PG_SZ 4096 + + int main(int argc, char **argv) + { + int i, nr_pages = 1; + int64_t *dump_ptr; + char *addr, *a; + int loop = 1; + + if (argc > 1) + nr_pages = atoi(argv[1]); + + printf("Allocating %d pages to swap in/out\n", nr_pages); + + /* allocate pages */ + addr = mmap(NULL, nr_pages * PG_SZ, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0); + *addr = 1; + + /* initialize data in page to all '*' chars */ + memset(addr, '*', nr_pages * PG_SZ); + + printf("Swapping out %d pages\n", nr_pages); + + /* Tell kernel to swap it out */ + madvise(addr, nr_pages * PG_SZ, MADV_PAGEOUT); + + while (loop > 0) { + /* Wait for swap out to finish */ + sleep(5); + + a = addr; + + printf("Swapping in %d pages\n", nr_pages); + + /* Access the page ... this will swap it back in again */ + for (i = 0; i < nr_pages; i++) { + if (a[0] != '*') { + printf("Bad data from decompress!!!!!\n"); + + dump_ptr = (int64_t *)a; + for (int j = 0; j < 100; j++) { + printf(" page %d data: %#llx\n", i, *dump_ptr); + dump_ptr++; + } + } + + a += PG_SZ; + } + + loop --; + } + + printf("Swapped out and in %d pages\n", nr_pages); diff --git a/Documentation/driver-api/crypto/iaa/index.rst b/Documentation/driver-api/crypto/iaa/index.rst new file mode 100644 index 000000000000..aa6837e27264 --- /dev/null +++ b/Documentation/driver-api/crypto/iaa/index.rst @@ -0,0 +1,20 @@ +.. SPDX-License-Identifier: GPL-2.0 + +================================= +IAA (Intel Analytics Accelerator) +================================= + +IAA provides hardware compression and decompression via the crypto +API. + +.. toctree:: + :maxdepth: 1 + + iaa-crypto + +.. only:: subproject and html + + Indices + ======= + + * :ref:`genindex` diff --git a/Documentation/driver-api/crypto/index.rst b/Documentation/driver-api/crypto/index.rst new file mode 100644 index 000000000000..fb9709b98bea --- /dev/null +++ b/Documentation/driver-api/crypto/index.rst @@ -0,0 +1,20 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============== +Crypto Drivers +============== + +Documentation for crypto drivers that may need more involved setup and +configuration. + +.. toctree:: + :maxdepth: 1 + + iaa/index + +.. only:: subproject and html + + Indices + ======= + + * :ref:`genindex` diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst index ff9aa1afdc62..2ad1237d5d5f 100644 --- a/Documentation/driver-api/index.rst +++ b/Documentation/driver-api/index.rst @@ -113,6 +113,7 @@ available subsections can be seen below. xillybus zorro hte/index + crypto/index .. only:: subproject and html From patchwork Mon Mar 6 18:52:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65061 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2016000wrd; Mon, 6 Mar 2023 10:57:02 -0800 (PST) X-Google-Smtp-Source: AK7set83fCoPqpvjwvSFiWOUO9HWr02wnzhGhRgScP59+rBQiKCbYwiNxhnmP8+ga7px90s/rEtQ X-Received: by 2002:a05:6402:31f8:b0:4ac:b602:3fa4 with SMTP id dy24-20020a05640231f800b004acb6023fa4mr11061213edb.17.1678129022862; Mon, 06 Mar 2023 10:57:02 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678129022; cv=none; d=google.com; s=arc-20160816; b=vFAgNrGLIYKmsa8oPElaSKOT1LtWwSpADs+QpksX3A79bfKPVsqOxE5cEcgdBRu7/R iLqV9Ha5w/AyTGW5/pqTvgA/2ICx5eMsxkO0+Tcxpqfcinh+9vkekpC6iVkn1/5/JVma /rpWAVVRVHgm216XECcoVCaPsWz48kel2vONjH6nMN9crmU4fMKBJOu33e5MBPqR/NsF SxWLa0ApsJOP4Qt5tuO5kfRMMRWzYPoc/RK2ADEDdg0hobT5lZAkloxfPGliNcISQS3s 1YEza04yMS+RNfgoww7dCFCyww0eKnVDuSHYoDJ7cpCmH52I26C9gc13HH3NMpQgWWrD cWIw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=GRueEIWKoIq2/93NOg74ipdnIc3zn1zuNZ4f+z1DifU=; b=XQptjoY+9AXklLdurW7+ZmBRun5ncm6gR0mT8L0OUuSXgoDOPz7BxWvTvAFksawk8S /OJya3yniEdKpv1PbR6LLlDZbSzT/BvMpnRUM/XxcDpP3ui+5NkMb0eRrYgknGfikUCl pHqccSKmEWp9/wlUcYO21PoJgaWIvKhvnV5lsztR7gGsQJu1+yWZPtlDxSoIiv5sUbrj BzS8fKukyUJ+YNc3TpDNZde9zQi7hEvlIw7GEGmaHIPCp1yw63eTNS/Wzt+YY1kYhoq3 5f9uoAw5kQgaXHpIMdrHG0GHTEvDEXZIhZDzCRoy6t6EmgbXiTtXVCaAFSIgt6rrNiIT HgNg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=jMI6kgek; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id lb16-20020a170907785000b008c673be3440si5231113ejc.987.2023.03.06.10.56.39; Mon, 06 Mar 2023 10:57:02 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=jMI6kgek; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230354AbjCFSxm (ORCPT + 99 others); Mon, 6 Mar 2023 13:53:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59956 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229564AbjCFSx0 (ORCPT ); Mon, 6 Mar 2023 13:53:26 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7EB2C271C; Mon, 6 Mar 2023 10:52:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128773; x=1709664773; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=b5HUmfxrM8xSxXSlb+w7H0MXJqXI9H3bd/PqVDb4Ba4=; b=jMI6kgekCbvJalWJ4dLxeHup3faSDSw1AxzIH9Xy7Fv1BfkPEJK7uySZ a5EbpdiP4uONC6C2egyGLYeOaR6zsATS2mhPS7aFk5FGTCPSoSnu7sZ8Z U65bYVwqEm8OnRRWHJgrHCMV94hNn/6EujC2rVQHmQTa5L1cETA2kQEYj +quwnKCh4LVJ+7A7cCg93HS/v5/vHTH+vG7qbM3SVdw3HxZ1x6gV+6SRJ gAPRJVbYlfvC3oTGcIXAiatCEiTS6LnpB6AePz6cgW73psdkGQbbu58We LMUmX+z6o32r30H/IqKWTvRZg8G6c7iCsGLqEIhGvuxZDS2ffluAUWwUV A==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227748" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227748" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:53 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255834" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255834" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:51 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 09/16] crypto: iaa - Add Intel IAA Compression Accelerator crypto driver core Date: Mon, 6 Mar 2023 12:52:19 -0600 Message-Id: <20230306185226.26483-10-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645818167588883?= X-GMAIL-MSGID: =?utf-8?q?1759645818167588883?= The Intel Analytics Accelerator (IAA) is a hardware accelerator that provides very high thoughput compression/decompression compatible with the DEFLATE compression standard described in RFC 1951, which is the compression/decompression algorithm exported by this module. Users can select IAA compress/decompress acceleration by specifying 'iaa_crypto' as the compression algorithm to use by whatever facility allows compression algorithms to be selected. For example, a zram drive can select iaa_crypto via: # echo iaa_crypto > /sys/block/zram1/comp_algorithm Similarly for zswap: # echo iaa_crypto > /sys/module/zswap/parameters/compressor This patch adds iaa_crypto as an idxd sub-driver and tracks iaa devices and workqueues as they are probed or removed. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi --- drivers/crypto/Kconfig | 1 + drivers/crypto/Makefile | 1 + drivers/crypto/iaa/Kconfig | 10 + drivers/crypto/iaa/Makefile | 10 + drivers/crypto/iaa/iaa_crypto.h | 30 +++ drivers/crypto/iaa/iaa_crypto_main.c | 326 +++++++++++++++++++++++++++ 6 files changed, 378 insertions(+) create mode 100644 drivers/crypto/iaa/Kconfig create mode 100644 drivers/crypto/iaa/Makefile create mode 100644 drivers/crypto/iaa/iaa_crypto.h create mode 100644 drivers/crypto/iaa/iaa_crypto_main.c diff --git a/drivers/crypto/Kconfig b/drivers/crypto/Kconfig index 3b2516d1433f..46123f59f9a6 100644 --- a/drivers/crypto/Kconfig +++ b/drivers/crypto/Kconfig @@ -506,6 +506,7 @@ source "drivers/crypto/qat/Kconfig" source "drivers/crypto/cavium/cpt/Kconfig" source "drivers/crypto/cavium/nitrox/Kconfig" source "drivers/crypto/marvell/Kconfig" +source "drivers/crypto/iaa/Kconfig" config CRYPTO_DEV_CAVIUM_ZIP tristate "Cavium ZIP driver" diff --git a/drivers/crypto/Makefile b/drivers/crypto/Makefile index 476f1a25ca32..d96fc84ce755 100644 --- a/drivers/crypto/Makefile +++ b/drivers/crypto/Makefile @@ -9,6 +9,7 @@ obj-$(CONFIG_CRYPTO_DEV_ATMEL_I2C) += atmel-i2c.o obj-$(CONFIG_CRYPTO_DEV_ATMEL_ECC) += atmel-ecc.o obj-$(CONFIG_CRYPTO_DEV_ATMEL_SHA204A) += atmel-sha204a.o obj-$(CONFIG_CRYPTO_DEV_CAVIUM_ZIP) += cavium/ +obj-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO) += iaa/ obj-$(CONFIG_CRYPTO_DEV_CCP) += ccp/ obj-$(CONFIG_CRYPTO_DEV_CCREE) += ccree/ obj-$(CONFIG_CRYPTO_DEV_CHELSIO) += chelsio/ diff --git a/drivers/crypto/iaa/Kconfig b/drivers/crypto/iaa/Kconfig new file mode 100644 index 000000000000..fcccb6ff7e29 --- /dev/null +++ b/drivers/crypto/iaa/Kconfig @@ -0,0 +1,10 @@ +config CRYPTO_DEV_IAA_CRYPTO + tristate "Support for Intel(R) IAA Compression Accelerator" + depends on CRYPTO_DEFLATE + depends on INTEL_IDXD + default n + help + This driver supports acceleration for compression and + decompression with the Intel Analytics Accelerator (IAA) + hardware using the cryptographic API. If you choose 'M' + here, the module will be called iaa_crypto. diff --git a/drivers/crypto/iaa/Makefile b/drivers/crypto/iaa/Makefile new file mode 100644 index 000000000000..03859431c897 --- /dev/null +++ b/drivers/crypto/iaa/Makefile @@ -0,0 +1,10 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Makefile for IAA crypto device drivers +# + +ccflags-y += -I $(srctree)/drivers/dma/idxd -DDEFAULT_SYMBOL_NAMESPACE=IDXD + +obj-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO) := iaa_crypto.o + +iaa_crypto-y := iaa_crypto_main.o diff --git a/drivers/crypto/iaa/iaa_crypto.h b/drivers/crypto/iaa/iaa_crypto.h new file mode 100644 index 000000000000..5d1fff7f4b8e --- /dev/null +++ b/drivers/crypto/iaa/iaa_crypto.h @@ -0,0 +1,30 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */ + +#ifndef __IAA_CRYPTO_H__ +#define __IAA_CRYPTO_H__ + +#include +#include +#include + +#define IDXD_SUBDRIVER_NAME "crypto" + +/* Representation of IAA workqueue */ +struct iaa_wq { + struct list_head list; + struct idxd_wq *wq; + + struct iaa_device *iaa_device; +}; + +/* Representation of IAA device with wqs, populated by probe */ +struct iaa_device { + struct list_head list; + struct idxd_device *idxd; + + int n_wq; + struct list_head wqs; +}; + +#endif diff --git a/drivers/crypto/iaa/iaa_crypto_main.c b/drivers/crypto/iaa/iaa_crypto_main.c new file mode 100644 index 000000000000..e7e7fbdd72e3 --- /dev/null +++ b/drivers/crypto/iaa/iaa_crypto_main.c @@ -0,0 +1,326 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "idxd.h" +#include "iaa_crypto.h" + +#ifdef pr_fmt +#undef pr_fmt +#endif + +#define pr_fmt(fmt) "idxd: " IDXD_SUBDRIVER_NAME ": " fmt + +/* number of iaa instances probed */ +static unsigned int nr_iaa; + +static LIST_HEAD(iaa_devices); +static DEFINE_MUTEX(iaa_devices_lock); + +static struct iaa_device *iaa_device_alloc(void) +{ + struct iaa_device *iaa_device; + + iaa_device = kzalloc(sizeof(*iaa_device), GFP_KERNEL); + if (!iaa_device) + return NULL; + + INIT_LIST_HEAD(&iaa_device->wqs); + + return iaa_device; +} + +static void iaa_device_free(struct iaa_device *iaa_device) +{ + struct iaa_wq *iaa_wq, *next; + + list_for_each_entry_safe(iaa_wq, next, &iaa_device->wqs, list) { + list_del(&iaa_wq->list); + kfree(iaa_wq); + } + + kfree(iaa_device); +} + +static bool iaa_has_wq(struct iaa_device *iaa_device, struct idxd_wq *wq) +{ + struct iaa_wq *iaa_wq; + + list_for_each_entry(iaa_wq, &iaa_device->wqs, list) { + if (iaa_wq->wq == wq) + return true; + } + + return false; +} + +static struct iaa_device *add_iaa_device(struct idxd_device *idxd) +{ + struct iaa_device *iaa_device; + + iaa_device = iaa_device_alloc(); + if (!iaa_device) + return NULL; + + iaa_device->idxd = idxd; + + list_add_tail(&iaa_device->list, &iaa_devices); + + nr_iaa++; + + return iaa_device; +} + +static void del_iaa_device(struct iaa_device *iaa_device) +{ + list_del(&iaa_device->list); + + iaa_device_free(iaa_device); + + nr_iaa--; +} + +static int add_iaa_wq(struct iaa_device *iaa_device, struct idxd_wq *wq, + struct iaa_wq **new_wq) +{ + struct idxd_device *idxd = iaa_device->idxd; + struct pci_dev *pdev = idxd->pdev; + struct device *dev = &pdev->dev; + struct iaa_wq *iaa_wq; + + iaa_wq = kzalloc(sizeof(*iaa_wq), GFP_KERNEL); + if (!iaa_wq) + return -ENOMEM; + + iaa_wq->wq = wq; + iaa_wq->iaa_device = iaa_device; + wq->private_data = iaa_wq; + + list_add_tail(&iaa_wq->list, &iaa_device->wqs); + + iaa_device->n_wq++; + + if (new_wq) + *new_wq = iaa_wq; + + dev_dbg(dev, "added wq %d to iaa device %d, n_wq %d\n", + wq->id, iaa_device->idxd->id, iaa_device->n_wq); + + return 0; +} + +static void del_iaa_wq(struct iaa_device *iaa_device, struct idxd_wq *wq) +{ + struct idxd_device *idxd = iaa_device->idxd; + struct pci_dev *pdev = idxd->pdev; + struct device *dev = &pdev->dev; + struct iaa_wq *iaa_wq; + + list_for_each_entry(iaa_wq, &iaa_device->wqs, list) { + if (iaa_wq->wq == wq) { + list_del(&iaa_wq->list); + iaa_device->n_wq--; + + dev_dbg(dev, "removed wq %d from iaa_device %d, n_wq %d, nr_iaa %d\n", + wq->id, iaa_device->idxd->id, + iaa_device->n_wq, nr_iaa); + + if (iaa_device->n_wq == 0) + del_iaa_device(iaa_device); + break; + } + } +} + +static int save_iaa_wq(struct idxd_wq *wq) +{ + struct iaa_device *iaa_device, *found = NULL; + struct idxd_device *idxd; + struct pci_dev *pdev; + struct device *dev; + int ret = 0; + + list_for_each_entry(iaa_device, &iaa_devices, list) { + if (iaa_device->idxd == wq->idxd) { + idxd = iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + /* + * Check to see that we don't already have this wq. + * Shouldn't happen but we don't control probing. + */ + if (iaa_has_wq(iaa_device, wq)) { + dev_dbg(dev, "same wq probed multiple times for iaa_device %p\n", + iaa_device); + goto out; + } + + found = iaa_device; + + ret = add_iaa_wq(iaa_device, wq, NULL); + if (ret) + goto out; + + break; + } + } + + if (!found) { + struct iaa_device *new_device; + struct iaa_wq *new_wq; + + new_device = add_iaa_device(wq->idxd); + if (!new_device) { + ret = -ENOMEM; + goto out; + } + + ret = add_iaa_wq(new_device, wq, &new_wq); + if (ret) { + del_iaa_device(new_device); + goto out; + } + } + + if (WARN_ON(nr_iaa == 0)) + return -EINVAL; + + idxd_wq_get(wq); +out: + return 0; +} + +static void remove_iaa_wq(struct idxd_wq *wq) +{ + struct iaa_device *iaa_device; + + list_for_each_entry(iaa_device, &iaa_devices, list) { + if (iaa_has_wq(iaa_device, wq)) { + del_iaa_wq(iaa_device, wq); + idxd_wq_put(wq); + break; + } + } +} + +static int iaa_crypto_probe(struct idxd_dev *idxd_dev) +{ + struct idxd_wq *wq = idxd_dev_to_wq(idxd_dev); + struct idxd_device *idxd = wq->idxd; + struct idxd_driver_data *data = idxd->data; + struct device *dev = &idxd_dev->conf_dev; + int ret = 0; + + if (idxd->state != IDXD_DEV_ENABLED) + return -ENXIO; + + if (data->type != IDXD_TYPE_IAX) + return -ENODEV; + + mutex_lock(&wq->wq_lock); + + if (!idxd_wq_driver_name_match(wq, dev)) { + dev_dbg(dev, "wq %d.%d driver_name match failed: wq driver_name %s, dev driver name %s\n", + idxd->id, wq->id, wq->driver_name, dev->driver->name); + idxd->cmd_status = IDXD_SCMD_WQ_NO_DRV_NAME; + ret = -ENODEV; + goto err; + } + + wq->type = IDXD_WQT_KERNEL; + + ret = drv_enable_wq(wq, IDXD_DTYPE_CRYPTO); + if (ret < 0) { + dev_dbg(dev, "enable wq %d.%d failed: %d\n", + idxd->id, wq->id, ret); + ret = -ENXIO; + goto err; + } + + mutex_lock(&iaa_devices_lock); + + ret = save_iaa_wq(wq); + if (ret) + goto err_save; + + mutex_unlock(&iaa_devices_lock); +out: + mutex_unlock(&wq->wq_lock); + + return ret; + +err_save: + drv_disable_wq(wq); +err: + wq->type = IDXD_WQT_NONE; + + goto out; +} + +static void iaa_crypto_remove(struct idxd_dev *idxd_dev) +{ + struct idxd_wq *wq = idxd_dev_to_wq(idxd_dev); + + idxd_wq_quiesce(wq); + + mutex_lock(&wq->wq_lock); + mutex_lock(&iaa_devices_lock); + + remove_iaa_wq(wq); + drv_disable_wq(wq); + + mutex_unlock(&iaa_devices_lock); + mutex_unlock(&wq->wq_lock); +} + +static enum idxd_dev_type dev_types[] = { + IDXD_DEV_WQ, + IDXD_DEV_NONE, +}; + +static struct idxd_device_driver iaa_crypto_driver = { + .probe = iaa_crypto_probe, + .remove = iaa_crypto_remove, + .name = IDXD_SUBDRIVER_NAME, + .type = dev_types, +}; + +static int __init iaa_crypto_init_module(void) +{ + int ret = 0; + + ret = idxd_driver_register(&iaa_crypto_driver); + if (ret) { + pr_debug("IAA wq sub-driver registration failed\n"); + goto out; + } + + pr_debug("initialized\n"); +out: + return ret; +} + +static void __exit iaa_crypto_cleanup_module(void) +{ + idxd_driver_unregister(&iaa_crypto_driver); + + pr_debug("cleaned up\n"); +} + +MODULE_IMPORT_NS(IDXD); +MODULE_LICENSE("GPL"); +MODULE_ALIAS_IDXD_DEVICE(0); +MODULE_AUTHOR("Intel Corporation"); +MODULE_DESCRIPTION("IAA Compression Accelerator Crypto Driver"); + +module_init(iaa_crypto_init_module); +module_exit(iaa_crypto_cleanup_module); From patchwork Mon Mar 6 18:52:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65072 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2025238wrd; Mon, 6 Mar 2023 11:15:11 -0800 (PST) X-Google-Smtp-Source: AK7set/6qBd5aiG3B2LO7EQYkixerjyCI88Y500JS0apH6FcvfpTVH3vjoCTD6HEtNIAYmkzFeE7 X-Received: by 2002:a17:906:80c1:b0:87b:d62c:d87 with SMTP id a1-20020a17090680c100b0087bd62c0d87mr11962414ejx.1.1678130111510; Mon, 06 Mar 2023 11:15:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678130111; cv=none; d=google.com; s=arc-20160816; b=fbnQFf0K+i41vVzOiTi8kOwy7Ps9VP98aJbRg1DEyUMZwkXvRvtb9zvKSs510mMGJi f30VYIulsX7CcenoNHrBSuXeagrRFVUuB/9y3oM4cyDWU3lqET3fn34xpJMSWvBt9pqa 6eUbKF9u7pF9y/odHNHzFCiprp1iAxm+8oQ2Fm9Fb8v6WWWezp4kn++n9M2hd8zMAnM9 5oHmEcbI1Mc+mTVIExKyHlNIskTTtrmMkRNl8n6iush2CaRy9oYJV74RoDs3klT9v2vi qyvEAHlVFwj8wGKwROKnWKvkO5sg+GOrW4GS21w56CcVIn3j8OwYx7VPEot7NTBHonQ3 xnwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AsMVuAFkcjabCYTyJbaCKcA9MbjpeX4M3hAJxHHwun4=; b=dMZQiECCpk5wKGux2WJXAqRFPloA/ZZPKgtlFZ3G2nILjbihb7UVHTSARs5jT+Kv6X NRZ4LCrNQhEEk3yi3NjC+BYHF83XTwtHQnCaX0zw2Fz6fcNXEFae3xRcGUNkOhRoOTCE JyP7mn0QD2qn4u1GZqgLqazOzNgtBMflkzz36p3ubC/df2Wi16YQ4oU3Hxu5mPF3Xavm uTG/U9HSeew7l0AWyQb3Q0N8Yl1zIjn1IVgKANlNqdGlQTKXguhLRkwytWd57iMk5I3u yUYGOTyEy/aawgtoangNKDeWewfUsNZQGBfI3LofNjHacEOIoJIOt7+5n8g39SwKQQxh THBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=BNd9uqmk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id va12-20020a17090711cc00b008b175a0717fsi7086278ejb.396.2023.03.06.11.14.47; Mon, 06 Mar 2023 11:15:11 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=BNd9uqmk; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230196AbjCFSyF (ORCPT + 99 others); Mon, 6 Mar 2023 13:54:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230447AbjCFSx2 (ORCPT ); Mon, 6 Mar 2023 13:53:28 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB7D32A6C4; Mon, 6 Mar 2023 10:52:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128776; x=1709664776; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MA13azLzVhBmCMnrgMiSTRMZ8iVJIgRnYu5CpPKzfi4=; b=BNd9uqmkk7tFIyv78FQHFyU9sR9pt/vEFQT9eWJvT/64bskN7IUtpyXH SVGJiKCc1Yo66NWJl1XOY85/nqCo/pGm0bXrPnCjRw5R30NRvlZhAOZmG kJL7erfyOU/4zxsEw+4UfJgosMjzL1u8JpQ24BOZBFI9F84gbKFUwnLIT 5YGMTtMHY8L9MFmbhnmI1tDEa1SeCTxeXVbPwPxKgP3bJ9UNUoHAYrHhY vEX7WRK+4ZDZgKCztKRqdozzvh9LH+mvKmfVq96Yn/66k+ZAhXB+qKc+G l3+aEp3hvxoDxU1sT4lYDNhKUkYuBpQfIzXLax8oTJaWMTfrP5zfMqWkj A==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227761" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227761" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:56 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255868" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255868" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:54 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 10/16] crypto: iaa - Add per-cpu workqueue table with rebalancing Date: Mon, 6 Mar 2023 12:52:20 -0600 Message-Id: <20230306185226.26483-11-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759646959815743171?= X-GMAIL-MSGID: =?utf-8?q?1759646959815743171?= The iaa compression/decompression algorithms in later patches need a way to retrieve an appropriate IAA workqueue depending on how close the associated IAA device is to the current cpu. For this purpose, add a per-cpu array of workqueues such that an appropriate workqueue can be retrieved by simply accessing the per-cpu array. Whenever a new workqueue is bound to or unbound from the iaa_crypto driver, the available workqueues are 'rebalanced' such that work submitted from a particular CPU is given to the most appropriate workqueue available. There currently isn't any way for the user to tweak the way this is done internally - if necessary, knobs can be added later for that purpose. Current best practice is to configure and bind at least one workqueue for each IAA device, but as long as there is at least one workqueue configured and bound to any IAA device in the system, the iaa_crypto driver will work, albeit most likely not as efficiently. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi --- drivers/crypto/iaa/iaa_crypto.h | 7 + drivers/crypto/iaa/iaa_crypto_main.c | 244 +++++++++++++++++++++++++++ 2 files changed, 251 insertions(+) diff --git a/drivers/crypto/iaa/iaa_crypto.h b/drivers/crypto/iaa/iaa_crypto.h index 5d1fff7f4b8e..c25546fa87f7 100644 --- a/drivers/crypto/iaa/iaa_crypto.h +++ b/drivers/crypto/iaa/iaa_crypto.h @@ -27,4 +27,11 @@ struct iaa_device { struct list_head wqs; }; +struct wq_table_entry { + struct idxd_wq **wqs; + int max_wqs; + int n_wqs; + int cur_wq; +}; + #endif diff --git a/drivers/crypto/iaa/iaa_crypto_main.c b/drivers/crypto/iaa/iaa_crypto_main.c index e7e7fbdd72e3..016e23aeb147 100644 --- a/drivers/crypto/iaa/iaa_crypto_main.c +++ b/drivers/crypto/iaa/iaa_crypto_main.c @@ -22,10 +22,92 @@ /* number of iaa instances probed */ static unsigned int nr_iaa; +static unsigned int nr_cpus; +static unsigned int nr_nodes; +static unsigned int nr_cpus_per_node; + +/* Number of physical cpus sharing each iaa instance */ +static unsigned int cpus_per_iaa; + +/* Per-cpu lookup table for balanced wqs */ +static struct wq_table_entry __percpu *wq_table; + +static void wq_table_add(int cpu, struct idxd_wq *wq) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + if (WARN_ON(entry->n_wqs == entry->max_wqs)) + return; + + entry->wqs[entry->n_wqs++] = wq; + + pr_debug("%s: added iaa wq %d.%d to idx %d of cpu %d\n", __func__, + entry->wqs[entry->n_wqs - 1]->idxd->id, + entry->wqs[entry->n_wqs - 1]->id, entry->n_wqs - 1, cpu); +} + +static void wq_table_free_entry(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + kfree(entry->wqs); + memset(entry, 0, sizeof(*entry)); +} + +static void wq_table_clear_entry(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + entry->n_wqs = 0; + entry->cur_wq = 0; + memset(entry->wqs, 0, entry->max_wqs * sizeof(struct idxd_wq *)); +} static LIST_HEAD(iaa_devices); static DEFINE_MUTEX(iaa_devices_lock); +/* + * Given a cpu, find the closest IAA instance. The idea is to try to + * choose the most appropriate IAA instance for a caller and spread + * available workqueues around to clients. + */ +static inline int cpu_to_iaa(int cpu) +{ + int node, n_cpus = 0, test_cpu, iaa = 0; + int nr_iaa_per_node, nr_cores_per_iaa; + const struct cpumask *node_cpus; + + if (!nr_nodes) + return 0; + + nr_iaa_per_node = nr_iaa / nr_nodes; + if (!nr_iaa_per_node) + return 0; + + nr_cores_per_iaa = nr_cpus_per_node / nr_iaa_per_node; + + for_each_online_node(node) { + node_cpus = cpumask_of_node(node); + if (!cpumask_test_cpu(cpu, node_cpus)) + continue; + + for_each_cpu(test_cpu, node_cpus) { + if ((n_cpus % nr_cpus_per_node) == 0) + iaa = node * nr_iaa_per_node; + + if (test_cpu == cpu) + return iaa; + + n_cpus++; + + if ((n_cpus % cpus_per_iaa) == 0) + iaa++; + } + } + + return -1; +} + static struct iaa_device *iaa_device_alloc(void) { struct iaa_device *iaa_device; @@ -141,6 +223,53 @@ static void del_iaa_wq(struct iaa_device *iaa_device, struct idxd_wq *wq) } } +static void clear_wq_table(void) +{ + int cpu; + + for (cpu = 0; cpu < nr_cpus; cpu++) + wq_table_clear_entry(cpu); + + pr_debug("cleared wq table\n"); +} + +static void free_wq_table(void) +{ + int cpu; + + for (cpu = 0; cpu < nr_cpus; cpu++) + wq_table_free_entry(cpu); + + free_percpu(wq_table); + + pr_debug("freed wq table\n"); +} + +static int alloc_wq_table(int max_wqs) +{ + struct wq_table_entry *entry; + int cpu; + + wq_table = alloc_percpu(struct wq_table_entry); + if (!wq_table) + return -ENOMEM; + + for (cpu = 0; cpu < nr_cpus; cpu++) { + entry = per_cpu_ptr(wq_table, cpu); + entry->wqs = kzalloc(GFP_KERNEL, max_wqs * sizeof(struct wq *)); + if (!entry->wqs) { + free_wq_table(); + return -ENOMEM; + } + + entry->max_wqs = max_wqs; + } + + pr_debug("initialized wq table\n"); + + return 0; +} + static int save_iaa_wq(struct idxd_wq *wq) { struct iaa_device *iaa_device, *found = NULL; @@ -195,6 +324,8 @@ static int save_iaa_wq(struct idxd_wq *wq) return -EINVAL; idxd_wq_get(wq); + + cpus_per_iaa = (nr_nodes * nr_cpus_per_node) / nr_iaa; out: return 0; } @@ -210,6 +341,97 @@ static void remove_iaa_wq(struct idxd_wq *wq) break; } } + + if (nr_iaa) + cpus_per_iaa = (nr_nodes * nr_cpus_per_node) / nr_iaa; + else + cpus_per_iaa = 0; +} + +static int wq_table_add_wqs(int iaa, int cpu) +{ + struct iaa_device *iaa_device, *found_device = NULL; + int ret = 0, cur_iaa = 0, n_wqs_added = 0; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + + list_for_each_entry(iaa_device, &iaa_devices, list) { + idxd = iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + if (cur_iaa != iaa) { + cur_iaa++; + continue; + } + + found_device = iaa_device; + dev_dbg(dev, "getting wq from iaa_device %d, cur_iaa %d\n", + found_device->idxd->id, cur_iaa); + break; + } + + if (!found_device) { + found_device = list_first_entry_or_null(&iaa_devices, + struct iaa_device, list); + if (!found_device) { + pr_debug("couldn't find any iaa devices with wqs!\n"); + ret = -EINVAL; + goto out; + } + cur_iaa = 0; + + idxd = found_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + dev_dbg(dev, "getting wq from only iaa_device %d, cur_iaa %d\n", + found_device->idxd->id, cur_iaa); + } + + list_for_each_entry(iaa_wq, &found_device->wqs, list) { + wq_table_add(cpu, iaa_wq->wq); + pr_debug("rebalance: added wq for cpu=%d: iaa wq %d.%d\n", + cpu, iaa_wq->wq->idxd->id, iaa_wq->wq->id); + n_wqs_added++; + }; + + if (!n_wqs_added) { + pr_debug("couldn't find any iaa wqs!\n"); + ret = -EINVAL; + goto out; + } +out: + return ret; +} + +static void rebalance_wq_table(void) +{ + int cpu, iaa; + + if (nr_iaa == 0) + return; + + clear_wq_table(); + + pr_debug("rebalance: nr_nodes=%d, nr_cpus %d, nr_iaa %d, cpus_per_iaa %d\n", + nr_nodes, nr_cpus, nr_iaa, cpus_per_iaa); + + for (cpu = 0; cpu < nr_cpus; cpu++) { + iaa = cpu_to_iaa(cpu); + pr_debug("rebalance: cpu=%d iaa=%d\n", cpu, iaa); + + if (WARN_ON(iaa == -1)) { + pr_debug("rebalance (cpu_to_iaa(%d)) failed!\n", cpu); + return; + } + + if (WARN_ON(wq_table_add_wqs(iaa, cpu))) { + pr_debug("could not add any wqs for iaa %d to cpu %d!\n", iaa, cpu); + return; + } + } } static int iaa_crypto_probe(struct idxd_dev *idxd_dev) @@ -218,6 +440,7 @@ static int iaa_crypto_probe(struct idxd_dev *idxd_dev) struct idxd_device *idxd = wq->idxd; struct idxd_driver_data *data = idxd->data; struct device *dev = &idxd_dev->conf_dev; + bool first_wq = false; int ret = 0; if (idxd->state != IDXD_DEV_ENABLED) @@ -248,10 +471,19 @@ static int iaa_crypto_probe(struct idxd_dev *idxd_dev) mutex_lock(&iaa_devices_lock); + if (list_empty(&iaa_devices)) { + ret = alloc_wq_table(wq->idxd->max_wqs); + if (ret) + goto err_alloc; + first_wq = true; + } + ret = save_iaa_wq(wq); if (ret) goto err_save; + rebalance_wq_table(); + mutex_unlock(&iaa_devices_lock); out: mutex_unlock(&wq->wq_lock); @@ -259,6 +491,10 @@ static int iaa_crypto_probe(struct idxd_dev *idxd_dev) return ret; err_save: + if (first_wq) + free_wq_table(); +err_alloc: + mutex_unlock(&iaa_devices_lock); drv_disable_wq(wq); err: wq->type = IDXD_WQT_NONE; @@ -277,6 +513,10 @@ static void iaa_crypto_remove(struct idxd_dev *idxd_dev) remove_iaa_wq(wq); drv_disable_wq(wq); + rebalance_wq_table(); + + if (nr_iaa == 0) + free_wq_table(); mutex_unlock(&iaa_devices_lock); mutex_unlock(&wq->wq_lock); @@ -298,6 +538,10 @@ static int __init iaa_crypto_init_module(void) { int ret = 0; + nr_cpus = num_online_cpus(); + nr_nodes = num_online_nodes(); + nr_cpus_per_node = boot_cpu_data.x86_max_cores; + ret = idxd_driver_register(&iaa_crypto_driver); if (ret) { pr_debug("IAA wq sub-driver registration failed\n"); From patchwork Mon Mar 6 18:52:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65062 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2016059wrd; Mon, 6 Mar 2023 10:57:09 -0800 (PST) X-Google-Smtp-Source: AK7set9FD8FgV0LUIKZ9J/Jjc+U2x7c7SGMx48ch3Duz20hqdwYEYO2wSoXhwUaHI63FQNdIeUye X-Received: by 2002:a17:906:374e:b0:8af:2af5:1191 with SMTP id e14-20020a170906374e00b008af2af51191mr12465675ejc.18.1678129029694; Mon, 06 Mar 2023 10:57:09 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678129029; cv=none; d=google.com; s=arc-20160816; b=mMmK7xKK4N47WuMaPkonT/j4mR6FYxDq4f4OZQFj70lnfmOq+XQCEKBE/TVWEqZBVF BZ+c5XAZwUufI8TXsi4lijENw4xscme2EySkaupnV3R/9uN4jE/UnRg+NQ5GzSVoGFmN Tk0nprlW3CxrBrmYDNruNo0pFunM/C/YhlG1VXWRYb0Gk8o5Bgp4NhdmWKvLSNnJbY/T 4LwJq1zi8SdxtaJdTaoWhkHdplwG4pqpfPjuT7ZmZtiFL3xTmykrGuA1sNUVOk8cyMVN yAvKDixj9bvecF0dmRe3kN4QpWt1LUrheArGwv3X9ZMLpNB2xx1SngoFiIZ+GdPnWer9 RMbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=y/R1c9knaNkeFemCkCbAYWY2aWIrrwye1/flys+SGKU=; b=Lr47r1ap8AvmlNQ2Gk1+eksqhBNKJFG70Jf+VjBovFD2uBGmayrZ+8TYMx2hVXVci0 nH31gAdG+BMNvy74D8eGeyeJpgW6KCu83o8TqkqMKUWoIV7QonwnBJ1H2iZKpNf4iiRW Xnd/d50idu06E6IvptGBHVe/0FR4iknEWFjtePnqZKsfnm5eQW81p0uoQ0vmiN715Yys HsnpTItVfAHquZER+zVLRUWqKZ4/GgZwcqIxwrv/KqHheohi/fRJoofz7KMwZEs7FJjT IwEHJvScNJCEBtGj0qRhW8nxrPixc9W2r3PyEJW/wxscULyli+lzUDufT+Ia4cmFa3FR /k8A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=K2liiGlM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i19-20020a170906a29300b008b138238f31si9934367ejz.970.2023.03.06.10.56.46; Mon, 06 Mar 2023 10:57:09 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=K2liiGlM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230373AbjCFSyQ (ORCPT + 99 others); Mon, 6 Mar 2023 13:54:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229874AbjCFSxi (ORCPT ); Mon, 6 Mar 2023 13:53:38 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0A4A3B67C; Mon, 6 Mar 2023 10:53:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128781; x=1709664781; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bHpQD+NzS3N+3eoiRG6yDBYPjwV/mVMktF7PXvykG94=; b=K2liiGlMcA6oXpoUECkmLdzeeIskoEeibJSXiSF1vyaIgnZg4NVt4hXX VHscR2Zdz6KjflBZc91Ar85dCNpN8hF/FnCpGa/LIVwEz54PZTf5T3S+6 DQQHMvKJkTm197khuTBPthASYUHfSKeaUb7DKNLpHQkuY+syOGWJQE2eQ JFfZyT+40GXQN3+Ru47aNY0I93sDnRXNQw9lN7Irs6PLaHIsBQ6Dsv2As Fhu3X3Ipq18oCNQXY/+hH2ZIR0q+Lt/H9ZBc2F68lbvPoBGGZYDUHx0bi z5H2E0mnHXC24QuV27Tser6BOWxFDXHBZk+uICTfNQql5UHUd668wTeww g==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227773" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227773" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255883" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255883" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:57 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 11/16] crypto: iaa - Add compression mode management along with fixed mode Date: Mon, 6 Mar 2023 12:52:21 -0600 Message-Id: <20230306185226.26483-12-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645825677607256?= X-GMAIL-MSGID: =?utf-8?q?1759645825677607256?= Additionally, it defines an in-kernel API for adding, removing, and setting compression schemes, which can be used by kernel modules or other kernel code that implements IAA compression schemes. It also adds a separate file, iaa_crypto_comp_fixed.c, containing huffman tables generated for a compression scheme named 'fixed'. Future compression schemes can be added in a similar fashion. The compression mode in effect can be selected by the user via the new iaa_crypto 'compression_mode' driver attribute. Currently, there is only one compression mode available, 'fixed' mode: echo "fixed" > /sys/bus/dsa/drivers/crypto/compression_mode Signed-off-by: Tom Zanussi --- drivers/crypto/iaa/Makefile | 2 +- drivers/crypto/iaa/iaa_crypto.h | 87 +++++ drivers/crypto/iaa/iaa_crypto_comp_fixed.c | 92 +++++ drivers/crypto/iaa/iaa_crypto_main.c | 421 +++++++++++++++++++++ 4 files changed, 601 insertions(+), 1 deletion(-) create mode 100644 drivers/crypto/iaa/iaa_crypto_comp_fixed.c diff --git a/drivers/crypto/iaa/Makefile b/drivers/crypto/iaa/Makefile index 03859431c897..cc87feffd059 100644 --- a/drivers/crypto/iaa/Makefile +++ b/drivers/crypto/iaa/Makefile @@ -7,4 +7,4 @@ ccflags-y += -I $(srctree)/drivers/dma/idxd -DDEFAULT_SYMBOL_NAMESPACE=IDXD obj-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO) := iaa_crypto.o -iaa_crypto-y := iaa_crypto_main.o +iaa_crypto-y := iaa_crypto_main.o iaa_crypto_comp_fixed.o diff --git a/drivers/crypto/iaa/iaa_crypto.h b/drivers/crypto/iaa/iaa_crypto.h index c25546fa87f7..2daa3522e073 100644 --- a/drivers/crypto/iaa/iaa_crypto.h +++ b/drivers/crypto/iaa/iaa_crypto.h @@ -10,6 +10,11 @@ #define IDXD_SUBDRIVER_NAME "crypto" +#define IAA_COMP_MODES_MAX 2 + +#define FIXED_HDR 0x2 +#define FIXED_HDR_SIZE 3 + /* Representation of IAA workqueue */ struct iaa_wq { struct list_head list; @@ -18,11 +23,24 @@ struct iaa_wq { struct iaa_device *iaa_device; }; +struct iaa_device_compression_mode { + const char *name; + + struct aecs_comp_table_record *aecs_comp_table; + struct aecs_decomp_table_record *aecs_decomp_table; + + dma_addr_t aecs_comp_table_dma_addr; + dma_addr_t aecs_decomp_table_dma_addr; +}; + /* Representation of IAA device with wqs, populated by probe */ struct iaa_device { struct list_head list; struct idxd_device *idxd; + struct iaa_device_compression_mode *compression_modes[IAA_COMP_MODES_MAX]; + struct iaa_device_compression_mode *active_compression_mode; + int n_wq; struct list_head wqs; }; @@ -34,4 +52,73 @@ struct wq_table_entry { int cur_wq; }; +#define IAA_AECS_ALIGN 32 + +/* + * Analytics Engine Configuration and State (AECS) contains parameters and + * internal state of the analytics engine. + */ +struct aecs_comp_table_record { + u32 crc; + u32 xor_checksum; + u32 reserved0[5]; + u32 num_output_accum_bits; + u8 output_accum[256]; + u32 ll_sym[286]; + u32 reserved1; + u32 reserved2; + u32 d_sym[30]; + u32 reserved_padding[2]; +} __packed; + +/* AECS for decompress */ +struct aecs_decomp_table_record { + u32 crc; + u32 xor_checksum; + u32 low_filter_param; + u32 high_filter_param; + u32 output_mod_idx; + u32 drop_init_decomp_out_bytes; + u32 reserved[36]; + u32 output_accum_data[2]; + u32 out_bits_valid; + u32 bit_off_indexing; + u32 input_accum_data[64]; + u8 size_qw[32]; + u32 decomp_state[1220]; +} __packed; + +int iaa_aecs_init_fixed(void); +void iaa_aecs_cleanup_fixed(void); + +typedef int (*iaa_dev_comp_init_fn_t) (struct iaa_device_compression_mode *mode); +typedef int (*iaa_dev_comp_free_fn_t) (struct iaa_device_compression_mode *mode); + +struct iaa_compression_mode { + const char *name; + u32 *ll_table; + int ll_table_size; + u32 *d_table; + int d_table_size; + u32 *header_table; + int header_table_size; + u16 gen_decomp_table_flags; + iaa_dev_comp_init_fn_t init; + iaa_dev_comp_free_fn_t free; +}; + +int add_iaa_compression_mode(const char *name, + const u32 *ll_table, + int ll_table_size, + const u32 *d_table, + int d_table_size, + const u8 *header_table, + int header_table_size, + u16 gen_decomp_table_flags, + iaa_dev_comp_init_fn_t init, + iaa_dev_comp_free_fn_t free); + +void remove_iaa_compression_mode(const char *name); +int set_iaa_compression_mode(const char *name); + #endif diff --git a/drivers/crypto/iaa/iaa_crypto_comp_fixed.c b/drivers/crypto/iaa/iaa_crypto_comp_fixed.c new file mode 100644 index 000000000000..e965da11b4d9 --- /dev/null +++ b/drivers/crypto/iaa/iaa_crypto_comp_fixed.c @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */ + +#include "idxd.h" +#include "iaa_crypto.h" + +/* + * Fixed Huffman tables the IAA hardware requires to implement RFC-1951. + */ +const u32 fixed_ll_sym[286] = { + 0x40030, 0x40031, 0x40032, 0x40033, 0x40034, 0x40035, 0x40036, 0x40037, + 0x40038, 0x40039, 0x4003A, 0x4003B, 0x4003C, 0x4003D, 0x4003E, 0x4003F, + 0x40040, 0x40041, 0x40042, 0x40043, 0x40044, 0x40045, 0x40046, 0x40047, + 0x40048, 0x40049, 0x4004A, 0x4004B, 0x4004C, 0x4004D, 0x4004E, 0x4004F, + 0x40050, 0x40051, 0x40052, 0x40053, 0x40054, 0x40055, 0x40056, 0x40057, + 0x40058, 0x40059, 0x4005A, 0x4005B, 0x4005C, 0x4005D, 0x4005E, 0x4005F, + 0x40060, 0x40061, 0x40062, 0x40063, 0x40064, 0x40065, 0x40066, 0x40067, + 0x40068, 0x40069, 0x4006A, 0x4006B, 0x4006C, 0x4006D, 0x4006E, 0x4006F, + 0x40070, 0x40071, 0x40072, 0x40073, 0x40074, 0x40075, 0x40076, 0x40077, + 0x40078, 0x40079, 0x4007A, 0x4007B, 0x4007C, 0x4007D, 0x4007E, 0x4007F, + 0x40080, 0x40081, 0x40082, 0x40083, 0x40084, 0x40085, 0x40086, 0x40087, + 0x40088, 0x40089, 0x4008A, 0x4008B, 0x4008C, 0x4008D, 0x4008E, 0x4008F, + 0x40090, 0x40091, 0x40092, 0x40093, 0x40094, 0x40095, 0x40096, 0x40097, + 0x40098, 0x40099, 0x4009A, 0x4009B, 0x4009C, 0x4009D, 0x4009E, 0x4009F, + 0x400A0, 0x400A1, 0x400A2, 0x400A3, 0x400A4, 0x400A5, 0x400A6, 0x400A7, + 0x400A8, 0x400A9, 0x400AA, 0x400AB, 0x400AC, 0x400AD, 0x400AE, 0x400AF, + 0x400B0, 0x400B1, 0x400B2, 0x400B3, 0x400B4, 0x400B5, 0x400B6, 0x400B7, + 0x400B8, 0x400B9, 0x400BA, 0x400BB, 0x400BC, 0x400BD, 0x400BE, 0x400BF, + 0x48190, 0x48191, 0x48192, 0x48193, 0x48194, 0x48195, 0x48196, 0x48197, + 0x48198, 0x48199, 0x4819A, 0x4819B, 0x4819C, 0x4819D, 0x4819E, 0x4819F, + 0x481A0, 0x481A1, 0x481A2, 0x481A3, 0x481A4, 0x481A5, 0x481A6, 0x481A7, + 0x481A8, 0x481A9, 0x481AA, 0x481AB, 0x481AC, 0x481AD, 0x481AE, 0x481AF, + 0x481B0, 0x481B1, 0x481B2, 0x481B3, 0x481B4, 0x481B5, 0x481B6, 0x481B7, + 0x481B8, 0x481B9, 0x481BA, 0x481BB, 0x481BC, 0x481BD, 0x481BE, 0x481BF, + 0x481C0, 0x481C1, 0x481C2, 0x481C3, 0x481C4, 0x481C5, 0x481C6, 0x481C7, + 0x481C8, 0x481C9, 0x481CA, 0x481CB, 0x481CC, 0x481CD, 0x481CE, 0x481CF, + 0x481D0, 0x481D1, 0x481D2, 0x481D3, 0x481D4, 0x481D5, 0x481D6, 0x481D7, + 0x481D8, 0x481D9, 0x481DA, 0x481DB, 0x481DC, 0x481DD, 0x481DE, 0x481DF, + 0x481E0, 0x481E1, 0x481E2, 0x481E3, 0x481E4, 0x481E5, 0x481E6, 0x481E7, + 0x481E8, 0x481E9, 0x481EA, 0x481EB, 0x481EC, 0x481ED, 0x481EE, 0x481EF, + 0x481F0, 0x481F1, 0x481F2, 0x481F3, 0x481F4, 0x481F5, 0x481F6, 0x481F7, + 0x481F8, 0x481F9, 0x481FA, 0x481FB, 0x481FC, 0x481FD, 0x481FE, 0x481FF, + 0x38000, 0x38001, 0x38002, 0x38003, 0x38004, 0x38005, 0x38006, 0x38007, + 0x38008, 0x38009, 0x3800A, 0x3800B, 0x3800C, 0x3800D, 0x3800E, 0x3800F, + 0x38010, 0x38011, 0x38012, 0x38013, 0x38014, 0x38015, 0x38016, 0x38017, + 0x400C0, 0x400C1, 0x400C2, 0x400C3, 0x400C4, 0x400C5 +}; + +const u32 fixed_d_sym[30] = { + 0x28000, 0x28001, 0x28002, 0x28003, 0x28004, 0x28005, 0x28006, 0x28007, + 0x28008, 0x28009, 0x2800A, 0x2800B, 0x2800C, 0x2800D, 0x2800E, 0x2800F, + 0x28010, 0x28011, 0x28012, 0x28013, 0x28014, 0x28015, 0x28016, 0x28017, + 0x28018, 0x28019, 0x2801A, 0x2801B, 0x2801C, 0x2801D +}; + +static int init_fixed_mode(struct iaa_device_compression_mode *mode) +{ + struct aecs_comp_table_record *comp_table = mode->aecs_comp_table; + u32 bfinal = 1; + u32 offset; + + /* Configure aecs table using fixed Huffman table */ + comp_table->crc = 0; + comp_table->xor_checksum = 0; + offset = comp_table->num_output_accum_bits / 8; + comp_table->output_accum[offset] = FIXED_HDR | bfinal; + comp_table->num_output_accum_bits = FIXED_HDR_SIZE; + + return 0; +} + +int iaa_aecs_init_fixed(void) +{ + int ret; + + ret = add_iaa_compression_mode("fixed", + fixed_ll_sym, + sizeof(fixed_ll_sym), + fixed_d_sym, + sizeof(fixed_d_sym), + NULL, 0, 0, + init_fixed_mode, NULL); + if (!ret) + pr_debug("IAA fixed compression mode initialized\n"); + + return ret; +} + +void iaa_aecs_cleanup_fixed(void) +{ + remove_iaa_compression_mode("fixed"); +} diff --git a/drivers/crypto/iaa/iaa_crypto_main.c b/drivers/crypto/iaa/iaa_crypto_main.c index 016e23aeb147..9bd42cfa949e 100644 --- a/drivers/crypto/iaa/iaa_crypto_main.c +++ b/drivers/crypto/iaa/iaa_crypto_main.c @@ -66,6 +66,381 @@ static void wq_table_clear_entry(int cpu) static LIST_HEAD(iaa_devices); static DEFINE_MUTEX(iaa_devices_lock); +static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; +static int active_compression_mode; + +static ssize_t compression_mode_show(struct device_driver *driver, char *buf) +{ + int ret = 0; + + ret = sprintf(buf, "%s\n", "fixed"); + + return ret; +} + +static ssize_t compression_mode_store(struct device_driver *driver, + const char *buf, size_t count) +{ + int ret = -EBUSY; + char *mode_name; + + mutex_lock(&iaa_devices_lock); + + mode_name = kstrndup(buf, count, GFP_KERNEL); + if (!mode_name) { + ret = -ENOMEM; + goto out; + } + + ret = set_iaa_compression_mode(strim(mode_name)); + if (ret == 0) + ret = count; + + kfree(mode_name); +out: + mutex_unlock(&iaa_devices_lock); + + return ret; +} +static DRIVER_ATTR_RW(compression_mode); + +static int find_empty_iaa_compression_mode(void) +{ + int i = -EINVAL; + + for (i = 0; i < IAA_COMP_MODES_MAX; i++) { + if (iaa_compression_modes[i]) + continue; + break; + } + + return i; +} + +static struct iaa_compression_mode *find_iaa_compression_mode(const char *name, int *idx) +{ + struct iaa_compression_mode *mode; + int i; + + for (i = 0; i < IAA_COMP_MODES_MAX; i++) { + mode = iaa_compression_modes[i]; + if (!mode) + continue; + + if (!strcmp(mode->name, name)) { + *idx = i; + return iaa_compression_modes[i]; + } + } + + return NULL; +} + +static void free_iaa_compression_mode(struct iaa_compression_mode *mode) +{ + kfree(mode->name); + kfree(mode->ll_table); + kfree(mode->d_table); + kfree(mode->header_table); + + kfree(mode); +} + +/* + * IAA Compression modes are defined by an ll_table, a d_table, and an + * optional header_table. These tables are typically generated and + * captured using statistics collected from running actual + * compress/decompress workloads. + * + * A module or other kernel code can add and remove compression modes + * with a given name using the exported @add_iaa_compression_mode() + * and @remove_iaa_compression_mode functions. + * + * Successfully added compression modes can be selected using the + * function @set_iaa_compression_mode(), passing in the name of the + * compression mode. Henceforth, all compressions and decompressions + * will use the given compression mode. Any in-flight decompressions + * using the old mode will subsequently fail. + * + * When a new compression mode is added, the tables are saved in a + * global compression mode list. When IAA devices are added, a + * per-IAA device dma mapping is created for each IAA device, for each + * compression mode. These are the tables used to do the actual + * compression/deccompression and are unmapped if/when the devices are + * removed. Currently, compression modes must be added before any + * device is added, and removed after all devices have been removed. + */ + +/** + * remove_iaa_compression_mode - Remove an IAA compression mode + * @name: The name the compression mode will be known as + * + * Remove the IAA compression mode named @name. + */ +void remove_iaa_compression_mode(const char *name) +{ + struct iaa_compression_mode *mode; + int idx; + + mutex_lock(&iaa_devices_lock); + + if (!list_empty(&iaa_devices)) + goto out; + + mode = find_iaa_compression_mode(name, &idx); + if (mode) { + free_iaa_compression_mode(mode); + iaa_compression_modes[idx] = NULL; + } +out: + mutex_unlock(&iaa_devices_lock); +} +EXPORT_SYMBOL_GPL(remove_iaa_compression_mode); + +/** + * add_iaa_compression_mode - Add an IAA compression mode + * @name: The name the compression mode will be known as + * @ll_table: The ll table + * @ll_table_size: The ll table size in bytes + * @d_table: The d table + * @d_table_size: The d table size in bytes + * @header_table: Optional header table + * @header_table_size: Optional header table size in bytes + * @gen_decomp_table_flags: Otional flags used to generate the decomp table + * @init: Optional callback function to init the compression mode data + * @free: Optional callback function to free the compression mode data + * + * Add a new IAA compression mode named @name. If successful, @name + * can subsequently be given to @set_iaa_compression_mode() to make + * that mode the current mode for iaa compression/decompression. + * + * Returns 0 if successful, errcode otherwise. + */ +int add_iaa_compression_mode(const char *name, + const u32 *ll_table, + int ll_table_size, + const u32 *d_table, + int d_table_size, + const u8 *header_table, + int header_table_size, + u16 gen_decomp_table_flags, + iaa_dev_comp_init_fn_t init, + iaa_dev_comp_free_fn_t free) +{ + struct iaa_compression_mode *mode; + int idx, ret = -ENOMEM; + + mutex_lock(&iaa_devices_lock); + + if (!list_empty(&iaa_devices)) { + ret = -EBUSY; + goto out; + } + + mode = kzalloc(sizeof(*mode), GFP_KERNEL); + if (!mode) + goto out; + + mode->name = kstrdup(name, GFP_KERNEL); + if (!mode->name) + goto free; + + if (ll_table) { + mode->ll_table = kzalloc(ll_table_size, GFP_KERNEL); + if (!mode->ll_table) + goto free; + memcpy(mode->ll_table, ll_table, ll_table_size); + mode->ll_table_size = ll_table_size; + } + + if (d_table) { + mode->d_table = kzalloc(d_table_size, GFP_KERNEL); + if (!mode->d_table) + goto free; + memcpy(mode->d_table, d_table, d_table_size); + mode->d_table_size = d_table_size; + } + + if (header_table) { + mode->header_table = kzalloc(header_table_size, GFP_KERNEL); + if (!mode->header_table) + goto free; + memcpy(mode->header_table, header_table, header_table_size); + mode->header_table_size = header_table_size; + } + + mode->gen_decomp_table_flags = gen_decomp_table_flags; + + mode->init = init; + mode->free = free; + + idx = find_empty_iaa_compression_mode(); + if (idx < 0) + goto free; + + pr_debug("IAA compression mode %s added at idx %d\n", + mode->name, idx); + + iaa_compression_modes[idx] = mode; + + ret = 0; +out: + mutex_unlock(&iaa_devices_lock); + + return ret; +free: + free_iaa_compression_mode(mode); + goto out; +} +EXPORT_SYMBOL_GPL(add_iaa_compression_mode); + +static void set_iaa_device_compression_mode(struct iaa_device *iaa_device, int idx) +{ + iaa_device->active_compression_mode = iaa_device->compression_modes[idx]; +} + +static void update_iaa_devices_compression_mode(void) +{ + struct iaa_device *iaa_device; + + list_for_each_entry(iaa_device, &iaa_devices, list) + set_iaa_device_compression_mode(iaa_device, active_compression_mode); +} + +/** + * set_iaa_compression_mode - Set an IAA compression mode + * @name: The name of the compression mode + * + * Make the IAA compression mode named @name the current compression + * mode used by compression/decompression. + */ + +int set_iaa_compression_mode(const char *name) +{ + struct iaa_compression_mode *mode; + int ret = -EINVAL; + int idx; + + mode = find_iaa_compression_mode(name, &idx); + if (mode) { + active_compression_mode = idx; + update_iaa_devices_compression_mode(); + pr_debug("compression mode set to: %s\n", name); + ret = 0; + } + + return ret; +} + +static void free_device_compression_mode(struct iaa_device *iaa_device, + struct iaa_device_compression_mode *device_mode) +{ + size_t size = sizeof(struct aecs_comp_table_record) + IAA_AECS_ALIGN; + struct device *dev = &iaa_device->idxd->pdev->dev; + + kfree(device_mode->name); + + if (device_mode->aecs_comp_table) + dma_free_coherent(dev, size, device_mode->aecs_comp_table, + device_mode->aecs_comp_table_dma_addr); + if (device_mode->aecs_decomp_table) + dma_free_coherent(dev, size, device_mode->aecs_decomp_table, + device_mode->aecs_decomp_table_dma_addr); + + kfree(device_mode); +} + +static int init_device_compression_mode(struct iaa_device *iaa_device, + struct iaa_compression_mode *mode, + int idx, struct idxd_wq *wq) +{ + size_t size = sizeof(struct aecs_comp_table_record) + IAA_AECS_ALIGN; + struct device *dev = &iaa_device->idxd->pdev->dev; + struct iaa_device_compression_mode *device_mode; + int ret = -ENOMEM; + + device_mode = kzalloc(sizeof(*device_mode), GFP_KERNEL); + if (!device_mode) + return -ENOMEM; + + device_mode->name = kstrdup(mode->name, GFP_KERNEL); + if (!device_mode->name) + goto free; + + device_mode->aecs_comp_table = dma_alloc_coherent(dev, size, + &device_mode->aecs_comp_table_dma_addr, GFP_KERNEL); + if (!device_mode->aecs_comp_table) + goto free; + + device_mode->aecs_decomp_table = dma_alloc_coherent(dev, size, + &device_mode->aecs_decomp_table_dma_addr, GFP_KERNEL); + if (!device_mode->aecs_decomp_table) + goto free; + + /* Add Huffman table to aecs */ + memset(device_mode->aecs_comp_table, 0, sizeof(*device_mode->aecs_comp_table)); + memcpy(device_mode->aecs_comp_table->ll_sym, mode->ll_table, mode->ll_table_size); + memcpy(device_mode->aecs_comp_table->d_sym, mode->d_table, mode->d_table_size); + + if (mode->init) { + ret = mode->init(device_mode); + if (ret) + goto free; + } + + /* mode index should match iaa_compression_modes idx */ + iaa_device->compression_modes[idx] = device_mode; + + pr_debug("IAA %s compression mode initialized for iaa device %d\n", + mode->name, iaa_device->idxd->id); + + ret = 0; +out: + return ret; +free: + pr_debug("IAA %s compression mode initialization failed for iaa device %d\n", + mode->name, iaa_device->idxd->id); + + free_device_compression_mode(iaa_device, device_mode); + goto out; +} + +static int init_device_compression_modes(struct iaa_device *iaa_device, + struct idxd_wq *wq) +{ + struct iaa_compression_mode *mode; + int i, ret = 0; + + for (i = 0; i < IAA_COMP_MODES_MAX; i++) { + mode = iaa_compression_modes[i]; + if (!mode) + continue; + + ret = init_device_compression_mode(iaa_device, mode, i, wq); + if (ret) + break; + } + + return ret; +} + +static void remove_device_compression_modes(struct iaa_device *iaa_device) +{ + struct iaa_device_compression_mode *device_mode; + int i; + + for (i = 0; i < IAA_COMP_MODES_MAX; i++) { + device_mode = iaa_device->compression_modes[i]; + if (!device_mode) + continue; + + free_device_compression_mode(iaa_device, device_mode); + iaa_device->compression_modes[i] = NULL; + if (iaa_compression_modes[i]->free) + iaa_compression_modes[i]->free(device_mode); + } +} + /* * Given a cpu, find the closest IAA instance. The idea is to try to * choose the most appropriate IAA instance for a caller and spread @@ -162,8 +537,23 @@ static struct iaa_device *add_iaa_device(struct idxd_device *idxd) return iaa_device; } +static int init_iaa_device(struct iaa_device *iaa_device, struct iaa_wq *iaa_wq) +{ + int ret = 0; + + ret = init_device_compression_modes(iaa_device, iaa_wq->wq); + if (ret) + return ret; + + set_iaa_device_compression_mode(iaa_device, active_compression_mode); + + return ret; +} + static void del_iaa_device(struct iaa_device *iaa_device) { + remove_device_compression_modes(iaa_device); + list_del(&iaa_device->list); iaa_device_free(iaa_device); @@ -318,6 +708,13 @@ static int save_iaa_wq(struct idxd_wq *wq) del_iaa_device(new_device); goto out; } + + ret = init_iaa_device(new_device, new_wq); + if (ret) { + del_iaa_wq(new_device, new_wq->wq); + del_iaa_device(new_device); + goto out; + } } if (WARN_ON(nr_iaa == 0)) @@ -548,14 +945,38 @@ static int __init iaa_crypto_init_module(void) goto out; } + ret = driver_create_file(&iaa_crypto_driver.drv, + &driver_attr_compression_mode); + if (ret) { + pr_debug("IAA compression mode attr creation failed\n"); + goto err_attr_create; + } + + ret = iaa_aecs_init_fixed(); + if (ret < 0) { + pr_debug("IAA fixed compression mode init failed\n"); + goto err_compression_mode; + } + pr_debug("initialized\n"); out: return ret; + +err_compression_mode: + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_compression_mode); +err_attr_create: + idxd_driver_unregister(&iaa_crypto_driver); + + goto out; } static void __exit iaa_crypto_cleanup_module(void) { + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_compression_mode); idxd_driver_unregister(&iaa_crypto_driver); + iaa_aecs_cleanup_fixed(); pr_debug("cleaned up\n"); } From patchwork Mon Mar 6 18:52:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65063 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2016105wrd; Mon, 6 Mar 2023 10:57:17 -0800 (PST) X-Google-Smtp-Source: AK7set+TgU6j/UIpQsK05KL7h5MxNWvxikHofDpyWiU3KzSFhDOSKtmY3kFd/TVasPpto5oSE8Ik X-Received: by 2002:aa7:d856:0:b0:4af:6e95:861b with SMTP id f22-20020aa7d856000000b004af6e95861bmr17382279eds.2.1678129036069; Mon, 06 Mar 2023 10:57:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678129036; cv=none; d=google.com; s=arc-20160816; b=usiKo/DHDsh7OBDIUQ/en94JS60r0FjiW8AVQfv60Esr0gZfCPl/CFWPS2mp1nmwgi +xEnPA/0NsceYY9ATbHIEnjJ20RybKMwR4zTDRaiDoBFaiVgcFFLbKIKWq4c/JhurMGT 6pCAB77zUu+h+3Ym9GZbcqO8URNM7WtCBa10UzrnfF6CdQ5ClwltwxsciAl7qx/aRIJe 05DDUunlzBXlcz8A+CA0KvLV/zSqDFWnI0JQANlZJoZWowGHPAVu6raNLnJASrcj3pwt c7oRSyepr8vssbszX677fduXvWqpuos0XreNInVAo+gv3HsB4D+0epRSMRNr/cytx1g6 +taQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=HrZqSvViETmOyCbHqejUVyVMvKJ/NKlJD+Y8Og63mWI=; b=LxHxlZOSNiZEywSc2zkxPpuV9NQTycdmYEFysQzqrHCwg0LNl/QZPKx+0Jr9t+YQio EkNxAMabF9hA+uE3/Z1t0MVrRRS1O6u4nsoQPFRZELnXOOB2qokBkEWKoJ1ai1HzUxtY BgT//DClk4pQPJZenqdJv5zaj+EcK0rGLzWxrWv5UP2/RusbefE/f2t4nxKBccIN/uqp PHd1aISU3jqiW1apFSpKbpAZuNjPbyattrZFAQqeDJO9w4FCOKOjyRI6QBTmzbhDobVp h5IOadJ3HiY6yqVqwxJWdCyXHVaaJUjIfMpyIr8U3ir76+kfg6Hj2L9d0ljFnTp8eXKZ vabA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="AoSwoll/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e22-20020a056402089600b004bca220ebcesi11287177edy.292.2023.03.06.10.56.52; Mon, 06 Mar 2023 10:57:16 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b="AoSwoll/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230121AbjCFSyd (ORCPT + 99 others); Mon, 6 Mar 2023 13:54:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34734 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229961AbjCFSyB (ORCPT ); Mon, 6 Mar 2023 13:54:01 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DBDF123137; Mon, 6 Mar 2023 10:53:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128785; x=1709664785; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uR+iWhHaiU+j4y6I4WFGPNMBDa5ZX0CzLHFXsMED2LY=; b=AoSwoll/01ew2uCQsAy255X7p24Cz2RfEza90Pr0bD1kQqzmclY3O7pZ mnw1zKmkTzd6H+N7bNy6VMhzihbF8kBG+pnGakoJib81/lvryXrDHksmJ J6NUCO7Vu/0KOpPBZc7HghokcTV+j/dpICCf1T9JDVkAArJWr2OWomH5H YiyCDAq0x4uDuN5zgHQPoY8WtGSV6IqlbXWd6X0k3tNtVHvAmYQsalNCh xC6HO+u0IGRBLaLR3bcYgKeg2kj83Zmv0xy0WC/EVvpk0d3+MHWJzj//d /qtg/7Qw0OHGAYRO+RlH7qJ4DTtiZ6lAVRUBUh+LQTxDQNTeaHCEte1kj g==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227788" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227788" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:53:01 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255903" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255903" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:52:59 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 12/16] crypto: iaa - Add support for iaa_crypto deflate compression algorithm Date: Mon, 6 Mar 2023 12:52:22 -0600 Message-Id: <20230306185226.26483-13-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645832148026755?= X-GMAIL-MSGID: =?utf-8?q?1759645832148026755?= This patch registers the iaa_crypto deflate compression algorithm and hooks it up to the IAA hardware. With this change, the IAA crypto algorithm is operational and compression and decompression operations are fully enabled following the successful binding of the first IAA workqueue to the iaa_crypto sub-driver, and similarly, the IAA crypto algorithm is automatically unregistered when there are no IAA workqueues bound to the driver, following their removal. A new iaa_crypto 'verify_compress' driver attribute is also added, allowing the user to toggle compression verification. If set, each compress will be internally decompressed and the contents verified, returning error codes if unsuccessful. This can be toggled with 0/1: echo 0 > /sys/bus/dsa/drivers/crypto/verify_compress The default setting is '1' - verify all compresses. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi --- drivers/crypto/iaa/iaa_crypto.h | 23 + drivers/crypto/iaa/iaa_crypto_main.c | 608 ++++++++++++++++++++++++++- 2 files changed, 630 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/iaa/iaa_crypto.h b/drivers/crypto/iaa/iaa_crypto.h index 2daa3522e073..176beb2afff6 100644 --- a/drivers/crypto/iaa/iaa_crypto.h +++ b/drivers/crypto/iaa/iaa_crypto.h @@ -10,11 +10,34 @@ #define IDXD_SUBDRIVER_NAME "crypto" +#define IAA_DECOMP_ENABLE BIT(0) +#define IAA_DECOMP_FLUSH_OUTPUT BIT(1) +#define IAA_DECOMP_CHECK_FOR_EOB BIT(2) +#define IAA_DECOMP_STOP_ON_EOB BIT(3) +#define IAA_DECOMP_SUPPRESS_OUTPUT BIT(9) + +#define IAA_COMP_FLUSH_OUTPUT BIT(1) +#define IAA_COMP_APPEND_EOB BIT(2) + +#define IAA_COMPLETION_TIMEOUT 1000000 + +#define IAA_ANALYTICS_ERROR 0x0a +#define IAA_ERROR_COMP_BUF_OVERFLOW 0x19 +#define IAA_ERROR_WATCHDOG_EXPIRED 0x24 + #define IAA_COMP_MODES_MAX 2 #define FIXED_HDR 0x2 #define FIXED_HDR_SIZE 3 +#define IAA_COMP_FLAGS (IAA_COMP_FLUSH_OUTPUT | \ + IAA_COMP_APPEND_EOB) + +#define IAA_DECOMP_FLAGS (IAA_DECOMP_ENABLE | \ + IAA_DECOMP_FLUSH_OUTPUT | \ + IAA_DECOMP_CHECK_FOR_EOB | \ + IAA_DECOMP_STOP_ON_EOB) + /* Representation of IAA workqueue */ struct iaa_wq { struct list_head list; diff --git a/drivers/crypto/iaa/iaa_crypto_main.c b/drivers/crypto/iaa/iaa_crypto_main.c index 9bd42cfa949e..7626e12e65c9 100644 --- a/drivers/crypto/iaa/iaa_crypto_main.c +++ b/drivers/crypto/iaa/iaa_crypto_main.c @@ -20,6 +20,8 @@ #define pr_fmt(fmt) "idxd: " IDXD_SUBDRIVER_NAME ": " fmt +#define IAA_ALG_PRIORITY 300 + /* number of iaa instances probed */ static unsigned int nr_iaa; static unsigned int nr_cpus; @@ -32,6 +34,20 @@ static unsigned int cpus_per_iaa; /* Per-cpu lookup table for balanced wqs */ static struct wq_table_entry __percpu *wq_table; +static struct idxd_wq *wq_table_next_wq(int cpu) +{ + struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); + + if (++entry->cur_wq >= entry->n_wqs) + entry->cur_wq = 0; + + pr_debug("%s: returning wq at idx %d (iaa wq %d.%d) from cpu %d\n", __func__, + entry->cur_wq, entry->wqs[entry->cur_wq]->idxd->id, + entry->wqs[entry->cur_wq]->id, cpu); + + return entry->wqs[entry->cur_wq]; +} + static void wq_table_add(int cpu, struct idxd_wq *wq) { struct wq_table_entry *entry = per_cpu_ptr(wq_table, cpu); @@ -66,6 +82,39 @@ static void wq_table_clear_entry(int cpu) static LIST_HEAD(iaa_devices); static DEFINE_MUTEX(iaa_devices_lock); +/* If enabled, IAA hw crypto algos are registered, unavailable otherwise */ +static bool iaa_crypto_enabled; + +/* Verify results of IAA compress or not */ +static bool iaa_verify_compress = true; + +static ssize_t verify_compress_show(struct device_driver *driver, char *buf) +{ + return sprintf(buf, "%d\n", iaa_verify_compress); +} + +static ssize_t verify_compress_store(struct device_driver *driver, + const char *buf, size_t count) +{ + int ret = -EBUSY; + + mutex_lock(&iaa_devices_lock); + + if (iaa_crypto_enabled) + goto out; + + ret = kstrtobool(buf, &iaa_verify_compress); + if (ret) + goto out; + + ret = count; +out: + mutex_unlock(&iaa_devices_lock); + + return ret; +} +static DRIVER_ATTR_RW(verify_compress); + static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; static int active_compression_mode; @@ -86,6 +135,9 @@ static ssize_t compression_mode_store(struct device_driver *driver, mutex_lock(&iaa_devices_lock); + if (iaa_crypto_enabled) + goto out; + mode_name = kstrndup(buf, count, GFP_KERNEL); if (!mode_name) { ret = -ENOMEM; @@ -350,6 +402,89 @@ static void free_device_compression_mode(struct iaa_device *iaa_device, kfree(device_mode); } +#define IDXD_OP_FLAG_AECS_RW_TGLS 0x400000 +#define IAX_AECS_DEFAULT_FLAG (IDXD_OP_FLAG_CRAV | IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CC) +#define IAX_AECS_COMPRESS_FLAG (IAX_AECS_DEFAULT_FLAG | IDXD_OP_FLAG_RD_SRC2_AECS) +#define IAX_AECS_DECOMPRESS_FLAG (IAX_AECS_DEFAULT_FLAG | IDXD_OP_FLAG_RD_SRC2_AECS) +#define IAX_AECS_GEN_FLAG (IAX_AECS_DEFAULT_FLAG | \ + IDXD_OP_FLAG_WR_SRC2_AECS_COMP | \ + IDXD_OP_FLAG_AECS_RW_TGLS) + +static int check_completion(struct device *dev, + struct iax_completion_record *comp, + bool compress, + bool only_once); + +static int decompress_header(struct iaa_device_compression_mode *device_mode, + struct iaa_compression_mode *mode, + struct idxd_wq *wq) +{ + dma_addr_t src_addr, src2_addr; + struct idxd_desc *idxd_desc; + struct iax_hw_desc *desc; + struct device *dev; + int ret = 0; + + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + if (IS_ERR(idxd_desc)) + return PTR_ERR(idxd_desc); + + desc = idxd_desc->iax_hw; + + dev = &wq->idxd->pdev->dev; + + src_addr = dma_map_single(dev, (void *)mode->header_table, + mode->header_table_size, DMA_TO_DEVICE); + dev_dbg(dev, "%s: mode->name %s, src_addr %llx, dev %p, src %p, slen %d\n", + __func__, mode->name, src_addr, dev, + mode->header_table, mode->header_table_size); + if (unlikely(dma_mapping_error(dev, src_addr))) { + dev_dbg(dev, "dma_map_single err, exiting\n"); + ret = -ENOMEM; + return ret; + } + + desc->flags = IAX_AECS_GEN_FLAG; + desc->opcode = IAX_OPCODE_DECOMPRESS; + + desc->src1_addr = (u64)src_addr; + desc->src1_size = mode->header_table_size; + + src2_addr = device_mode->aecs_decomp_table_dma_addr; + desc->src2_addr = (u64)src2_addr; + desc->src2_size = 1088; + dev_dbg(dev, "%s: mode->name %s, src2_addr %llx, dev %p, src2_size %d\n", + __func__, mode->name, desc->src2_addr, dev, desc->src2_size); + desc->max_dst_size = 0; // suppressed output + + desc->decompr_flags = mode->gen_decomp_table_flags; + +#ifdef SPR_E0 + desc->priv = 1; +#else + desc->priv = 0; +#endif + desc->completion_addr = idxd_desc->compl_dma; + + ret = idxd_submit_desc(wq, idxd_desc); + if (ret) { + pr_err("%s: submit_desc failed ret=0x%x\n", __func__, ret); + goto out; + } + + ret = check_completion(dev, idxd_desc->iax_completion, false, false); + if (ret) + dev_dbg(dev, "%s: mode->name %s check_completion failed ret=%d\n", + __func__, mode->name, ret); + else + dev_dbg(dev, "%s: mode->name %s succeeded\n", __func__, + mode->name); +out: + dma_unmap_single(dev, src2_addr, 1088, DMA_TO_DEVICE); + + return ret; +} + static int init_device_compression_mode(struct iaa_device *iaa_device, struct iaa_compression_mode *mode, int idx, struct idxd_wq *wq) @@ -382,6 +517,14 @@ static int init_device_compression_mode(struct iaa_device *iaa_device, memcpy(device_mode->aecs_comp_table->ll_sym, mode->ll_table, mode->ll_table_size); memcpy(device_mode->aecs_comp_table->d_sym, mode->d_table, mode->d_table_size); + if (mode->header_table) { + ret = decompress_header(device_mode, mode, wq); + if (ret) { + pr_debug("iaa header decompression failed: ret=%d\n", ret); + goto free; + } + } + if (mode->init) { ret = mode->init(device_mode); if (ret) @@ -831,6 +974,436 @@ static void rebalance_wq_table(void) } } +static inline int check_completion(struct device *dev, + struct iax_completion_record *comp, + bool compress, + bool only_once) +{ + char *op_str = compress ? "compress" : "decompress"; + int ret = 0; + + while (!comp->status) { + if (only_once) + return -EAGAIN; + cpu_relax(); + } + + if (comp->status != IAX_COMP_SUCCESS) { + if (comp->status == IAA_ERROR_WATCHDOG_EXPIRED) { + ret = -ETIMEDOUT; + dev_dbg(dev, "%s timed out, size=0x%x\n", + op_str, comp->output_size); + goto out; + } + + if (comp->status == IAA_ANALYTICS_ERROR && + comp->error_code == IAA_ERROR_COMP_BUF_OVERFLOW && compress) { + ret = -E2BIG; + dev_dbg(dev, "compressed > uncompressed size," + " not compressing, size=0x%x\n", + comp->output_size); + goto out; + } + + ret = -EINVAL; + dev_dbg(dev, "iaa %s status=0x%x, error=0x%x, size=0x%x\n", + op_str, comp->status, comp->error_code, comp->output_size); + print_hex_dump(KERN_INFO, "cmp-rec: ", DUMP_PREFIX_OFFSET, 8, 1, comp, 64, 0); + + goto out; + } +out: + return ret; +} + +static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, + struct idxd_wq *wq, + dma_addr_t src_addr, unsigned int slen, + dma_addr_t dst_addr, unsigned int *dlen, + u32 *compression_crc, + bool disable_async) +{ + struct idxd_desc *idxd_desc; + struct iax_hw_desc *desc; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + int ret = 0; + + iaa_wq = wq->private_data; + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + if (IS_ERR(idxd_desc)) { + dev_dbg(dev, "idxd descriptor allocation failed\n"); + dev_dbg(dev, "iaa compress failed: ret=%ld\n", PTR_ERR(idxd_desc)); + return PTR_ERR(idxd_desc); + } + desc = idxd_desc->iax_hw; + + desc->flags = IDXD_OP_FLAG_CRAV | IDXD_OP_FLAG_RCR | + IDXD_OP_FLAG_RD_SRC2_AECS | IDXD_OP_FLAG_CC; + desc->opcode = IAX_OPCODE_COMPRESS; + desc->compr_flags = IAA_COMP_FLAGS; + desc->priv = 1; + + desc->src1_addr = (u64)src_addr; + desc->src1_size = slen; + desc->dst_addr = (u64)dst_addr; + desc->max_dst_size = *dlen; + desc->src2_addr = iaa_wq->iaa_device->active_compression_mode->aecs_comp_table_dma_addr; + desc->src2_size = sizeof(struct aecs_comp_table_record); + desc->completion_addr = idxd_desc->compl_dma; + + dev_dbg(dev, "%s: compression mode %s," + " desc->src1_addr %llx, desc->src1_size %d," + " desc->dst_addr %llx, desc->max_dst_size %d," + " desc->src2_addr %llx, desc->src2_size %d\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + desc->src1_addr, desc->src1_size, desc->dst_addr, + desc->max_dst_size, desc->src2_addr, desc->src2_size); + + ret = idxd_submit_desc(wq, idxd_desc); + if (ret) { + dev_dbg(dev, "submit_desc failed ret=%d\n", ret); + goto err; + } + + ret = check_completion(dev, idxd_desc->iax_completion, true, false); + if (ret) { + dev_dbg(dev, "check_completion failed ret=%d\n", ret); + goto err; + } + + *dlen = idxd_desc->iax_completion->output_size; + + *compression_crc = idxd_desc->iax_completion->crc; + + idxd_free_desc(wq, idxd_desc); +out: + return ret; +err: + idxd_free_desc(wq, idxd_desc); + dev_dbg(dev, "iaa compress failed: ret=%d\n", ret); + + goto out; +} + +static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, + struct idxd_wq *wq, + dma_addr_t src_addr, unsigned int slen, + dma_addr_t dst_addr, unsigned int *dlen, + u32 compression_crc) +{ + struct idxd_desc *idxd_desc; + struct iax_hw_desc *desc; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + int ret = 0; + + iaa_wq = wq->private_data; + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + if (IS_ERR(idxd_desc)) { + dev_dbg(dev, "idxd descriptor allocation failed\n"); + dev_dbg(dev, "iaa compress failed: ret=%ld\n", + PTR_ERR(idxd_desc)); + return PTR_ERR(idxd_desc); + } + desc = idxd_desc->iax_hw; + + /* Verify (optional) - decompress and check crc, suppress dest write */ + + desc->flags = IDXD_OP_FLAG_CRAV | IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CC; + desc->opcode = IAX_OPCODE_DECOMPRESS; + desc->decompr_flags = IAA_DECOMP_FLAGS | IAA_DECOMP_SUPPRESS_OUTPUT; + desc->priv = 1; + + desc->src1_addr = (u64)dst_addr; + desc->src1_size = *dlen; + desc->dst_addr = (u64)src_addr; + desc->max_dst_size = slen; + desc->completion_addr = idxd_desc->compl_dma; + + dev_dbg(dev, "(verify) compression mode %s," + " desc->src1_addr %llx, desc->src1_size %d," + " desc->dst_addr %llx, desc->max_dst_size %d," + " desc->src2_addr %llx, desc->src2_size %d\n", + iaa_wq->iaa_device->active_compression_mode->name, + desc->src1_addr, desc->src1_size, desc->dst_addr, + desc->max_dst_size, desc->src2_addr, desc->src2_size); + + ret = idxd_submit_desc(wq, idxd_desc); + if (ret) { + dev_dbg(dev, "submit_desc (verify) failed ret=%d\n", ret); + goto err; + } + + ret = check_completion(dev, idxd_desc->iax_completion, false, false); + if (ret) { + dev_dbg(dev, "(verify) check_completion failed ret=%d\n", ret); + goto err; + } + + if (compression_crc != idxd_desc->iax_completion->crc) { + ret = -EINVAL; + dev_dbg(dev, "(verify) iaa comp/decomp crc mismatch:" + " comp=0x%x, decomp=0x%x\n", compression_crc, + idxd_desc->iax_completion->crc); + print_hex_dump(KERN_INFO, "cmp-rec: ", DUMP_PREFIX_OFFSET, + 8, 1, idxd_desc->iax_completion, 64, 0); + goto err; + } + + idxd_free_desc(wq, idxd_desc); +out: + return ret; +err: + idxd_free_desc(wq, idxd_desc); + dev_dbg(dev, "iaa compress failed: ret=%d\n", ret); + + goto out; +} + +static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, + struct idxd_wq *wq, + dma_addr_t src_addr, unsigned int slen, + dma_addr_t dst_addr, unsigned int *dlen, + bool disable_async) +{ + struct idxd_desc *idxd_desc; + struct iax_hw_desc *desc; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + int ret = 0; + + iaa_wq = wq->private_data; + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + idxd_desc = idxd_alloc_desc(wq, IDXD_OP_BLOCK); + if (IS_ERR(idxd_desc)) { + dev_dbg(dev, "idxd descriptor allocation failed\n"); + dev_dbg(dev, "iaa decompress failed: ret=%ld\n", + PTR_ERR(idxd_desc)); + return PTR_ERR(idxd_desc); + } + desc = idxd_desc->iax_hw; + + desc->flags = IDXD_OP_FLAG_CRAV | IDXD_OP_FLAG_RCR | IDXD_OP_FLAG_CC; + desc->opcode = IAX_OPCODE_DECOMPRESS; + desc->max_dst_size = PAGE_SIZE; + desc->decompr_flags = IAA_DECOMP_FLAGS; + desc->priv = 1; + + desc->src1_addr = (u64)src_addr; + desc->dst_addr = (u64)dst_addr; + desc->max_dst_size = *dlen; + desc->src1_size = slen; + desc->completion_addr = idxd_desc->compl_dma; + + dev_dbg(dev, "%s decompression mode %s," + " desc->src1_addr %llx, desc->src1_size %d," + " desc->dst_addr %llx, desc->max_dst_size %d," + " desc->src2_addr %llx, desc->src2_size %d\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + desc->src1_addr, desc->src1_size, desc->dst_addr, + desc->max_dst_size, desc->src2_addr, desc->src2_size); + + ret = idxd_submit_desc(wq, idxd_desc); + if (ret) { + dev_dbg(dev, "submit_desc failed ret=%d\n", ret); + goto err; + } + + ret = check_completion(dev, idxd_desc->iax_completion, false, false); + if (ret) { + dev_dbg(dev, "check_completion failed ret=%d\n", ret); + goto err; + } + + *dlen = idxd_desc->iax_completion->output_size; + + idxd_free_desc(wq, idxd_desc); +out: + return ret; +err: + idxd_free_desc(wq, idxd_desc); + dev_dbg(dev, "iaa decompress failed: ret=%d\n", ret); + + goto out; +} + +static int iaa_comp_compress(struct crypto_tfm *tfm, + const u8 *src, unsigned int slen, + u8 *dst, unsigned int *dlen) +{ + dma_addr_t src_addr, dst_addr; + u32 compression_crc; + struct idxd_wq *wq; + struct device *dev; + int cpu, ret = 0; + + if (!iaa_crypto_enabled) { + pr_debug("iaa_crypto disabled, not compressing\n"); + return -ENODEV; + } + + cpu = get_cpu(); + wq = wq_table_next_wq(cpu); + put_cpu(); + if (!wq) { + pr_debug("no wq configured for cpu=%d\n", cpu); + ret = -ENODEV; + goto out; + } + dev = &wq->idxd->pdev->dev; + + src_addr = dma_map_single(dev, (void *)src, slen, DMA_TO_DEVICE); + dev_dbg(dev, "dma_map_single, src_addr %llx, dev %p," + " src %p, slen %d\n", src_addr, dev, src, slen); + if (unlikely(dma_mapping_error(dev, src_addr))) { + dev_dbg(dev, "dma_map_single err, exiting\n"); + ret = -ENOMEM; + goto out; + } + + dst_addr = dma_map_single(dev, (void *)dst, *dlen, DMA_FROM_DEVICE); + dev_dbg(dev, "dma_map_single, dst_addr %llx, dev %p," + " dst %p, *dlen %d\n", dst_addr, dev, dst, *dlen); + if (unlikely(dma_mapping_error(dev, dst_addr))) { + dev_dbg(dev, "dma_map_single err, exiting\n"); + ret = -ENOMEM; + goto err_map_dst; + } + + dev_dbg(dev, "src %p, src_addr %llx, slen %d, dst %p," + " dst_addr %llx, dlen %u\n", src, src_addr, + slen, dst, dst_addr, *dlen); + + ret = iaa_compress(tfm, NULL, wq, src_addr, slen, dst_addr, + dlen, &compression_crc, true); + if (iaa_verify_compress) { + dma_sync_single_for_device(dev, dst_addr, *dlen, DMA_FROM_DEVICE); + dma_sync_single_for_device(dev, src_addr, slen, DMA_TO_DEVICE); + ret = iaa_compress_verify(tfm, NULL, wq, src_addr, + slen, dst_addr, dlen, compression_crc); + } + + if (ret != 0) + dev_dbg(dev, "synchronous compress failed ret=%d\n", ret); + + dma_unmap_single(dev, dst_addr, *dlen, DMA_FROM_DEVICE); +err_map_dst: + dma_unmap_single(dev, src_addr, slen, DMA_TO_DEVICE); +out: + return ret; +} + +static int iaa_comp_decompress(struct crypto_tfm *tfm, + const u8 *src, unsigned int slen, + u8 *dst, unsigned int *dlen) +{ + dma_addr_t src_addr, dst_addr; + struct idxd_wq *wq; + struct device *dev; + int cpu, ret = 0; + + if (!iaa_crypto_enabled) { + pr_debug("iaa_crypto disabled, not decompressing\n"); + return -ENODEV; + } + + cpu = get_cpu(); + wq = wq_table_next_wq(cpu); + put_cpu(); + if (!wq) { + pr_debug("no wq configured for cpu=%d\n", cpu); + ret = -ENODEV; + goto out; + } + dev = &wq->idxd->pdev->dev; + + dev_dbg(dev, "using wq for cpu=%d = wq %p\n", cpu, wq); + + src_addr = dma_map_single(dev, (void *)src, slen, DMA_TO_DEVICE); + dev_dbg(dev, "dma_map_single, src_addr %llx, dev %p," + " src %p, slen %d\n", src_addr, dev, src, slen); + if (unlikely(dma_mapping_error(dev, src_addr))) { + dev_dbg(dev, "dma_map_single err, exiting\n"); + ret = -ENOMEM; + goto out; + } + + dst_addr = dma_map_single(dev, (void *)dst, *dlen, DMA_FROM_DEVICE); + dev_dbg(dev, "dma_map_single, dst_addr %llx, dev %p," + " dst %p, *dlen %d\n", dst_addr, dev, dst, *dlen); + if (unlikely(dma_mapping_error(dev, dst_addr))) { + dev_dbg(dev, "dma_map_single err, exiting\n"); + ret = -ENOMEM; + goto err_map_dst; + } + + dev_dbg(dev, "src %p, src_addr %llx, slen %d, dst %p," + " dst_addr %llx, dlen %u\n", src, src_addr, + slen, dst, dst_addr, *dlen); + + ret = iaa_decompress(tfm, NULL, wq, src_addr, slen, dst_addr, dlen, true); + if (ret != 0) + dev_dbg(dev, "synchronous decompress failed ret=%d\n", ret); + + dma_unmap_single(dev, dst_addr, *dlen, DMA_FROM_DEVICE); +err_map_dst: + dma_unmap_single(dev, src_addr, slen, DMA_TO_DEVICE); +out: + return ret; +} + +static struct crypto_alg iaa_comp_deflate = { + .cra_name = "deflate", + .cra_driver_name = "iaa_crypto", + .cra_flags = CRYPTO_ALG_TYPE_COMPRESS, + .cra_priority = IAA_ALG_PRIORITY, + .cra_module = THIS_MODULE, + .cra_u = { + .compress = { + .coa_compress = iaa_comp_compress, + .coa_decompress = iaa_comp_decompress + } + } +}; + +static int iaa_register_compression_device(void) +{ + int ret; + + ret = crypto_register_alg(&iaa_comp_deflate); + if (ret < 0) { + pr_debug("deflate algorithm registration failed\n"); + return ret; + } + + return ret; +} + +static int iaa_unregister_compression_device(void) +{ + crypto_unregister_alg(&iaa_comp_deflate); + + return 0; +} + static int iaa_crypto_probe(struct idxd_dev *idxd_dev) { struct idxd_wq *wq = idxd_dev_to_wq(idxd_dev); @@ -881,12 +1454,26 @@ static int iaa_crypto_probe(struct idxd_dev *idxd_dev) rebalance_wq_table(); + if (first_wq) { + ret = iaa_register_compression_device(); + if (ret == 0) { + iaa_crypto_enabled = true; + } else { + dev_dbg(dev, "IAA compression device registration failed\n"); + goto err_register; + } + + pr_info("iaa_crypto now ENABLED\n"); + } + mutex_unlock(&iaa_devices_lock); out: mutex_unlock(&wq->wq_lock); return ret; +err_register: + remove_iaa_wq(wq); err_save: if (first_wq) free_wq_table(); @@ -902,6 +1489,7 @@ static int iaa_crypto_probe(struct idxd_dev *idxd_dev) static void iaa_crypto_remove(struct idxd_dev *idxd_dev) { struct idxd_wq *wq = idxd_dev_to_wq(idxd_dev); + struct device *dev = &idxd_dev->conf_dev; idxd_wq_quiesce(wq); @@ -912,9 +1500,16 @@ static void iaa_crypto_remove(struct idxd_dev *idxd_dev) drv_disable_wq(wq); rebalance_wq_table(); - if (nr_iaa == 0) + if (nr_iaa == 0) { + iaa_crypto_enabled = false; free_wq_table(); + if (iaa_unregister_compression_device()) + dev_dbg(dev, "IAA compression device unregister failed\n"); + + pr_info("iaa_crypto now DISABLED\n"); + } + mutex_unlock(&iaa_devices_lock); mutex_unlock(&wq->wq_lock); } @@ -952,6 +1547,13 @@ static int __init iaa_crypto_init_module(void) goto err_attr_create; } + ret = driver_create_file(&iaa_crypto_driver.drv, + &driver_attr_verify_compress); + if (ret) { + pr_debug("IAA verify_compress attr creation failed\n"); + goto err_attr_create; + } + ret = iaa_aecs_init_fixed(); if (ret < 0) { pr_debug("IAA fixed compression mode init failed\n"); @@ -965,6 +1567,8 @@ static int __init iaa_crypto_init_module(void) err_compression_mode: driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_verify_compress); err_attr_create: idxd_driver_unregister(&iaa_crypto_driver); @@ -975,6 +1579,8 @@ static void __exit iaa_crypto_cleanup_module(void) { driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_verify_compress); idxd_driver_unregister(&iaa_crypto_driver); iaa_aecs_cleanup_fixed(); From patchwork Mon Mar 6 18:52:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65064 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2016166wrd; Mon, 6 Mar 2023 10:57:23 -0800 (PST) X-Google-Smtp-Source: AK7set9B3WSBpbdQJOzgwKINmscqTQ69lMNEnARHSOih5PjV4WVz8AL4hS3BU/weUogioGrBQ+Uv X-Received: by 2002:aa7:d347:0:b0:4ae:e51e:9e3e with SMTP id m7-20020aa7d347000000b004aee51e9e3emr9527915edr.24.1678129042971; Mon, 06 Mar 2023 10:57:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678129042; cv=none; d=google.com; s=arc-20160816; b=O41QmAC2yMPvq0Ob8sKVtTLHzBzNLv5sAuw4+fjY4I5pEKtukneEiiPqHL299/w5Bq B2Q7Mk5IFw1wf8zfWbpDFyktLhf02ywINW5+faA6rDj7PDfKq7bzQB03C5hV5vtaxyTN S9mKe40CLxclsYO3jQtvI16KRJe9AatgJdxDAo4xmEhdKb/qRTu+lRU2B1aWGfnn8Gc8 O1jhHx4A3q4rX+OhQKOSf0Gxh6tyK9vfw/7T2Ejg1RfBOAZdoHgimtaP0YACUoLpfFBv 0Qhv2hIGu7UsziWHGFJ++/oDedWnRwg7RCTo0VsJRe8Bi7W2aMIXsLXbL7/79uvYY65e 6BjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4u4tDvhKQs1baS0k/urJ69K9aiyVsMEkP+mfyOsZN7w=; b=IzuTi7zex+M6yydpSPt7zl96XKEIXplCid/5fnds8QrYKaKU8DI2Hcr/e1nI17jmrC a22v7kQVSKDBMHyLnyfOkMA2xa5kxA/39HQFKoFtCFavzf+J+6qiU32wtEfXyFxy53QA SyHa5m0hvN2UlcmDYLGw1UOaJZZMzUoTicL+woyKThBwmI/vrUtT2Uo0O28ddDZC4XW3 Qry7ocnQtmXsBbcyaCLgecUnugbXG5gacUqEJIUvIRpUhkKnZpgvyZaoWVHlZrPWp+yR QsH51BeqU5TVRSE6JOOYXb+F0q02u6raTMcB/755WE5TL4AADafjPAjW9ni+Nk71OGQ5 OFHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=mnTvcZ3H; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id o5-20020aa7c505000000b004ad7c1f4d14si10269045edq.268.2023.03.06.10.56.59; Mon, 06 Mar 2023 10:57:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=mnTvcZ3H; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230023AbjCFSyg (ORCPT + 99 others); Mon, 6 Mar 2023 13:54:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60670 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230183AbjCFSyD (ORCPT ); Mon, 6 Mar 2023 13:54:03 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F3F3922001; Mon, 6 Mar 2023 10:53:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128788; x=1709664788; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kSzEf6OwOfgI6CQ2H6vteMQ32tTZCCg2KIQ8SLoZxO4=; b=mnTvcZ3H1TNyCMGo7G3H+eJ5FJQbCZr9KS/7ndOTbJvyEQ4/3JDoJKcG wtFnM19+Rg9A1LLK4lkXF+ya0vcNhY5GO1JpFT2sS/duqQrW2MwLRhq3x WvS1XJGPjs6Co87ZXwuKNdk5t7W4VsTgOApdQ2RPRpheFTHg4NwXSvgr/ PURzMC/gC1/tYIZ6Yo5mbfpe6kKNG/rPU5uwzAJK4hVdtK7leSSZgszue qccfAWHetAxyMFYjFvV0SXEUt8OLXWCD4vPZivQwZXoHFNOd61APl4Mfr QWCQQYdvLSvQ9WrOXiVNGUu9qSU1xUo5bLQZ2JpZx+YfGIr2/9FsUAWhj w==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227804" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227804" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:53:03 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255917" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255917" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:53:01 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 13/16] crypto: iaa - Add support for default IAA 'canned' compression mode Date: Mon, 6 Mar 2023 12:52:23 -0600 Message-Id: <20230306185226.26483-14-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645839373678060?= X-GMAIL-MSGID: =?utf-8?q?1759645839373678060?= Add support for a 'canned' compression mode using the IAA compression mode in-kernel API. The IAA 'canned' compression mode is added alongside the existing 'fixed' compression mode and is named simply 'canned'. It implements a good general-purpose compression scheme whose tables were generated from statistics derived from a wide variety of SPEC17 workloads. It provides much better overall characteristics than the existing deflate-1951 tables implemented by 'fixed'. Either 'fixed' or 'canned' modes can be chosen as the mode to be used for compression/decompression via the iaa_crypto compression_mode iaa_crypto driver attribute: To choose 'fixed' mode: echo "fixed" > /sys/bus/dsa/drivers/crypto/compression_mode To choose 'canned' mode: echo "canned" > /sys/bus/dsa/drivers/crypto/compression_mode [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi --- drivers/crypto/iaa/Makefile | 2 +- drivers/crypto/iaa/iaa_crypto.h | 2 + drivers/crypto/iaa/iaa_crypto_comp_canned.c | 110 ++++++++++++++++++++ drivers/crypto/iaa/iaa_crypto_main.c | 39 ++++++- 4 files changed, 150 insertions(+), 3 deletions(-) create mode 100644 drivers/crypto/iaa/iaa_crypto_comp_canned.c diff --git a/drivers/crypto/iaa/Makefile b/drivers/crypto/iaa/Makefile index cc87feffd059..ff6ab1d0bc13 100644 --- a/drivers/crypto/iaa/Makefile +++ b/drivers/crypto/iaa/Makefile @@ -7,4 +7,4 @@ ccflags-y += -I $(srctree)/drivers/dma/idxd -DDEFAULT_SYMBOL_NAMESPACE=IDXD obj-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO) := iaa_crypto.o -iaa_crypto-y := iaa_crypto_main.o iaa_crypto_comp_fixed.o +iaa_crypto-y := iaa_crypto_main.o iaa_crypto_comp_canned.o iaa_crypto_comp_fixed.o diff --git a/drivers/crypto/iaa/iaa_crypto.h b/drivers/crypto/iaa/iaa_crypto.h index 176beb2afff6..ed910201e2fc 100644 --- a/drivers/crypto/iaa/iaa_crypto.h +++ b/drivers/crypto/iaa/iaa_crypto.h @@ -111,6 +111,8 @@ struct aecs_decomp_table_record { u32 decomp_state[1220]; } __packed; +int iaa_aecs_init_canned(void); +void iaa_aecs_cleanup_canned(void); int iaa_aecs_init_fixed(void); void iaa_aecs_cleanup_fixed(void); diff --git a/drivers/crypto/iaa/iaa_crypto_comp_canned.c b/drivers/crypto/iaa/iaa_crypto_comp_canned.c new file mode 100644 index 000000000000..aff0899ffb9e --- /dev/null +++ b/drivers/crypto/iaa/iaa_crypto_comp_canned.c @@ -0,0 +1,110 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */ + +#include "idxd.h" +#include "iaa_crypto.h" + +#define IAA_AECS_ALIGN 32 + +/* + * These tables were generated from statistics derived from a wide + * variety of SPEC17 workloads and implement a good general-purpose + * compression scheme called simply 'canned'. + */ + +static const u32 canned_ll_iaa[286] = { +0x28002, 0x38024, 0x40066, 0x40067, 0x40068, 0x48144, 0x40069, 0x48145, +0x4006a, 0x48146, 0x4006b, 0x48147, 0x48148, 0x48149, 0x4814a, 0x4814b, +0x4006c, 0x4814c, 0x4814d, 0x4814e, 0x4814f, 0x48150, 0x48151, 0x48152, +0x4006d, 0x48153, 0x48154, 0x48155, 0x48156, 0x48157, 0x48158, 0x48159, +0x38025, 0x4815a, 0x4815b, 0x4815c, 0x4815d, 0x4815e, 0x4815f, 0x48160, +0x4006e, 0x48161, 0x48162, 0x48163, 0x48164, 0x48165, 0x4006f, 0x48166, +0x38026, 0x38027, 0x40070, 0x40071, 0x40072, 0x40073, 0x40074, 0x40075, +0x38028, 0x40076, 0x40077, 0x48167, 0x40078, 0x40079, 0x4007a, 0x38029, +0x3802a, 0x4007b, 0x48168, 0x48169, 0x4007c, 0x4816a, 0x4007d, 0x4816b, +0x4007e, 0x4816c, 0x4816d, 0x4816e, 0x4816f, 0x48170, 0x48171, 0x48172, +0x4007f, 0x48173, 0x48174, 0x48175, 0x48176, 0x48177, 0x48178, 0x48179, +0x40080, 0x4817a, 0x4817b, 0x4817c, 0x4817d, 0x4817e, 0x4817f, 0x48180, +0x40081, 0x3802b, 0x40082, 0x3802c, 0x3802d, 0x3802e, 0x40083, 0x48181, +0x40084, 0x40085, 0x48182, 0x48183, 0x40086, 0x40087, 0x40088, 0x40089, +0x4008a, 0x48184, 0x4008b, 0x4008c, 0x4008d, 0x4008e, 0x48185, 0x48186, +0x4008f, 0x48187, 0x48188, 0x48189, 0x4818a, 0x4818b, 0x4818c, 0x4818d, +0x40090, 0x4818e, 0x4818f, 0x48190, 0x48191, 0x48192, 0x48193, 0x48194, +0x40091, 0x48195, 0x48196, 0x48197, 0x48198, 0x48199, 0x4819a, 0x4819b, +0x40092, 0x4819c, 0x4819d, 0x4819e, 0x4819f, 0x481a0, 0x481a1, 0x481a2, +0x40093, 0x481a3, 0x481a4, 0x481a5, 0x481a6, 0x481a7, 0x481a8, 0x481a9, +0x40094, 0x481aa, 0x481ab, 0x481ac, 0x481ad, 0x481ae, 0x481af, 0x481b0, +0x481b1, 0x481b2, 0x481b3, 0x481b4, 0x481b5, 0x481b6, 0x481b7, 0x481b8, +0x40095, 0x481b9, 0x481ba, 0x481bb, 0x481bc, 0x481bd, 0x481be, 0x481bf, +0x40096, 0x481c0, 0x481c1, 0x481c2, 0x481c3, 0x481c4, 0x481c5, 0x40097, +0x40098, 0x481c6, 0x481c7, 0x481c8, 0x481c9, 0x481ca, 0x481cb, 0x481cc, +0x40099, 0x481cd, 0x481ce, 0x481cf, 0x481d0, 0x481d1, 0x481d2, 0x481d3, +0x4009a, 0x481d4, 0x481d5, 0x481d6, 0x481d7, 0x481d8, 0x481d9, 0x481da, +0x481db, 0x481dc, 0x481dd, 0x481de, 0x481df, 0x481e0, 0x481e1, 0x481e2, +0x4009b, 0x481e3, 0x481e4, 0x481e5, 0x481e6, 0x481e7, 0x481e8, 0x481e9, +0x481ea, 0x481eb, 0x481ec, 0x481ed, 0x481ee, 0x481ef, 0x481f0, 0x481f1, +0x4009c, 0x481f2, 0x481f3, 0x481f4, 0x481f5, 0x481f6, 0x481f7, 0x481f8, +0x481f9, 0x481fa, 0x481fb, 0x503fe, 0x481fc, 0x481fd, 0x481fe, 0x4009d, +0x503ff, 0x20000, 0x28003, 0x30010, 0x28004, 0x28005, 0x28006, 0x4009e, +0x4009f, 0x3802f, 0x38030, 0x30011, 0x400a0, 0x38031, 0x38032, 0x400a1, +0x28007, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, +0x0, 0x0, 0x0, 0x0, 0x0, 0x0, +}; + +static const u32 canned_d_iaa[30] = { +0x3807e, 0x20004, 0x481fe, 0x18000, 0x400fe, 0x18001, 0x3003c, 0x20005, +0x20006, 0x28016, 0x20007, 0x28017, 0x20008, 0x28018, 0x28019, 0x20009, +0x2000a, 0x2801a, 0x2801b, 0x2801c, 0x2801d, 0x3003d, 0x3003e, 0x481ff, +0x0, 0x0, 0x0, 0x0, 0x0, 0x0, +}; + +#define CANNED_HEADER_SIZE (71) + +static const u8 canned_header[] = { +0x85, 0xd7, 0x05, 0x40, 0x54, 0x4d, 0x10, 0x06, +0x80, 0x3d, 0x40, 0x44, 0x2c, 0x6c, 0xec, 0xb3, +0xb1, 0xf0, 0x8e, 0x3c, 0x10, 0xb8, 0x43, 0x11, +0xb1, 0xb1, 0xb1, 0x39, 0xee, 0x0e, 0x41, 0x29, +0x09, 0xc5, 0xc6, 0xc6, 0xc6, 0xc6, 0xc6, 0x6e, +0xc5, 0xc6, 0x04, 0x1b, 0x1b, 0xbb, 0x15, 0xbb, +0x15, 0x3b, 0x7e, 0x95, 0x7f, 0xf6, 0xed, 0x2e, +0xdc, 0x3d, 0xee, 0xd8, 0x1b, 0x3f, 0xbe, 0x37, +0xb3, 0xb3, 0xb3, 0x3b, 0xb3, 0xf1, 0x9e, +}; + +#define HEADER_SIZE_IN_BITS 568 + +#define CEIL(a, b) (((a) + ((b) - 1)) / (b)) + +int iaa_aecs_init_canned(void) +{ + u16 gen_decomp_table_flags; + unsigned int slen; + int ret; + + slen = CEIL(HEADER_SIZE_IN_BITS, 8); + + gen_decomp_table_flags = 0x1; + gen_decomp_table_flags |= 1 << 9; // suppress output + gen_decomp_table_flags |= (((slen * 8) - HEADER_SIZE_IN_BITS) << 6); + + ret = add_iaa_compression_mode("canned", + canned_ll_iaa, + sizeof(canned_ll_iaa), + canned_d_iaa, + sizeof(canned_d_iaa), + canned_header, + sizeof(canned_header), + gen_decomp_table_flags, + NULL, NULL); + + if (!ret) + pr_debug("IAA canned compression mode initialized\n"); + + return ret; +} + +void iaa_aecs_cleanup_canned(void) +{ + remove_iaa_compression_mode("canned"); +} diff --git a/drivers/crypto/iaa/iaa_crypto_main.c b/drivers/crypto/iaa/iaa_crypto_main.c index 7626e12e65c9..cf7c0169cd7b 100644 --- a/drivers/crypto/iaa/iaa_crypto_main.c +++ b/drivers/crypto/iaa/iaa_crypto_main.c @@ -118,11 +118,16 @@ static DRIVER_ATTR_RW(verify_compress); static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; static int active_compression_mode; +static bool canned_mode; + static ssize_t compression_mode_show(struct device_driver *driver, char *buf) { int ret = 0; - ret = sprintf(buf, "%s\n", "fixed"); + if (canned_mode) + ret = sprintf(buf, "%s\n", "canned"); + else + ret = sprintf(buf, "%s\n", "fixed"); return ret; } @@ -381,6 +386,12 @@ int set_iaa_compression_mode(const char *name) ret = 0; } + if (ret == 0 && !strcmp(name, "canned")) + canned_mode = true; + + if (ret == 0 && !strcmp(name, "fixed")) + canned_mode = false; + return ret; } @@ -1103,6 +1114,7 @@ static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_device *idxd; struct iaa_wq *iaa_wq; struct pci_dev *pdev; + dma_addr_t src2_addr; struct device *dev; int ret = 0; @@ -1133,6 +1145,13 @@ static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, desc->max_dst_size = slen; desc->completion_addr = idxd_desc->compl_dma; + if (canned_mode) { + src2_addr = iaa_wq->iaa_device->active_compression_mode->aecs_decomp_table_dma_addr; + desc->src2_addr = (u64)src2_addr; + desc->src2_size = 1088; + desc->flags |= IDXD_OP_FLAG_RD_SRC2_AECS; + } + dev_dbg(dev, "(verify) compression mode %s," " desc->src1_addr %llx, desc->src1_size %d," " desc->dst_addr %llx, desc->max_dst_size %d," @@ -1184,6 +1203,7 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_device *idxd; struct iaa_wq *iaa_wq; struct pci_dev *pdev; + dma_addr_t src2_addr; struct device *dev; int ret = 0; @@ -1213,7 +1233,14 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, desc->src1_size = slen; desc->completion_addr = idxd_desc->compl_dma; - dev_dbg(dev, "%s decompression mode %s," + if (canned_mode) { + src2_addr = iaa_wq->iaa_device->active_compression_mode->aecs_decomp_table_dma_addr; + desc->src2_addr = (u64)src2_addr; + desc->src2_size = 1088; + desc->flags |= IDXD_OP_FLAG_RD_SRC2_AECS; + } + + dev_dbg(dev, "%s: decompression mode %s," " desc->src1_addr %llx, desc->src1_size %d," " desc->dst_addr %llx, desc->max_dst_size %d," " desc->src2_addr %llx, desc->src2_size %d\n", __func__, @@ -1554,8 +1581,15 @@ static int __init iaa_crypto_init_module(void) goto err_attr_create; } + ret = iaa_aecs_init_canned(); + if (ret < 0) { + pr_debug("IAA canned compression mode init failed\n"); + goto err_compression_mode; + } + ret = iaa_aecs_init_fixed(); if (ret < 0) { + iaa_aecs_cleanup_canned(); pr_debug("IAA fixed compression mode init failed\n"); goto err_compression_mode; } @@ -1582,6 +1616,7 @@ static void __exit iaa_crypto_cleanup_module(void) driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_verify_compress); idxd_driver_unregister(&iaa_crypto_driver); + iaa_aecs_cleanup_canned(); iaa_aecs_cleanup_fixed(); pr_debug("cleaned up\n"); From patchwork Mon Mar 6 18:52:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65066 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2016302wrd; Mon, 6 Mar 2023 10:57:45 -0800 (PST) X-Google-Smtp-Source: AK7set8tbxgsbyTMmscrtLwF3pHCvnvzLSBwiQTEo7ix/QaCZc/P+kJF5jXYSML+Al5/xIxxl9DX X-Received: by 2002:a17:907:c706:b0:8b1:7ae9:647 with SMTP id ty6-20020a170907c70600b008b17ae90647mr10050449ejc.76.1678129065296; Mon, 06 Mar 2023 10:57:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678129065; cv=none; d=google.com; s=arc-20160816; b=Ri0nFdmXwJ2yCTxSedTMQtgD3rdBVPQEYm6CnHNPLf1m47ilACyoLPVXA5sjMI7TP8 v0cvu0Z9yJOhex74YSEXQKhO2nuYfKiHVlroxkJrpH4PlzUD1TjXRbOAk9SBZDdj0RpW 81AvKLT9+34CIxeVKM3x4lC5mW7vceR9lfxlnrKXqDt96hfSv7WCRvVlFq96/kNlnVqk 1gbnlnTFcE5tCtqeR41bbYO/UpWFQSgjL9i+4c1JjwWtU4bcIAQXCdPy33nfhR4XVRGe FYVhK30fPSd1KzNYN29Bf/Fndwk9vKeTdyhj4XLjUOXUhXXj8+sTKO66y+AEAQShEWA5 GOjA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QZkvLfPmVgPr/u10p/hmIKiEuf7Kx9Fuv+my6FaV3bg=; b=NiWcQyzLOuYHi1VBrUnIt1IbHxUZTtwzyqCCQDVMdFEUB7Zu96r1txVECpvulhdEdn /J4a75UtSpTvoRwqzFecRBzcG2TT2neDWRpMidIrj63LX5RWcYiq5Q+uPiLcECm8gp/+ g6OYUVFjfyg0y/eTqWuGzplfbZDvuDIz1EL4j2z4VCm/QM0NlXV6fYUqgxaPnh2vRbSf sIn9zo1iB4qtx6ZGJcXyj7uYCI/KAQQn1KhykCKs3qNoacsSDCYqjEcu79So1dT8Ynij exqQG5Ucvo/Yy877SNaCJmaWcixd7ozkMe89IFsgnNQnD2k/o+AaiMclfH6TeBUoXaHP ACHA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Jhb9nMW7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id bl15-20020a170906c24f00b008bc21b16e13si10160124ejb.871.2023.03.06.10.57.21; Mon, 06 Mar 2023 10:57:45 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=Jhb9nMW7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229557AbjCFSzR (ORCPT + 99 others); Mon, 6 Mar 2023 13:55:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32884 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230244AbjCFSyN (ORCPT ); Mon, 6 Mar 2023 13:54:13 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B20D74BE92; Mon, 6 Mar 2023 10:53:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128796; x=1709664796; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LJpqELhw+jvd5hGy7AG+RytE6z5raCmAekjOLSm7cWs=; b=Jhb9nMW7taB+VkkGqwdjHlpjBUeSQ1rXTZ6BLC8rkfn/lYNafl818RNs bl4OMe1ROgko2ajJz4zrGJqBNUk/cuusiuduUghSEwfzPwmC7O347nZqJ hU96IGKQe83mob4JB8e2LiRshIv5798ygrOWxs7ONydygPZU63A3NSoq4 88BX09NjFWTkRN/Mk/ey59RUHx5Puzllrf5K5B1umBQGKJ7YHT53VUWG+ p8Zmq+s08Kh0JkaihA5oYUYuJKfZgvtTPztBX4janU6eSU/KIQD5pUryh eIPQRJMTyYJ7sTLyPFAxn6msqCSU9LVNymBPEUPs1s31+44e4Ba3hOpxU Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227816" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227816" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:53:05 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255937" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255937" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:53:04 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 14/16] crypto: iaa - Add support for iaa_crypto async deflate compression algorithm Date: Mon, 6 Mar 2023 12:52:24 -0600 Message-Id: <20230306185226.26483-15-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645862807097573?= X-GMAIL-MSGID: =?utf-8?q?1759645862807097573?= Add support for the iax_crypto async deflate algorithm, so that the IAA crypto hardware can be used with facilities that require it, such as zswap. [ Based on work originally by George Powley, Jing Lin and Kyung Min Park ] Signed-off-by: Tom Zanussi --- drivers/crypto/iaa/iaa_crypto_main.c | 169 +++++++++++++++++++++++++++ 1 file changed, 169 insertions(+) diff --git a/drivers/crypto/iaa/iaa_crypto_main.c b/drivers/crypto/iaa/iaa_crypto_main.c index cf7c0169cd7b..39f72eda43ff 100644 --- a/drivers/crypto/iaa/iaa_crypto_main.c +++ b/drivers/crypto/iaa/iaa_crypto_main.c @@ -10,6 +10,7 @@ #include #include #include +#include #include "idxd.h" #include "iaa_crypto.h" @@ -1411,6 +1412,159 @@ static struct crypto_alg iaa_comp_deflate = { } }; +static int iaa_comp_acompress(struct acomp_req *req) +{ + struct crypto_tfm *tfm = req->base.tfm; + dma_addr_t src_addr, dst_addr; + int nr_sgs, cpu, ret = 0; + struct iaa_wq *iaa_wq; + u32 compression_crc; + struct idxd_wq *wq; + struct device *dev; + + if (!iaa_crypto_enabled) { + pr_debug("iaa_crypto disabled, not compressing\n"); + return -ENODEV; + } + + cpu = get_cpu(); + wq = wq_table_next_wq(cpu); + put_cpu(); + if (!wq) { + pr_debug("no wq configured for cpu=%d\n", cpu); + ret = -ENODEV; + goto out; + } + iaa_wq = wq->private_data; + + dev = &wq->idxd->pdev->dev; + + nr_sgs = dma_map_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); + if (nr_sgs <= 0 || nr_sgs > 1) { + dev_dbg(dev, "couldn't map src sg for iaa device %d," + " wq %d: ret=%d\n", iaa_wq->iaa_device->idxd->id, + iaa_wq->wq->id, ret); + ret = -EIO; + goto out; + } + src_addr = sg_dma_address(req->src); + dev_dbg(dev, "dma_map_sg, src_addr %llx, nr_sgs %d, req->src %p," + " req->slen %d, sg_dma_len(sg) %d\n", src_addr, nr_sgs, + req->src, req->slen, sg_dma_len(req->src)); + + nr_sgs = dma_map_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); + if (nr_sgs <= 0 || nr_sgs > 1) { + dev_dbg(dev, "couldn't map dst sg for iaa device %d," + " wq %d: ret=%d\n", iaa_wq->iaa_device->idxd->id, + iaa_wq->wq->id, ret); + ret = -EIO; + goto err_map_dst; + } + dst_addr = sg_dma_address(req->dst); + dev_dbg(dev, "dma_map_sg, dst_addr %llx, nr_sgs %d, req->dst %p," + " req->dlen %d, sg_dma_len(sg) %d\n", dst_addr, nr_sgs, + req->dst, req->dlen, sg_dma_len(req->dst)); + + ret = iaa_compress(tfm, req, wq, src_addr, req->slen, dst_addr, + &req->dlen, &compression_crc, false); + if (ret == -EINPROGRESS) + goto out; + + if (iaa_verify_compress) { + dma_sync_sg_for_device(dev, req->dst, 1, DMA_FROM_DEVICE); + dma_sync_sg_for_device(dev, req->src, 1, DMA_TO_DEVICE); + ret = iaa_compress_verify(tfm, req, wq, src_addr, req->slen, + dst_addr, &req->dlen, compression_crc); + } + + if (ret != 0) + dev_dbg(dev, "asynchronous compress failed ret=%d\n", ret); + + dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); +err_map_dst: + dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); +out: + return ret; +} + +static int iaa_comp_adecompress(struct acomp_req *req) +{ + struct crypto_tfm *tfm = req->base.tfm; + dma_addr_t src_addr, dst_addr; + int nr_sgs, cpu, ret = 0; + struct iaa_wq *iaa_wq; + struct device *dev; + struct idxd_wq *wq; + + if (!iaa_crypto_enabled) { + pr_debug("iaa_crypto disabled, not decompressing\n"); + return -ENODEV; + } + + cpu = get_cpu(); + wq = wq_table_next_wq(cpu); + put_cpu(); + if (!wq) { + pr_debug("no wq configured for cpu=%d\n", cpu); + ret = -ENODEV; + goto out; + } + iaa_wq = wq->private_data; + + dev = &wq->idxd->pdev->dev; + + nr_sgs = dma_map_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); + if (nr_sgs <= 0 || nr_sgs > 1) { + dev_dbg(dev, "couldn't map src sg for iaa device %d," + " wq %d: ret=%d\n", iaa_wq->iaa_device->idxd->id, + iaa_wq->wq->id, ret); + ret = -EIO; + goto out; + } + src_addr = sg_dma_address(req->src); + dev_dbg(dev, "dma_map_sg, src_addr %llx, nr_sgs %d, req->src %p," + " req->slen %d, sg_dma_len(sg) %d\n", src_addr, nr_sgs, + req->src, req->slen, sg_dma_len(req->src)); + + nr_sgs = dma_map_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); + if (nr_sgs <= 0 || nr_sgs > 1) { + dev_dbg(dev, "couldn't map dst sg for iaa device %d," + " wq %d: ret=%d\n", iaa_wq->iaa_device->idxd->id, + iaa_wq->wq->id, ret); + ret = -EIO; + goto err_map_dst; + } + dst_addr = sg_dma_address(req->dst); + dev_dbg(dev, "dma_map_sg, dst_addr %llx, nr_sgs %d, req->dst %p," + " req->dlen %d, sg_dma_len(sg) %d\n", dst_addr, nr_sgs, + req->dst, req->dlen, sg_dma_len(req->dst)); + + ret = iaa_decompress(tfm, req, wq, src_addr, req->slen, + dst_addr, &req->dlen, false); + if (ret == -EINPROGRESS) + goto out; + + if (ret != 0) + dev_dbg(dev, "asynchronous decompress failed ret=%d\n", ret); + + dma_unmap_sg(dev, req->dst, sg_nents(req->dst), DMA_FROM_DEVICE); +err_map_dst: + dma_unmap_sg(dev, req->src, sg_nents(req->src), DMA_TO_DEVICE); +out: + return ret; +} + +static struct acomp_alg iaa_acomp_deflate = { + .compress = iaa_comp_acompress, + .decompress = iaa_comp_adecompress, + .base = { + .cra_name = "deflate", + .cra_driver_name = "iaa_crypto", + .cra_module = THIS_MODULE, + .cra_priority = IAA_ALG_PRIORITY, + } +}; + static int iaa_register_compression_device(void) { int ret; @@ -1421,12 +1575,27 @@ static int iaa_register_compression_device(void) return ret; } + ret = crypto_register_acomp(&iaa_acomp_deflate); + if (ret) { + pr_err("deflate algorithm acomp registration failed (%d)\n", ret); + goto err_unregister_alg_deflate; + } + + return ret; + +err_unregister_alg_deflate: + crypto_unregister_alg(&iaa_comp_deflate); + return ret; } static int iaa_unregister_compression_device(void) { + if (refcount_read(&iaa_acomp_deflate.base.cra_refcnt) > 1) + return -EBUSY; + crypto_unregister_alg(&iaa_comp_deflate); + crypto_unregister_acomp(&iaa_acomp_deflate); return 0; } From patchwork Mon Mar 6 18:52:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65065 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2016270wrd; Mon, 6 Mar 2023 10:57:40 -0800 (PST) X-Google-Smtp-Source: AK7set/F2Yx3kVtV0YbxPJYVCsr5fsZNRmUvDVLm5QKfhkfpC5BTc7G+ElRf89S5em79nqgoibgW X-Received: by 2002:a17:906:6c84:b0:8aa:33c4:87d5 with SMTP id s4-20020a1709066c8400b008aa33c487d5mr10996469ejr.10.1678129060717; Mon, 06 Mar 2023 10:57:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678129060; cv=none; d=google.com; s=arc-20160816; b=Zezr3JO5d/J9HgMrIHCHyXq/DMdmEfoKbNt2JUSySaOoqTWpkff1nfFZvzuWPMsYWY 6oKFbDYtbT9naLOkZTbesL1jSbY/NRajIV1McFq3WnYY6c89UEnIIqiFPMCCwamWsanA uT+jjmAKq8d9IqmTLHigdOFW9cV1FkhcJSYGK5K6Sr3SzsG/Zprnla4X+MSxgylLQT4G 2bmOqIhfzA+bhPnd99jH3O/KZFmWx/h+wJgizuANHu7cJxf+fRPboM8/7J7P/U95Ljgd skvOWrjArEmNsGF5kVtTDDcsOsWZshxg47x9pK9/MPgO6t7yLNGb2xFyQTPaGTQzw6GC 5pEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=PaMI70F7+PtEPtpJKnI36M6oy6JVMuMq1gd5VTkfnC4=; b=AfxEM6gbGtioM7y4MPAKx+Dcm662zhS0q20Oo+NyYLsdvco8XjkToMmfda5uEbmNxK ZyTq4bNKlrieA0HMPo978dqu7D5piBFpQq3gZ9Gm1pm7394VyBeYedxFiFLR2lMohltX HeSSwgx3TSuYwI8ua/Tih6SRnk62wkF5G2B34rSsbWBF24WvyfkV8hfhJmiBS75s96FD Id/7CeAJ+0fO1fGSuue6iknGeC9DT5bjj4KhybAY6OtLmjq2REdDJD5dsBXtja/w/HKK 6fgkFxmus7QKRVv2Srck2RqUNWVwi45sKQrx/F17Gy4C6Jb3HHI0TDRzZJlV9LuLJMhH b0gw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=fThthSYQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id kb15-20020a1709070f8f00b008c9eb8758desi265559ejc.915.2023.03.06.10.57.17; Mon, 06 Mar 2023 10:57:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=fThthSYQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229639AbjCFSzK (ORCPT + 99 others); Mon, 6 Mar 2023 13:55:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230325AbjCFSye (ORCPT ); Mon, 6 Mar 2023 13:54:34 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C1BD78CAF; Mon, 6 Mar 2023 10:53:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128803; x=1709664803; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KHrdduABqoYk7FYqgdOqGwEE+ECLY4d79jW5db6sXVE=; b=fThthSYQUfRK9rm0AUQ5zvJV3c1abFgo0IlHn+iwWQ0boTo+LACd5enq dMMGseVHjtOLEQykQiok3e2T0wmPGja3QrGL93q54Tcl66VWKyn6c2TCk go6/i3m6d7N8ipc/WQzLK8cx5o8OEZz0y1QWxvyAaGmknI3I3rms/EIdv NXva8/zRXwDAZw1y89QROI7dhOZeXB/+HG09GcBQ2LMYWmPo4zgp6MjO/ n8d4wt5I5zmWSYB+PKVx68cRCHCwGMV/4+V0F7T0aN5KAcYV94QRCOkyE OWZ49KwgOKGM0Yf7GVvUUuwMG7Jit3OjZ5NJ+6WbLYgHLgsKz+GaGpohM Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227831" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227831" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:53:07 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255949" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255949" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:53:06 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 15/16] crypto: iaa - Add irq support for the crypto async interface Date: Mon, 6 Mar 2023 12:52:25 -0600 Message-Id: <20230306185226.26483-16-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645858003036737?= X-GMAIL-MSGID: =?utf-8?q?1759645858003036737?= The existing iaa crypto async support provides an implementation that satisfies the interface but does so in a synchronous manner - it fills and submits the IDXD descriptor and then waits for it to complete before returning. This isn't a problem at the moment, since all existing callers (e.g. zswap) wrap any asynchronous callees in a synchronous wrapper anyway. This change makes the iaa crypto async implementation truly asynchronous: it fills and submits the IDXD descriptor, then returns immediately with -EINPROGRESS. It also sets the descriptor's 'request completion irq' bit and sets up a callback with the IDXD driver which is called when the operation completes and the irq fires. The existing callers such as zswap use synchronous wrappers to deal with -EINPROGRESS and so work as expected without any changes. This mode can be enabled by writing 'async_irq' to the sync_mode iaa_crypto driver attribute: echo async_irq > /sys/bus/dsa/drivers/crypto/sync_mode Async mode without interrupts (caller must poll) can be enabled by writing 'async' to it: echo async > /sys/bus/dsa/drivers/crypto/sync_mode The default sync mode can be enabled by writing 'sync' to it: echo sync > /sys/bus/dsa/drivers/crypto/sync_mode Signed-off-by: Tom Zanussi --- drivers/crypto/iaa/iaa_crypto_main.c | 216 ++++++++++++++++++++++++++- 1 file changed, 215 insertions(+), 1 deletion(-) diff --git a/drivers/crypto/iaa/iaa_crypto_main.c b/drivers/crypto/iaa/iaa_crypto_main.c index 39f72eda43ff..aedf06a0c1ec 100644 --- a/drivers/crypto/iaa/iaa_crypto_main.c +++ b/drivers/crypto/iaa/iaa_crypto_main.c @@ -116,6 +116,102 @@ static ssize_t verify_compress_store(struct device_driver *driver, } static DRIVER_ATTR_RW(verify_compress); +/* + * The iaa crypto driver supports three 'sync' methods determining how + * compressions and decompressions are performed: + * + * - sync: the compression or decompression completes before + * returning. This is the mode used by the async crypto + * interface when the sync mode is set to 'sync' and by + * the sync crypto interface regardless of setting. + * + * - async: the compression or decompression is submitted and returns + * immediately. Completion interrupts are not used so + * the caller is responsible for polling the descriptor + * for completion. This mode is applicable to only the + * async crypto interface and is ignored for anything + * else. + * + * - async_irq: the compression or decompression is submitted and + * returns immediately. Completion interrupts are + * enabled so the caller can wait for the completion and + * yield to other threads. When the compression or + * decompression completes, the completion is signaled + * and the caller awakened. This mode is applicable to + * only the async crypto interface and is ignored for + * anything else. + * + * These modes can be set using the iaa_crypto sync_mode driver + * attribute. + */ + +/* Use async mode */ +static bool async_mode; +/* Use interrupts */ +static bool use_irq; + +/** + * set_iaa_sync_mode - Set IAA sync mode + * @name: The name of the sync mode + * + * Make the IAA sync mode named @name the current sync mode used by + * compression/decompression. + */ + +static int set_iaa_sync_mode(const char *name) +{ + int ret = 0; + + if (sysfs_streq(name, "sync")) { + async_mode = false; + use_irq = false; + } else if (sysfs_streq(name, "async")) { + async_mode = true; + use_irq = false; + } else if (sysfs_streq(name, "async_irq")) { + async_mode = true; + use_irq = true; + } else { + ret = -EINVAL; + } + + return ret; +} + +static ssize_t sync_mode_show(struct device_driver *driver, char *buf) +{ + int ret = 0; + + if (!async_mode && !use_irq) + ret = sprintf(buf, "%s\n", "sync"); + else if (async_mode && !use_irq) + ret = sprintf(buf, "%s\n", "async"); + else if (async_mode && use_irq) + ret = sprintf(buf, "%s\n", "async_irq"); + + return ret; +} + +static ssize_t sync_mode_store(struct device_driver *driver, + const char *buf, size_t count) +{ + int ret = -EBUSY; + + mutex_lock(&iaa_devices_lock); + + if (iaa_crypto_enabled) + goto out; + + ret = set_iaa_sync_mode(buf); + if (ret == 0) + ret = count; +out: + mutex_unlock(&iaa_devices_lock); + + return ret; +} +static DRIVER_ATTR_RW(sync_mode); + static struct iaa_compression_mode *iaa_compression_modes[IAA_COMP_MODES_MAX]; static int active_compression_mode; @@ -1028,6 +1124,71 @@ static inline int check_completion(struct device *dev, return ret; } +static int iaa_compress_verify(struct crypto_tfm *tfm, struct acomp_req *req, + struct idxd_wq *wq, + dma_addr_t src_addr, unsigned int slen, + dma_addr_t dst_addr, unsigned int *dlen, + u32 compression_crc); + +static void iaa_desc_complete(struct idxd_desc *idxd_desc, + enum idxd_complete_type comp_type, + bool free_desc, void *__ctx, + u32 *status) +{ + struct crypto_ctx *ctx = __ctx; + struct idxd_device *idxd; + struct iaa_wq *iaa_wq; + struct pci_dev *pdev; + struct device *dev; + int ret, err = 0; + + iaa_wq = idxd_desc->wq->private_data; + idxd = iaa_wq->iaa_device->idxd; + pdev = idxd->pdev; + dev = &pdev->dev; + + dev_dbg(dev, "%s: compression mode %s," + " ctx->src_addr %llx, ctx->dst_addr %llx\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + ctx->src_addr, ctx->dst_addr); + + ret = check_completion(dev, idxd_desc->iax_completion, + ctx->compress, false); + if (ret) { + dev_dbg(dev, "%s: check_completion failed ret=%d\n", __func__, ret); + err = -EIO; + } + + ctx->req->dlen = idxd_desc->iax_completion->output_size; + + if (ctx->compress && iaa_verify_compress) { + u32 compression_crc; + + compression_crc = idxd_desc->iax_completion->crc; + dma_sync_sg_for_device(dev, ctx->req->dst, 1, DMA_FROM_DEVICE); + dma_sync_sg_for_device(dev, ctx->req->src, 1, DMA_TO_DEVICE); + ret = iaa_compress_verify(ctx->tfm, ctx->req, iaa_wq->wq, ctx->src_addr, + ctx->req->slen, ctx->dst_addr, &ctx->req->dlen, + compression_crc); + if (ret) { + dev_dbg(dev, "%s: compress verify failed ret=%d\n", __func__, ret); + err = -EIO; + } + } + + if (ctx->req->base.complete) + acomp_request_complete(ctx->req, err); + + dma_unmap_sg(dev, ctx->req->dst, sg_nents(ctx->req->dst), DMA_FROM_DEVICE); + dma_unmap_sg(dev, ctx->req->src, sg_nents(ctx->req->src), DMA_TO_DEVICE); + + if (ret != 0) + dev_dbg(dev, "asynchronous compress failed ret=%d\n", ret); + + if (free_desc) + idxd_free_desc(idxd_desc->wq, idxd_desc); +} + static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, struct idxd_wq *wq, dma_addr_t src_addr, unsigned int slen, @@ -1070,6 +1231,22 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, desc->src2_size = sizeof(struct aecs_comp_table_record); desc->completion_addr = idxd_desc->compl_dma; + if (use_irq) { + desc->flags |= IDXD_OP_FLAG_RCI; + + idxd_desc->crypto.req = req; + idxd_desc->crypto.tfm = tfm; + idxd_desc->crypto.src_addr = src_addr; + idxd_desc->crypto.dst_addr = dst_addr; + idxd_desc->crypto.compress = true; + + dev_dbg(dev, "%s use_async_irq: compression mode %s," + " src_addr %llx, dst_addr %llx\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + src_addr, dst_addr); + } else if (req && async_mode && !disable_async) + req->base.data = idxd_desc; + dev_dbg(dev, "%s: compression mode %s," " desc->src1_addr %llx, desc->src1_size %d," " desc->dst_addr %llx, desc->max_dst_size %d," @@ -1084,6 +1261,12 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, goto err; } + if (req && async_mode && !disable_async) { + ret = -EINPROGRESS; + dev_dbg(dev, "%s: returning -EINPROGRESS\n", __func__); + goto out; + } + ret = check_completion(dev, idxd_desc->iax_completion, true, false); if (ret) { dev_dbg(dev, "check_completion failed ret=%d\n", ret); @@ -1241,6 +1424,19 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, desc->flags |= IDXD_OP_FLAG_RD_SRC2_AECS; } + if (use_irq) { + desc->flags |= IDXD_OP_FLAG_RCI; + + idxd_desc->crypto.req = req; + idxd_desc->crypto.compress = false; + + dev_dbg(dev, "%s: use_async_irq compression mode %s," + " src_addr %llx, dst_addr %llx\n", __func__, + iaa_wq->iaa_device->active_compression_mode->name, + src_addr, dst_addr); + } else if (req && async_mode && !disable_async) + req->base.data = idxd_desc; + dev_dbg(dev, "%s: decompression mode %s," " desc->src1_addr %llx, desc->src1_size %d," " desc->dst_addr %llx, desc->max_dst_size %d," @@ -1255,6 +1451,12 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, goto err; } + if (req && async_mode && !disable_async) { + ret = -EINPROGRESS; + dev_dbg(dev, "%s: returning -EINPROGRESS\n", __func__); + goto out; + } + ret = check_completion(dev, idxd_desc->iax_completion, false, false); if (ret) { dev_dbg(dev, "check_completion failed ret=%d\n", ret); @@ -1627,7 +1829,7 @@ static int iaa_crypto_probe(struct idxd_dev *idxd_dev) wq->type = IDXD_WQT_KERNEL; - ret = drv_enable_wq(wq, IDXD_DTYPE_CRYPTO); + ret = drv_enable_wq(wq); if (ret < 0) { dev_dbg(dev, "enable wq %d.%d failed: %d\n", idxd->id, wq->id, ret); @@ -1720,6 +1922,7 @@ static struct idxd_device_driver iaa_crypto_driver = { .remove = iaa_crypto_remove, .name = IDXD_SUBDRIVER_NAME, .type = dev_types, + .desc_complete = iaa_desc_complete, }; static int __init iaa_crypto_init_module(void) @@ -1736,6 +1939,13 @@ static int __init iaa_crypto_init_module(void) goto out; } + ret = driver_create_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); + if (ret) { + pr_debug("IAA sync mode attr creation failed\n"); + goto err_attr_create; + } + ret = driver_create_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); if (ret) { @@ -1768,6 +1978,8 @@ static int __init iaa_crypto_init_module(void) return ret; err_compression_mode: + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); driver_remove_file(&iaa_crypto_driver.drv, @@ -1780,6 +1992,8 @@ static int __init iaa_crypto_init_module(void) static void __exit iaa_crypto_cleanup_module(void) { + driver_remove_file(&iaa_crypto_driver.drv, + &driver_attr_sync_mode); driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_compression_mode); driver_remove_file(&iaa_crypto_driver.drv, From patchwork Mon Mar 6 18:52:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tom Zanussi X-Patchwork-Id: 65067 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a5d:5915:0:0:0:0:0 with SMTP id v21csp2016326wrd; Mon, 6 Mar 2023 10:57:49 -0800 (PST) X-Google-Smtp-Source: AK7set/tMuhB/m8InKMytBQAQ1XUB+48spxj7veww4TMqaXwAPDNdR7imblaBX/Z/Nxm9rJ2Szo6 X-Received: by 2002:a17:906:af01:b0:8b2:3748:3db9 with SMTP id lx1-20020a170906af0100b008b237483db9mr11901149ejb.51.1678129069484; Mon, 06 Mar 2023 10:57:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1678129069; cv=none; d=google.com; s=arc-20160816; b=f4IEvCTK8uekiZQZ0sDRTO7LnyfaOzvLejYstdkMT78XfYtfdP7QyntX/I7AxKyMNh PTBQcLvuBlfG5TefXR5Mbqep3X79hvN9OpftOVy3tAkJkgepL0c/vrEpDYbQtC2ClMwa owXqmUyxwo2Cn5TWtr3eUGWEy42LDPwLHOEselIJZlAKEeHSq/8gm1gleckSnxjfxLxW mQlSr/XftP0K1redqYqCHz+k72CHlQqErk38CnkDA/PWGNOG0/GrvWVa+i31ZILe7ii9 eH68ZvBaJkUjBdauLsr7fa/UF1PY1cdt5i0IIjDp85SCfkLGZU6FjWIYpU4l0uCnz2bf ilXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ORO9Zzh3jacHI8DGf71q3ZtUWI0A7SP34CL+cPdmFvU=; b=zO7rbkCohYpGAIT/hvHDfuWcYhoB9ahzH8M5pTO3nTSVAO2Ezr2QdcjBQpvb9DLKpB Vr+e59rwECJB16eDbi/joao+KQ4VO0dNUz/3teZe0IJO9kgirc3ip5dRed2ljkpadPnR D6uuBL/njyliCeB76Qcf/sYONsxJFkv2lfsbfVFgtUZp8gJD1SueoF+r472SskhnckoW mr9rRSmHmBzj2OUNK8CoS7js1jSPTGQAcCqZBAugTE2UumbsESC1H7Fl4MviV8RI3J2h EHGZbpa1wt8j22o3oP+MBDkPEEyGPrldmFnWwmoT2JNYe68Y0dTu5tT16UseiaPhr5GB FdIA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=h709Kk8i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gu25-20020a170906f29900b008d65603e880si9893763ejb.583.2023.03.06.10.57.26; Mon, 06 Mar 2023 10:57:49 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=h709Kk8i; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229717AbjCFSzU (ORCPT + 99 others); Mon, 6 Mar 2023 13:55:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230403AbjCFSyu (ORCPT ); Mon, 6 Mar 2023 13:54:50 -0500 Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 655ED79B07; Mon, 6 Mar 2023 10:53:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1678128805; x=1709664805; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=quzK9fB7Fr3XWUDpEtz1+NyY16J6Kbj/fqzstjTLfrE=; b=h709Kk8iA/lDdYAoAnbaa8oNGYolVyE4NEalpM83dymRS3aPqzYBex0b HjYeRboGxkvFAdDk91QuJu/ptObRbIWhy3z4oxGwp3XxYTLv77HNp0q73 IkosIGEKyjXC0y8lM7qB+QEXXmvTZKwAUH3DlSP83FJYAvF8cxwx6M+1g 6u3finwjQczWBJ5Dx3+O665A1LbqZziwuvifCWWZkHESlUGwOTBbnQ1gX b/Uk59TH1V6c3RKe6kvp1xiVWuOZuZg2gEl9Chp4iEBngFjRt5F18rDQM SS4H15aaoTZmTqV/IZzSwEyGe1GasYxI+xRHpJXYd4Gj1Ggw/8RlRJQ45 w==; X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="398227849" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="398227849" Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:53:09 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10641"; a="626255963" X-IronPort-AV: E=Sophos;i="5.98,238,1673942400"; d="scan'208";a="626255963" Received: from jeblanco-mobl.amr.corp.intel.com (HELO tzanussi-mobl1.hsd1.il.comcast.net) ([10.212.118.26]) by orsmga003-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Mar 2023 10:53:08 -0800 From: Tom Zanussi To: herbert@gondor.apana.org.au, davem@davemloft.net, fenghua.yu@intel.com, vkoul@kernel.org Cc: dave.jiang@intel.com, tony.luck@intel.com, wajdi.k.feghali@intel.com, james.guilford@intel.com, kanchana.p.sridhar@intel.com, linux-kernel@vger.kernel.org, linux-crypto@vger.kernel.org, dmaengine@vger.kernel.org Subject: [PATCH 16/16] crypto: iaa - Add IAA Compression Accelerator stats Date: Mon, 6 Mar 2023 12:52:26 -0600 Message-Id: <20230306185226.26483-17-tom.zanussi@linux.intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230306185226.26483-1-tom.zanussi@linux.intel.com> References: <20230306185226.26483-1-tom.zanussi@linux.intel.com> MIME-Version: 1.0 X-Spam-Status: No, score=-4.3 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_EF,RCVD_IN_DNSWL_MED,SPF_HELO_NONE, SPF_NONE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1759645866748471600?= X-GMAIL-MSGID: =?utf-8?q?1759645866748471600?= Add support for optional debugfs statistics support for the IAA Compression Accelerator. This is enabled by the kernel config item: CRYPTO_DEV_IAA_CRYPTO_STATS When enabled, the IAA crypto driver will generate statistics which can be accessed at /sys/kernel/debug/iaa-crypto/. See Documentation/driver-api/crypto/iax/iax-crypto.rst for details. Signed-off-by: Tom Zanussi --- drivers/crypto/iaa/Kconfig | 9 + drivers/crypto/iaa/Makefile | 2 + drivers/crypto/iaa/iaa_crypto.h | 22 +++ drivers/crypto/iaa/iaa_crypto_main.c | 71 +++++++ drivers/crypto/iaa/iaa_crypto_stats.c | 271 ++++++++++++++++++++++++++ drivers/crypto/iaa/iaa_crypto_stats.h | 58 ++++++ 6 files changed, 433 insertions(+) create mode 100644 drivers/crypto/iaa/iaa_crypto_stats.c create mode 100644 drivers/crypto/iaa/iaa_crypto_stats.h diff --git a/drivers/crypto/iaa/Kconfig b/drivers/crypto/iaa/Kconfig index fcccb6ff7e29..cffb3a4359fc 100644 --- a/drivers/crypto/iaa/Kconfig +++ b/drivers/crypto/iaa/Kconfig @@ -8,3 +8,12 @@ config CRYPTO_DEV_IAA_CRYPTO decompression with the Intel Analytics Accelerator (IAA) hardware using the cryptographic API. If you choose 'M' here, the module will be called iaa_crypto. + +config CRYPTO_DEV_IAA_CRYPTO_STATS + bool "Enable Intel(R) IAA Compression Accelerator Statistics" + depends on CRYPTO_DEV_IAA_CRYPTO + default n + help + Enable statistics for the IAA compression accelerator. + These include per-device and per-workqueue statistics in + addition to global driver statistics. diff --git a/drivers/crypto/iaa/Makefile b/drivers/crypto/iaa/Makefile index ff6ab1d0bc13..2a1bee385932 100644 --- a/drivers/crypto/iaa/Makefile +++ b/drivers/crypto/iaa/Makefile @@ -8,3 +8,5 @@ ccflags-y += -I $(srctree)/drivers/dma/idxd -DDEFAULT_SYMBOL_NAMESPACE=IDXD obj-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO) := iaa_crypto.o iaa_crypto-y := iaa_crypto_main.o iaa_crypto_comp_canned.o iaa_crypto_comp_fixed.o + +iaa_crypto-$(CONFIG_CRYPTO_DEV_IAA_CRYPTO_STATS) += iaa_crypto_stats.o diff --git a/drivers/crypto/iaa/iaa_crypto.h b/drivers/crypto/iaa/iaa_crypto.h index ed910201e2fc..b7842d2df682 100644 --- a/drivers/crypto/iaa/iaa_crypto.h +++ b/drivers/crypto/iaa/iaa_crypto.h @@ -44,6 +44,11 @@ struct iaa_wq { struct idxd_wq *wq; struct iaa_device *iaa_device; + + u64 comp_calls; + u64 comp_bytes; + u64 decomp_calls; + u64 decomp_bytes; }; struct iaa_device_compression_mode { @@ -66,6 +71,11 @@ struct iaa_device { int n_wq; struct list_head wqs; + + u64 comp_calls; + u64 comp_bytes; + u64 decomp_calls; + u64 decomp_bytes; }; struct wq_table_entry { @@ -146,4 +156,16 @@ int add_iaa_compression_mode(const char *name, void remove_iaa_compression_mode(const char *name); int set_iaa_compression_mode(const char *name); +#if defined(CONFIG_CRYPTO_DEV_IAA_CRYPTO_STATS) +void global_stats_show(struct seq_file *m); +void device_stats_show(struct seq_file *m, struct iaa_device *iaa_device); +void reset_iaa_crypto_stats(void); +void reset_device_stats(struct iaa_device *iaa_device); +#else +static inline void global_stats_show(struct seq_file *m) {} +static inline void device_stats_show(struct seq_file *m, struct iaa_device *iaa_device) {} +static inline void reset_iaa_crypto_stats(void) {} +static inline void reset_device_stats(struct iaa_device *iaa_device) {} +#endif + #endif diff --git a/drivers/crypto/iaa/iaa_crypto_main.c b/drivers/crypto/iaa/iaa_crypto_main.c index aedf06a0c1ec..5292f9f0480d 100644 --- a/drivers/crypto/iaa/iaa_crypto_main.c +++ b/drivers/crypto/iaa/iaa_crypto_main.c @@ -14,6 +14,7 @@ #include "idxd.h" #include "iaa_crypto.h" +#include "iaa_crypto_stats.h" #ifdef pr_fmt #undef pr_fmt @@ -1101,6 +1102,7 @@ static inline int check_completion(struct device *dev, ret = -ETIMEDOUT; dev_dbg(dev, "%s timed out, size=0x%x\n", op_str, comp->output_size); + update_completion_timeout_errs(); goto out; } @@ -1110,6 +1112,7 @@ static inline int check_completion(struct device *dev, dev_dbg(dev, "compressed > uncompressed size," " not compressing, size=0x%x\n", comp->output_size); + update_completion_comp_buf_overflow_errs(); goto out; } @@ -1117,6 +1120,7 @@ static inline int check_completion(struct device *dev, dev_dbg(dev, "iaa %s status=0x%x, error=0x%x, size=0x%x\n", op_str, comp->status, comp->error_code, comp->output_size); print_hex_dump(KERN_INFO, "cmp-rec: ", DUMP_PREFIX_OFFSET, 8, 1, comp, 64, 0); + update_completion_einval_errs(); goto out; } @@ -1161,6 +1165,15 @@ static void iaa_desc_complete(struct idxd_desc *idxd_desc, ctx->req->dlen = idxd_desc->iax_completion->output_size; + /* Update stats */ + if (ctx->compress) { + update_total_comp_bytes_out(ctx->req->dlen); + update_wq_comp_bytes(iaa_wq->wq, ctx->req->dlen); + } else { + update_total_decomp_bytes_in(ctx->req->dlen); + update_wq_decomp_bytes(iaa_wq->wq, ctx->req->dlen); + } + if (ctx->compress && iaa_verify_compress) { u32 compression_crc; @@ -1261,6 +1274,10 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, goto err; } + /* Update stats */ + update_total_comp_calls(); + update_wq_comp_calls(wq); + if (req && async_mode && !disable_async) { ret = -EINPROGRESS; dev_dbg(dev, "%s: returning -EINPROGRESS\n", __func__); @@ -1275,6 +1292,10 @@ static int iaa_compress(struct crypto_tfm *tfm, struct acomp_req *req, *dlen = idxd_desc->iax_completion->output_size; + /* Update stats */ + update_total_comp_bytes_out(*dlen); + update_wq_comp_bytes(wq, *dlen); + *compression_crc = idxd_desc->iax_completion->crc; idxd_free_desc(wq, idxd_desc); @@ -1451,6 +1472,10 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, goto err; } + /* Update stats */ + update_total_decomp_calls(); + update_wq_decomp_calls(wq); + if (req && async_mode && !disable_async) { ret = -EINPROGRESS; dev_dbg(dev, "%s: returning -EINPROGRESS\n", __func__); @@ -1466,6 +1491,10 @@ static int iaa_decompress(struct crypto_tfm *tfm, struct acomp_req *req, *dlen = idxd_desc->iax_completion->output_size; idxd_free_desc(wq, idxd_desc); + + /* Update stats */ + update_total_decomp_bytes_in(slen); + update_wq_decomp_bytes(wq, slen); out: return ret; err: @@ -1483,6 +1512,7 @@ static int iaa_comp_compress(struct crypto_tfm *tfm, u32 compression_crc; struct idxd_wq *wq; struct device *dev; + u64 start_time_ns; int cpu, ret = 0; if (!iaa_crypto_enabled) { @@ -1522,8 +1552,10 @@ static int iaa_comp_compress(struct crypto_tfm *tfm, " dst_addr %llx, dlen %u\n", src, src_addr, slen, dst, dst_addr, *dlen); + start_time_ns = iaa_get_ts(); ret = iaa_compress(tfm, NULL, wq, src_addr, slen, dst_addr, dlen, &compression_crc, true); + update_max_comp_delay_ns(start_time_ns); if (iaa_verify_compress) { dma_sync_single_for_device(dev, dst_addr, *dlen, DMA_FROM_DEVICE); dma_sync_single_for_device(dev, src_addr, slen, DMA_TO_DEVICE); @@ -1548,6 +1580,7 @@ static int iaa_comp_decompress(struct crypto_tfm *tfm, dma_addr_t src_addr, dst_addr; struct idxd_wq *wq; struct device *dev; + u64 start_time_ns; int cpu, ret = 0; if (!iaa_crypto_enabled) { @@ -1589,7 +1622,9 @@ static int iaa_comp_decompress(struct crypto_tfm *tfm, " dst_addr %llx, dlen %u\n", src, src_addr, slen, dst, dst_addr, *dlen); + start_time_ns = iaa_get_ts(); ret = iaa_decompress(tfm, NULL, wq, src_addr, slen, dst_addr, dlen, true); + update_max_decomp_delay_ns(start_time_ns); if (ret != 0) dev_dbg(dev, "synchronous decompress failed ret=%d\n", ret); @@ -1925,6 +1960,38 @@ static struct idxd_device_driver iaa_crypto_driver = { .desc_complete = iaa_desc_complete, }; +int wq_stats_show(struct seq_file *m, void *v) +{ + struct iaa_device *iaa_device; + + mutex_lock(&iaa_devices_lock); + + global_stats_show(m); + + list_for_each_entry(iaa_device, &iaa_devices, list) + device_stats_show(m, iaa_device); + + mutex_unlock(&iaa_devices_lock); + + return 0; +} + +int iaa_crypto_stats_reset(void *data, u64 value) +{ + struct iaa_device *iaa_device; + + reset_iaa_crypto_stats(); + + mutex_lock(&iaa_devices_lock); + + list_for_each_entry(iaa_device, &iaa_devices, list) + reset_device_stats(iaa_device); + + mutex_unlock(&iaa_devices_lock); + + return 0; +} + static int __init iaa_crypto_init_module(void) { int ret = 0; @@ -1973,6 +2040,9 @@ static int __init iaa_crypto_init_module(void) goto err_compression_mode; } + if (iaa_crypto_debugfs_init()) + pr_warn("debugfs init failed, stats not available\n"); + pr_debug("initialized\n"); out: return ret; @@ -1992,6 +2062,7 @@ static int __init iaa_crypto_init_module(void) static void __exit iaa_crypto_cleanup_module(void) { + iaa_crypto_debugfs_cleanup(); driver_remove_file(&iaa_crypto_driver.drv, &driver_attr_sync_mode); driver_remove_file(&iaa_crypto_driver.drv, diff --git a/drivers/crypto/iaa/iaa_crypto_stats.c b/drivers/crypto/iaa/iaa_crypto_stats.c new file mode 100644 index 000000000000..336847b83085 --- /dev/null +++ b/drivers/crypto/iaa/iaa_crypto_stats.c @@ -0,0 +1,271 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include "../../dma/idxd/idxd.h" +#include +#include +#include "iaa_crypto.h" +#include "iaa_crypto_stats.h" + +static u64 total_comp_calls; +static u64 total_decomp_calls; +static u64 max_comp_delay_ns; +static u64 max_decomp_delay_ns; +static u64 max_acomp_delay_ns; +static u64 max_adecomp_delay_ns; +static u64 total_comp_bytes_out; +static u64 total_decomp_bytes_in; +static u64 total_completion_einval_errors; +static u64 total_completion_timeout_errors; +static u64 total_completion_comp_buf_overflow_errors; + +static struct dentry *iaa_crypto_debugfs_root; + +void update_total_comp_calls(void) +{ + total_comp_calls++; +} + +void update_total_comp_bytes_out(int n) +{ + total_comp_bytes_out += n; +} + +void update_total_decomp_calls(void) +{ + total_decomp_calls++; +} + +void update_total_decomp_bytes_in(int n) +{ + total_decomp_bytes_in += n; +} + +void update_completion_einval_errs(void) +{ + total_completion_einval_errors++; +} + +void update_completion_timeout_errs(void) +{ + total_completion_timeout_errors++; +} + +void update_completion_comp_buf_overflow_errs(void) +{ + total_completion_comp_buf_overflow_errors++; +} + +void update_max_comp_delay_ns(u64 start_time_ns) +{ + u64 time_diff; + + time_diff = ktime_get_ns() - start_time_ns; + + if (time_diff > max_comp_delay_ns) + max_comp_delay_ns = time_diff; +} + +void update_max_decomp_delay_ns(u64 start_time_ns) +{ + u64 time_diff; + + time_diff = ktime_get_ns() - start_time_ns; + + if (time_diff > max_decomp_delay_ns) + max_decomp_delay_ns = time_diff; +} + +void update_max_acomp_delay_ns(u64 start_time_ns) +{ + u64 time_diff; + + time_diff = ktime_get_ns() - start_time_ns; + + if (time_diff > max_acomp_delay_ns) + max_acomp_delay_ns = time_diff; +} + +void update_max_adecomp_delay_ns(u64 start_time_ns) +{ + u64 time_diff; + + time_diff = ktime_get_ns() - start_time_ns; + + if (time_diff > max_adecomp_delay_ns) + + max_adecomp_delay_ns = time_diff; +} + +void update_wq_comp_calls(struct idxd_wq *idxd_wq) +{ + struct iaa_wq *wq = idxd_wq->private_data; + + wq->comp_calls++; + wq->iaa_device->comp_calls++; +} + +void update_wq_comp_bytes(struct idxd_wq *idxd_wq, int n) +{ + struct iaa_wq *wq = idxd_wq->private_data; + + wq->comp_bytes += n; + wq->iaa_device->comp_bytes += n; +} + +void update_wq_decomp_calls(struct idxd_wq *idxd_wq) +{ + struct iaa_wq *wq = idxd_wq->private_data; + + wq->decomp_calls++; + wq->iaa_device->decomp_calls++; +} + +void update_wq_decomp_bytes(struct idxd_wq *idxd_wq, int n) +{ + struct iaa_wq *wq = idxd_wq->private_data; + + wq->decomp_bytes += n; + wq->iaa_device->decomp_bytes += n; +} + +void reset_iaa_crypto_stats(void) +{ + total_comp_calls = 0; + total_decomp_calls = 0; + max_comp_delay_ns = 0; + max_decomp_delay_ns = 0; + max_acomp_delay_ns = 0; + max_adecomp_delay_ns = 0; + total_comp_bytes_out = 0; + total_decomp_bytes_in = 0; + total_completion_einval_errors = 0; + total_completion_timeout_errors = 0; + total_completion_comp_buf_overflow_errors = 0; +} + +static void reset_wq_stats(struct iaa_wq *wq) +{ + wq->comp_calls = 0; + wq->comp_bytes = 0; + wq->decomp_calls = 0; + wq->decomp_bytes = 0; +} + +void reset_device_stats(struct iaa_device *iaa_device) +{ + struct iaa_wq *iaa_wq; + + iaa_device->comp_calls = 0; + iaa_device->comp_bytes = 0; + iaa_device->decomp_calls = 0; + iaa_device->decomp_bytes = 0; + + list_for_each_entry(iaa_wq, &iaa_device->wqs, list) + reset_wq_stats(iaa_wq); +} + +static void wq_show(struct seq_file *m, struct iaa_wq *iaa_wq) +{ + seq_printf(m, " name: %s\n", iaa_wq->wq->name); + seq_printf(m, " comp_calls: %llu\n", iaa_wq->comp_calls); + seq_printf(m, " comp_bytes: %llu\n", iaa_wq->comp_bytes); + seq_printf(m, " decomp_calls: %llu\n", iaa_wq->decomp_calls); + seq_printf(m, " decomp_bytes: %llu\n\n", iaa_wq->decomp_bytes); +} + +void device_stats_show(struct seq_file *m, struct iaa_device *iaa_device) +{ + struct iaa_wq *iaa_wq; + + seq_puts(m, "iaa device:\n"); + seq_printf(m, " id: %d\n", iaa_device->idxd->id); + seq_printf(m, " n_wqs: %d\n", iaa_device->n_wq); + seq_printf(m, " comp_calls: %llu\n", iaa_device->comp_calls); + seq_printf(m, " comp_bytes: %llu\n", iaa_device->comp_bytes); + seq_printf(m, " decomp_calls: %llu\n", iaa_device->decomp_calls); + seq_printf(m, " decomp_bytes: %llu\n", iaa_device->decomp_bytes); + seq_puts(m, " wqs:\n"); + + list_for_each_entry(iaa_wq, &iaa_device->wqs, list) + wq_show(m, iaa_wq); +} + +void global_stats_show(struct seq_file *m) +{ + seq_puts(m, "global stats:\n"); + seq_printf(m, " total_comp_calls: %llu\n", total_comp_calls); + seq_printf(m, " total_decomp_calls: %llu\n", total_decomp_calls); + seq_printf(m, " total_comp_bytes_out: %llu\n", total_comp_bytes_out); + seq_printf(m, " total_decomp_bytes_in: %llu\n", total_decomp_bytes_in); + seq_printf(m, " total_completion_einval_errors: %llu\n", + total_completion_einval_errors); + seq_printf(m, " total_completion_timeout_errors: %llu\n", + total_completion_timeout_errors); + seq_printf(m, " total_completion_comp_buf_overflow_errors: %llu\n\n", + total_completion_comp_buf_overflow_errors); +} + +static int wq_stats_open(struct inode *inode, struct file *file) +{ + return single_open(file, wq_stats_show, file); +} + +const struct file_operations wq_stats_fops = { + .open = wq_stats_open, + .read = seq_read, + .llseek = seq_lseek, + .release = single_release, +}; + +DEFINE_DEBUGFS_ATTRIBUTE(wq_stats_reset_fops, NULL, iaa_crypto_stats_reset, "%llu\n"); + +int __init iaa_crypto_debugfs_init(void) +{ + if (!debugfs_initialized()) + return -ENODEV; + + iaa_crypto_debugfs_root = debugfs_create_dir("iaa_crypto", NULL); + if (!iaa_crypto_debugfs_root) + return -ENOMEM; + + debugfs_create_u64("max_comp_delay_ns", 0644, + iaa_crypto_debugfs_root, &max_comp_delay_ns); + debugfs_create_u64("max_decomp_delay_ns", 0644, + iaa_crypto_debugfs_root, &max_decomp_delay_ns); + debugfs_create_u64("max_acomp_delay_ns", 0644, + iaa_crypto_debugfs_root, &max_comp_delay_ns); + debugfs_create_u64("max_adecomp_delay_ns", 0644, + iaa_crypto_debugfs_root, &max_decomp_delay_ns); + debugfs_create_u64("total_comp_calls", 0644, + iaa_crypto_debugfs_root, &total_comp_calls); + debugfs_create_u64("total_decomp_calls", 0644, + iaa_crypto_debugfs_root, &total_decomp_calls); + debugfs_create_u64("total_comp_bytes_out", 0644, + iaa_crypto_debugfs_root, &total_comp_bytes_out); + debugfs_create_u64("total_decomp_bytes_in", 0644, + iaa_crypto_debugfs_root, &total_decomp_bytes_in); + debugfs_create_file("wq_stats", 0644, iaa_crypto_debugfs_root, NULL, + &wq_stats_fops); + debugfs_create_file("stats_reset", 0644, iaa_crypto_debugfs_root, NULL, + &wq_stats_reset_fops); + + return 0; +} + +void __exit iaa_crypto_debugfs_cleanup(void) +{ + debugfs_remove_recursive(iaa_crypto_debugfs_root); +} + +MODULE_LICENSE("GPL"); diff --git a/drivers/crypto/iaa/iaa_crypto_stats.h b/drivers/crypto/iaa/iaa_crypto_stats.h new file mode 100644 index 000000000000..ad8333329fa6 --- /dev/null +++ b/drivers/crypto/iaa/iaa_crypto_stats.h @@ -0,0 +1,58 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright(c) 2021 Intel Corporation. All rights rsvd. */ + +#ifndef __CRYPTO_DEV_IAA_CRYPTO_STATS_H__ +#define __CRYPTO_DEV_IAA_CRYPTO_STATS_H__ + +#if defined(CONFIG_CRYPTO_DEV_IAA_CRYPTO_STATS) +int iaa_crypto_debugfs_init(void); +void iaa_crypto_debugfs_cleanup(void); + +void update_total_comp_calls(void); +void update_total_comp_bytes_out(int n); +void update_total_decomp_calls(void); +void update_total_decomp_bytes_in(int n); +void update_max_comp_delay_ns(u64 start_time_ns); +void update_max_decomp_delay_ns(u64 start_time_ns); +void update_max_acomp_delay_ns(u64 start_time_ns); +void update_max_adecomp_delay_ns(u64 start_time_ns); +void update_completion_einval_errs(void); +void update_completion_timeout_errs(void); +void update_completion_comp_buf_overflow_errs(void); + +void update_wq_comp_calls(struct idxd_wq *idxd_wq); +void update_wq_comp_bytes(struct idxd_wq *idxd_wq, int n); +void update_wq_decomp_calls(struct idxd_wq *idxd_wq); +void update_wq_decomp_bytes(struct idxd_wq *idxd_wq, int n); + +int wq_stats_show(struct seq_file *m, void *v); +int iaa_crypto_stats_reset(void *data, u64 value); + +static inline u64 iaa_get_ts(void) { return ktime_get_ns(); } + +#else +static inline int iaa_crypto_debugfs_init(void) { return 0; } +static inline void iaa_crypto_debugfs_cleanup(void) {} + +static inline void update_total_comp_calls(void) {} +static inline void update_total_comp_bytes_out(int n) {} +static inline void update_total_decomp_calls(void) {} +static inline void update_total_decomp_bytes_in(int n) {} +static inline void update_max_comp_delay_ns(u64 start_time_ns) {} +static inline void update_max_decomp_delay_ns(u64 start_time_ns) {} +static inline void update_max_acomp_delay_ns(u64 start_time_ns) {} +static inline void update_max_adecomp_delay_ns(u64 start_time_ns) {} +static inline void update_completion_einval_errs(void) {} +static inline void update_completion_timeout_errs(void) {} +static inline void update_completion_comp_buf_overflow_errs(void) {} + +static inline void update_wq_comp_calls(struct idxd_wq *idxd_wq) {} +static inline void update_wq_comp_bytes(struct idxd_wq *idxd_wq, int n) {} +static inline void update_wq_decomp_calls(struct idxd_wq *idxd_wq) {} +static inline void update_wq_decomp_bytes(struct idxd_wq *idxd_wq, int n) {} + +static inline u64 iaa_get_ts(void) { return 0; } + +#endif // CONFIG_CRYPTO_DEV_IAA_CRYPTO_STATS + +#endif