From patchwork Wed Apr 19 10:03:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Tesarik X-Patchwork-Id: 85298 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp264908vqo; Wed, 19 Apr 2023 03:06:27 -0700 (PDT) X-Google-Smtp-Source: AKy350bkyX2ovOa6OmESr58zDbA76ByWUloyFWT830j9GnKfnDJ4vec7azIsUch5RKpX9glJ72Zz X-Received: by 2002:a05:6a20:d398:b0:ee:aa9e:945a with SMTP id iq24-20020a056a20d39800b000eeaa9e945amr2954630pzb.26.1681898786826; Wed, 19 Apr 2023 03:06:26 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681898786; cv=none; d=google.com; s=arc-20160816; b=tf4pJJ7UTf/TmR0qqreBK5ztYrOdJaTur3sL+v02DWbjvplqdUXCdfYLWIWH9H89Gs hCqQTuWEkg6oqj5bSNj8S6VQ9QeUDKk+gpccrf0U9qcz81GsmthbL2eq+PRCfz7zLXME XM3Bii04JqhG5JzZU28HyQNQhsiXv7ntDOIHA7gwWVmT7+qxw00iYs0hWQztJ6jN1SJX z0qZKoQyJEPcnJIOso2Cn1rG0xQxzeoj5ue0fekFRZ2wL3786jzepc2mVt6/5HtwwobQ mz2d45opKSUukDRlfv2H6kEH4k3raxVWdFra2Uz3nGdX2O54EWY3rW9uTVkj5cIw4rbh bqpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=Qn5lJU0GLFbJx5/ZN6QSf8mW1JOyc3paXyHmDcFcAuA=; b=Fr7gIN8HHXO6aku6u+hzmozACQ2R/Tmq2JJnrMg4ksm35tm2C8MgPtKYFs5IwvnRx+ 98+fP/+NApbIkyReQsYFotu1OdzBnWFh/h23tclTzsYpi9vkiWYx9R8Yw/sPCkft5FiP adDOvSHMgva882SUWminqS+bXstWgeQszh/m3UOIWHk+1sE3aVdVg1lV7NRxCracAQ7C lJoHDT3xnXHBYAyugxcQgvOkGS3Y77RiVW1kfAothDT4dhkASj+1oVN5Hln3XPrdSnxv pRtVryeuGwg5/KrbdHrEUpkCX5jAO9axsGFO44bNjUmZheL5zBiFYw4cOgc9+Z0rYTo0 +4iA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id t25-20020aa79479000000b0063d27f81f05si5983880pfq.247.2023.04.19.03.06.13; Wed, 19 Apr 2023 03:06:26 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232518AbjDSKFi (ORCPT + 99 others); Wed, 19 Apr 2023 06:05:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232513AbjDSKFf (ORCPT ); Wed, 19 Apr 2023 06:05:35 -0400 Received: from frasgout11.his.huawei.com (frasgout11.his.huawei.com [14.137.139.23]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 65DA0106; Wed, 19 Apr 2023 03:05:33 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.18.147.228]) by frasgout11.his.huawei.com (SkyGuard) with ESMTP id 4Q1blL1yGHz9xqwr; Wed, 19 Apr 2023 17:55:58 +0800 (CST) Received: from A2101119013HW2.china.huawei.com (unknown [10.45.152.239]) by APP1 (Coremail) with SMTP id LxC2BwAXi_aYvD9k7TY6Ag--.1944S3; Wed, 19 Apr 2023 11:04:48 +0100 (CET) From: Petr Tesarik To: Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Zhen Lei , Randy Dunlap , Damien Le Moal , Kim Phillips , "Steven Rostedt (Google)" , Muchun Song , Ondrej Zary , "Jason A. Donenfeld" , Petr Tesarik , Hans de Goede , Dan Williams , Andy Shevchenko , Kees Cook , Thomas Gleixner , Won Chung , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), dri-devel@lists.freedesktop.org (open list:DRM DRIVERS), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Kefeng Wang , petr@tesarici.cz Subject: [PATCH v2 1/7] swiotlb: Use a helper to initialize swiotlb fields in struct device Date: Wed, 19 Apr 2023 12:03:53 +0200 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-CM-TRANSID: LxC2BwAXi_aYvD9k7TY6Ag--.1944S3 X-Coremail-Antispam: 1UD129KBjvJXoW7Ww4DuF1Utry5Cry7Jw1xuFg_yoW8ZrWUpF 97Aay5KFW2gFZ7CrW7C3s7JF1fKan7Ca4xCrya9wn5CFnxJF9xJFyDJrW5Arn5Jr409F4f JFySvry5CF17Aw7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmmb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUGw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVWUJVWUCwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV WxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW8JVWxJwA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_ Gr1j6F4UJwAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ew Av7VC0I7IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY 6r1j6r4UM4x0Y48IcxkI7VAKI48JM4IIrI8v6xkF7I0E8cxan2IY04v7MxkF7I0En4kS14 v26r1q6r43MxkF7I0Ew4C26cxK6c8Ij28IcwCF04k20xvY0x0EwIxGrwCFx2IqxVCFs4IE 7xkEbVWUJVW8JwC20s026c02F40E14v26r1j6r18MI8I3I0E7480Y4vE14v26r106r1rMI 8E67AF67kF1VAFwI0_Wrv_Gr1UMIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_ JF4lIxAIcVC0I7IYx2IY6xkF7I0E14v26F4j6r4UJwCI42IY6xAIw20EY4v20xvaj40_Jr 0_JF4lIxAIcVC2z280aVAFwI0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8Jr0_Cr1U YxBIdaVFxhVjvjDU0xZFpf9x07j50eQUUUUU= X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763598702276962335?= X-GMAIL-MSGID: =?utf-8?q?1763598702276962335?= From: Petr Tesarik Move swiotlb initialization code to swiotlb.h. This change also allows to provide a stub implementation if swiotlb is not configured, getting rid of an #ifdef in driver core. Signed-off-by: Petr Tesarik --- drivers/base/core.c | 4 +--- include/linux/swiotlb.h | 12 ++++++++++++ 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/drivers/base/core.c b/drivers/base/core.c index 6878dfcbf0d6..a5dc7c673102 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -3060,9 +3060,7 @@ void device_initialize(struct device *dev) defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) dev->dma_coherent = dma_default_coherent; #endif -#ifdef CONFIG_SWIOTLB - dev->dma_io_tlb_mem = &io_tlb_default_mem; -#endif + swiotlb_dev_init(dev); } EXPORT_SYMBOL_GPL(device_initialize); diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index bcef10e20ea4..b65b7330f7e5 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -119,6 +119,15 @@ static inline bool is_swiotlb_force_bounce(struct device *dev) return mem && mem->force_bounce; } +/** + * swiotlb_dev_init() - initialize swiotlb fields in &struct device + * @dev: device to be initialized + */ +static inline void swiotlb_dev_init(struct device *dev) +{ + dev->dma_io_tlb_mem = &io_tlb_default_mem; +} + void swiotlb_init(bool addressing_limited, unsigned int flags); void __init swiotlb_exit(void); size_t swiotlb_max_mapping_size(struct device *dev); @@ -128,6 +137,9 @@ void __init swiotlb_adjust_size(unsigned long size); static inline void swiotlb_init(bool addressing_limited, unsigned int flags) { } +static inline void swiotlb_dev_init(struct device *dev) +{ +} static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr) { return false; From patchwork Wed Apr 19 10:03:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Tesarik X-Patchwork-Id: 85305 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp277048vqo; Wed, 19 Apr 2023 03:30:03 -0700 (PDT) X-Google-Smtp-Source: AKy350b0cF1tTHLGhUCgN1JGFwAwuyPOo5duGa7wnPUw2X9OaROrfXH0BOilq42psrEUhcSWftbj X-Received: by 2002:a17:90a:c095:b0:247:4e73:cbdd with SMTP id o21-20020a17090ac09500b002474e73cbddmr2253448pjs.9.1681900203621; Wed, 19 Apr 2023 03:30:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681900203; cv=none; d=google.com; s=arc-20160816; b=w6CBm2VTjfsy1hNkl2MENDEmXAaxCqPEZgmH6SL7DYf5yo+GcDgwgsPrC1fgFTzBsq ibOrtskRRT3PW/EbgNeFD3RBTiGugkLHZquwQJyXi5KRHG9TdgWYcw5WZ+j+wnMm83lM Bq2LqQvrmwbg1bcK0g1oaZonn5pPfPsABqdMhrARUwIaz41Gpx+4Vw6lCE2XzFJ79LPU O0T64OoZ/WOKTu/mf4VWtke6oJ7xfcv4wNwYZempcgLT4DUiooLKf55WqPYbuBGALVcr ttA4q1Wp8PFqb0t0PYohVMxO6ob8qBspFX4BBFA7gmU1xFr/nl5yCQuO9Y6hGSXCDQ5J /4dg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=c+ERw3mYXyNKyEeB/M13vMpvYtQsEtTJYF8qE8xZc6c=; b=der9m5bgVSW6cq695ZPQ4mOpZoZbxsidpqA03vI9TPGjcK1YAU0OKrDM5kEyNVoNi5 hMqEP6y5FLS4uzY3uqAmBDN6l/ZUlk5/Uzi0HNnroNX9zTNGyoS5wC7S3T9+cLldHqpq 06kiMDltAn6O4a4LC0jVjXdHvbTSdKGaaFJ3tB3vlD+xE2W7e0TrL7pQUkVodPbVdjFO qYXEoGdGVg2nQeyDfFTQ/KJR6TCyzfkuai22ZjsiiVwGwe4FUJYGnTWrqq1ufonWOO5F dyrzoPHeCW5BhmxIGA15sIwjEqZhC4Nbt477NsMa42CBi+9kJDzcTH1s8KqhLtxt0vkO 7MeA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h1-20020a17090aa88100b0024730b93a6asi1472714pjq.147.2023.04.19.03.29.51; Wed, 19 Apr 2023 03:30:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232706AbjDSKF5 (ORCPT + 99 others); Wed, 19 Apr 2023 06:05:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232860AbjDSKFx (ORCPT ); Wed, 19 Apr 2023 06:05:53 -0400 Received: from frasgout11.his.huawei.com (frasgout11.his.huawei.com [14.137.139.23]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4ECE2106; Wed, 19 Apr 2023 03:05:51 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.18.147.227]) by frasgout11.his.huawei.com (SkyGuard) with ESMTP id 4Q1blj4lqNz9xqwp; Wed, 19 Apr 2023 17:56:17 +0800 (CST) Received: from A2101119013HW2.china.huawei.com (unknown [10.45.152.239]) by APP1 (Coremail) with SMTP id LxC2BwAXi_aYvD9k7TY6Ag--.1944S4; Wed, 19 Apr 2023 11:05:07 +0100 (CET) From: Petr Tesarik To: Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Zhen Lei , Randy Dunlap , Damien Le Moal , Kim Phillips , "Steven Rostedt (Google)" , Muchun Song , Ondrej Zary , "Jason A. Donenfeld" , Petr Tesarik , Hans de Goede , Dan Williams , Andy Shevchenko , Kees Cook , Thomas Gleixner , Won Chung , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), dri-devel@lists.freedesktop.org (open list:DRM DRIVERS), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Kefeng Wang , petr@tesarici.cz Subject: [PATCH v2 2/7] swiotlb: Move code around in preparation for dynamic bounce buffers Date: Wed, 19 Apr 2023 12:03:54 +0200 Message-Id: <2954d90769bb926271c22553ef57ec805fea7865.1681898595.git.petr.tesarik.ext@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-CM-TRANSID: LxC2BwAXi_aYvD9k7TY6Ag--.1944S4 X-Coremail-Antispam: 1UD129KBjvJXoW3JryUur45Aw47Kw4fKF4kZwb_yoW3uFW8pF 1ft3WrtFsxJF1xC397uw48JF1Fkw1kG343Caya9ryF9Fy3Xrn0vFZ8CrW5WayFvFWv9F4U Xr98uF4rKF47Jr7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmGb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUXw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVWUJVWUCwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV W8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv6xkF7I0E14v2 6r4UJVWxJr1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2 WlYx0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE7xkE bVWUJVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCY1x0262kKe7 AKxVWUtVW8ZwCY1x0264kExVAvwVAq07x20xyl42xK82IYc2Ij64vIr41l4I8I3I0E4IkC 6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWw C2zVAF1VAY17CE14v26rWY6r4UJwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j 6r1xMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UMIIF0xvE42xK8VAvwI8IcIk0rV WUJVWUCwCI42IY6I8E87Iv67AKxVW8JVWxJwCI42IY6I8E87Iv6xkF7I0E14v26r4UJVWx JrUvcSsGvfC2KfnxnUUI43ZEXa7IU8go7tUUUUU== X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763600188160429972?= X-GMAIL-MSGID: =?utf-8?q?1763600188160429972?= From: Petr Tesarik To prepare for the introduction of dynamically allocated bounce buffers, separate out common code and code which handles non-dynamic (aka fixed) bounce buffers. No functional change, but this commit should make the addition of dynamic allocations easier to review. Signed-off-by: Petr Tesarik --- include/linux/swiotlb.h | 31 ++++++++++- kernel/dma/swiotlb.c | 110 +++++++++++++++++++++++++++++++++------- 2 files changed, 122 insertions(+), 19 deletions(-) diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index b65b7330f7e5..a0bb6b30b17c 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -105,11 +105,40 @@ struct io_tlb_mem { }; extern struct io_tlb_mem io_tlb_default_mem; +/** + * is_swiotlb_fixed() - check if a physical address belongs to a swiotlb slot + * @mem: relevant swiotlb pool + * @paddr: physical address within the DMA buffer + * + * Check if @paddr points into a fixed bounce buffer slot. + * This check should be as fast as possible. + * + * Return: + * * %true if @paddr points into a @mem fixed slot + * * %false otherwise + */ +static inline bool is_swiotlb_fixed(struct io_tlb_mem *mem, phys_addr_t paddr) +{ + return paddr >= mem->start && paddr < mem->end; +} + +/** + * is_swiotlb_buffer() - check if a physical address is allocated from the + * swiotlb pool + * @dev: device which has mapped the buffer + * @paddr: physical address within the DMA buffer + * + * Check if @paddr points into a bounce buffer. + * + * Return: + * * %true if @paddr points into a bounce buffer + * * %false otherwise + */ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr) { struct io_tlb_mem *mem = dev->dma_io_tlb_mem; - return mem && paddr >= mem->start && paddr < mem->end; + return mem && is_swiotlb_fixed(mem, paddr); } static inline bool is_swiotlb_force_bounce(struct device *dev) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index dac42a2ad588..2f09a4ed4215 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -78,6 +78,10 @@ phys_addr_t swiotlb_unencrypted_base; static unsigned long default_nslabs = IO_TLB_DEFAULT_SIZE >> IO_TLB_SHIFT; static unsigned long default_nareas; +static void swiotlb_copy(struct device *dev, phys_addr_t orig_addr, + unsigned char *vaddr, size_t size, size_t alloc_size, + unsigned int tlb_offset, enum dma_data_direction dir); + /** * struct io_tlb_area - IO TLB memory area descriptor * @@ -523,7 +527,6 @@ static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size int index = (tlb_addr - mem->start) >> IO_TLB_SHIFT; phys_addr_t orig_addr = mem->slots[index].orig_addr; size_t alloc_size = mem->slots[index].alloc_size; - unsigned long pfn = PFN_DOWN(orig_addr); unsigned char *vaddr = mem->vaddr + tlb_addr - mem->start; unsigned int tlb_offset, orig_addr_offset; @@ -540,6 +543,34 @@ static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size } tlb_offset -= orig_addr_offset; + swiotlb_copy(dev, orig_addr, vaddr, size, alloc_size, tlb_offset, dir); +} + +/** + * swiotlb_copy() - copy swiotlb buffer content, checking for overflows. + * @dev: device which has mapped the bounce buffer + * @orig_addr: physical address of the original buffer + * @vaddr: virtual address inside the bounce buffer + * @size: number of bytes to copy + * @alloc_size: total allocated size of the bounce buffer + * @tlb_offset: offset within the bounce buffer + * @dir: direction of the data transfer + * + * If @dir is %DMA_TO_DEVICE, copy data from the original buffer to the + * bounce buffer, otherwise copy from the bounce buffer to the original + * buffer. + * + * The original buffer may be in high memory; that's why @orig_addr is + * a physical address. Note that this is the address of the beginning + * of the bounce buffer. Copying starts at offset @tlb_offset. This is + * needed to check accesses beyond the allocated size. + */ +static void swiotlb_copy(struct device *dev, phys_addr_t orig_addr, + unsigned char *vaddr, size_t size, size_t alloc_size, + unsigned int tlb_offset, enum dma_data_direction dir) +{ + unsigned long pfn = PFN_DOWN(orig_addr); + if (tlb_offset > alloc_size) { dev_WARN_ONCE(dev, 1, "Buffer overflow detected. Allocation size: %zu. Mapping size: %zu+%u.\n", @@ -734,15 +765,65 @@ static unsigned long mem_used(struct io_tlb_mem *mem) return used; } +/** + * swiotlb_fixed_map() - allocate a bounce buffer from fixed slots + * @dev: device which maps the buffer + * @orig_addr: address of the original buffer + * @alloc_size: total size of the original buffer + * @alloc_align_mask: + * required physical alignment of the I/O buffer + * @attrs: optional DMA attributes for the map operation + * + * Search for a suitable slot or sequence of slots and initialize them + * for use with the original buffer. + * + * Return: Physical address of the bounce buffer, or %DMA_MAPPING_ERROR. + */ +static phys_addr_t swiotlb_fixed_map(struct device *dev, phys_addr_t orig_addr, + size_t alloc_size, unsigned int alloc_align_mask, + unsigned long attrs) +{ + struct io_tlb_mem *mem = dev->dma_io_tlb_mem; + unsigned int offset = swiotlb_align_offset(dev, orig_addr); + int index = swiotlb_find_slots(dev, orig_addr, + alloc_size + offset, alloc_align_mask); + unsigned int i; + + if (index == -1) + return (phys_addr_t)DMA_MAPPING_ERROR; + + /* + * Save away the mapping from the original address to the DMA address. + * This is needed when we sync the memory. Then we sync the buffer if + * needed. + */ + for (i = 0; i < nr_slots(alloc_size + offset); i++) + mem->slots[index + i].orig_addr = slot_addr(orig_addr, i); + return slot_addr(mem->start, index) + offset; +} + +/** + * swiotlb_tbl_map_single() - map DMA buffer to a bounce buffer + * @dev: device which maps the buffer + * @orig_addr: address of the original buffer + * @mapping_size: size of the original buffer to be synced now + * @alloc_size: total size of the original buffer + * @alloc_align_mask: + * required physical alignment of the I/O buffer + * @dir: direction of the data transfer + * @attrs: optional DMA attributes for the map operation + * + * Create a mapping of the DMA buffer into a bounce buffer and copy the + * original data. + * + * Return: Physical address of the bounce buffer, or %DMA_MAPPING_ERROR. + */ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, size_t mapping_size, size_t alloc_size, unsigned int alloc_align_mask, enum dma_data_direction dir, unsigned long attrs) { struct io_tlb_mem *mem = dev->dma_io_tlb_mem; - unsigned int offset = swiotlb_align_offset(dev, orig_addr); - unsigned int i; - int index; phys_addr_t tlb_addr; if (!mem || !mem->nslabs) { @@ -760,24 +841,17 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, return (phys_addr_t)DMA_MAPPING_ERROR; } - index = swiotlb_find_slots(dev, orig_addr, - alloc_size + offset, alloc_align_mask); - if (index == -1) { + tlb_addr = swiotlb_fixed_map(dev, orig_addr, alloc_size, + alloc_align_mask, attrs); + + if (tlb_addr == (phys_addr_t)DMA_MAPPING_ERROR) { if (!(attrs & DMA_ATTR_NO_WARN)) dev_warn_ratelimited(dev, - "swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slots)\n", - alloc_size, mem->nslabs, mem_used(mem)); - return (phys_addr_t)DMA_MAPPING_ERROR; + "swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slots)\n", + alloc_size, mem->nslabs, mem_used(mem)); + return tlb_addr; } - /* - * Save away the mapping from the original address to the DMA address. - * This is needed when we sync the memory. Then we sync the buffer if - * needed. - */ - for (i = 0; i < nr_slots(alloc_size + offset); i++) - mem->slots[index + i].orig_addr = slot_addr(orig_addr, i); - tlb_addr = slot_addr(mem->start, index) + offset; /* * When dir == DMA_FROM_DEVICE we could omit the copy from the orig * to the tlb buffer, if we knew for sure the device will From patchwork Wed Apr 19 10:03:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Tesarik X-Patchwork-Id: 85299 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp265750vqo; Wed, 19 Apr 2023 03:07:46 -0700 (PDT) X-Google-Smtp-Source: AKy350bEZ/f5BqU3PxoNSfCqP9gvccSnL0BWmo4S8eDhRzL1aDDIQ9Tl0XUvxG0sz3j7eKA1yMU0 X-Received: by 2002:a17:90a:7f92:b0:247:4877:2172 with SMTP id m18-20020a17090a7f9200b0024748772172mr2274532pjl.47.1681898865875; Wed, 19 Apr 2023 03:07:45 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681898865; cv=none; d=google.com; s=arc-20160816; b=WJ0PhxWukG73JoikAOvXaT4hwocPuHIRkLe1xRelMbK16kdtq0k2f2HDEGNDIWkrNd HHAwPRq0MCs1V6RkYKXSUDrxQBlpH1qNiv+MD2NQYarFwVq4y9f5o3zeUtHkiGM7kjWY /rR3hF6MV9JCjgE4AxUa9LiEiGHSpyKl6c0QPl8tQD0ZmqmoNPFgspogrdkCAP1n/0Wa N3APTNpT7OkeGe8fzMpL9H4aK9mEOYrLXIZdvebAIDDLbIatjI9QQdmFNSmnIZ+uaJsq Mwp4sEhBty7THdl0kur0QTNZHmB8rY+Y4FjwUFYH714W204m3pkVZRdK2Iw9RF5dSw3i R2kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=Y45VeXFx+dKJiHYvCdPStPPQMCBgSu0eRYMwrvbrj/o=; b=xk+GIt/fwuqEejZCcdq4zGAqDOKqUCPTD6LmqE3qC83iEvC+fSvazTumsV3Qn+ud9i s7MSpQZ5CI4lMBhOSedQrGC/Wci8/tZPDNWB5k5ZeMMc9pubpim73b2HeQssM73WFCJM uhYHqUFb16x+h+EZpo1XF1dr2qoVSHFuurXd3OcAGwhSCTI9nEkMKin98ubaYU1nrQbR VLV+77VyKpYIPp7IvAS76gleMARCXYnPk/6D8nmPu/KrKos1VVJix/sEbTGAQwU4jSve azuoMGPciDLlsKRaP9EfFJwg5uebr38I5N55RredFY5MSWKSmvNJGADoL953Xz76iy1L 1u0A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id y17-20020a17090aa41100b0023f000be67dsi1486599pjp.13.2023.04.19.03.07.31; Wed, 19 Apr 2023 03:07:45 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233008AbjDSKGd (ORCPT + 99 others); Wed, 19 Apr 2023 06:06:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233007AbjDSKGU (ORCPT ); Wed, 19 Apr 2023 06:06:20 -0400 Received: from frasgout11.his.huawei.com (frasgout11.his.huawei.com [14.137.139.23]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 180A0B746; Wed, 19 Apr 2023 03:06:10 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.18.147.227]) by frasgout11.his.huawei.com (SkyGuard) with ESMTP id 4Q1bm42MkWz9v7GZ; Wed, 19 Apr 2023 17:56:36 +0800 (CST) Received: from A2101119013HW2.china.huawei.com (unknown [10.45.152.239]) by APP1 (Coremail) with SMTP id LxC2BwAXi_aYvD9k7TY6Ag--.1944S5; Wed, 19 Apr 2023 11:05:26 +0100 (CET) From: Petr Tesarik To: Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Zhen Lei , Randy Dunlap , Damien Le Moal , Kim Phillips , "Steven Rostedt (Google)" , Muchun Song , Ondrej Zary , "Jason A. Donenfeld" , Petr Tesarik , Hans de Goede , Dan Williams , Andy Shevchenko , Kees Cook , Thomas Gleixner , Won Chung , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), dri-devel@lists.freedesktop.org (open list:DRM DRIVERS), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Kefeng Wang , petr@tesarici.cz Subject: [PATCH v2 3/7] dma-mapping: introduce the DMA_ATTR_MAY_SLEEP attribute Date: Wed, 19 Apr 2023 12:03:55 +0200 Message-Id: <5116beca58c18b73611a88063c290e56f38bb441.1681898595.git.petr.tesarik.ext@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-CM-TRANSID: LxC2BwAXi_aYvD9k7TY6Ag--.1944S5 X-Coremail-Antispam: 1UD129KBjvJXoW7ZF17ur1rGrWUZw18GFy5XFb_yoW8Cw48p3 ZagFyfGrZ2gr1xCr1kG3W29r4UXa18u345GF40vr1rZFW5A3Z29rZ8Kw1Yq3s8XryxCFWF vrW29ry5CryqyrDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmGb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUWw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVWUCVW8JwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV W8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv6xkF7I0E14v2 6r4UJVWxJr1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2 WlYx0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE7xkE bVWUJVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCY1x0262kKe7 AKxVWUtVW8ZwCY1x0264kExVAvwVAq07x20xyl42xK82IYc2Ij64vIr41l4I8I3I0E4IkC 6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWw C2zVAF1VAY17CE14v26rWY6r4UJwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j 6r1xMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UMIIF0xvE42xK8VAvwI8IcIk0rV WUJVWUCwCI42IY6I8E87Iv67AKxVW8JVWxJwCI42IY6I8E87Iv6xkF7I0E14v26r4UJVWx JrUvcSsGvfC2KfnxnUUI43ZEXa7IU8D5r7UUUUU== X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763598785449703468?= X-GMAIL-MSGID: =?utf-8?q?1763598785449703468?= From: Petr Tesarik Introduce a DMA attribute to tell the DMA-mapping subsystem that the operation is allowed to sleep. This patch merely adds the flag, but it does not do anything at the moment. Signed-off-by: Petr Tesarik --- Documentation/core-api/dma-attributes.rst | 10 ++++++++++ include/linux/dma-mapping.h | 6 ++++++ 2 files changed, 16 insertions(+) diff --git a/Documentation/core-api/dma-attributes.rst b/Documentation/core-api/dma-attributes.rst index 1887d92e8e92..9ce00926455f 100644 --- a/Documentation/core-api/dma-attributes.rst +++ b/Documentation/core-api/dma-attributes.rst @@ -130,3 +130,13 @@ accesses to DMA buffers in both privileged "supervisor" and unprivileged subsystem that the buffer is fully accessible at the elevated privilege level (and ideally inaccessible or at least read-only at the lesser-privileged levels). + +DMA_ATTR_MAY_SLEEP +------------------ + +This tells the DMA-mapping subsystem that it is allowed to sleep. For example, +if mapping needs a bounce buffer, software IO TLB may use CMA for the +allocation if this flag is given. + +This attribute is not used for dma_alloc\* functions. Instead, the provided +GFP flags are used to determine whether the allocation may sleep. diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 0ee20b764000..7a75c503ac38 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -61,6 +61,12 @@ */ #define DMA_ATTR_PRIVILEGED (1UL << 9) +/* + * DMA_ATTR_MAY_SLEEP: This tells the DMA-mapping subsystem that it is allowed + * to sleep. + */ +#define DMA_ATTR_MAY_SLEEP (1UL << 10) + /* * A dma_addr_t can hold any valid DMA or bus address for the platform. It can * be given to a device to use as a DMA source or target. It is specific to a From patchwork Wed Apr 19 10:03:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Tesarik X-Patchwork-Id: 85302 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp273996vqo; Wed, 19 Apr 2023 03:23:37 -0700 (PDT) X-Google-Smtp-Source: AKy350bW9NKDBNzQ9xhcrWHKwGHZ1oy2k3U8SMpKtlCttIVyPLYXgFhbPIQBoQJrh5BKTgjO0z8m X-Received: by 2002:a17:90a:b78e:b0:247:4c28:311f with SMTP id m14-20020a17090ab78e00b002474c28311fmr2333902pjr.34.1681899817401; Wed, 19 Apr 2023 03:23:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681899817; cv=none; d=google.com; s=arc-20160816; b=U9FflILfrKoSlEnBZRrbSKLij09Hga+kfFKyyYhmPlf5eKz7ES4kGbFHsP95IchVMw 4KoOm39kXUvtaqE2SiwRCKgXyN0OckgG7IxMlZaLWwBp7cz7KUtNVQTvDuZWtUCDXBSl i7wUD/ZVe4ypXutLRQrmPY1KdpB8FgjbknHGGIvnkzbwGtCTS38RZwonavRKPuN1H5v1 qTsVvMDoudxyrEmOTqKPeAvFT2vS+1CzbVjeCAmTzz6hT6YRy1ZUROQTHa0p6m5WVsKH Fv5Uj6A+f3wZ6qO0GA6YuTQ21NP8D33oZfkyiigtYo7h0/hZu5/N77AVReV//OLMCh/9 D9Jw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=hW2fvO/qfY03vPEXeejMNqKWCVHQ77INywVuBC9V/hw=; b=JGhbKUYBxtlmgFwM7J39YBengABd7D0ew24FL14p+83I4aSgFgwzXpZAXr1zC2RoOc mLNzkrUPbAts24YB+Hf6lhCI5zzSkUCqV418IC3spyMVspbEBZyrMGs5OMgplT2y/sAn tLiARLZDky8viucpTqRl3Lij+YFAke1LyC5fiLRPVt/3NPQd5M7duFhTRjPyRlnERp7u A4gL7D0lzcKm/IfSJLx11MO6fyamuHuH8M13JCcDMGW5WcDNVy2R4EmqbEabSeHOQywH C9qDWP/ng9jSpNLbxKhzlLzwvK7u9XV85wqZM8+gg0BlxxfXn0nguZUJHm7NFCFAn7JY Gq9g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q13-20020a17090aa00d00b002472da38b3asi1571267pjp.60.2023.04.19.03.23.25; Wed, 19 Apr 2023 03:23:37 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232977AbjDSKHE (ORCPT + 99 others); Wed, 19 Apr 2023 06:07:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232993AbjDSKGr (ORCPT ); Wed, 19 Apr 2023 06:06:47 -0400 Received: from frasgout13.his.huawei.com (frasgout13.his.huawei.com [14.137.139.46]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4434B106; Wed, 19 Apr 2023 03:06:30 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.18.147.228]) by frasgout13.his.huawei.com (SkyGuard) with ESMTP id 4Q1bmT52rrz9v7ZN; Wed, 19 Apr 2023 17:56:57 +0800 (CST) Received: from A2101119013HW2.china.huawei.com (unknown [10.45.152.239]) by APP1 (Coremail) with SMTP id LxC2BwAXi_aYvD9k7TY6Ag--.1944S6; Wed, 19 Apr 2023 11:05:45 +0100 (CET) From: Petr Tesarik To: Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Zhen Lei , Randy Dunlap , Damien Le Moal , Kim Phillips , "Steven Rostedt (Google)" , Muchun Song , Ondrej Zary , "Jason A. Donenfeld" , Petr Tesarik , Hans de Goede , Dan Williams , Andy Shevchenko , Kees Cook , Thomas Gleixner , Won Chung , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), dri-devel@lists.freedesktop.org (open list:DRM DRIVERS), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Kefeng Wang , petr@tesarici.cz Subject: [PATCH v2 4/7] swiotlb: Dynamically allocated bounce buffers Date: Wed, 19 Apr 2023 12:03:56 +0200 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-CM-TRANSID: LxC2BwAXi_aYvD9k7TY6Ag--.1944S6 X-Coremail-Antispam: 1UD129KBjvAXoWfGFW5GryrJrWDGr13JF15twb_yoW8Ar4kXo Wfuw43Xw1xtw1UCF9YkFWkGF47Za1kKFWfZr4rXayjgay7Zryjg3y7tF15X3sxWw18KFyx AFyYqFW8WF4xArWkn29KB7ZKAUJUUUU8529EdanIXcx71UUUUU7v73VFW2AGmfu7bjvjm3 AaLaJ3UjIYCTnIWjp_UUUO67kC6x804xWl14x267AKxVWrJVCq3wAFc2x0x2IEx4CE42xK 8VAvwI8IcIk0rVWrJVCq3wAFIxvE14AKwVWUJVWUGwA2048vs2IY020E87I2jVAFwI0_JF 0E3s1l82xGYIkIc2x26xkF7I0E14v26ryj6s0DM28lY4IEw2IIxxk0rwA2F7IY1VAKz4vE j48ve4kI8wA2z4x0Y4vE2Ix0cI8IcVAFwI0_JFI_Gr1l84ACjcxK6xIIjxv20xvEc7CjxV AFwI0_Gr1j6F4UJwA2z4x0Y4vEx4A2jsIE14v26r4j6F4UM28EF7xvwVC2z280aVCY1x02 67AKxVW8Jr0_Cr1UM2AIxVAIcxkEcVAq07x20xvEncxIr21l5I8CrVACY4xI64kE6c02F4 0Ex7xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv67AKxVWUJVW8JwAm72CE4IkC 6x0Yz7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lFIxGxcIEc7CjxVA2Y2ka0xkIwI1lc7CjxV Aaw2AFwI0_Jw0_GFylc7CjxVAKzI0EY4vE52x082I5MxAIw28IcxkI7VAKI48JMxC20s02 6xCaFVCjc4AY6r1j6r4UMI8I3I0E5I8CrVAFwI0_Jr0_Jr4lx2IqxVCjr7xvwVAFwI0_Jr I_JrWlx4CE17CEb7AF67AKxVWrXVW8Jr1lIxkGc2Ij64vIr41lIxAIcVC0I7IYx2IY67AK xVWUJVWUCwCI42IY6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F4UJwCI42IY6xAIw20EY4v20x vaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_Gr0_Cr1lIxAIcVC2z280aVCY1x0267AKxVW8 Jr0_Cr1UYxBIdaVFxhVjvjDU0xZFpf9x07j5CzZUUUUU= X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763599782474995539?= X-GMAIL-MSGID: =?utf-8?q?1763599782474995539?= From: Petr Tesarik The software IO TLB was designed with the assumption that it is not used much, especially on 64-bit systems, so a small fixed memory area (currently 64 MiB) is sufficient to handle the few cases which still require a bounce buffer. However, these cases are not so rare in some circumstances. First, if SEV is active, all DMA must be done through shared unencrypted pages, and SWIOTLB is used to make this happen without changing device drivers. The software IO TLB size is increased to 6% of total memory in sev_setup_arch(), but that is more of an approximation. The actual requirements may vary depending on which drivers are used and the amount of I/O. Second, some embedded devices have very little RAM, so 64 MiB is not negligible. Sadly, these are exactly the devices that also often need a software IO TLB. Although minimum swiotlb size can be found empirically by extensive testing, it would be easier to allocate a small swiotlb at boot and let it grow on demand. Growing the SWIOTLB data structures at run time is impossible. The whole SWIOTLB region is contiguous in physical memory to allow combining adjacent slots and also to ensure that alignment constraints can be met. The SWIOTLB is too big for the buddy allocator (cf. MAX_ORDER). More importantly, even if a new SWIOTLB could be allocated (e.g. from CMA), it cannot be extended in-place (because surrounding pages may be already allocated for other purposes), and there is no mechanism for relocating already mapped bounce buffers: The DMA API gets only the address of a buffer, and the implementation (direct or IOMMU) checks whether it belongs to the software IO TLB. It is possible to allocate multiple smaller struct io_tlb_mem instances. However, they would have to be stored in a non-constant container (list or tree), which needs synchronization between readers and writers, creating contention in a hot path for all devices, not only those which need software IO TLB. Another option is to allocate a very large SWIOTLB at boot, but allow migrating pages to other users (like CMA does). This approach might work, but there are many open issues: 1. After a page is migrated away from SWIOTLB, it must not be used as a (direct) DMA buffer. Otherwise SWIOTLB code would have to check which pages have been migrated to determine whether a given buffer address belongs to a bounce buffer or not, effectively introducing all the issues of multiple SWIOTLB instances. 2. Unlike SWIOTLB, CMA cannot be used from atomic contexts, and that for many different reasons. This might be changed in theory, but it would take a lot of investigation and time. OTOH improvement to the SWIOTLB is needed now. 3. If SWIOTLB is implemented separately from CMA and not as its part, users have to solve the dilemma of how to distribute precious DMA-able memory between them. The present patch is a simplistic solution. Bounce buffers are allocated using the non-coherent DMA API, removing the need to implement a new custom allocator. These buffers are then tracked in a per-device linked list, reducing the impact on devices that do not need the SWIOTLB. Analysis of real-world I/O patterns has shown that the same buffer is typically looked up repeatedly (for further sync operations, or to be unmapped). The most recently used bounce buffer is therefore always moved to the beginning of the list. The list performed better than a maple tree when tested with fio against a QEMU SATA drive backed by a RAM block device in the host (4 cores, 16 iodepth). Other scenarios are also likely to benefit from this MRU order but have not been tested. Operations on the list are serialized with a spinlock. It is unfortunately not possible to use an RCU list, because quiescent state is not guaranteed to happen before an asynchronous event (e.g. hardware interrupt) on another CPU. If that CPU used an old version of the list, it would fail to copy data to and/or from the newly allocated bounce buffer. Last but not least, bounce buffers are never allocated dynamically if the SWIOTLB is in fact a DMA restricted pool, because that would defeat the purpose of a restricted pool. Signed-off-by: Petr Tesarik --- include/linux/device.h | 8 ++ include/linux/swiotlb.h | 8 +- kernel/dma/swiotlb.c | 252 ++++++++++++++++++++++++++++++++++++++-- 3 files changed, 259 insertions(+), 9 deletions(-) diff --git a/include/linux/device.h b/include/linux/device.h index 1508e637bb26..e12d6092bb9c 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -505,6 +505,12 @@ struct device_physical_location { * @dma_mem: Internal for coherent mem override. * @cma_area: Contiguous memory area for dma allocations * @dma_io_tlb_mem: Pointer to the swiotlb pool used. Not for driver use. + * @dma_io_tlb_dyn_lock: + * Spinlock to protect the list of dynamically allocated bounce + * buffers. + * @dma_io_tlb_dyn_slots: + * Dynamically allocated bounce buffers for this device. + * Not for driver use. * @archdata: For arch-specific additions. * @of_node: Associated device tree node. * @fwnode: Associated device node supplied by platform firmware. @@ -610,6 +616,8 @@ struct device { #endif #ifdef CONFIG_SWIOTLB struct io_tlb_mem *dma_io_tlb_mem; + spinlock_t dma_io_tlb_dyn_lock; + struct list_head dma_io_tlb_dyn_slots; #endif /* arch specific additions */ struct dev_archdata archdata; diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index a0bb6b30b17c..0856eddb9063 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -105,6 +105,8 @@ struct io_tlb_mem { }; extern struct io_tlb_mem io_tlb_default_mem; +bool is_swiotlb_dyn(struct device *dev, phys_addr_t paddr); + /** * is_swiotlb_fixed() - check if a physical address belongs to a swiotlb slot * @mem: relevant swiotlb pool @@ -138,7 +140,9 @@ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr) { struct io_tlb_mem *mem = dev->dma_io_tlb_mem; - return mem && is_swiotlb_fixed(mem, paddr); + return mem && + (is_swiotlb_fixed(mem, paddr) || + is_swiotlb_dyn(dev, paddr)); } static inline bool is_swiotlb_force_bounce(struct device *dev) @@ -155,6 +159,8 @@ static inline bool is_swiotlb_force_bounce(struct device *dev) static inline void swiotlb_dev_init(struct device *dev) { dev->dma_io_tlb_mem = &io_tlb_default_mem; + spin_lock_init(&dev->dma_io_tlb_dyn_lock); + INIT_LIST_HEAD(&dev->dma_io_tlb_dyn_slots); } void swiotlb_init(bool addressing_limited, unsigned int flags); diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 2f09a4ed4215..f4faee38ead9 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -68,6 +68,22 @@ struct io_tlb_slot { unsigned int list; }; +/** + * struct io_tlb_dyn_slot - dynamically allocated swiotlb slot + * @node: node in the per-device list + * @orig_addr: physical address of the original DMA buffer + * @alloc_size: total size of the DMA buffer + * @page: first page of the bounce buffer + * @dma_addr: DMA address of the bounce buffer + */ +struct io_tlb_dyn_slot { + struct list_head node; + phys_addr_t orig_addr; + size_t alloc_size; + struct page *page; + dma_addr_t dma_addr; +}; + static bool swiotlb_force_bounce; static bool swiotlb_force_disable; @@ -509,6 +525,191 @@ void __init swiotlb_exit(void) memset(mem, 0, sizeof(*mem)); } +/** + * lookup_dyn_slot_locked() - look up a dynamically allocated bounce buffer + * @dev: device which has mapped the buffer + * @paddr: physical address within the bounce buffer + * + * Walk through the list of bounce buffers dynamically allocated for @dev, + * looking for a buffer which contains @paddr. + * + * Context: Any context. Expects dma_io_tlb_dyn_lock lock to be held. + * Return: + * Address of a &struct io_tlb_dyn_slot, or %NULL if not found. + */ +static struct io_tlb_dyn_slot *lookup_dyn_slot_locked(struct device *dev, + phys_addr_t paddr) +{ + struct io_tlb_dyn_slot *slot; + + list_for_each_entry(slot, &dev->dma_io_tlb_dyn_slots, node) { + phys_addr_t start = page_to_phys(slot->page); + phys_addr_t end = start + slot->alloc_size - 1; + + if (start <= paddr && paddr <= end) + return slot; + } + return NULL; +} + +/** + * lookup_dyn_slot() - look up a dynamically allocated bounce buffer + * @dev: device which has mapped the buffer + * @paddr: physical address within the bounce buffer + * + * Search for a dynamically allocated bounce buffer which contains + * @paddr. If found, the buffer is moved to the beginning of the linked + * list. Use lookup_dyn_slot_locked() directly where this behavior is not + * required/desired. + * + * Context: Any context. Takes and releases dma_io_tlb_dyn_lock. + * Return: + * Address of a &struct io_tlb_dyn_slot, or %NULL if not found. + */ +static struct io_tlb_dyn_slot *lookup_dyn_slot(struct device *dev, + phys_addr_t paddr) +{ + struct io_tlb_dyn_slot *slot; + unsigned long flags; + + spin_lock_irqsave(&dev->dma_io_tlb_dyn_lock, flags); + slot = lookup_dyn_slot_locked(dev, paddr); + list_move(&slot->node, &dev->dma_io_tlb_dyn_slots); + spin_unlock_irqrestore(&dev->dma_io_tlb_dyn_lock, flags); + return slot; +} + +/** + * is_swiotlb_dyn() - check if a physical address belongs to a dynamically + * allocated bounce buffer + * @dev: device which has mapped the buffer + * @paddr: physical address within the bounce buffer + * + * Check whether there is a dynamically allocated bounce buffer which + * contains @paddr. If found, the buffer is moved to the beginning of + * the MRU list. + * + * Return: + * * %true if @paddr points into a dynamically allocated bounce buffer + * * %false otherwise + */ +bool is_swiotlb_dyn(struct device *dev, phys_addr_t paddr) +{ + return !!lookup_dyn_slot(dev, paddr); +} + +/** + * swiotlb_dyn_bounce() - copy a dynamically allocated buffer from or back + * to the original dma location + * @dev: device which has mapped the buffer + * @tlb_addr: physical address inside the bounce buffer + * @size: size of the region to be copied + * @dir: direction of the data transfer + * + * Copy data to or from a buffer of @size bytes starting at @tlb_addr. + * This function works only for dynamically allocated bounce buffers. + */ +static void swiotlb_dyn_bounce(struct device *dev, phys_addr_t tlb_addr, + size_t size, enum dma_data_direction dir) +{ + struct io_tlb_dyn_slot *slot = lookup_dyn_slot(dev, tlb_addr); + unsigned int tlb_offset; + unsigned char *vaddr; + + if (!slot) + return; + + tlb_offset = tlb_addr - page_to_phys(slot->page); + vaddr = page_address(slot->page) + tlb_offset; + + swiotlb_copy(dev, slot->orig_addr, vaddr, size, slot->alloc_size, + tlb_offset, dir); +} + +/** + * swiotlb_dyn_map() - allocate a bounce buffer dynamically + * @dev: device which maps the buffer + * @orig_addr: address of the original buffer + * @alloc_size: total size of the original buffer + * @alloc_align_mask: + * required physical alignment of the I/O buffer + * @dir: direction of the data transfer + * @attrs: optional DMA attributes for the map operation + * + * Allocate a new bounce buffer using DMA non-coherent API. This function + * assumes that there is a fallback allocation scheme if the allocation + * fails. In fact, it always fails for buffers smaller than a page and + * for address constraints that are not (yet) correctly handled by + * dma_direct_alloc_pages(). + * + * Return: Physical address of the bounce buffer, or %DMA_MAPPING_ERROR. + */ +static phys_addr_t swiotlb_dyn_map(struct device *dev, phys_addr_t orig_addr, + size_t alloc_size, unsigned int alloc_align_mask, + enum dma_data_direction dir, unsigned long attrs) +{ + struct io_tlb_dyn_slot *slot; + unsigned long flags; + gfp_t gfp; + + /* Allocation has page granularity. Avoid small buffers. */ + if (alloc_size < PAGE_SIZE) + goto err; + + /* DMA direct does not deal with physical address constraints. */ + if (alloc_align_mask || dma_get_min_align_mask(dev)) + goto err; + + gfp = (attrs & DMA_ATTR_MAY_SLEEP) ? GFP_NOIO : GFP_NOWAIT; + slot = kmalloc(sizeof(*slot), gfp | __GFP_NOWARN); + if (!slot) + goto err; + + slot->orig_addr = orig_addr; + slot->alloc_size = alloc_size; + slot->page = dma_direct_alloc_pages(dev, PAGE_ALIGN(alloc_size), + &slot->dma_addr, dir, + gfp | __GFP_NOWARN); + if (!slot->page) + goto err_free_slot; + + spin_lock_irqsave(&dev->dma_io_tlb_dyn_lock, flags); + list_add(&slot->node, &dev->dma_io_tlb_dyn_slots); + spin_unlock_irqrestore(&dev->dma_io_tlb_dyn_lock, flags); + + return page_to_phys(slot->page); + +err_free_slot: + kfree(slot); +err: + return (phys_addr_t)DMA_MAPPING_ERROR; +} + +/** + * swiotlb_dyn_unmap() - unmap a dynamically allocated bounce buffer + * @dev: device which mapped the buffer + * @tlb_addr: physical address of the bounce buffer + * @dir: direction of the data transfer + * + * Release all resources associated with a dynamically allocated bounce + * buffer. + */ +static void swiotlb_dyn_unmap(struct device *dev, phys_addr_t tlb_addr, + enum dma_data_direction dir) +{ + struct io_tlb_dyn_slot *slot; + unsigned long flags; + + spin_lock_irqsave(&dev->dma_io_tlb_dyn_lock, flags); + slot = lookup_dyn_slot_locked(dev, tlb_addr); + list_del(&slot->node); + spin_unlock_irqrestore(&dev->dma_io_tlb_dyn_lock, flags); + + dma_direct_free_pages(dev, slot->alloc_size, slot->page, + slot->dma_addr, dir); + kfree(slot); +} + /* * Return the offset into a iotlb slot required to keep the device happy. */ @@ -517,11 +718,19 @@ static unsigned int swiotlb_align_offset(struct device *dev, u64 addr) return addr & dma_get_min_align_mask(dev) & (IO_TLB_SIZE - 1); } -/* - * Bounce: copy the swiotlb buffer from or back to the original dma location +/** + * swiotlb_fixed_bounce() - copy a fixed-slot swiotlb buffer from or back + * to the original dma location + * @dev: device which has mapped the buffer + * @tlb_addr: physical address inside the bounce buffer + * @size: size of the region to be copied + * @dir: direction of the data transfer + * + * Copy data to or from a buffer of @size bytes starting at @tlb_addr. + * This function works only for fixed bounce buffers. */ -static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size, - enum dma_data_direction dir) +static void swiotlb_fixed_bounce(struct device *dev, phys_addr_t tlb_addr, + size_t size, enum dma_data_direction dir) { struct io_tlb_mem *mem = dev->dma_io_tlb_mem; int index = (tlb_addr - mem->start) >> IO_TLB_SHIFT; @@ -617,6 +826,25 @@ static void swiotlb_copy(struct device *dev, phys_addr_t orig_addr, } } +/** + * swiotlb_bounce() - copy the swiotlb buffer from or back to the original + * dma location + * @dev: device which has mapped the buffer + * @tlb_addr: physical address inside the bounce buffer + * @size: size of the region to be copied + * @dir: direction of the data transfer + * + * Copy data to or from a buffer of @size bytes starting at @tlb_addr. + */ +static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size, + enum dma_data_direction dir) +{ + if (is_swiotlb_fixed(dev->dma_io_tlb_mem, tlb_addr)) + swiotlb_fixed_bounce(dev, tlb_addr, size, dir); + else + swiotlb_dyn_bounce(dev, tlb_addr, size, dir); +} + static inline phys_addr_t slot_addr(phys_addr_t start, phys_addr_t idx) { return start + (idx << IO_TLB_SHIFT); @@ -841,8 +1069,13 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, return (phys_addr_t)DMA_MAPPING_ERROR; } - tlb_addr = swiotlb_fixed_map(dev, orig_addr, alloc_size, - alloc_align_mask, attrs); + tlb_addr = (phys_addr_t)DMA_MAPPING_ERROR; + if (!is_swiotlb_for_alloc(dev)) + tlb_addr = swiotlb_dyn_map(dev, orig_addr, alloc_size, + alloc_align_mask, dir, attrs); + if (tlb_addr == (phys_addr_t)DMA_MAPPING_ERROR) + tlb_addr = swiotlb_fixed_map(dev, orig_addr, alloc_size, + alloc_align_mask, attrs); if (tlb_addr == (phys_addr_t)DMA_MAPPING_ERROR) { if (!(attrs & DMA_ATTR_NO_WARN)) @@ -924,7 +1157,10 @@ void swiotlb_tbl_unmap_single(struct device *dev, phys_addr_t tlb_addr, (dir == DMA_FROM_DEVICE || dir == DMA_BIDIRECTIONAL)) swiotlb_bounce(dev, tlb_addr, mapping_size, DMA_FROM_DEVICE); - swiotlb_release_slots(dev, tlb_addr); + if (is_swiotlb_fixed(dev->dma_io_tlb_mem, tlb_addr)) + swiotlb_release_slots(dev, tlb_addr); + else + swiotlb_dyn_unmap(dev, tlb_addr, dir); } void swiotlb_sync_single_for_device(struct device *dev, phys_addr_t tlb_addr, @@ -1055,7 +1291,7 @@ bool swiotlb_free(struct device *dev, struct page *page, size_t size) { phys_addr_t tlb_addr = page_to_phys(page); - if (!is_swiotlb_buffer(dev, tlb_addr)) + if (!is_swiotlb_fixed(dev->dma_io_tlb_mem, tlb_addr)) return false; swiotlb_release_slots(dev, tlb_addr); From patchwork Wed Apr 19 10:03:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Tesarik X-Patchwork-Id: 85300 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp267271vqo; Wed, 19 Apr 2023 03:10:24 -0700 (PDT) X-Google-Smtp-Source: AKy350ZuDpzceU3pzkLM/mxWN8KhEp8ibqitr1sMOcVlK7NFtPtOiBQLIaUiXY5Z27r1MuZhUQQ6 X-Received: by 2002:a05:6a00:9a1:b0:63d:3765:dc8e with SMTP id u33-20020a056a0009a100b0063d3765dc8emr3946923pfg.32.1681899024308; Wed, 19 Apr 2023 03:10:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681899024; cv=none; d=google.com; s=arc-20160816; b=bCyo7jUVxkLnnZKygh6BFdXMFHBbuTBenQL7GIh6c6OBhxzL6Wwl5v8M9FbAUVoaSr cqLClAlB3UhyN0UQlqGhDWiUeM04IUM5VKXSgRM46yyeCo1qzEClNMn39CA3VPz+McVl /9Eq0KX6NiNc6bIDzFO4gzxBjxbqjT18tfN/TNmVi4cod8FPMVGk1RZHvUSG6hEq0ZFb 1JNciHIvj4YeUviM4wdpIKqU91kchNM9t7+3Q0My99+3oRj94nXOitMNZ6/iIotMi/WS ee21/4Tx+WLTi3DT+aCjoL+ke8uogU5VhIAqKfP588dtxjTMqeuo2rFIiva05yImNAao AK4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=fLV3g8ms2uLbHinc5MklYijOORZFhxBs3iA23C1yRTo=; b=jlAp5IRUn5rfHyKukfFHzhYGLnvHXmCyYt8uvwEXYjnE1K2XymJRXsQpk4hR9tl8MX UWR51FXaURekCFH7rBhquKrsVskfNpZVQzXEFWqbYO2YBVQceM7yQ6StPYim1V4hMVYc TSzSZ657e66exlcSJwdsznTDM4rIULabE4OUu7rkPxgXWXVbaKT5lcAimhki9Ec7fpnM M9nFR7TcPet26cUFnUQb6l6mkcedJZg4kyv4NO8wj9f/ZE6GfLh3UC6UZjmySpn0pjo+ MebZl4Ke4HLwnqQ0bHKqqiuC6E37nrRFQTzmUf64skDa5I956WfjX9leXrIT2JQeHYfj HzIQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g1-20020aa79f01000000b0063d334beed6si4563004pfr.341.2023.04.19.03.10.08; Wed, 19 Apr 2023 03:10:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230026AbjDSKHW (ORCPT + 99 others); Wed, 19 Apr 2023 06:07:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60418 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232988AbjDSKG7 (ORCPT ); Wed, 19 Apr 2023 06:06:59 -0400 Received: from frasgout11.his.huawei.com (frasgout11.his.huawei.com [14.137.139.23]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0B9614445; Wed, 19 Apr 2023 03:06:48 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.18.147.228]) by frasgout11.his.huawei.com (SkyGuard) with ESMTP id 4Q1bmp6WsQz9v7Hh; Wed, 19 Apr 2023 17:57:14 +0800 (CST) Received: from A2101119013HW2.china.huawei.com (unknown [10.45.152.239]) by APP1 (Coremail) with SMTP id LxC2BwAXi_aYvD9k7TY6Ag--.1944S7; Wed, 19 Apr 2023 11:06:04 +0100 (CET) From: Petr Tesarik To: Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Zhen Lei , Randy Dunlap , Damien Le Moal , Kim Phillips , "Steven Rostedt (Google)" , Muchun Song , Ondrej Zary , "Jason A. Donenfeld" , Petr Tesarik , Hans de Goede , Dan Williams , Andy Shevchenko , Kees Cook , Thomas Gleixner , Won Chung , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), dri-devel@lists.freedesktop.org (open list:DRM DRIVERS), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Kefeng Wang , petr@tesarici.cz Subject: [PATCH v2 5/7] swiotlb: Add a boot option to enable dynamic bounce buffers Date: Wed, 19 Apr 2023 12:03:57 +0200 Message-Id: <05e2e9f6bcde2a184844206f4d586328c967652e.1681898595.git.petr.tesarik.ext@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-CM-TRANSID: LxC2BwAXi_aYvD9k7TY6Ag--.1944S7 X-Coremail-Antispam: 1UD129KBjvJXoW3Gw1kXFWrJw13ArWrGry8Xwb_yoW7Ww17pr W09a4fKFs7JF18Z34DCa17GF1Fka1I9ay3JFWrWryFyr98WrnYqwnrKr4YqF1Fy3y0vF47 ZFy5ZF4Ykr17t3DanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmEb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWUCVW8JwA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv6xkF7I0E 14v26r4UJVWxJr1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrV C2j2WlYx0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE 7xkEbVWUJVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCY1x0262 kKe7AKxVWUtVW8ZwCY1x0264kExVAvwVAq07x20xyl42xK82IYc2Ij64vIr41l4I8I3I0E 4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGV WUWwC2zVAF1VAY17CE14v26rWY6r4UJwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v2 6r1I6r4UMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UMIIF0xvE42xK8VAvwI8IcI k0rVWUJVWUCwCI42IY6I8E87Iv67AKxVW8JVWxJwCI42IY6I8E87Iv6xkF7I0E14v26r4U JVWxJrUvcSsGvfC2KfnxnUUI43ZEXa7IU8yxR3UUUUU== X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763598951348942624?= X-GMAIL-MSGID: =?utf-8?q?1763598951348942624?= From: Petr Tesarik The main goal of allocating bounce buffers dynamically is to allow allocating a minimal fixed swiotlb at boot time but avoid hard limits on the amount of I/O that can be handled later. Compared to fixed IO TLB slots, dynamic allocation of bounce buffers typically increases the worst-case I/O latency and may also reduce performance for some workloads. I did some basic testing with fio against a QEMU SATA drive backed by a RAM block device in the host to minimize external factors. The kernel was booted with "swiotlb=force,dynamic". I performed testing of single-threaded I/O of 4-KiB segments, single-threaded I/O of 1-MiB segments, and 4-core parallel I/O of 64-KiB segments. The last column is the coefficient of variance in 5 runs of the test: Read Write Coeff single 4-KiB +1.9% +1.9% 1.7% single 1-MiB -8.1% -8.2% 2.2% parallel -9.4% -9.5% 2.6% There is a slight increase in bandwidth for single-threaded 4-KiB segments. This is because the buddy allocator is quite efficient for order-0 allocations, so the overhead is offset by faster allocation from an almost empty fixed swiotlb (which is still used for buffers smaller than one page). Anyway, since the feature is new and does not benefit all workloads, make it disabled by default and let people turn it on with "swiotlb=dynamic" if needed. Since this option can be combined with "force", the parser is modified to allow multiple options separated by commas. A new bool field is added to struct io_tlb_mem to tell whether dynamic allocations are allowed. This field is always false for DMA restricted pools. It is also false for other software IO TLBs unless "swiotlb=dynamic" was specified. Signed-off-by: Petr Tesarik --- .../admin-guide/kernel-parameters.txt | 6 +++++- include/linux/swiotlb.h | 3 ++- kernel/dma/swiotlb.c | 20 ++++++++++++++----- 3 files changed, 22 insertions(+), 7 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 6221a1d057dd..c8bc0c8b8df6 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -6110,14 +6110,18 @@ Execution Facility on pSeries. swiotlb= [ARM,IA-64,PPC,MIPS,X86] - Format: { [,] | force | noforce } + Format: { [,] [,option-list] | option-list } -- Number of I/O TLB slabs -- Second integer after comma. Number of swiotlb areas with their own lock. Will be rounded up to a power of 2. + -- Comma-separated list of options. + + Available options: force -- force using of bounce buffers even if they wouldn't be automatically used by the kernel noforce -- Never use bounce buffers (for debugging) + dynamic -- allow dynamic allocation of bounce buffers switches= [HW,M68k] diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index 0856eddb9063..e614aa0f4f64 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -98,6 +98,7 @@ struct io_tlb_mem { bool late_alloc; bool force_bounce; bool for_alloc; + bool allow_dyn; unsigned int nareas; unsigned int area_nslabs; struct io_tlb_area *areas; @@ -142,7 +143,7 @@ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr) return mem && (is_swiotlb_fixed(mem, paddr) || - is_swiotlb_dyn(dev, paddr)); + (mem->allow_dyn && is_swiotlb_dyn(dev, paddr))); } static inline bool is_swiotlb_force_bounce(struct device *dev) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index f4faee38ead9..4899fb0e4331 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -86,6 +86,7 @@ struct io_tlb_dyn_slot { static bool swiotlb_force_bounce; static bool swiotlb_force_disable; +static bool swiotlb_dynamic; struct io_tlb_mem io_tlb_default_mem; @@ -167,10 +168,18 @@ setup_io_tlb_npages(char *str) swiotlb_adjust_nareas(simple_strtoul(str, &str, 0)); if (*str == ',') ++str; - if (!strcmp(str, "force")) - swiotlb_force_bounce = true; - else if (!strcmp(str, "noforce")) - swiotlb_force_disable = true; + while (str && *str) { + char *opt = strsep(&str, ","); + + if (!strcmp(opt, "force")) + swiotlb_force_bounce = true; + else if (!strcmp(opt, "noforce")) + swiotlb_force_disable = true; + else if (!strcmp(opt, "dynamic")) + swiotlb_dynamic = true; + else + pr_warn("Invalid swiotlb option: %s", opt); + } return 0; } @@ -287,6 +296,7 @@ static void swiotlb_init_io_tlb_mem(struct io_tlb_mem *mem, phys_addr_t start, mem->area_nslabs = nslabs / mem->nareas; mem->force_bounce = swiotlb_force_bounce || (flags & SWIOTLB_FORCE); + mem->allow_dyn = swiotlb_dynamic; for (i = 0; i < mem->nareas; i++) { spin_lock_init(&mem->areas[i].lock); @@ -1070,7 +1080,7 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, } tlb_addr = (phys_addr_t)DMA_MAPPING_ERROR; - if (!is_swiotlb_for_alloc(dev)) + if (mem->allow_dyn) tlb_addr = swiotlb_dyn_map(dev, orig_addr, alloc_size, alloc_align_mask, dir, attrs); if (tlb_addr == (phys_addr_t)DMA_MAPPING_ERROR) From patchwork Wed Apr 19 10:03:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Tesarik X-Patchwork-Id: 85306 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp277185vqo; Wed, 19 Apr 2023 03:30:15 -0700 (PDT) X-Google-Smtp-Source: AKy350ZLmO8IA02eciR8mTt4JDn2NNd2CCLJDwqQ7CKWv0shK4pgEnHn/8vGE/vcUC+EfJiR1JGE X-Received: by 2002:a05:6a20:a110:b0:f0:b6e3:90c9 with SMTP id q16-20020a056a20a11000b000f0b6e390c9mr3854620pzk.4.1681900215728; Wed, 19 Apr 2023 03:30:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681900215; cv=none; d=google.com; s=arc-20160816; b=gmPnd7/Hg7mCN0ZkGYWLW1n5REUs2OjzmeuqAt+FJ0Ds9BxUlX05v5aJ25SzcpuNFo G26mhe/ky1CKv6AapPA5fFp0k3eI82KNO/bPew5YmXWGzGgBCV6GggPxCatlKMABOFNL csVt8H13NaCW4e2/fhyD2qRJjv09V1sWoDmodaiXFP/ugiM4dFWQ1hSdk2TVdxR9SEah q8P63gmKfXFOJVMcCvBvM9fg6yAlH6yPbpmFn/s0XkTMQrTTGTYG/mvvnnR0rwlEc6kP zf9MDJa6nNcnVSCMIzA800Fj4FyZLuZ97QIiNaOGfp564hLwnwnXzRr7quaIA/m+lASC 2lOA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=dsfei6RIDqaRKOiTQmd6lsGkK/Y7dA2m2bM55k3pgpo=; b=jNderZTTUajai542fUk9gx+gmj1YsPSZkyApMNCxj9/MnLaj3JceibPRqn74lVUcBd +EQgir0EbmmQWbTR9qEPq6J9KxB7RZ6AR3cgVOp7DCBxK+v1oygznFMAoaIFgJm6xkFi SCTUr6U3u9mgU+sXUNtTQSw1Oszep2F1M9mAZaB0MUDiTw7nA48+RQ51Z16MxyXPWJR5 ffp7mA3lDD0V7tasHC8m/7K2QYkF+fVYU686ljdzT/UfMQ2cIrEPqkSMDsON/higdVIV n+/NTL7dlrVYzh4JfhgxIIs5nhcQCppIuyMmDnZIgySoTukjJEl+l5wDXrBSkHVUiniQ PaYQ== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id r10-20020a638f4a000000b0051ba204e005si8578741pgn.199.2023.04.19.03.30.03; Wed, 19 Apr 2023 03:30:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233073AbjDSKHr (ORCPT + 99 others); Wed, 19 Apr 2023 06:07:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233049AbjDSKHN (ORCPT ); Wed, 19 Apr 2023 06:07:13 -0400 Received: from frasgout12.his.huawei.com (frasgout12.his.huawei.com [14.137.139.154]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C1BDB1445D; Wed, 19 Apr 2023 03:07:08 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.18.147.228]) by frasgout12.his.huawei.com (SkyGuard) with ESMTP id 4Q1bm5169Xz9xFm3; Wed, 19 Apr 2023 17:56:37 +0800 (CST) Received: from A2101119013HW2.china.huawei.com (unknown [10.45.152.239]) by APP1 (Coremail) with SMTP id LxC2BwAXi_aYvD9k7TY6Ag--.1944S8; Wed, 19 Apr 2023 11:06:23 +0100 (CET) From: Petr Tesarik To: Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Zhen Lei , Randy Dunlap , Damien Le Moal , Kim Phillips , "Steven Rostedt (Google)" , Muchun Song , Ondrej Zary , "Jason A. Donenfeld" , Petr Tesarik , Hans de Goede , Dan Williams , Andy Shevchenko , Kees Cook , Thomas Gleixner , Won Chung , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), dri-devel@lists.freedesktop.org (open list:DRM DRIVERS), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Kefeng Wang , petr@tesarici.cz Subject: [PATCH v2 6/7] drm: Use DMA_ATTR_MAY_SLEEP from process context Date: Wed, 19 Apr 2023 12:03:58 +0200 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-CM-TRANSID: LxC2BwAXi_aYvD9k7TY6Ag--.1944S8 X-Coremail-Antispam: 1UD129KBjvJXoW7ZF4kXFWDJF1Duw1rGFWrGrg_yoW8XFWkpF 4xAFyqvr40qFZYv39rZ3WkZasxua12qFWxua4UJws5u3WYyF9F9ryYyFW2vrWDAr97Xr4f tr9FyryUAry2kFJanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmEb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWUCVW8JwA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv6xkF7I0E 14v26r4UJVWxJr1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrV C2j2WlYx0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE 7xkEbVWUJVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCY1x0262 kKe7AKxVWUtVW8ZwCY1x0264kExVAvwVAq07x20xyl42xK82IYc2Ij64vIr41l4I8I3I0E 4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGV WUWwC2zVAF1VAY17CE14v26rWY6r4UJwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v2 6r1I6r4UMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UMIIF0xvE42xK8VAvwI8IcI k0rVWUJVWUCwCI42IY6I8E87Iv67AKxVW8JVWxJwCI42IY6I8E87Iv6xkF7I0E14v26r4U JVWxJrUvcSsGvfC2KfnxnUUI43ZEXa7IU8yxR3UUUUU== X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,RCVD_IN_MSPIKE_H2, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763600200263313626?= X-GMAIL-MSGID: =?utf-8?q?1763600200263313626?= From: Petr Tesarik These mappings are never done from atomic context. If a dynamically allocated bounce buffer is used for the mapping, this change allows to allocate from CMA. Signed-off-by: Petr Tesarik --- drivers/gpu/drm/drm_gem_shmem_helper.c | 2 +- drivers/gpu/drm/drm_prime.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c b/drivers/gpu/drm/drm_gem_shmem_helper.c index 2b2163c8138e..b5bb4f9c130a 100644 --- a/drivers/gpu/drm/drm_gem_shmem_helper.c +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c @@ -702,7 +702,7 @@ static struct sg_table *drm_gem_shmem_get_pages_sgt_locked(struct drm_gem_shmem_ goto err_put_pages; } /* Map the pages for use by the h/w. */ - ret = dma_map_sgtable(obj->dev->dev, sgt, DMA_BIDIRECTIONAL, 0); + ret = dma_map_sgtable(obj->dev->dev, sgt, DMA_BIDIRECTIONAL, DMA_ATTR_MAY_SLEEP); if (ret) goto err_free_sgt; diff --git a/drivers/gpu/drm/drm_prime.c b/drivers/gpu/drm/drm_prime.c index f924b8b4ab6b..f32e12445570 100644 --- a/drivers/gpu/drm/drm_prime.c +++ b/drivers/gpu/drm/drm_prime.c @@ -635,7 +635,7 @@ struct sg_table *drm_gem_map_dma_buf(struct dma_buf_attachment *attach, return sgt; ret = dma_map_sgtable(attach->dev, sgt, dir, - DMA_ATTR_SKIP_CPU_SYNC); + DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MAY_SLEEP); if (ret) { sg_free_table(sgt); kfree(sgt); From patchwork Wed Apr 19 10:03:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Tesarik X-Patchwork-Id: 85303 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp274368vqo; Wed, 19 Apr 2023 03:24:24 -0700 (PDT) X-Google-Smtp-Source: AKy350b6/QPvAsc7JhQrMpWQEqWhDGTM0DxM7sDuv53hT4c7Tf6mEM+A5J8dZLHWALP4ygGDLbAW X-Received: by 2002:a05:6a00:2305:b0:63d:3411:f9eb with SMTP id h5-20020a056a00230500b0063d3411f9ebmr3386418pfh.1.1681899864386; Wed, 19 Apr 2023 03:24:24 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681899864; cv=none; d=google.com; s=arc-20160816; b=z99a9fyFO8CeVw3XnMDzBSf4hcJPxswowd01FkorJejtQRI2D7mlfkzmfvEALsImdO gR81W/R7/tJu7yHNa5tdCNenyhLWFdkityR3hJejYMRgXEHx3vaqfIjG7Cad69OKH0YE FGT6j93KnLJA6cYUAPIlkTIn5mKUFo7+2WHUclQ6cdjgROKujnKB6xIj2zDVE8aWsBF9 nVqcy6X69Rf0Mgt4kPUxSQQrJF6BzWt5/NhmVp3R8ihFXH37uWlF7HT6lhFhkjNacZ0A kjZrzsMDCESYxh8otU4TYePADOKQr5NL7Egi5raAYcp8frmhaELuuG/oMaoMat7fJh61 SvRQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=U9mWzwFUnbp/5A+eD/y/ICit7IFdi18a7mivtaU7+IM=; b=uNaQ1Pss+hMPDzpaDQCyz6dB1nn9CGL5aMtHxiPqfUV/Dn+azxT5xGrF7tZ1zTfn2r zl/qPu4BiXEj0KHZxRKw0Unor504EDnC3Lw/bgPtSx96PXizi/wXlVwwoFlpT2Waj2qN GapBi3SGZmDTtMUswksviqPsfAwb9BxOLb7w/22ilpUO+O6LIahplCZpjLeYR2LkNqBj HZQVeuFo+ilArlkk+E8Ak4j3rhkPRhJeNxOYnG/JZTaiE7AnhslXeXfseyg+K62LhjHi kkIxwwVVUgYL5z+IfRusTq3Gi9P1D6a9n4TEV7UI0eFuPvyZ/zI8jgWSSnTq/0ZBk8jh Q87Q== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a25-20020aa795b9000000b0063d5fffd8b1si1033229pfk.228.2023.04.19.03.24.12; Wed, 19 Apr 2023 03:24:24 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233088AbjDSKIS (ORCPT + 99 others); Wed, 19 Apr 2023 06:08:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233065AbjDSKHe (ORCPT ); Wed, 19 Apr 2023 06:07:34 -0400 Received: from frasgout11.his.huawei.com (frasgout11.his.huawei.com [14.137.139.23]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27A5F14443; Wed, 19 Apr 2023 03:07:26 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.18.147.227]) by frasgout11.his.huawei.com (SkyGuard) with ESMTP id 4Q1bnX3fTjz9v7V0; Wed, 19 Apr 2023 17:57:52 +0800 (CST) Received: from A2101119013HW2.china.huawei.com (unknown [10.45.152.239]) by APP1 (Coremail) with SMTP id LxC2BwAXi_aYvD9k7TY6Ag--.1944S9; Wed, 19 Apr 2023 11:06:42 +0100 (CET) From: Petr Tesarik To: Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Zhen Lei , Randy Dunlap , Damien Le Moal , Kim Phillips , "Steven Rostedt (Google)" , Muchun Song , Ondrej Zary , "Jason A. Donenfeld" , Petr Tesarik , Hans de Goede , Dan Williams , Andy Shevchenko , Kees Cook , Thomas Gleixner , Won Chung , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), dri-devel@lists.freedesktop.org (open list:DRM DRIVERS), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Kefeng Wang , petr@tesarici.cz Subject: [PATCH v2 7/7] swiotlb: per-device flag if there are dynamically allocated buffers Date: Wed, 19 Apr 2023 12:03:59 +0200 Message-Id: <08ec2c2baa9a24d2eae5d5107868fd6df0f7dd1c.1681898595.git.petr.tesarik.ext@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-CM-TRANSID: LxC2BwAXi_aYvD9k7TY6Ag--.1944S9 X-Coremail-Antispam: 1UD129KBjvJXoWxWr1UWr47Jw1kZw4UKF48Zwb_yoW5uFW3pF y8uF98KF4qqFykZ3sF9w47uF47uw4q93y3CrWFgr1Fkry5X34rWF4kCry2y34rJr409F4x XryjvrWrAF17Ww7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmEb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUAV Cq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0 rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVW8JVW5JwA2z4x0Y4vE2Ix0cI8IcVCY1x0267 AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv6xkF7I0E 14v26r4UJVWxJr1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrV C2j2WlYx0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE 7xkEbVWUJVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCY1x0262 kKe7AKxVWUtVW8ZwCY1x0264kExVAvwVAq07x20xyl42xK82IYc2Ij64vIr41l4I8I3I0E 4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGV WUWwC2zVAF1VAY17CE14v26rWY6r4UJwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v2 6r1I6r4UMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UMIIF0xvE42xK8VAvwI8IcI k0rVWUJVWUCwCI42IY6I8E87Iv67AKxVW8JVWxJwCI42IY6I8E87Iv6xkF7I0E14v26r4U JVWxJrUvcSsGvfC2KfnxnUUI43ZEXa7IU8yxR3UUUUU== X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763599832306720935?= X-GMAIL-MSGID: =?utf-8?q?1763599832306720935?= From: Petr Tesarik Do not walk the list of dynamically allocated bounce buffers if the list is empty. This avoids taking dma_io_tlb_dyn_lock for devices which do not use any dynamically allocated bounce buffers. When unmapping the last dynamically allocated bounce buffer, the flag is set to false as soon as possible to allow skipping the spinlock even before the list itself is updated. Signed-off-by: Petr Tesarik --- include/linux/device.h | 4 ++++ include/linux/swiotlb.h | 6 +++++- kernel/dma/swiotlb.c | 6 ++++++ 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/include/linux/device.h b/include/linux/device.h index e12d6092bb9c..131b6db7fb3f 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -511,6 +511,9 @@ struct device_physical_location { * @dma_io_tlb_dyn_slots: * Dynamically allocated bounce buffers for this device. * Not for driver use. + * @dma_io_tlb_have_dyn: + * Does this device have any dynamically allocated bounce + * buffers? Not for driver use. * @archdata: For arch-specific additions. * @of_node: Associated device tree node. * @fwnode: Associated device node supplied by platform firmware. @@ -618,6 +621,7 @@ struct device { struct io_tlb_mem *dma_io_tlb_mem; spinlock_t dma_io_tlb_dyn_lock; struct list_head dma_io_tlb_dyn_slots; + bool dma_io_tlb_have_dyn; #endif /* arch specific additions */ struct dev_archdata archdata; diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index e614aa0f4f64..9ca4812b5977 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -143,7 +143,11 @@ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr) return mem && (is_swiotlb_fixed(mem, paddr) || - (mem->allow_dyn && is_swiotlb_dyn(dev, paddr))); + /* Pairs with smp_store_release() in swiotlb_dyn_map() + * and swiotlb_dyn_unmap(). + */ + (smp_load_acquire(&dev->dma_io_tlb_have_dyn) && + is_swiotlb_dyn(dev, paddr))); } static inline bool is_swiotlb_force_bounce(struct device *dev) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 4899fb0e4331..9b4faed7ef8f 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -685,6 +685,9 @@ static phys_addr_t swiotlb_dyn_map(struct device *dev, phys_addr_t orig_addr, spin_lock_irqsave(&dev->dma_io_tlb_dyn_lock, flags); list_add(&slot->node, &dev->dma_io_tlb_dyn_slots); + if (!dev->dma_io_tlb_have_dyn) + /* Pairs with smp_load_acquire() in is_swiotlb_buffer() */ + smp_store_release(&dev->dma_io_tlb_have_dyn, true); spin_unlock_irqrestore(&dev->dma_io_tlb_dyn_lock, flags); return page_to_phys(slot->page); @@ -711,6 +714,9 @@ static void swiotlb_dyn_unmap(struct device *dev, phys_addr_t tlb_addr, unsigned long flags; spin_lock_irqsave(&dev->dma_io_tlb_dyn_lock, flags); + if (list_is_singular(&dev->dma_io_tlb_dyn_slots)) + /* Pairs with smp_load_acquire() in is_swiotlb_buffer() */ + smp_store_release(&dev->dma_io_tlb_have_dyn, false); slot = lookup_dyn_slot_locked(dev, tlb_addr); list_del(&slot->node); spin_unlock_irqrestore(&dev->dma_io_tlb_dyn_lock, flags);