From patchwork Wed Apr 19 10:03:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Tesarik X-Patchwork-Id: 85305 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp277048vqo; Wed, 19 Apr 2023 03:30:03 -0700 (PDT) X-Google-Smtp-Source: AKy350b0cF1tTHLGhUCgN1JGFwAwuyPOo5duGa7wnPUw2X9OaROrfXH0BOilq42psrEUhcSWftbj X-Received: by 2002:a17:90a:c095:b0:247:4e73:cbdd with SMTP id o21-20020a17090ac09500b002474e73cbddmr2253448pjs.9.1681900203621; Wed, 19 Apr 2023 03:30:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681900203; cv=none; d=google.com; s=arc-20160816; b=w6CBm2VTjfsy1hNkl2MENDEmXAaxCqPEZgmH6SL7DYf5yo+GcDgwgsPrC1fgFTzBsq ibOrtskRRT3PW/EbgNeFD3RBTiGugkLHZquwQJyXi5KRHG9TdgWYcw5WZ+j+wnMm83lM Bq2LqQvrmwbg1bcK0g1oaZonn5pPfPsABqdMhrARUwIaz41Gpx+4Vw6lCE2XzFJ79LPU O0T64OoZ/WOKTu/mf4VWtke6oJ7xfcv4wNwYZempcgLT4DUiooLKf55WqPYbuBGALVcr ttA4q1Wp8PFqb0t0PYohVMxO6ob8qBspFX4BBFA7gmU1xFr/nl5yCQuO9Y6hGSXCDQ5J /4dg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=c+ERw3mYXyNKyEeB/M13vMpvYtQsEtTJYF8qE8xZc6c=; b=der9m5bgVSW6cq695ZPQ4mOpZoZbxsidpqA03vI9TPGjcK1YAU0OKrDM5kEyNVoNi5 hMqEP6y5FLS4uzY3uqAmBDN6l/ZUlk5/Uzi0HNnroNX9zTNGyoS5wC7S3T9+cLldHqpq 06kiMDltAn6O4a4LC0jVjXdHvbTSdKGaaFJ3tB3vlD+xE2W7e0TrL7pQUkVodPbVdjFO qYXEoGdGVg2nQeyDfFTQ/KJR6TCyzfkuai22ZjsiiVwGwe4FUJYGnTWrqq1ufonWOO5F dyrzoPHeCW5BhmxIGA15sIwjEqZhC4Nbt477NsMa42CBi+9kJDzcTH1s8KqhLtxt0vkO 7MeA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id h1-20020a17090aa88100b0024730b93a6asi1472714pjq.147.2023.04.19.03.29.51; Wed, 19 Apr 2023 03:30:03 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232706AbjDSKF5 (ORCPT + 99 others); Wed, 19 Apr 2023 06:05:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59482 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232860AbjDSKFx (ORCPT ); Wed, 19 Apr 2023 06:05:53 -0400 Received: from frasgout11.his.huawei.com (frasgout11.his.huawei.com [14.137.139.23]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4ECE2106; Wed, 19 Apr 2023 03:05:51 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.18.147.227]) by frasgout11.his.huawei.com (SkyGuard) with ESMTP id 4Q1blj4lqNz9xqwp; Wed, 19 Apr 2023 17:56:17 +0800 (CST) Received: from A2101119013HW2.china.huawei.com (unknown [10.45.152.239]) by APP1 (Coremail) with SMTP id LxC2BwAXi_aYvD9k7TY6Ag--.1944S4; Wed, 19 Apr 2023 11:05:07 +0100 (CET) From: Petr Tesarik To: Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter , Christoph Hellwig , Marek Szyprowski , Robin Murphy , Borislav Petkov , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Zhen Lei , Randy Dunlap , Damien Le Moal , Kim Phillips , "Steven Rostedt (Google)" , Muchun Song , Ondrej Zary , "Jason A. Donenfeld" , Petr Tesarik , Hans de Goede , Dan Williams , Andy Shevchenko , Kees Cook , Thomas Gleixner , Won Chung , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), dri-devel@lists.freedesktop.org (open list:DRM DRIVERS), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Kefeng Wang , petr@tesarici.cz Subject: [PATCH v2 2/7] swiotlb: Move code around in preparation for dynamic bounce buffers Date: Wed, 19 Apr 2023 12:03:54 +0200 Message-Id: <2954d90769bb926271c22553ef57ec805fea7865.1681898595.git.petr.tesarik.ext@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-CM-TRANSID: LxC2BwAXi_aYvD9k7TY6Ag--.1944S4 X-Coremail-Antispam: 1UD129KBjvJXoW3JryUur45Aw47Kw4fKF4kZwb_yoW3uFW8pF 1ft3WrtFsxJF1xC397uw48JF1Fkw1kG343Caya9ryF9Fy3Xrn0vFZ8CrW5WayFvFWv9F4U Xr98uF4rKF47Jr7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUmGb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUXw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVWUJVWUCwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV W8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_Gr0_Cr1l84ACjcxK6I8E87Iv6xkF7I0E14v2 6r4UJVWxJr1le2I262IYc4CY6c8Ij28IcVAaY2xG8wAqx4xG64xvF2IEw4CE5I8CrVC2j2 WlYx0E2Ix0cI8IcVAFwI0_Jr0_Jr4lYx0Ex4A2jsIE14v26r1j6r4UMcvjeVCFs4IE7xkE bVWUJVW8JwACjcxG0xvY0x0EwIxGrwACI402YVCY1x02628vn2kIc2xKxwCY1x0262kKe7 AKxVWUtVW8ZwCY1x0264kExVAvwVAq07x20xyl42xK82IYc2Ij64vIr41l4I8I3I0E4IkC 6x0Yz7v_Jr0_Gr1lx2IqxVAqx4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWw C2zVAF1VAY17CE14v26rWY6r4UJwCIc40Y0x0EwIxGrwCI42IY6xIIjxv20xvE14v26r1j 6r1xMIIF0xvE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UMIIF0xvE42xK8VAvwI8IcIk0rV WUJVWUCwCI42IY6I8E87Iv67AKxVW8JVWxJwCI42IY6I8E87Iv6xkF7I0E14v26r4UJVWx JrUvcSsGvfC2KfnxnUUI43ZEXa7IU8go7tUUUUU== X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763600188160429972?= X-GMAIL-MSGID: =?utf-8?q?1763600188160429972?= From: Petr Tesarik To prepare for the introduction of dynamically allocated bounce buffers, separate out common code and code which handles non-dynamic (aka fixed) bounce buffers. No functional change, but this commit should make the addition of dynamic allocations easier to review. Signed-off-by: Petr Tesarik --- include/linux/swiotlb.h | 31 ++++++++++- kernel/dma/swiotlb.c | 110 +++++++++++++++++++++++++++++++++------- 2 files changed, 122 insertions(+), 19 deletions(-) diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index b65b7330f7e5..a0bb6b30b17c 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -105,11 +105,40 @@ struct io_tlb_mem { }; extern struct io_tlb_mem io_tlb_default_mem; +/** + * is_swiotlb_fixed() - check if a physical address belongs to a swiotlb slot + * @mem: relevant swiotlb pool + * @paddr: physical address within the DMA buffer + * + * Check if @paddr points into a fixed bounce buffer slot. + * This check should be as fast as possible. + * + * Return: + * * %true if @paddr points into a @mem fixed slot + * * %false otherwise + */ +static inline bool is_swiotlb_fixed(struct io_tlb_mem *mem, phys_addr_t paddr) +{ + return paddr >= mem->start && paddr < mem->end; +} + +/** + * is_swiotlb_buffer() - check if a physical address is allocated from the + * swiotlb pool + * @dev: device which has mapped the buffer + * @paddr: physical address within the DMA buffer + * + * Check if @paddr points into a bounce buffer. + * + * Return: + * * %true if @paddr points into a bounce buffer + * * %false otherwise + */ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr) { struct io_tlb_mem *mem = dev->dma_io_tlb_mem; - return mem && paddr >= mem->start && paddr < mem->end; + return mem && is_swiotlb_fixed(mem, paddr); } static inline bool is_swiotlb_force_bounce(struct device *dev) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index dac42a2ad588..2f09a4ed4215 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -78,6 +78,10 @@ phys_addr_t swiotlb_unencrypted_base; static unsigned long default_nslabs = IO_TLB_DEFAULT_SIZE >> IO_TLB_SHIFT; static unsigned long default_nareas; +static void swiotlb_copy(struct device *dev, phys_addr_t orig_addr, + unsigned char *vaddr, size_t size, size_t alloc_size, + unsigned int tlb_offset, enum dma_data_direction dir); + /** * struct io_tlb_area - IO TLB memory area descriptor * @@ -523,7 +527,6 @@ static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size int index = (tlb_addr - mem->start) >> IO_TLB_SHIFT; phys_addr_t orig_addr = mem->slots[index].orig_addr; size_t alloc_size = mem->slots[index].alloc_size; - unsigned long pfn = PFN_DOWN(orig_addr); unsigned char *vaddr = mem->vaddr + tlb_addr - mem->start; unsigned int tlb_offset, orig_addr_offset; @@ -540,6 +543,34 @@ static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size } tlb_offset -= orig_addr_offset; + swiotlb_copy(dev, orig_addr, vaddr, size, alloc_size, tlb_offset, dir); +} + +/** + * swiotlb_copy() - copy swiotlb buffer content, checking for overflows. + * @dev: device which has mapped the bounce buffer + * @orig_addr: physical address of the original buffer + * @vaddr: virtual address inside the bounce buffer + * @size: number of bytes to copy + * @alloc_size: total allocated size of the bounce buffer + * @tlb_offset: offset within the bounce buffer + * @dir: direction of the data transfer + * + * If @dir is %DMA_TO_DEVICE, copy data from the original buffer to the + * bounce buffer, otherwise copy from the bounce buffer to the original + * buffer. + * + * The original buffer may be in high memory; that's why @orig_addr is + * a physical address. Note that this is the address of the beginning + * of the bounce buffer. Copying starts at offset @tlb_offset. This is + * needed to check accesses beyond the allocated size. + */ +static void swiotlb_copy(struct device *dev, phys_addr_t orig_addr, + unsigned char *vaddr, size_t size, size_t alloc_size, + unsigned int tlb_offset, enum dma_data_direction dir) +{ + unsigned long pfn = PFN_DOWN(orig_addr); + if (tlb_offset > alloc_size) { dev_WARN_ONCE(dev, 1, "Buffer overflow detected. Allocation size: %zu. Mapping size: %zu+%u.\n", @@ -734,15 +765,65 @@ static unsigned long mem_used(struct io_tlb_mem *mem) return used; } +/** + * swiotlb_fixed_map() - allocate a bounce buffer from fixed slots + * @dev: device which maps the buffer + * @orig_addr: address of the original buffer + * @alloc_size: total size of the original buffer + * @alloc_align_mask: + * required physical alignment of the I/O buffer + * @attrs: optional DMA attributes for the map operation + * + * Search for a suitable slot or sequence of slots and initialize them + * for use with the original buffer. + * + * Return: Physical address of the bounce buffer, or %DMA_MAPPING_ERROR. + */ +static phys_addr_t swiotlb_fixed_map(struct device *dev, phys_addr_t orig_addr, + size_t alloc_size, unsigned int alloc_align_mask, + unsigned long attrs) +{ + struct io_tlb_mem *mem = dev->dma_io_tlb_mem; + unsigned int offset = swiotlb_align_offset(dev, orig_addr); + int index = swiotlb_find_slots(dev, orig_addr, + alloc_size + offset, alloc_align_mask); + unsigned int i; + + if (index == -1) + return (phys_addr_t)DMA_MAPPING_ERROR; + + /* + * Save away the mapping from the original address to the DMA address. + * This is needed when we sync the memory. Then we sync the buffer if + * needed. + */ + for (i = 0; i < nr_slots(alloc_size + offset); i++) + mem->slots[index + i].orig_addr = slot_addr(orig_addr, i); + return slot_addr(mem->start, index) + offset; +} + +/** + * swiotlb_tbl_map_single() - map DMA buffer to a bounce buffer + * @dev: device which maps the buffer + * @orig_addr: address of the original buffer + * @mapping_size: size of the original buffer to be synced now + * @alloc_size: total size of the original buffer + * @alloc_align_mask: + * required physical alignment of the I/O buffer + * @dir: direction of the data transfer + * @attrs: optional DMA attributes for the map operation + * + * Create a mapping of the DMA buffer into a bounce buffer and copy the + * original data. + * + * Return: Physical address of the bounce buffer, or %DMA_MAPPING_ERROR. + */ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, size_t mapping_size, size_t alloc_size, unsigned int alloc_align_mask, enum dma_data_direction dir, unsigned long attrs) { struct io_tlb_mem *mem = dev->dma_io_tlb_mem; - unsigned int offset = swiotlb_align_offset(dev, orig_addr); - unsigned int i; - int index; phys_addr_t tlb_addr; if (!mem || !mem->nslabs) { @@ -760,24 +841,17 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, return (phys_addr_t)DMA_MAPPING_ERROR; } - index = swiotlb_find_slots(dev, orig_addr, - alloc_size + offset, alloc_align_mask); - if (index == -1) { + tlb_addr = swiotlb_fixed_map(dev, orig_addr, alloc_size, + alloc_align_mask, attrs); + + if (tlb_addr == (phys_addr_t)DMA_MAPPING_ERROR) { if (!(attrs & DMA_ATTR_NO_WARN)) dev_warn_ratelimited(dev, - "swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slots)\n", - alloc_size, mem->nslabs, mem_used(mem)); - return (phys_addr_t)DMA_MAPPING_ERROR; + "swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slots)\n", + alloc_size, mem->nslabs, mem_used(mem)); + return tlb_addr; } - /* - * Save away the mapping from the original address to the DMA address. - * This is needed when we sync the memory. Then we sync the buffer if - * needed. - */ - for (i = 0; i < nr_slots(alloc_size + offset); i++) - mem->slots[index + i].orig_addr = slot_addr(orig_addr, i); - tlb_addr = slot_addr(mem->start, index) + offset; /* * When dir == DMA_FROM_DEVICE we could omit the copy from the orig * to the tlb buffer, if we knew for sure the device will