From patchwork Tue May 9 09:18:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Petr Tesarik X-Patchwork-Id: 91472 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2742450vqo; Tue, 9 May 2023 02:34:15 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6FCN/1i5XTgV+Q+qLMHENCoZg5h/+UmDlUhjDIMryMA+iryj2yohkD2pru/MP7FPhio5Ev X-Received: by 2002:a05:6a20:428e:b0:101:4d10:e406 with SMTP id o14-20020a056a20428e00b001014d10e406mr1899733pzj.16.1683624855619; Tue, 09 May 2023 02:34:15 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1683624855; cv=none; d=google.com; s=arc-20160816; b=iDbM07VpTFwozQkVCYvBYEo8c2iohEQTT5FhxqbWZ1mjgq0tSRthMAcgfBf8h95ra7 D0/+C9bUZ2RGOq+HhA1tgbNPwryRn6g/hbV3WhAVWESq5MWew81i43fXJzOsKJbSYanf wUeXN3e3Yi9GjH5qEhMnjFWCohmilfJX8JJvUnffSfv3UVXqvaQV1YSvkogFsKGGV2b0 CX8/XAwpNcDgjTTcyoPobIAfBZEGm4o1SugyItyQlnWrmY4NKrG1Kb6ZSyxX1k/Vakv7 iWkQ9fqwtsL/Obf51V789hBlaxPvNH2kxN3FpAn0wzKaXhAXxY/mXx7fJpdmwtadBLUA gH1w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from; bh=NXLVwqAaNVWHTnVTR4zVdvEZj3wGVRyaeyRmqxsVcbA=; b=uPxp4pCm4VX77Uzl/DL+ouPfCr2itBe/I3zR51xmBpBvqAEhoZIadg8bzZ+mBEERaK EXzfska5jMH2+c0iNNhQl5V8KclxEUkMlq6RTNyYeeRx3gYIecKJiePjHYyY8DzEthJ6 y+qcpQ8TmLVjP5z8uWictPZWP5LiuN8XSJy5zZdcZq9eWsEUqtSOqA8XMlJMoygzSNYC IsoSH2ESugW7+xPMIBtgWlkX8Z+/YoDt0NYpg9uCDDmoydJLGf+NkuycG8qcP5iGmqIx QIyqev/EdRuGeLLkm+jDSqHkl37KX+xzu5AuD9INJ55e6qBHIkPfU/BiV/FkfbDB64Y+ Tn3w== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id qe2-20020a17090b4f8200b0024e09da51e9si16121434pjb.58.2023.05.09.02.34.03; Tue, 09 May 2023 02:34:15 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234880AbjEIJUF (ORCPT + 99 others); Tue, 9 May 2023 05:20:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42042 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234687AbjEIJUD (ORCPT ); Tue, 9 May 2023 05:20:03 -0400 Received: from frasgout13.his.huawei.com (frasgout13.his.huawei.com [14.137.139.46]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E1D3DDDBD; Tue, 9 May 2023 02:19:51 -0700 (PDT) Received: from mail02.huawei.com (unknown [172.18.147.228]) by frasgout13.his.huawei.com (SkyGuard) with ESMTP id 4QFsn16k6nz9v7GY; Tue, 9 May 2023 17:09:57 +0800 (CST) Received: from A2101119013HW2.china.huawei.com (unknown [10.81.212.219]) by APP2 (Coremail) with SMTP id GxC2BwDHdz7lD1pksN2OAg--.5S4; Tue, 09 May 2023 10:19:13 +0100 (CET) From: Petr Tesarik To: Jonathan Corbet , Greg Kroah-Hartman , "Rafael J. Wysocki" , Maarten Lankhorst , Maxime Ripard , Thomas Zimmermann , David Airlie , Daniel Vetter , Christoph Hellwig , Marek Szyprowski , Robin Murphy , "Paul E. McKenney" , Borislav Petkov , Randy Dunlap , Catalin Marinas , Damien Le Moal , Kim Phillips , "Steven Rostedt (Google)" , Andy Shevchenko , Hans de Goede , Jason Gunthorpe , Kees Cook , Thomas Gleixner , linux-doc@vger.kernel.org (open list:DOCUMENTATION), linux-kernel@vger.kernel.org (open list), dri-devel@lists.freedesktop.org (open list:DRM DRIVERS), iommu@lists.linux.dev (open list:DMA MAPPING HELPERS) Cc: Roberto Sassu , Kefeng Wang , petr@tesarici.cz Subject: [PATCH v2 RESEND 2/7] swiotlb: Move code around in preparation for dynamic bounce buffers Date: Tue, 9 May 2023 11:18:14 +0200 Message-Id: <74fc11b6f9422a0d7688c68ed13ee35a0fe1ff13.1683623618.git.petr.tesarik.ext@huawei.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: MIME-Version: 1.0 X-CM-TRANSID: GxC2BwDHdz7lD1pksN2OAg--.5S4 X-Coremail-Antispam: 1UD129KBjvJXoW3JryUur45Aw47Kw4fKF4kZwb_yoW3uFWUpF 1rtF1rtFsxJF1xC397uw48JF1rCw1kCry3Caya9ryF9Fy3Xrn0vFs8CrW5WayFvFWv9F4U Xr98uF4fGa17Jr7anT9S1TB71UUUUUUqnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUPSb4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M28IrcIa0xkI8VA2jI8067AKxVWUXw A2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAvFVAK0II2c7xJM28CjxkF64kEwVA0rcxS w2x7M28EF7xvwVC0I7IYx2IY67AKxVWUJVWUCwA2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxV WxJVW8Jr1l84ACjcxK6I8E87Iv67AKxVW8JVWxJwA2z4x0Y4vEx4A2jsIEc7CjxVAFwI0_ Gr1j6F4UJwAS0I0E0xvYzxvE52x082IY62kv0487Mc02F40EFcxC0VAKzVAqx4xG6I80ew Av7VC0I7IYx2IY67AKxVWUJVWUGwAv7VC2z280aVAFwI0_Jr0_Gr1lOx8S6xCaFVCjc4AY 6r1j6r4UM4x0Y48IcxkI7VAKI48JM4IIrI8v6xkF7I0E8cxan2IY04v7MxkF7I0Ew4C26c xK6c8Ij28IcwCF04k20xvY0x0EwIxGrwCFx2IqxVCFs4IE7xkEbVWUJVW8JwC20s026c02 F40E14v26r1j6r18MI8I3I0E7480Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Wrv_Gr 1UMIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_Jr0_JF4lIxAIcVC0I7IYx2IY6xkF 7I0E14v26F4j6r4UJwCI42IY6xAIw20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI 0_Jr0_Gr1lIxAIcVC2z280aVCY1x0267AKxVW8JVW8JrUvcSsGvfC2KfnxnUUI43ZEXa7I U8qQ6JUUUUU== X-CM-SenderInfo: hshw23xhvd2x3n6k3tpzhluzxrxghudrp/ X-CFilter-Loop: Reflected X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_NONE, SPF_NONE,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765408616987970885?= X-GMAIL-MSGID: =?utf-8?q?1765408616987970885?= From: Petr Tesarik To prepare for the introduction of dynamically allocated bounce buffers, separate out common code and code which handles non-dynamic (aka fixed) bounce buffers. No functional change, but this commit should make the addition of dynamic allocations easier to review. Signed-off-by: Petr Tesarik --- include/linux/swiotlb.h | 31 ++++++++++- kernel/dma/swiotlb.c | 110 +++++++++++++++++++++++++++++++++------- 2 files changed, 122 insertions(+), 19 deletions(-) diff --git a/include/linux/swiotlb.h b/include/linux/swiotlb.h index d851dbce1143..281ecc6b9bcc 100644 --- a/include/linux/swiotlb.h +++ b/include/linux/swiotlb.h @@ -114,11 +114,40 @@ struct io_tlb_mem { }; extern struct io_tlb_mem io_tlb_default_mem; +/** + * is_swiotlb_fixed() - check if a physical address belongs to a swiotlb slot + * @mem: relevant swiotlb pool + * @paddr: physical address within the DMA buffer + * + * Check if @paddr points into a fixed bounce buffer slot. + * This check should be as fast as possible. + * + * Return: + * * %true if @paddr points into a @mem fixed slot + * * %false otherwise + */ +static inline bool is_swiotlb_fixed(struct io_tlb_mem *mem, phys_addr_t paddr) +{ + return paddr >= mem->start && paddr < mem->end; +} + +/** + * is_swiotlb_buffer() - check if a physical address is allocated from the + * swiotlb pool + * @dev: device which has mapped the buffer + * @paddr: physical address within the DMA buffer + * + * Check if @paddr points into a bounce buffer. + * + * Return: + * * %true if @paddr points into a bounce buffer + * * %false otherwise + */ static inline bool is_swiotlb_buffer(struct device *dev, phys_addr_t paddr) { struct io_tlb_mem *mem = dev->dma_io_tlb_mem; - return mem && paddr >= mem->start && paddr < mem->end; + return mem && is_swiotlb_fixed(mem, paddr); } static inline bool is_swiotlb_force_bounce(struct device *dev) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index af2e304c672c..96ba93be6772 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -76,6 +76,10 @@ struct io_tlb_mem io_tlb_default_mem; static unsigned long default_nslabs = IO_TLB_DEFAULT_SIZE >> IO_TLB_SHIFT; static unsigned long default_nareas; +static void swiotlb_copy(struct device *dev, phys_addr_t orig_addr, + unsigned char *vaddr, size_t size, size_t alloc_size, + unsigned int tlb_offset, enum dma_data_direction dir); + /** * struct io_tlb_area - IO TLB memory area descriptor * @@ -480,7 +484,6 @@ static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size int index = (tlb_addr - mem->start) >> IO_TLB_SHIFT; phys_addr_t orig_addr = mem->slots[index].orig_addr; size_t alloc_size = mem->slots[index].alloc_size; - unsigned long pfn = PFN_DOWN(orig_addr); unsigned char *vaddr = mem->vaddr + tlb_addr - mem->start; unsigned int tlb_offset, orig_addr_offset; @@ -497,6 +500,34 @@ static void swiotlb_bounce(struct device *dev, phys_addr_t tlb_addr, size_t size } tlb_offset -= orig_addr_offset; + swiotlb_copy(dev, orig_addr, vaddr, size, alloc_size, tlb_offset, dir); +} + +/** + * swiotlb_copy() - copy swiotlb buffer content, checking for overflows. + * @dev: device which has mapped the bounce buffer + * @orig_addr: physical address of the original buffer + * @vaddr: virtual address inside the bounce buffer + * @size: number of bytes to copy + * @alloc_size: total allocated size of the bounce buffer + * @tlb_offset: offset within the bounce buffer + * @dir: direction of the data transfer + * + * If @dir is %DMA_TO_DEVICE, copy data from the original buffer to the + * bounce buffer, otherwise copy from the bounce buffer to the original + * buffer. + * + * The original buffer may be in high memory; that's why @orig_addr is + * a physical address. Note that this is the address of the beginning + * of the bounce buffer. Copying starts at offset @tlb_offset. This is + * needed to check accesses beyond the allocated size. + */ +static void swiotlb_copy(struct device *dev, phys_addr_t orig_addr, + unsigned char *vaddr, size_t size, size_t alloc_size, + unsigned int tlb_offset, enum dma_data_direction dir) +{ + unsigned long pfn = PFN_DOWN(orig_addr); + if (tlb_offset > alloc_size) { dev_WARN_ONCE(dev, 1, "Buffer overflow detected. Allocation size: %zu. Mapping size: %zu+%u.\n", @@ -727,15 +758,65 @@ static unsigned long mem_used(struct io_tlb_mem *mem) return used; } +/** + * swiotlb_fixed_map() - allocate a bounce buffer from fixed slots + * @dev: device which maps the buffer + * @orig_addr: address of the original buffer + * @alloc_size: total size of the original buffer + * @alloc_align_mask: + * required physical alignment of the I/O buffer + * @attrs: optional DMA attributes for the map operation + * + * Search for a suitable slot or sequence of slots and initialize them + * for use with the original buffer. + * + * Return: Physical address of the bounce buffer, or %DMA_MAPPING_ERROR. + */ +static phys_addr_t swiotlb_fixed_map(struct device *dev, phys_addr_t orig_addr, + size_t alloc_size, unsigned int alloc_align_mask, + unsigned long attrs) +{ + struct io_tlb_mem *mem = dev->dma_io_tlb_mem; + unsigned int offset = swiotlb_align_offset(dev, orig_addr); + int index = swiotlb_find_slots(dev, orig_addr, + alloc_size + offset, alloc_align_mask); + unsigned int i; + + if (index == -1) + return (phys_addr_t)DMA_MAPPING_ERROR; + + /* + * Save away the mapping from the original address to the DMA address. + * This is needed when we sync the memory. Then we sync the buffer if + * needed. + */ + for (i = 0; i < nr_slots(alloc_size + offset); i++) + mem->slots[index + i].orig_addr = slot_addr(orig_addr, i); + return slot_addr(mem->start, index) + offset; +} + +/** + * swiotlb_tbl_map_single() - map DMA buffer to a bounce buffer + * @dev: device which maps the buffer + * @orig_addr: address of the original buffer + * @mapping_size: size of the original buffer to be synced now + * @alloc_size: total size of the original buffer + * @alloc_align_mask: + * required physical alignment of the I/O buffer + * @dir: direction of the data transfer + * @attrs: optional DMA attributes for the map operation + * + * Create a mapping of the DMA buffer into a bounce buffer and copy the + * original data. + * + * Return: Physical address of the bounce buffer, or %DMA_MAPPING_ERROR. + */ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, size_t mapping_size, size_t alloc_size, unsigned int alloc_align_mask, enum dma_data_direction dir, unsigned long attrs) { struct io_tlb_mem *mem = dev->dma_io_tlb_mem; - unsigned int offset = swiotlb_align_offset(dev, orig_addr); - unsigned int i; - int index; phys_addr_t tlb_addr; if (!mem || !mem->nslabs) { @@ -753,24 +834,17 @@ phys_addr_t swiotlb_tbl_map_single(struct device *dev, phys_addr_t orig_addr, return (phys_addr_t)DMA_MAPPING_ERROR; } - index = swiotlb_find_slots(dev, orig_addr, - alloc_size + offset, alloc_align_mask); - if (index == -1) { + tlb_addr = swiotlb_fixed_map(dev, orig_addr, alloc_size, + alloc_align_mask, attrs); + + if (tlb_addr == (phys_addr_t)DMA_MAPPING_ERROR) { if (!(attrs & DMA_ATTR_NO_WARN)) dev_warn_ratelimited(dev, - "swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slots)\n", - alloc_size, mem->nslabs, mem_used(mem)); - return (phys_addr_t)DMA_MAPPING_ERROR; + "swiotlb buffer is full (sz: %zd bytes), total %lu (slots), used %lu (slots)\n", + alloc_size, mem->nslabs, mem_used(mem)); + return tlb_addr; } - /* - * Save away the mapping from the original address to the DMA address. - * This is needed when we sync the memory. Then we sync the buffer if - * needed. - */ - for (i = 0; i < nr_slots(alloc_size + offset); i++) - mem->slots[index + i].orig_addr = slot_addr(orig_addr, i); - tlb_addr = slot_addr(mem->start, index) + offset; /* * When dir == DMA_FROM_DEVICE we could omit the copy from the orig * to the tlb buffer, if we knew for sure the device will