From patchwork Mon Nov 28 18:02:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 26908 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp5861220wrr; Mon, 28 Nov 2022 10:21:09 -0800 (PST) X-Google-Smtp-Source: AA0mqf7O9fTqwHg8vEim/7fmASU4q6FJaEAOpagVoQ+LhgeB1bSDItTvkqx4IOysn4jIqWmlbY32 X-Received: by 2002:a17:902:d545:b0:189:6ab3:9e5c with SMTP id z5-20020a170902d54500b001896ab39e5cmr16490150plf.5.1669659668875; Mon, 28 Nov 2022 10:21:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669659668; cv=none; d=google.com; s=arc-20160816; b=KkJEm8wHQoNp6zDvxWE/NR8LuQc4Ka4l48dFMVfquk8XrajrxdwyhF64LLo+Vx/sUh LlfSIrQUaiLBIx3Xd7tbxSBIfaYdQjomfoLh8uWTHe89mFwriUNhSxLEgeXIUDOcE2R8 p6mOukvmdPJLk8zlH75YY4XBHjq5L4tRNtN7XD4nw5uD5/iiCFFYKdwyZ+D0AzdxAVx2 lL4KREYSwHN3pCIwPIfLxqR3nmYgButl6JpzgOQV4eQtTKe+xxDiuwXAya2T3yEjfPul ZWDqldCiXdbFDlRUOF6UubqxJ4SPAPR+R2OMoLA1ZosmMSldC1HU3LOQO1ZfKB47l3oy N+iA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=/Fh+crFYr0bdZ0CEDCCytk9X8KD6hynMr1Yq9paj0G4=; b=BJLiWCNp/HXsDE+a+l386sFIXH8ox2/kAMU8ey6sc88kPJf4GxQyt7ronhL5fNGFvo tjXmadcFvFL0kPOpL6cqYbylPlFifPTz0uc/lUqXiEh0i4lBkIYTwDO2J5eM1Jzx7c5D 9DRMr1JvCmWI9n2ErnAmlVv0MKXPprtiqJmSZT6ffT7Na3xvVxCjtLCcjweY6UdBRccC NwAliMTacadOPQY4qekfR5rbhSlx0C3F/ER5A9iThw62XNxPt7R/it9hVg03ub+9l2UH x6qa2N7tf34J4KF9hlaNXDpEWd0zLl3ucr3XnU5yXYY9flkUXfTTxNyIkn52Po+1PQer L/8Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=f8Ck9XrD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n27-20020aa7985b000000b0056d789ba707si10798259pfq.294.2022.11.28.10.20.55; Mon, 28 Nov 2022 10:21:08 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=f8Ck9XrD; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234184AbiK1ST1 (ORCPT + 99 others); Mon, 28 Nov 2022 13:19:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234196AbiK1SSy (ORCPT ); Mon, 28 Nov 2022 13:18:54 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 493FC2AC40 for ; Mon, 28 Nov 2022 10:03:05 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id q7so17319153wrr.8 for ; Mon, 28 Nov 2022 10:03:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=/Fh+crFYr0bdZ0CEDCCytk9X8KD6hynMr1Yq9paj0G4=; b=f8Ck9XrDcaB/E4hGC+FYMlnArzd3Sq82oo1yzaQNVZIBLt+2yTGRY7PB7dZdnKgaE/ /5XnlfYGvyoKSXRlzluNuKc31hOtz6cGxMjxUGm+kXC8IjtorJ39gGjCjfnn0uJaOxj0 V//Q4KL6qSyDcXprZMrHn52tOkCDJS3LAjPJdLsOK94Pbbf1INGO83C6HitBcDp0Tyxl bADOVWR84N4yp17bCUCe7GkV5NBS79AZG4dr++FvDEAW4Iy4PTixlNmXLIS8jcqhjMRL fKJ8YOCkV6miNWCDVcXbxGZdoESFHA1SQMVr0g/xuXhsCioyVU0uuTrx+OR2YBCvBSlO EJ+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/Fh+crFYr0bdZ0CEDCCytk9X8KD6hynMr1Yq9paj0G4=; b=ZWqcday6svwN7VK0rmMv1TAROcyZGH7+zX18vsQQ9IEvFedLjLYrMyqdQzufJssbcv NjT5kHxrf5Old6LCPYxbmWK7cSNoUkXJK33uCGtodjMP0kwjCkHDJgLFl2W1NacqTssf eI0J9lbdjN8vDeeFdDP+crW05trpG4INDLOKGwuCgaecnyRlXFp9xi8GP9pcBQY7hMfU WdpavAsYkuHZbkOnxzbHtUcu1z4eKqelnKqaHpKvs8zslhsFsx3/cA2vb6mYhNe3xZPg WeoyQO2se8oe3S6pxh9/IHgmIEGWr36AH5pVZ7oSfkkdrQBwZjTYkPDMcjBxqdr5h0FG L2Dw== X-Gm-Message-State: ANoB5pkrx7VbBm/MeOUzuqgJQmsLfkxe1jQ6PDOOxhYKmaLr27c5jd8U p27ZA0Oivk5DWVztppTiqJyhXQ== X-Received: by 2002:adf:de0e:0:b0:241:6f01:ad47 with SMTP id b14-20020adfde0e000000b002416f01ad47mr24231647wrm.222.1669658583556; Mon, 28 Nov 2022 10:03:03 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:f4d1:b340:8675:e840]) by smtp.gmail.com with ESMTPSA id m18-20020adffa12000000b00241c6729c2bsm11350836wrr.26.2022.11.28.10.03.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Nov 2022 10:03:02 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 1/3] mm/khugepaged: Take the right locks for page table retraction Date: Mon, 28 Nov 2022 19:02:50 +0100 Message-Id: <20221128180252.1684965-1-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog MIME-Version: 1.0 X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750765056790736416?= X-GMAIL-MSGID: =?utf-8?q?1750765056790736416?= pagetable walks on address ranges mapped by VMAs can be done under the mmap lock, the lock of an anon_vma attached to the VMA, or the lock of the VMA's address_space. Only one of these needs to be held, and it does not need to be held in exclusive mode. Under those circumstances, the rules for concurrent access to page table entries are: - Terminal page table entries (entries that don't point to another page table) can be arbitrarily changed under the page table lock, with the exception that they always need to be consistent for hardware page table walks and lockless_pages_from_mm(). This includes that they can be changed into non-terminal entries. - Non-terminal page table entries (which point to another page table) can not be modified; readers are allowed to READ_ONCE() an entry, verify that it is non-terminal, and then assume that its value will stay as-is. Retracting a page table involves modifying a non-terminal entry, so page-table-level locks are insufficient to protect against concurrent page table traversal; it requires taking all the higher-level locks under which it is possible to start a page walk in the relevant range in exclusive mode. The collapse_huge_page() path for anonymous THP already follows this rule, but the shmem/file THP path was getting it wrong, making it possible for concurrent rmap-based operations to cause corruption. Cc: stable@kernel.org Fixes: 27e1f8273113 ("khugepaged: enable collapse pmd for pte-mapped THP") Acked-by: David Hildenbrand Signed-off-by: Jann Horn Reviewed-by: Yang Shi --- v4: added ack by David Hildenbrand mm/khugepaged.c | 55 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 4 deletions(-) base-commit: eb7081409f94a9a8608593d0fb63a1aa3d6f95d8 diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4734315f79407..674b111a24fa7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1384,16 +1384,37 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, return SCAN_SUCCEED; } +/* + * A note about locking: + * Trying to take the page table spinlocks would be useless here because those + * are only used to synchronize: + * + * - modifying terminal entries (ones that point to a data page, not to another + * page table) + * - installing *new* non-terminal entries + * + * Instead, we need roughly the same kind of protection as free_pgtables() or + * mm_take_all_locks() (but only for a single VMA): + * The mmap lock together with this VMA's rmap locks covers all paths towards + * the page table entries we're messing with here, except for hardware page + * table walks and lockless_pages_from_mm(). + */ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp) { - spinlock_t *ptl; pmd_t pmd; mmap_assert_write_locked(mm); - ptl = pmd_lock(vma->vm_mm, pmdp); + if (vma->vm_file) + lockdep_assert_held_write(&vma->vm_file->f_mapping->i_mmap_rwsem); + /* + * All anon_vmas attached to the VMA have the same root and are + * therefore locked by the same lock. + */ + if (vma->anon_vma) + lockdep_assert_held_write(&vma->anon_vma->root->rwsem); + pmd = pmdp_collapse_flush(vma, addr, pmdp); - spin_unlock(ptl); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); @@ -1444,6 +1465,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false)) return SCAN_VMA_CHECK; + /* + * Symmetry with retract_page_tables(): Exclude MAP_PRIVATE mappings + * that got written to. Without this, we'd have to also lock the + * anon_vma if one exists. + */ + if (vma->anon_vma) + return SCAN_VMA_CHECK; + /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ if (userfaultfd_wp(vma)) return SCAN_PTE_UFFD_WP; @@ -1477,6 +1506,20 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* + * We need to lock the mapping so that from here on, only GUP-fast and + * hardware page walks can access the parts of the page tables that + * we're operating on. + * See collapse_and_free_pmd(). + */ + i_mmap_lock_write(vma->vm_file->f_mapping); + + /* + * This spinlock should be unnecessary: Nobody else should be accessing + * the page tables under spinlock protection here, only + * lockless_pages_from_mm() and the hardware page walker can access page + * tables while all the high-level locks are held in write mode. + */ start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); result = SCAN_FAIL; @@ -1531,6 +1574,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove pte entries */ collapse_and_free_pmd(mm, vma, haddr, pmd); + i_mmap_unlock_write(vma->vm_file->f_mapping); + maybe_install_pmd: /* step 5: install pmd entry */ result = install_pmd @@ -1544,6 +1589,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, abort: pte_unmap_unlock(start_pte, ptl); + i_mmap_unlock_write(vma->vm_file->f_mapping); goto drop_hpage; } @@ -1600,7 +1646,8 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, * An alternative would be drop the check, but check that page * table is clear before calling pmdp_collapse_flush() under * ptl. It has higher chance to recover THP for the VMA, but - * has higher cost too. + * has higher cost too. It would also probably require locking + * the anon_vma. */ if (vma->anon_vma) { result = SCAN_PAGE_ANON; From patchwork Mon Nov 28 18:02:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 26909 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp5861363wrr; Mon, 28 Nov 2022 10:21:23 -0800 (PST) X-Google-Smtp-Source: AA0mqf4zWB9DaVyo1tVdplt1e9rMLmhcNAU1Z9vokxa1ldl91F2gszc6oe7soOBgN/yUF9f6dOBm X-Received: by 2002:a17:90a:8592:b0:219:2e8c:a775 with SMTP id m18-20020a17090a859200b002192e8ca775mr8279182pjn.102.1669659683243; Mon, 28 Nov 2022 10:21:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669659683; cv=none; d=google.com; s=arc-20160816; b=zg+pJ/eLll9zf/N2liUZgGl4FVGFjgcwOe6XrJq/uea5ZHqMXBFaXpnFUioPEoU0S9 VC3gOPJhiua84aJYueBzd/myd1qX/qnbCeolwYhrw8iAGP3AxbdqNw3ohsFxaNJtmrJZ 0SpPur7GHfK/w4l8FsRAe92oW1VsSJKmyBbf0owOB5CN8O5H+Q+ZHEQpipSxTZDQg9UX y78Dx1spwIXrtr8ru42L8iJFMxDjcQ/qEBZF3GgbnQl/B3YToKsQz4NiRHmNddmRdJO5 OUwQECOQo3tFNbQDGjwMREcHCYvEodLELDnL0uRKdNsjjg7R/ellHsyj2KhC1SXRNHCu AH2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=RUow5Vogs/7PnC9nrxFlLM/L16QCcApD5uJXFy4USuY=; b=XpIkZUJ7I2SekNsmiutO/QTbhlG0RyVWdYwpdldxRL+pCeev5j3F1Gjy3rC/En3RvO 5I58vnEHtrkK+9vLlCOTbta6S7zWBHqyGEfnsLWW1zPGwCXEpPSclgj7lSsTk7yJ3mNk M5cbqeIvPfcKQTY9Dh/EL4ssgI8jHAplYmF5paudbzsiavyhmyHdHaL+hXW1G/VJOBLV DQINwrFwsLhJPGWvO3DLyyVIUWdefLQWcSkafD76q5j7W7fEt1NkPn3JmJ9h5z0pQuov 7KZe26hMUg+gSzdpEgEaN5GDE11Nm4w1L8QxNZWoJDE1sAe/jAIo7TSu5hwcuzJnJFWV nV8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="rU/5WuLN"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k16-20020a635a50000000b0046ec7b397e1si13309569pgm.759.2022.11.28.10.21.07; Mon, 28 Nov 2022 10:21:23 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="rU/5WuLN"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232008AbiK1STc (ORCPT + 99 others); Mon, 28 Nov 2022 13:19:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53384 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234199AbiK1SSy (ORCPT ); Mon, 28 Nov 2022 13:18:54 -0500 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B84212A70E for ; Mon, 28 Nov 2022 10:03:08 -0800 (PST) Received: by mail-wm1-x333.google.com with SMTP id t1so9112431wmi.4 for ; Mon, 28 Nov 2022 10:03:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RUow5Vogs/7PnC9nrxFlLM/L16QCcApD5uJXFy4USuY=; b=rU/5WuLN/iM6g1+U2lBomEDv/g6EreorKfHwKtqRkF8SblU0VsIoZV8lMj7la44xOo RFhODdlKSvqR653yK+HRv3a22IBJ7tLBK5Z6ja7TLaM28AE7gzd8hZGHoFRg1XxIlqjJ WzZLuZdJo6sJsyS+ZZSUvjrHnUPfa6Ac26czHLanj7NP3nju9ONX3HUkVJHKHv9yZ1BF xmWKi8CRGLpW4bKOJghlq05QZvrVOUN3iDX+uFzjp2XCnvE1dkTCbY2bZ7loNCgKJ9US 4XoznfiWY3eu0D8pWYcvLjab/VADOYaVOhIRlWOyHDbaAxmQm1iUD8calDDGZ6mT7Dcv kYVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RUow5Vogs/7PnC9nrxFlLM/L16QCcApD5uJXFy4USuY=; b=7pevsiCixJcaiiIkg6tv6sTqqPS+WiB1erRUEPNLltroUDhDRcC9dtcKwqMSlmyVmO PYrzg3v1R+4x68B4FEARKrS73taHPkaFibMIhtjZHaEjTEmmn8C90SMruv3iQpDK2n88 O4ifi1CsOoO/o44jdMNaRtv65FqMq1wMe4tBigMegyDvQkRylBG7G5kz06Detr+Si28e hzaAf9oUt6z4zWaLJj5YyGEH+ks4QXORBv2u+KdW9pJyhkbj8HNp5sNJZ9b6XPmilOQT L3wL48+hpYRgiGPGNH5+lNJfkNDpGEX4SQ1/fvfwIFZmOz+k360twTZCfJECFIwLAWqc Da9w== X-Gm-Message-State: ANoB5plUJ6uJkSjTJsN3y+BjSonYfOYQH23WMKnnPQaK6IEo5hbVIMvC mnkwk0G9AXhDJwEnlXkdnD7FTA== X-Received: by 2002:a7b:cd18:0:b0:3cf:a359:de90 with SMTP id f24-20020a7bcd18000000b003cfa359de90mr28732913wmj.122.1669658587158; Mon, 28 Nov 2022 10:03:07 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:f4d1:b340:8675:e840]) by smtp.gmail.com with ESMTPSA id l11-20020adff48b000000b002420dba6447sm6628727wro.59.2022.11.28.10.03.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Nov 2022 10:03:05 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 2/3] mm/khugepaged: Fix GUP-fast interaction by sending IPI Date: Mon, 28 Nov 2022 19:02:51 +0100 Message-Id: <20221128180252.1684965-2-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog In-Reply-To: <20221128180252.1684965-1-jannh@google.com> References: <20221128180252.1684965-1-jannh@google.com> MIME-Version: 1.0 X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750765072056227213?= X-GMAIL-MSGID: =?utf-8?q?1750765072056227213?= Since commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THP collapse"), the lockless_pages_from_mm() fastpath rechecks the pmd_t to ensure that the page table was not removed by khugepaged in between. However, lockless_pages_from_mm() still requires that the page table is not concurrently freed or reused to store non-PTE data. Otherwise, problems can occur because: - deposited page tables can be freed when a THP page somewhere in the mm is removed - some architectures store non-PTE information inside deposited page tables (see radix__pgtable_trans_huge_deposit()) Additionally, lockless_pages_from_mm() is also somewhat brittle with regards to page tables being repeatedly moved back and forth, but that shouldn't be an issue in practice. Fix it by sending IPIs (if the architecture uses semi-RCU-style page table freeing) before freeing/reusing page tables. As noted in mm/gup.c, on configs that define CONFIG_HAVE_FAST_GUP, there are two possible cases: 1. CONFIG_MMU_GATHER_RCU_TABLE_FREE is set, causing tlb_remove_table_sync_one() to send an IPI to synchronize with lockless_pages_from_mm(). 2. CONFIG_MMU_GATHER_RCU_TABLE_FREE is unset, indicating that all TLB flushes are already guaranteed to send IPIs. tlb_remove_table_sync_one() will do nothing, but we've already run pmdp_collapse_flush(), which did a TLB flush, which must have involved IPIs. Cc: stable@kernel.org Fixes: ba76149f47d8 ("thp: khugepaged") Acked-by: David Hildenbrand Signed-off-by: Jann Horn Reviewed-by: Yang Shi --- v4: - added ack from David Hildenbrand - made commit message more verbose include/asm-generic/tlb.h | 4 ++++ mm/khugepaged.c | 2 ++ mm/mmu_gather.c | 4 +--- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 492dce43236ea..cab7cfebf40bd 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -222,12 +222,16 @@ extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #define tlb_needs_table_invalidate() (true) #endif +void tlb_remove_table_sync_one(void); + #else #ifdef tlb_needs_table_invalidate #error tlb_needs_table_invalidate() requires MMU_GATHER_RCU_TABLE_FREE #endif +static inline void tlb_remove_table_sync_one(void) { } + #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 674b111a24fa7..c3d3ce596bff7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1057,6 +1057,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl); mmu_notifier_invalidate_range_end(&range); + tlb_remove_table_sync_one(); spin_lock(pte_ptl); result = __collapse_huge_page_isolate(vma, address, pte, cc, @@ -1415,6 +1416,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v lockdep_assert_held_write(&vma->anon_vma->root->rwsem); pmd = pmdp_collapse_flush(vma, addr, pmdp); + tlb_remove_table_sync_one(); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index add4244e5790d..3a2c3f8cad2fe 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -153,7 +153,7 @@ static void tlb_remove_table_smp_sync(void *arg) /* Simply deliver the interrupt */ } -static void tlb_remove_table_sync_one(void) +void tlb_remove_table_sync_one(void) { /* * This isn't an RCU grace period and hence the page-tables cannot be @@ -177,8 +177,6 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ -static void tlb_remove_table_sync_one(void) { } - static void tlb_remove_table_free(struct mmu_table_batch *batch) { __tlb_remove_table_free(batch); From patchwork Mon Nov 28 18:02:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 26910 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp5861467wrr; Mon, 28 Nov 2022 10:21:31 -0800 (PST) X-Google-Smtp-Source: AA0mqf7RzwoR6QEIhlKpPI+C5E8jKnIFoEsOGikX7Ke91IEHCaMX4QINB5hZEZ4zrFK10WJ3Dw9G X-Received: by 2002:a17:902:7b84:b0:189:6623:61c9 with SMTP id w4-20020a1709027b8400b00189662361c9mr18413138pll.121.1669659691706; Mon, 28 Nov 2022 10:21:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669659691; cv=none; d=google.com; s=arc-20160816; b=cDYqW6aeDU1tCS/dgkGYV9xyDjnFldVdF8guhbfQ/V6M22kNkTdMssW1U8OOpC4LRc sIeWte67G9yLzgKTmklWq7aLbhrhPaqi6e+wyMDOpIkqZEJ4NyvTzis0ReD+Qt+2N1E+ 6ploak7NwV/aPMFh7YUGC92mtUHjcj+RRIXk+rLpZ2YBoUc2yBwv6nuSG5WOZZr+Rmet Y62wfTNRqvbJHXvxmoY3N9EfJGCJ3NJHcZfdZfSvcvt4gzljjIF0H3VKWJ+JuGMXHFAC IG5qLhWhqWTXCo50y/70MBHwm3n8wzUZ0jBHItdeUeQKp4MVmmbCRYggGDQXwv14Jihp MsVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=l98v59BwmjV3ozOfAPw93XrEkUrol6K2zteQO/pxMQA=; b=sVKhqF6Thn0R5TA/S1y9WPk4xToYyrO7X2epDepnKhjjmxevmG0yQXXQ0zS7Hfan9F 9YQ922HIgqX7JgxfnUl/+YFnCYp4J4kWf62GZGURDfQHDZJ66k8VH0UpfbXKDEqxvruL OzM+5zylrxxbOXqIeYDTOFRxpZVpiKUCQ+0zdODAXnUebDSjUUWFcH08Jj8eMWH1BNZq UzeHba0yQgv9flybUL/G9qQC21xD8+GZrMjhKnQu4XOe/2Y/FPisTuTEuJaqbV4NxhzM ZzfB0BuCI/CVNkxPC02ipxKd/DRteavDgu6tgEiMpwqdaOU9tkmi8SUPbKrgZf8DIrvz zpqQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="e/bGIA0H"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c11-20020a17090a8d0b00b002192566b3e3si6127884pjo.146.2022.11.28.10.21.17; Mon, 28 Nov 2022 10:21:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="e/bGIA0H"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234221AbiK1STj (ORCPT + 99 others); Mon, 28 Nov 2022 13:19:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234201AbiK1SSz (ORCPT ); Mon, 28 Nov 2022 13:18:55 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82933391CC for ; Mon, 28 Nov 2022 10:03:11 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id 5so9111604wmo.1 for ; Mon, 28 Nov 2022 10:03:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l98v59BwmjV3ozOfAPw93XrEkUrol6K2zteQO/pxMQA=; b=e/bGIA0HwGB+g6h3LY4eVczqCQQA7q3lac86AA8ayrG5RmwDmE42BKfG03HoYIQKsz fMIN5YuLApIg7+jeOJkwev91+mwmAodAoBnqKNmf+ZyiCiDkOu2GYqHqzKzhbJWp/ecI vWsg6LX5RDFYedCD530UHz52LBU5F7BC2+HnuTEORVeZrSQlOPbK4fD7O+EJmC+ChSqT 4/QrQNGQYxmjFVHe4CjMFUxzDdcOEnJOVqPakZboLMtfqHUntjEYD8GQnRWIeBPl84lH 7h9OmgR2dU+cq7dQy/wrZmcmqRNiPwgNox6e/naAqPQyFZei0ecGmQ4QHiMSLPBUWKVm BCUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l98v59BwmjV3ozOfAPw93XrEkUrol6K2zteQO/pxMQA=; b=GDyfVLrVqxvFqEOrZfFzBGo4M0lvRRcdLqeyHLhQoIxKXFOAiHaBTRXu5u5zZHyOyi LHnILqOEhP/MtnNQRCN+kb+KY7MQkG8rT42MhjFGSi0hRpiVrBNXJ+M+4SZV+3QwiOfk M0DyAkBTPxed+hFPEeVXOAoHqzpier1xldf6YhneNvL6heDkiV72vhHSsKSeaePUcW9J z9P2sqp7PocFtIZCIH6QrZyaXWUy42wBlXELWe4yrdqJmqkkVR4aacTO6epuT1y8uE6j hvZEYnllbKrxWxePNrJnzuO/cuwkEqeQSnu5icSQSbsi7Sm0mo+uC68uzt6UW0Esiqhk RaIw== X-Gm-Message-State: ANoB5pmrvU8l5lzUJzfelhoV5qH/1rxFVQHJIY1TH226oVC2ZcAdn4HU x3PIj/Cu+39U6rS7nMJQyVhL3g== X-Received: by 2002:a05:600c:4fd0:b0:3cf:70a0:f689 with SMTP id o16-20020a05600c4fd000b003cf70a0f689mr43238540wmq.161.1669658589915; Mon, 28 Nov 2022 10:03:09 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:f4d1:b340:8675:e840]) by smtp.gmail.com with ESMTPSA id d8-20020a5d6dc8000000b002415dd45320sm11359859wrz.112.2022.11.28.10.03.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Nov 2022 10:03:09 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 3/3] mm/khugepaged: Invoke MMU notifiers in shmem/file collapse paths Date: Mon, 28 Nov 2022 19:02:52 +0100 Message-Id: <20221128180252.1684965-3-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog In-Reply-To: <20221128180252.1684965-1-jannh@google.com> References: <20221128180252.1684965-1-jannh@google.com> MIME-Version: 1.0 X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750765080889588638?= X-GMAIL-MSGID: =?utf-8?q?1750765080889588638?= Any codepath that zaps page table entries must invoke MMU notifiers to ensure that secondary MMUs (like KVM) don't keep accessing pages which aren't mapped anymore. Secondary MMUs don't hold their own references to pages that are mirrored over, so failing to notify them can lead to page use-after-free. I'm marking this as addressing an issue introduced in commit f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages"), but most of the security impact of this only came in commit 27e1f8273113 ("khugepaged: enable collapse pmd for pte-mapped THP"), which actually omitted flushes for the removal of present PTEs, not just for the removal of empty page tables. Cc: stable@kernel.org Fixes: f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages") Signed-off-by: Jann Horn Acked-by: David Hildenbrand Reviewed-by: Yang Shi --- v4: no changes mm/khugepaged.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index c3d3ce596bff7..49eb4b4981d88 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1404,6 +1404,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v unsigned long addr, pmd_t *pmdp) { pmd_t pmd; + struct mmu_notifier_range range; mmap_assert_write_locked(mm); if (vma->vm_file) @@ -1415,8 +1416,12 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v if (vma->anon_vma) lockdep_assert_held_write(&vma->anon_vma->root->rwsem); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, addr, + addr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); pmd = pmdp_collapse_flush(vma, addr, pmdp); tlb_remove_table_sync_one(); + mmu_notifier_invalidate_range_end(&range); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd));