From patchwork Tue Nov 29 15:47:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 27304 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp419607wrr; Tue, 29 Nov 2022 07:51:03 -0800 (PST) X-Google-Smtp-Source: AA0mqf68TtQUBeBzODLOmRxErxFobl4s1T1vY2yPd+j/3A/idQdbua2+/Hx4i+D+1UVrEQFR8y/C X-Received: by 2002:a17:906:1614:b0:78d:4e5b:ffaa with SMTP id m20-20020a170906161400b0078d4e5bffaamr31374095ejd.455.1669737063125; Tue, 29 Nov 2022 07:51:03 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669737063; cv=none; d=google.com; s=arc-20160816; b=eiT/WP0upKUWUOq3Ku33qYzU2C0suJ1tlS1glRTlVZ+Aro9RMiN/XVkOk7kU3+sQTW F0DoLUe2yQe3tbjXo+ZOWCI5gCzX07slX49JUyClQdIDyROlvyAzAiPTHRsWqo07/SS8 zEevCpWmkmhpqjqTd4/MxfWzvRIeI0aQd051/bVPxG3H2SaHntS9RFEOlm2Lp6KA88MV itliSPHoNRTlfvxKyJlNdD5oABwWDgOUEvX1ylMZ1YzeM8ZiHUGmWGaVAW4ux29mYGRs BXCkOk94ju0ZL5PKrLdP6+gg4XUkBmYaxD7LbEyNl8MDqpnk9XsdWEq89Bt+sZq6g+wx bCbw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=zcNi41xvTMS4NGTUpyHD2vhE6PXJ5qCDMr6KFHcPQag=; b=DRjogQaZEvRrp0lB3dK5aoRqamgi86wHHH4lQOR+YpS1ewKAN+m+I7UgFF36gobPSo qP+AG1Td9rAXlkhYX+1H872/BSZrdbvJRVOWohmNt6UMR1JyA/lvqxcIGw1WG+ne02Ii SWEM9fr0f0Tz3QdHndIj+wfFYWYzYBe9BtJeLx3A1mkDZeSNxOfl8PGx0PIEPxRrIVNu dzsPYA93uXKxTG2aUb6sL8SJJ25vWWdVSkUvWFDaH6tt1iQsz91q5JVyc77kK1yFvCwP KGfUzSCcF4sXeJjTx6o9hdPMVwMYHHFw1FK5YhVNCZM9yJ37W1SnQCoyOSOlbVMPq9Ke pczA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="Ymj/TX58"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v2-20020aa7dbc2000000b0045d15f39bf4si10780429edt.479.2022.11.29.07.50.30; Tue, 29 Nov 2022 07:51:03 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b="Ymj/TX58"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236021AbiK2Pru (ORCPT + 99 others); Tue, 29 Nov 2022 10:47:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235775AbiK2Prs (ORCPT ); Tue, 29 Nov 2022 10:47:48 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A9CB6357 for ; Tue, 29 Nov 2022 07:47:42 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id v1so22734607wrt.11 for ; Tue, 29 Nov 2022 07:47:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=zcNi41xvTMS4NGTUpyHD2vhE6PXJ5qCDMr6KFHcPQag=; b=Ymj/TX58LmBHMlJm2tvaAwNPklk0OGDZu/h2bpwb8PEHBxKhk4ajUV93di2Grz8TVV 9lMyxfkKDAp24o03K5+1FRcgoCKVXZjlBGbIytbuviIem7nF4rYyG6KooxotGqK+hP1E oOi+BBXCoaw4iA3xFq6+2BtTm31okvGXqpDLHXAVrSQF4e3INEQh3xc0r9WI6qWJs9vr cn8nSzY+otKe+LsoUuNa8GtXB+bJo3a/2ezw2yBL/OKQ89GnW5aHwVJlGaUwwQWVVlUi OfYz0qsbjPaz3Sq039jK3ZktX72U89/hNuW+ZLmHuz+gzgfB998b5wYXplTrOYLmT1UV HOGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=zcNi41xvTMS4NGTUpyHD2vhE6PXJ5qCDMr6KFHcPQag=; b=agypwEglpyK4xut2vNW72TMMNiRJNKNKEpttnV5CHd+R1Q3ChM6PaPxCL9gcGOY0RM jqGOF9fJNRYKhKxT6sY/qseQ174qmRqvkH53Aw5KE2M9ztgLVNMuKSGL7App88h2Vl7J K5s2SqG7Hb7NJcAYhTOID4VvpLu+m5B0MNmwQ4/GboUgJkXLIn/QZZGy/YoJqeTsFcRv tsCbYFhhX4Na/MKJYoGMhSxMWYjQHQer6InbxRK0OLnt6bfnMuTJf3V2xEUmsBOxYRQw DRvMz0l0JeLTSqPiDO7Yqfk2qKljZjC1YE8MU7PGdnX9RwzXsIWzFJ1LIfV4dNqu0kgR BP8Q== X-Gm-Message-State: ANoB5pnIRwbpjAfDBvansrb20AX8LM03b2KlWbDmknz6Onua4vtxvt82 k9rjVcFPd5KRPCICTB6G8+mEpw== X-Received: by 2002:a05:6000:70f:b0:22e:41c5:7ef7 with SMTP id bs15-20020a056000070f00b0022e41c57ef7mr35635050wrb.332.1669736860806; Tue, 29 Nov 2022 07:47:40 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:5011:adcc:fddd:accf]) by smtp.gmail.com with ESMTPSA id x9-20020a5d6b49000000b002366e3f1497sm14090293wrw.6.2022.11.29.07.47.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Nov 2022 07:47:40 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v5 1/3] mm/khugepaged: Take the right locks for page table retraction Date: Tue, 29 Nov 2022 16:47:28 +0100 Message-Id: <20221129154730.2274278-1-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog MIME-Version: 1.0 X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750846210625273661?= X-GMAIL-MSGID: =?utf-8?q?1750846210625273661?= pagetable walks on address ranges mapped by VMAs can be done under the mmap lock, the lock of an anon_vma attached to the VMA, or the lock of the VMA's address_space. Only one of these needs to be held, and it does not need to be held in exclusive mode. Under those circumstances, the rules for concurrent access to page table entries are: - Terminal page table entries (entries that don't point to another page table) can be arbitrarily changed under the page table lock, with the exception that they always need to be consistent for hardware page table walks and lockless_pages_from_mm(). This includes that they can be changed into non-terminal entries. - Non-terminal page table entries (which point to another page table) can not be modified; readers are allowed to READ_ONCE() an entry, verify that it is non-terminal, and then assume that its value will stay as-is. Retracting a page table involves modifying a non-terminal entry, so page-table-level locks are insufficient to protect against concurrent page table traversal; it requires taking all the higher-level locks under which it is possible to start a page walk in the relevant range in exclusive mode. The collapse_huge_page() path for anonymous THP already follows this rule, but the shmem/file THP path was getting it wrong, making it possible for concurrent rmap-based operations to cause corruption. Cc: stable@kernel.org Fixes: 27e1f8273113 ("khugepaged: enable collapse pmd for pte-mapped THP") Acked-by: David Hildenbrand Reviewed-by: Yang Shi Signed-off-by: Jann Horn --- Notes: v4: added ack by David Hildenbrand v5: added reviewed-by by Yang Shi mm/khugepaged.c | 55 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 4 deletions(-) base-commit: eb7081409f94a9a8608593d0fb63a1aa3d6f95d8 diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4734315f79407..674b111a24fa7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1384,16 +1384,37 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, return SCAN_SUCCEED; } +/* + * A note about locking: + * Trying to take the page table spinlocks would be useless here because those + * are only used to synchronize: + * + * - modifying terminal entries (ones that point to a data page, not to another + * page table) + * - installing *new* non-terminal entries + * + * Instead, we need roughly the same kind of protection as free_pgtables() or + * mm_take_all_locks() (but only for a single VMA): + * The mmap lock together with this VMA's rmap locks covers all paths towards + * the page table entries we're messing with here, except for hardware page + * table walks and lockless_pages_from_mm(). + */ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp) { - spinlock_t *ptl; pmd_t pmd; mmap_assert_write_locked(mm); - ptl = pmd_lock(vma->vm_mm, pmdp); + if (vma->vm_file) + lockdep_assert_held_write(&vma->vm_file->f_mapping->i_mmap_rwsem); + /* + * All anon_vmas attached to the VMA have the same root and are + * therefore locked by the same lock. + */ + if (vma->anon_vma) + lockdep_assert_held_write(&vma->anon_vma->root->rwsem); + pmd = pmdp_collapse_flush(vma, addr, pmdp); - spin_unlock(ptl); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); @@ -1444,6 +1465,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false)) return SCAN_VMA_CHECK; + /* + * Symmetry with retract_page_tables(): Exclude MAP_PRIVATE mappings + * that got written to. Without this, we'd have to also lock the + * anon_vma if one exists. + */ + if (vma->anon_vma) + return SCAN_VMA_CHECK; + /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ if (userfaultfd_wp(vma)) return SCAN_PTE_UFFD_WP; @@ -1477,6 +1506,20 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* + * We need to lock the mapping so that from here on, only GUP-fast and + * hardware page walks can access the parts of the page tables that + * we're operating on. + * See collapse_and_free_pmd(). + */ + i_mmap_lock_write(vma->vm_file->f_mapping); + + /* + * This spinlock should be unnecessary: Nobody else should be accessing + * the page tables under spinlock protection here, only + * lockless_pages_from_mm() and the hardware page walker can access page + * tables while all the high-level locks are held in write mode. + */ start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); result = SCAN_FAIL; @@ -1531,6 +1574,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove pte entries */ collapse_and_free_pmd(mm, vma, haddr, pmd); + i_mmap_unlock_write(vma->vm_file->f_mapping); + maybe_install_pmd: /* step 5: install pmd entry */ result = install_pmd @@ -1544,6 +1589,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, abort: pte_unmap_unlock(start_pte, ptl); + i_mmap_unlock_write(vma->vm_file->f_mapping); goto drop_hpage; } @@ -1600,7 +1646,8 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, * An alternative would be drop the check, but check that page * table is clear before calling pmdp_collapse_flush() under * ptl. It has higher chance to recover THP for the VMA, but - * has higher cost too. + * has higher cost too. It would also probably require locking + * the anon_vma. */ if (vma->anon_vma) { result = SCAN_PAGE_ANON; From patchwork Tue Nov 29 15:47:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 27306 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp419779wrr; Tue, 29 Nov 2022 07:51:25 -0800 (PST) X-Google-Smtp-Source: AA0mqf5Tig3pgcnQvi2+4Sy8snztpBMAIwyLD402caYoaCsQ0KaQltHk9VxhI3BFax/R+UFICFF1 X-Received: by 2002:a17:906:c052:b0:79a:101a:7e57 with SMTP id bm18-20020a170906c05200b0079a101a7e57mr48394901ejb.368.1669737084533; Tue, 29 Nov 2022 07:51:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669737084; cv=none; d=google.com; s=arc-20160816; b=loZXqt3Z4z/00usclSsqupIaMgO50fLtGVdx6eKE4stPp5EU5fRhrnAfUq3e6psDQ8 rQTRJBMuqP0ogP+FpxufbgtHrML8qC2Sg/xmNoQVF7yy0Dy65IZv4impCGMNLtHS8fm7 o3X1aEIhJJwx1SZwp+4eO3I/v04R7C0WnWnrJ2nBITDHUF9qW9AYq04Vi+JopPkEVJC0 XgGTCtMoOQHM/ckSLGUpnZgqfUsfchCZiLPtlq8XZvPh5uMDP+7vpDoWG4dBg8ADvU/2 UTiQt9ocyyHfi98bVTjxCodvOEZBD7oeoe2P1hkMfM8+F1GR8L5439QQK9GkpBnlJW7Y qPTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kT8IZrjdmDIMAqdPXo5e6xGOYanI26oKv2vQ1WDKtQk=; b=JEaNjFy87f8jxWurmiOIEmQSdZfdQ6QNE3LvNSPgxr4JBAWV8qvp3inNhgz/ZsOJSr NdBR2JyOyt7I3Jep3xNzaKxQZSEUvRyUgGG6b8xVd6leYO0rzHNTfNX5gXJOKVBKEJUm RnS8H5h2/3j3x7uOp2YtUX3LePF0RYKVtTgmlhjexsd2s4TT0vpoiGXrPM7qUl39AN6a kTYYr4kgmF+a0SrDRNq0rUaJHKpgbIrYsUE4k5s6mIWD9GlqhpHtIt77FsHXLMfF8PCD fADxGWzPEFXWO9mCfGhWHRD8SATd1jHM4i2dSKsJ3CqPNihH5nJaCbozYlkAmzwL7Hs/ BI3g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=BL0l4dA+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x6-20020aa7d386000000b004571c47b13csi10568865edq.418.2022.11.29.07.50.47; Tue, 29 Nov 2022 07:51:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=BL0l4dA+; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236041AbiK2PsB (ORCPT + 99 others); Tue, 29 Nov 2022 10:48:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236031AbiK2Prw (ORCPT ); Tue, 29 Nov 2022 10:47:52 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10E4BC1E for ; Tue, 29 Nov 2022 07:47:44 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id o7-20020a05600c510700b003cffc0b3374so11165956wms.0 for ; Tue, 29 Nov 2022 07:47:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kT8IZrjdmDIMAqdPXo5e6xGOYanI26oKv2vQ1WDKtQk=; b=BL0l4dA+hwXaxDagId779Kfqnra35IhbXewFqGa9/iFcvBB3etkTunBBW79/i36Z7W FfYpfWUnFoFz59NOcvGwEnZS33GiDNsps7XypTQfxe7OCHjHXk+z0tOPFjYOE+cACcsZ BXOa2b+J8eBTp5BN0CToCCnb2NeAqGP9snzI7hhPMzlf2jIo2SFynaVZjlfQYYi0iCSw GPCs7ry1zENWaAccRHHdPcJ6eAZekj2Pr5KT7h76W9ibjnORNwIxH00ODyhVgzvlN1vh CW16wRyiAnWnHq7GeBxH72cDAsZOijNEWdnWbcouFDuLz3dJ+2kRu6/4vmBtPncSAits V0Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kT8IZrjdmDIMAqdPXo5e6xGOYanI26oKv2vQ1WDKtQk=; b=B3u2TXTzGODRPrCoz6rlFDe4o47QUNLPuXQfOHP+zazWljwEn4qwa0Tb0d85X8X2H1 oMEc3hQAtckUDnzE2oy7NxUuAd9RMNfQrBBbisN/7HedjkACJ2UZYv/9cenKMzkkqp6I N1wvL5lzS8X9veNNH4PvXGz8VCl5mav6e16F8sbxsQGgyU2AsxuxsiDkkZuh9AxKz4uK neLrsC5oZrB32HOjO1oNQw1FTvusgxkqfAfu7eNBGlgStmXL7Q0oFb5Y3y2b/eEcVe20 m4H4fSMt+4Si6T6wKTdc3lS8ndAW0aFYQe4dr3w92PyUmHBsrEvpRRNMbXzJZVFOxoLS YwYA== X-Gm-Message-State: ANoB5pmB8UEE2QRonjwDIh5gRlep0fbWAL8HE6goE7zF4XC7EyVb0flF 1VL6uFJUbsdFV3rdaylNDt88Tw== X-Received: by 2002:a05:600c:19ce:b0:3d0:5160:e0e2 with SMTP id u14-20020a05600c19ce00b003d05160e0e2mr11570874wmq.147.1669736863055; Tue, 29 Nov 2022 07:47:43 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:5011:adcc:fddd:accf]) by smtp.gmail.com with ESMTPSA id o9-20020a5d4a89000000b00241fde8fe04sm14090930wrq.7.2022.11.29.07.47.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Nov 2022 07:47:42 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v5 2/3] mm/khugepaged: Fix GUP-fast interaction by sending IPI Date: Tue, 29 Nov 2022 16:47:29 +0100 Message-Id: <20221129154730.2274278-2-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog In-Reply-To: <20221129154730.2274278-1-jannh@google.com> References: <20221129154730.2274278-1-jannh@google.com> MIME-Version: 1.0 X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750846233101151918?= X-GMAIL-MSGID: =?utf-8?q?1750846233101151918?= The khugepaged paths that remove page tables have to be careful to synchronize against the lockless_pages_from_mm() path, which traverses page tables while only being protected by disabled IRQs. lockless_pages_from_mm() must not: 1. interpret the contents of freed memory as page tables (and once a page table has been deposited, it can be freed) 2. interpret the contents of deposited page tables as PTEs, since some architectures will store non-PTE data inside deposited page tables (see radix__pgtable_trans_huge_deposit()) 3. create new page references from PTEs after the containing page table has been detached and: 3a. __collapse_huge_page_isolate() has checked the page refcount 3b. the page table has been reused at another virtual address and populated with new PTEs ("new page references" here refer to stable references returned to the caller; speculative references that are dropped on an error path are fine) commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THP collapse") addressed issue 3 by making the lockless_pages_from_mm() fastpath recheck the pmd_t to ensure that the page table was not removed by khugepaged in between (under the assumption that the page table is not repeatedly moving back and forth between two addresses, with one PTE repeatedly being populated with the same value). But to address issues 1 and 2, we need to send IPIs before freeing/reusing page tables. By doing that, issue 3 is also automatically addressed, so the fix from commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THP collapse") becomes redundant. We can ensure that the necessary IPI is sent by calling tlb_remove_table_sync_one() because, as noted in mm/gup.c, under configurations that define CONFIG_HAVE_FAST_GUP, there are two possible cases: 1. CONFIG_MMU_GATHER_RCU_TABLE_FREE is set, causing tlb_remove_table_sync_one() to send an IPI to synchronize with lockless_pages_from_mm(). 2. CONFIG_MMU_GATHER_RCU_TABLE_FREE is unset, indicating that all TLB flushes are already guaranteed to send IPIs. tlb_remove_table_sync_one() will do nothing, but we've already run pmdp_collapse_flush(), which did a TLB flush, which must have involved IPIs. Cc: stable@kernel.org Fixes: ba76149f47d8 ("thp: khugepaged") Reviewed-by: Yang Shi Acked-by: David Hildenbrand Signed-off-by: Jann Horn --- Notes: v4: - added ack from David Hildenbrand - made commit message more verbose v5: - added reviewed-by from Yang Shi - rewrote commit message based on feedback from Yang Shi include/asm-generic/tlb.h | 4 ++++ mm/khugepaged.c | 2 ++ mm/mmu_gather.c | 4 +--- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 492dce43236ea..cab7cfebf40bd 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -222,12 +222,16 @@ extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #define tlb_needs_table_invalidate() (true) #endif +void tlb_remove_table_sync_one(void); + #else #ifdef tlb_needs_table_invalidate #error tlb_needs_table_invalidate() requires MMU_GATHER_RCU_TABLE_FREE #endif +static inline void tlb_remove_table_sync_one(void) { } + #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 674b111a24fa7..c3d3ce596bff7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1057,6 +1057,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl); mmu_notifier_invalidate_range_end(&range); + tlb_remove_table_sync_one(); spin_lock(pte_ptl); result = __collapse_huge_page_isolate(vma, address, pte, cc, @@ -1415,6 +1416,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v lockdep_assert_held_write(&vma->anon_vma->root->rwsem); pmd = pmdp_collapse_flush(vma, addr, pmdp); + tlb_remove_table_sync_one(); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index add4244e5790d..3a2c3f8cad2fe 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -153,7 +153,7 @@ static void tlb_remove_table_smp_sync(void *arg) /* Simply deliver the interrupt */ } -static void tlb_remove_table_sync_one(void) +void tlb_remove_table_sync_one(void) { /* * This isn't an RCU grace period and hence the page-tables cannot be @@ -177,8 +177,6 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ -static void tlb_remove_table_sync_one(void) { } - static void tlb_remove_table_free(struct mmu_table_batch *batch) { __tlb_remove_table_free(batch); From patchwork Tue Nov 29 15:47:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 27309 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp420657wrr; Tue, 29 Nov 2022 07:53:24 -0800 (PST) X-Google-Smtp-Source: AA0mqf7ZFGkbm6jCJe3X5nqlZVjaBKbYpK5fnTQNnwtpjpNvLM3Bv4EykPfyz0nY0KlKrcArXtaq X-Received: by 2002:a17:906:2416:b0:7ad:4a55:d6f5 with SMTP id z22-20020a170906241600b007ad4a55d6f5mr49686684eja.220.1669737204415; Tue, 29 Nov 2022 07:53:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669737204; cv=none; d=google.com; s=arc-20160816; b=FQVyImfdJ/9gep3cq+bbUf6O3PyfQ6sVq80Wz2O49VNOnys8VgUHEsFGUgYUYacHEM Z/U/apu+e+PdGouR/9Muzu3wLx9is1FRkFA2CTNW8mMSziinc4Ozk/ZHOMeKkNZdxrbl DByX+wwjhIMJHdvjrbAAHyqmdMD06qZ6O54AXKCmt7HInHywqdxg+Icu0rwxqqdHjPvg aDHN7ygsKS//U7RL310CutGFoRfDLN1RPc6eDOoPxsXk7lUann7xXjyS5qO3r+7x3UEu 1JP+qBfmZwhepcn/7n+rujAysftWGmnGkanazPpBpC0L/abe7lLcsKPCMc2w5CvD66I7 apAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=YSdw+42zFeQjk2DGkc/5JCpwWvHgOHoXpertIATsPMg=; b=k2NHsjxgJHecBE25ZoADi8XH/pXvjE9Nx+BSTMEbdrNVcgYwxtdbNnk2vKy1I3kosj MOum8wFtGmsLMYnlDjXwOBjhbpA7kjYyZ7+bM952L3rC6iHM/Sy3+6zT6aDwLimaRIa2 WpJfJI9BHTAR1V2tJvgiB0vv5lwGTG33O+l2qfsu0iEVNVQx+qy/Umpq07BUu9u55Yik hrgE/SnICVwFC1yyAKNPqeT8FjfDCfpEFRu2waCQRChV5B2kxWJYP03jHfDhJsCcz9Ww Ts1wm/GjmvkAsv3u7iksxUF//jMLQDBAeG4D0s9e/z91HwXQ0hmQ8kHI8RqIkiP6Bt+y V4ug== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=kyY6Gy0J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dp14-20020a170906c14e00b007b27afde85csi2118157ejc.707.2022.11.29.07.53.00; Tue, 29 Nov 2022 07:53:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=kyY6Gy0J; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236027AbiK2PsE (ORCPT + 99 others); Tue, 29 Nov 2022 10:48:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235775AbiK2Pr6 (ORCPT ); Tue, 29 Nov 2022 10:47:58 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63FFC5F42 for ; Tue, 29 Nov 2022 07:47:49 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id x17so22742639wrn.6 for ; Tue, 29 Nov 2022 07:47:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YSdw+42zFeQjk2DGkc/5JCpwWvHgOHoXpertIATsPMg=; b=kyY6Gy0JKC+0AUgiybPvuNjn+SU6DV8iudSjb8b55zB8kB4CqhVidPvZfk6mww3B8n 6tPPi7aTD28MO7gKzrHO+ZFZTkMmX5YCdYvsqIG/OXyXs9ykyLPP5wRNXHSRX0t6RVsh IZX4qH8FwqDWynlcIRmrySVkQJImf/gX52fCnR1vlGWe2x9tFDFP0yt7ZfZIHsBm6j36 kYSmD90/AkP2Qg+2MXUuwV1NFk3p2dA+7dZpdiWsjPuLXQCQpLBU6xTlqxUMgxjOTIMH /nQkV9fl8zI6rpYGdiluQ5g7ciMaMfjc9nt2qlBPLrhiHOj/rqFidysvQv8pXgIoMM3b SNCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YSdw+42zFeQjk2DGkc/5JCpwWvHgOHoXpertIATsPMg=; b=O5k9gBUM+S7uiSiqTn1iH78+joylTFyzFVgYr5M6U79Gf7eZei1hNMJ7lJSjpy21+4 SC8tVwOFtt8jw0ZaCst+gQsSVB2ncdL4CnkQy2JenRuxKXihuJ0T07T+ON1evWBbhBw1 abj1MrPKlFtiKqMqsqN0oXG8KnFXAVpzGfZiRuypJjvi/UJleQYJe1J33czTuWhclOus vRN8ml8qdJjoOJzgt0+hzpNwdlhjxu/ixcbxrzrb3g74HzAjVblD9bdKKgk+ROUXHGOt B2ib+vhek+LKobwNSDkVPig5pN7oxcK89z1ZTB9uxcA9DaRgjkFRrweqTIm71R6Q5q7I qsiQ== X-Gm-Message-State: ANoB5pmSr4sRGsHwAF02GxBmVsOZXm5C/VjibwfK7CBVSWsQrC1dnFPU UNnR18EwonFCVFox34pqpg1a7Q== X-Received: by 2002:a5d:5709:0:b0:241:d71c:5dde with SMTP id a9-20020a5d5709000000b00241d71c5ddemr27080262wrv.678.1669736867789; Tue, 29 Nov 2022 07:47:47 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:5011:adcc:fddd:accf]) by smtp.gmail.com with ESMTPSA id x9-20020a5d6b49000000b002366e3f1497sm14090560wrw.6.2022.11.29.07.47.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 29 Nov 2022 07:47:47 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v5 3/3] mm/khugepaged: Invoke MMU notifiers in shmem/file collapse paths Date: Tue, 29 Nov 2022 16:47:30 +0100 Message-Id: <20221129154730.2274278-3-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog In-Reply-To: <20221129154730.2274278-1-jannh@google.com> References: <20221129154730.2274278-1-jannh@google.com> MIME-Version: 1.0 X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750846358692769437?= X-GMAIL-MSGID: =?utf-8?q?1750846358692769437?= Any codepath that zaps page table entries must invoke MMU notifiers to ensure that secondary MMUs (like KVM) don't keep accessing pages which aren't mapped anymore. Secondary MMUs don't hold their own references to pages that are mirrored over, so failing to notify them can lead to page use-after-free. I'm marking this as addressing an issue introduced in commit f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages"), but most of the security impact of this only came in commit 27e1f8273113 ("khugepaged: enable collapse pmd for pte-mapped THP"), which actually omitted flushes for the removal of present PTEs, not just for the removal of empty page tables. Cc: stable@kernel.org Fixes: f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages") Acked-by: David Hildenbrand Reviewed-by: Yang Shi Signed-off-by: Jann Horn --- Notes: v4: no changes v5: - added ack and reviewed-by mm/khugepaged.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index c3d3ce596bff7..49eb4b4981d88 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1404,6 +1404,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v unsigned long addr, pmd_t *pmdp) { pmd_t pmd; + struct mmu_notifier_range range; mmap_assert_write_locked(mm); if (vma->vm_file) @@ -1415,8 +1416,12 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v if (vma->anon_vma) lockdep_assert_held_write(&vma->anon_vma->root->rwsem); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, addr, + addr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); pmd = pmdp_collapse_flush(vma, addr, pmdp); tlb_remove_table_sync_one(); + mmu_notifier_invalidate_range_end(&range); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd));