From patchwork Fri Nov 25 21:37:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 26142 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp4304546wrr; Fri, 25 Nov 2022 13:41:48 -0800 (PST) X-Google-Smtp-Source: AA0mqf5ahxBg3vL/y6K8nPRNAVjYAYQgc8YNwmw3VcysL2wwwoR8cLXQnS3VLJ+v81T4uFRsBZO7 X-Received: by 2002:a17:906:2854:b0:7ae:3684:84b0 with SMTP id s20-20020a170906285400b007ae368484b0mr25579891ejc.622.1669412508380; Fri, 25 Nov 2022 13:41:48 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669412508; cv=none; d=google.com; s=arc-20160816; b=SJ3rinQjlfs4YOOSdLshccTptatrSkykANOZg/JlEiPfs25zoMa3rGp+iv2GXK5lX+ AjBmxS5bxOgwU60c2M/6TuadMvfacDkjFTvjfWthd1JIgnIgKFv9DxGAZvh3bfX1MYhS jpx0jyr/jwcmvlfajPFQd7Ww5azg61PUNQSGk0fu/UTKMtZ5HG6U6oblp+U0GTcI+NuI OQP6f4KkFceXqe72rXYXEBKM55/v4GvITcdvKStQBngZuqOaZP0nZw0K8FILEm9YvrgL zYctXIJH1Pdc75haKdodS2E7CzNlCrwmK9TVsnVsX87DAk68uFLwZWDjpVqYHZfAsdqT dpuA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=PMWbLEOm6/iOpsMSbMIlf3wKmoaEQMgHyilsDSmEvuU=; b=uvlNrmdMO3Z1/fmmrM3j5JLqt3ajjqnCZ+fwHi45+DRiZ1rwfIul1/NMkgpkd08uUT bV+XM4OEc5bocuYE0sb7gcyIpIB8wJIDQuIPTcZ38w1wYaZAWgL/NiOfBV6Q1vubT63W Za0STjGMoKaX3yhqgDPDSAqJwS8WstTzqFGP3e4MqLq25/uAO6W6Re9XkrqSoLkhCA6F E0Ot5LFzzPNZqhJnOAQ33T02RCWgyJeLf9pWNcedJrUusuCNo4TReGOnfGScIJJsYLBn +AY1RFtWyWJ9rXtJwidjNhyArhoh+ioY30aisFyVpwRdD2ZDaj4Jsqp9KQgctHmgavsW tPLg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=BOwP2Wul; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ho16-20020a1709070e9000b0078dffe01cbesi5312019ejc.4.2022.11.25.13.41.24; Fri, 25 Nov 2022 13:41:48 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=BOwP2Wul; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230001AbiKYVhg (ORCPT + 99 others); Fri, 25 Nov 2022 16:37:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229757AbiKYVhe (ORCPT ); Fri, 25 Nov 2022 16:37:34 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91C25B77 for ; Fri, 25 Nov 2022 13:37:29 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id t4so4347504wmj.5 for ; Fri, 25 Nov 2022 13:37:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=PMWbLEOm6/iOpsMSbMIlf3wKmoaEQMgHyilsDSmEvuU=; b=BOwP2WulGHimVFJJg7+ebwT37QuzaJXKKAxDMC8zTNWnHnzJf2ND6hQMz/nOtsfhwN SVLCkwkcI/KBIkCvoJkKYM4ANqNF6LaD5FDk0AmSrqCMfutBwoARiJE/okhrLW8H0bGB eXccaf9QfTyuqkGMGCaf+dkyO0PFK80zE3qZKfJmmh2FWIlmF4SROyUtCJakLCbmKrP7 2SJ3MOj13n5eiM6BKCkmagmWdUZop6J/JTsNRAwsJbeplXifx9bjzPiTC5iPkspldCfy XYXSlgBDapyGf1nLp1uUgVYchH23PG2dyO3eSI59trh0Votmjm9Dvvp9q0sXiON5vRfK zv5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PMWbLEOm6/iOpsMSbMIlf3wKmoaEQMgHyilsDSmEvuU=; b=Pm+sPkIL3lGAF01dWibJGmx2qxi7zKYvjL5B/Lk8EiEnlnNIABQ6MCIPXwQX/jQnUr 6S8QBSZH6Y/BfBW0t8kpkVjF6Pl3LlM66NfkCkrnroARfSgc7mS/2koHRh1so66moWkl vB6V8v4ezYVZ52aS1TS3FTqUSsyrecnITowbEbUMdVUjxDmWwtO0zrTQFPR9Mzlgjdnb HvGUG39uX9lrr9QsbITXH8uJXSvIFiqmtXzhSovKWIphXh5he1cOuFQBtj2Vv9FzA+Nk jdIDZlgOXOzR6TWAi1kXHX6YMRwPbC7rKeel1t02wizHszgMTD95mLhifBB3SpKml5EH yYsg== X-Gm-Message-State: ANoB5pn/P2mM6/Hnawc2FuR4qW0KavHewA9ltUL9PFb+E4J1sT5/rBNQ /aRu39IgN9ocIjdqnllysVgiFg== X-Received: by 2002:a7b:c859:0:b0:3cf:dcee:c349 with SMTP id c25-20020a7bc859000000b003cfdceec349mr16949063wml.82.1669412247939; Fri, 25 Nov 2022 13:37:27 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:9104:e316:5846:858e]) by smtp.gmail.com with ESMTPSA id p11-20020a05600c358b00b003cf71b1f66csm7025668wmq.0.2022.11.25.13.37.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Nov 2022 13:37:27 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 1/3] mm/khugepaged: Take the right locks for page table retraction Date: Fri, 25 Nov 2022 22:37:12 +0100 Message-Id: <20221125213714.4115729-1-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog MIME-Version: 1.0 X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750505890362075013?= X-GMAIL-MSGID: =?utf-8?q?1750505890362075013?= pagetable walks on address ranges mapped by VMAs can be done under the mmap lock, the lock of an anon_vma attached to the VMA, or the lock of the VMA's address_space. Only one of these needs to be held, and it does not need to be held in exclusive mode. Under those circumstances, the rules for concurrent access to page table entries are: - Terminal page table entries (entries that don't point to another page table) can be arbitrarily changed under the page table lock, with the exception that they always need to be consistent for hardware page table walks and lockless_pages_from_mm(). This includes that they can be changed into non-terminal entries. - Non-terminal page table entries (which point to another page table) can not be modified; readers are allowed to READ_ONCE() an entry, verify that it is non-terminal, and then assume that its value will stay as-is. Retracting a page table involves modifying a non-terminal entry, so page-table-level locks are insufficient to protect against concurrent page table traversal; it requires taking all the higher-level locks under which it is possible to start a page walk in the relevant range in exclusive mode. The collapse_huge_page() path for anonymous THP already follows this rule, but the shmem/file THP path was getting it wrong, making it possible for concurrent rmap-based operations to cause corruption. Cc: stable@kernel.org Fixes: 27e1f8273113 ("khugepaged: enable collapse pmd for pte-mapped THP") Signed-off-by: Jann Horn Acked-by: David Hildenbrand --- mm/khugepaged.c | 55 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 51 insertions(+), 4 deletions(-) base-commit: eb7081409f94a9a8608593d0fb63a1aa3d6f95d8 diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4734315f79407..674b111a24fa7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1384,16 +1384,37 @@ static int set_huge_pmd(struct vm_area_struct *vma, unsigned long addr, return SCAN_SUCCEED; } +/* + * A note about locking: + * Trying to take the page table spinlocks would be useless here because those + * are only used to synchronize: + * + * - modifying terminal entries (ones that point to a data page, not to another + * page table) + * - installing *new* non-terminal entries + * + * Instead, we need roughly the same kind of protection as free_pgtables() or + * mm_take_all_locks() (but only for a single VMA): + * The mmap lock together with this VMA's rmap locks covers all paths towards + * the page table entries we're messing with here, except for hardware page + * table walks and lockless_pages_from_mm(). + */ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp) { - spinlock_t *ptl; pmd_t pmd; mmap_assert_write_locked(mm); - ptl = pmd_lock(vma->vm_mm, pmdp); + if (vma->vm_file) + lockdep_assert_held_write(&vma->vm_file->f_mapping->i_mmap_rwsem); + /* + * All anon_vmas attached to the VMA have the same root and are + * therefore locked by the same lock. + */ + if (vma->anon_vma) + lockdep_assert_held_write(&vma->anon_vma->root->rwsem); + pmd = pmdp_collapse_flush(vma, addr, pmdp); - spin_unlock(ptl); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); @@ -1444,6 +1465,14 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false)) return SCAN_VMA_CHECK; + /* + * Symmetry with retract_page_tables(): Exclude MAP_PRIVATE mappings + * that got written to. Without this, we'd have to also lock the + * anon_vma if one exists. + */ + if (vma->anon_vma) + return SCAN_VMA_CHECK; + /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ if (userfaultfd_wp(vma)) return SCAN_PTE_UFFD_WP; @@ -1477,6 +1506,20 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, goto drop_hpage; } + /* + * We need to lock the mapping so that from here on, only GUP-fast and + * hardware page walks can access the parts of the page tables that + * we're operating on. + * See collapse_and_free_pmd(). + */ + i_mmap_lock_write(vma->vm_file->f_mapping); + + /* + * This spinlock should be unnecessary: Nobody else should be accessing + * the page tables under spinlock protection here, only + * lockless_pages_from_mm() and the hardware page walker can access page + * tables while all the high-level locks are held in write mode. + */ start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); result = SCAN_FAIL; @@ -1531,6 +1574,8 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, /* step 4: remove pte entries */ collapse_and_free_pmd(mm, vma, haddr, pmd); + i_mmap_unlock_write(vma->vm_file->f_mapping); + maybe_install_pmd: /* step 5: install pmd entry */ result = install_pmd @@ -1544,6 +1589,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, abort: pte_unmap_unlock(start_pte, ptl); + i_mmap_unlock_write(vma->vm_file->f_mapping); goto drop_hpage; } @@ -1600,7 +1646,8 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, * An alternative would be drop the check, but check that page * table is clear before calling pmdp_collapse_flush() under * ptl. It has higher chance to recover THP for the VMA, but - * has higher cost too. + * has higher cost too. It would also probably require locking + * the anon_vma. */ if (vma->anon_vma) { result = SCAN_PAGE_ANON; From patchwork Fri Nov 25 21:37:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 26143 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp4305793wrr; Fri, 25 Nov 2022 13:45:24 -0800 (PST) X-Google-Smtp-Source: AA0mqf41DSMLAnnpaVVyLYgVjtcxRFUCMMM6tDn9sLH48hIFoXQvWVeEE8nUB/g96pAybThOxVLJ X-Received: by 2002:a17:906:9618:b0:7ae:38a:fd with SMTP id s24-20020a170906961800b007ae038a00fdmr34307338ejx.501.1669412724465; Fri, 25 Nov 2022 13:45:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669412724; cv=none; d=google.com; s=arc-20160816; b=rJJFdykg6XRVOFs3J2IsXvrE16+BeIggnyu22M0F+YloHXuzI5RazzQ/cDtFNMcyen GzkirzlhbOCJcWG7GYx/ybPUyFWLFduEvjiGslebxKZ5iyKbbCny3oIQ80OyD7mTDRb/ AOpvAzRjBYAVc7TVy7vpqGgtWLtyvVKw97i+SRPyWV2H63L5djZgc2OqmrYBUaF6Ns7g iEqdw71bTAQlypI7rdfveY6JILD454xA2cuPOut13xYwzN1qV6c9VssyWzhD6KDSSeqg GUjVlFW8j7CRsN7HJfE0/QsD1gicr4CEg3KTkmGSnoNZSXDKYCltgQHoTC44FPOkB9UX ijkw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=SXD6TekHCwWyaJjEnEUCIQwhKm8R4syqj2EShSG9CG4=; b=D6zcqLN1sCArmqnYU9tuutMcymoJR9Q/uHu6oxa4K1WOF8Z3qojqC/BktcYXjumD/f Usso5pAGeUqIZDmRmjUUcWQf7fP8NuxckidkQMt7AgdUl5AJobzx13xi/z+jiCRTPyIa fXSD0GQ4Naz+QTBtPo3oGTEndEDJLMMxAtoTZKgEGNVUOs25+u5jnGYN7COJIINUB5L2 /E8lTQxPJXA4yflV8yAdfsrYj0daHCeRkJKX3M94nAusWOH1beGnCWgXT4GoOJtMKsjU kJtMyQfsMxgEbIYTkkgg/d3lDA64wkUVxbnix0mU6KWJxVt/Ws05y/pBsLsqGR49P21v G5pQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=LbycPTZa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id dp19-20020a170906c15300b007306ac0faa0si5238465ejc.615.2022.11.25.13.45.01; Fri, 25 Nov 2022 13:45:24 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=LbycPTZa; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229957AbiKYVhx (ORCPT + 99 others); Fri, 25 Nov 2022 16:37:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230075AbiKYVhu (ORCPT ); Fri, 25 Nov 2022 16:37:50 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9FDD101C4 for ; Fri, 25 Nov 2022 13:37:46 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id cl5so8508359wrb.9 for ; Fri, 25 Nov 2022 13:37:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SXD6TekHCwWyaJjEnEUCIQwhKm8R4syqj2EShSG9CG4=; b=LbycPTZagQ51nPLmloFB/7NtnvTiSeUQAuZZ2EkPPlQgtWmKZxxpz1La+0Sg5Wa4Rr QPK0+z28OGsP2RhtKkhY6uwU9FRTAt/NqXjZ7MjCe3YETc37AnNoh2YcVnwLSIA/A2in ozUtyauDDVkAhz9m29tRAE0JI3b3myHPK1RVcbBAl0asvNRR5Vy+CDZXCw/8tuXxqhjS yrv6noxZgbshgZx84/f8bQc2LkkMA/Z9vXXk6Bo0tqg2j/ykUC5HaVHtCvv4sb+G8fPj 1eJ657mhGUY43wuEPXURLQhISq427MEO8FteYgZbLF00C0vgmD5VPh021QAYYnfkDmNV 3Caw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SXD6TekHCwWyaJjEnEUCIQwhKm8R4syqj2EShSG9CG4=; b=MLE4Ci77Q8PNMjsKox8N5BRgmBjNoqqQoku45H7Mc5zJVhiRL7zPZtur6UjaYdBLq7 e2dTsG4zSogod1v8T4rsP92tu//0wGnDw0Uyxx6DfF9wdxBe6L7Y/K3TPWvIXkB7vAdH QIXQ8jLr899lLkvqhzsvNHoukxfiEGO2Fe57Z8f+SiakTcKb1DivZm8ps0028QJiU7Qq DoVvt9ACgut1aDKBu2g8I9fmP6v6coE0Xe3VIi+SaxgSlioHbINiI0bzwcZQR1WOdW8x VhNgqhwqUWM8MdOV8MrDJ08b+2b0WYl5fcNZ/NsYBGw8ReCNx8YjgN/uWqClmHEARZFD WFRA== X-Gm-Message-State: ANoB5pnL9gaOT7mhlsZevx8LViVtJZSZAp1++BY4zDzUJn7rVKi4nBKG LVByXgd5wvCaqKyDAe0jf4wpaw== X-Received: by 2002:a05:6000:235:b0:241:bd2b:f718 with SMTP id l21-20020a056000023500b00241bd2bf718mr23371777wrz.89.1669412265015; Fri, 25 Nov 2022 13:37:45 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:9104:e316:5846:858e]) by smtp.gmail.com with ESMTPSA id o5-20020a05600c510500b003b4ff30e566sm16931581wms.3.2022.11.25.13.37.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Nov 2022 13:37:44 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 2/3] mm/khugepaged: Fix GUP-fast interaction by sending IPI Date: Fri, 25 Nov 2022 22:37:13 +0100 Message-Id: <20221125213714.4115729-2-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog In-Reply-To: <20221125213714.4115729-1-jannh@google.com> References: <20221125213714.4115729-1-jannh@google.com> MIME-Version: 1.0 X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750506117145398576?= X-GMAIL-MSGID: =?utf-8?q?1750506117145398576?= Since commit 70cbc3cc78a99 ("mm: gup: fix the fast GUP race against THP collapse"), the lockless_pages_from_mm() fastpath rechecks the pmd_t to ensure that the page table was not removed by khugepaged in between. However, lockless_pages_from_mm() still requires that the page table is not concurrently freed. Fix it by sending IPIs (if the architecture uses semi-RCU-style page table freeing) before freeing/reusing page tables. Cc: stable@kernel.org Fixes: ba76149f47d8 ("thp: khugepaged") Signed-off-by: Jann Horn Acked-by: David Hildenbrand --- replaced the mmu_gather-based scheme with an RCU call as suggested by Peter Xu include/asm-generic/tlb.h | 4 ++++ mm/khugepaged.c | 2 ++ mm/mmu_gather.c | 4 +--- 3 files changed, 7 insertions(+), 3 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 492dce43236ea..cab7cfebf40bd 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -222,12 +222,16 @@ extern void tlb_remove_table(struct mmu_gather *tlb, void *table); #define tlb_needs_table_invalidate() (true) #endif +void tlb_remove_table_sync_one(void); + #else #ifdef tlb_needs_table_invalidate #error tlb_needs_table_invalidate() requires MMU_GATHER_RCU_TABLE_FREE #endif +static inline void tlb_remove_table_sync_one(void) { } + #endif /* CONFIG_MMU_GATHER_RCU_TABLE_FREE */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 674b111a24fa7..c3d3ce596bff7 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1057,6 +1057,7 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, _pmd = pmdp_collapse_flush(vma, address, pmd); spin_unlock(pmd_ptl); mmu_notifier_invalidate_range_end(&range); + tlb_remove_table_sync_one(); spin_lock(pte_ptl); result = __collapse_huge_page_isolate(vma, address, pte, cc, @@ -1415,6 +1416,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v lockdep_assert_held_write(&vma->anon_vma->root->rwsem); pmd = pmdp_collapse_flush(vma, addr, pmdp); + tlb_remove_table_sync_one(); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd)); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index add4244e5790d..3a2c3f8cad2fe 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -153,7 +153,7 @@ static void tlb_remove_table_smp_sync(void *arg) /* Simply deliver the interrupt */ } -static void tlb_remove_table_sync_one(void) +void tlb_remove_table_sync_one(void) { /* * This isn't an RCU grace period and hence the page-tables cannot be @@ -177,8 +177,6 @@ static void tlb_remove_table_free(struct mmu_table_batch *batch) #else /* !CONFIG_MMU_GATHER_RCU_TABLE_FREE */ -static void tlb_remove_table_sync_one(void) { } - static void tlb_remove_table_free(struct mmu_table_batch *batch) { __tlb_remove_table_free(batch); From patchwork Fri Nov 25 21:37:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jann Horn X-Patchwork-Id: 26144 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:f944:0:0:0:0:0 with SMTP id q4csp4306611wrr; Fri, 25 Nov 2022 13:47:40 -0800 (PST) X-Google-Smtp-Source: AA0mqf4EqNXWz4RyyXEk8W1Vrtayj2qg5h2gkUPjAU5C3MJqDDTu6I0HFsxry90OU11B6VB0gCWR X-Received: by 2002:aa7:cb49:0:b0:468:f307:3014 with SMTP id w9-20020aa7cb49000000b00468f3073014mr28872596edt.321.1669412860222; Fri, 25 Nov 2022 13:47:40 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1669412860; cv=none; d=google.com; s=arc-20160816; b=iowe4MTkgXQQMdaZpEKLnEzmEvCDcHivNH9NtZsse4Jk7bzVsjRCUxirOCrCCNLfyW wZnutawwzRL8EoSJOdiN7tivusev1EYoS3JTylfpLsqdQyZiT42nnctzRMz++bw6ZUJk ELSSTJEFHkz6JgAQANvFCVFb+4lpKkR+/OvG65Ywip/GTvXmvKUnWJUIsom/KTAlP35r pQ8Lzn7yEh6zipC+Xg0wOBoAOvdkwu+pQh1N4O08nSjWofGxFisKZd26AR3PAZcoxIH7 aeYENO7+ibTM+tT03K3i6Lqh5coBXXqrUyY9jv/FhGY53TjFyXkX7SRvFk9QwZENcr5j uvDQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=NOnwsTARUoatlgwSRumO8KX+asIUrP0Tz9G19w9mrYE=; b=P9jeY7jsrmm9f+cUPFfJvd0aeXrbE702e7BqPdx652gxiFrBFwXkc17CrFxT7b5RY4 p7d6oHUnZcW4ZWMSDitBjNGr8iXIFBFWd9rRFVJx6dgYMzRLNiDLAdVwmCQMnd8l3FSs VMfWu9pPvqImG3+4DdADQwxrj4lK7QwX/3v3HhPQ7gR/lTwsBN2VoFvdalLPtxUeXsi3 BBTtuAOE0jBEIgJy8OkFZHkyJQJMKvRVkLrbFi6t0P8bapVQPgYrF/JJcDbrQi+jXIJn xvEU/DpJ2A0ThGN1/q60c55Jn+5kHnJ2RCKwBQVHQgDdGATiJ+fBjByHn3Tc32lr1tB6 cxIg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=F18wqhUO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id x11-20020a1709064bcb00b007707b853e46si3014821ejv.882.2022.11.25.13.47.15; Fri, 25 Nov 2022 13:47:40 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20210112 header.b=F18wqhUO; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230097AbiKYVhy (ORCPT + 99 others); Fri, 25 Nov 2022 16:37:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41298 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230102AbiKYVhu (ORCPT ); Fri, 25 Nov 2022 16:37:50 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7512FC745 for ; Fri, 25 Nov 2022 13:37:48 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id q7so7627492wrr.8 for ; Fri, 25 Nov 2022 13:37:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NOnwsTARUoatlgwSRumO8KX+asIUrP0Tz9G19w9mrYE=; b=F18wqhUOYlIX6J6ZsQFtBJSCT9eGiwguPQmjVFDZS7DKaV/hKfBW3zmQdVbyWmz0uh 1qn6FwsVbEYLb+xyfH7ZZDkMM6uZzqnljapoiX58tD5RA3MCm3fzaxHGzkfPd3KCl72H eLo3OgIuOrR6ZHT0e5Vs6f3N7tbNhtobUBQo7V1CM1IAM8bmN6iM8aOJPykUrX6giarR xd4kwOdZ+Eq1KakuVhnZd5qSnVHmGaIw7gDT+yJUNeydDyPLcBSlzYbaId82H7eBHzlc fn5z8Iu3LtcKkbFACMjoo5wH0GZrIpPMcDx6MXGiwHzlNQBftGPREzMGphe0eLA01xqh 4L7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NOnwsTARUoatlgwSRumO8KX+asIUrP0Tz9G19w9mrYE=; b=B1q7HXoj8ohQVxaNwv3QcBQD+c3V8mVl3OkRUvoPe4lcaLqUxPFX/KuVnSWvKZjAKD q+SByo2z2qt6Azw2uK9aIqYcxsCVQ9oAlqRoisQZhTwyZl59NikqSdSicr6yj1JAFEgf /FJi8IuntgC4PrsXxDnqxUVNvxE9Dm7jvb8tlCyp/W9cVGfLhXhjykrRuTahqKtr73Wx QQf2Pl6UNC8U1w/frIm5b/lBIa6uPezmozjQMnV1yGhmAHqAw8U+KGr2OBepnFWePzGG Cqvpev+Sh6Kutw/xY+hSNazbhlAPkTm70H+claP9N7n1jDRHuyAENqj9kz8JgRfOLrYe Br0w== X-Gm-Message-State: ANoB5plctGCZoPmWZ+SvAE23htHysZ03ndv7SlsF1NX1phDSPuCNHXY6 4NiNT8trBI0NUKdd7SWnoXFpxQ== X-Received: by 2002:a5d:6f02:0:b0:241:fdb5:50f7 with SMTP id ay2-20020a5d6f02000000b00241fdb550f7mr6563623wrb.170.1669412266904; Fri, 25 Nov 2022 13:37:46 -0800 (PST) Received: from localhost ([2a00:79e0:9d:4:9104:e316:5846:858e]) by smtp.gmail.com with ESMTPSA id z6-20020adfe546000000b0023655e51c33sm4675291wrm.4.2022.11.25.13.37.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Nov 2022 13:37:46 -0800 (PST) From: Jann Horn To: security@kernel.org, Andrew Morton Cc: Yang Shi , David Hildenbrand , Peter Xu , John Hubbard , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v3 3/3] mm/khugepaged: Invoke MMU notifiers in shmem/file collapse paths Date: Fri, 25 Nov 2022 22:37:14 +0100 Message-Id: <20221125213714.4115729-3-jannh@google.com> X-Mailer: git-send-email 2.38.1.584.g0f3c55d4c2-goog In-Reply-To: <20221125213714.4115729-1-jannh@google.com> References: <20221125213714.4115729-1-jannh@google.com> MIME-Version: 1.0 X-Spam-Status: No, score=-17.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF, ENV_AND_HDR_SPF_MATCH,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL,USER_IN_DEF_SPF_WL autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1750506259230028646?= X-GMAIL-MSGID: =?utf-8?q?1750506259230028646?= Any codepath that zaps page table entries must invoke MMU notifiers to ensure that secondary MMUs (like KVM) don't keep accessing pages which aren't mapped anymore. Secondary MMUs don't hold their own references to pages that are mirrored over, so failing to notify them can lead to page use-after-free. I'm marking this as addressing an issue introduced in commit f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages"), but most of the security impact of this only came in commit 27e1f8273113 ("khugepaged: enable collapse pmd for pte-mapped THP"), which actually omitted flushes for the removal of present PTEs, not just for the removal of empty page tables. Cc: stable@kernel.org Fixes: f3f0e1d2150b ("khugepaged: add support of collapse for tmpfs/shmem pages") Signed-off-by: Jann Horn --- mm/khugepaged.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index c3d3ce596bff7..49eb4b4981d88 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1404,6 +1404,7 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v unsigned long addr, pmd_t *pmdp) { pmd_t pmd; + struct mmu_notifier_range range; mmap_assert_write_locked(mm); if (vma->vm_file) @@ -1415,8 +1416,12 @@ static void collapse_and_free_pmd(struct mm_struct *mm, struct vm_area_struct *v if (vma->anon_vma) lockdep_assert_held_write(&vma->anon_vma->root->rwsem); + mmu_notifier_range_init(&range, MMU_NOTIFY_CLEAR, 0, NULL, mm, addr, + addr + HPAGE_PMD_SIZE); + mmu_notifier_invalidate_range_start(&range); pmd = pmdp_collapse_flush(vma, addr, pmdp); tlb_remove_table_sync_one(); + mmu_notifier_invalidate_range_end(&range); mm_dec_nr_ptes(mm); page_table_check_pte_clear_range(mm, addr, pmd); pte_free(mm, pmd_pgtable(pmd));