From patchwork Mon May 15 14:35:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mathieu Desnoyers X-Patchwork-Id: 94174 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp6970375vqo; Mon, 15 May 2023 07:37:29 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6bGVdzfqkUVq5Vzyqmznwobu5BVG5baUDu/oekQZaDFw3aQD4uChTTr65G87c/9kQXJCfr X-Received: by 2002:a05:6a20:748a:b0:105:a24f:cff1 with SMTP id p10-20020a056a20748a00b00105a24fcff1mr6665353pzd.24.1684161449117; Mon, 15 May 2023 07:37:29 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684161449; cv=none; d=google.com; s=arc-20160816; b=y4DtlD7ZPopQ/l2euhCsq98IJZ8MBcgiAai64pJ6WHVcWU/Win7gBGHr2Nkz6WgOBz q3ygrom8fTBDmBp095Aj41s6rRvkadF30nMXRY8uo5j1sjlYtiTXkIE0hpGfVzxN5VQm 9Trok9rb/GctL2Q3xp+7cPMtOqsNPlYq+lCWtdCwq9ZU94C/WeCGYlup0npiK8St/yl2 fRqyoU0m225mu1WzmWeUuzO2ZXQUnhl/moqPsafX+SOpgCqwd4OfKGps8J72t6m6HdMZ c9HzeAnete/MBiYTOy/siXJ4oROh/K43t4EsU32HzV8IO/zcDgef5GbXeLiv5Q7pb1Z2 inJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=ZYEArWolIZt4sa6YXFI25cdVhoWmld9ONGA/zqZyRNw=; b=TO3uu7jg+6Au1p++43+ppFGk2isXjaQcBPVOu6BsTnO1HhZFzNm1SMsTxOl0Wygnaa d7cBwexwyl/4DBhgv9m+9tLRnN1MmcWMdGiDWL1spNWWHyj/VKfnZ/SsfetWGkO/f0B2 vfDuB34WLxGa1B4pkuZnBLQoT+Ai/WYrW0DzJKp/XA1IZXuTUtC+Y5FTNd1zSbIfojKd yEw76pDzRikp96l909O4WEYbKpBNK7BHAZeUj9NPtAgR6K++OnIlnB8uZLtC1YG5gdZe kVZbbY1hHobDw/YfrbDo3zWgOdNwb+mzE6Xv9FIhEcj7CeRHLkWdKp+e/iBM1QJOGoyg nDfA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=Ygbu0By2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a13-20020aa7970d000000b006281ec442absi17576991pfg.309.2023.05.15.07.37.13; Mon, 15 May 2023 07:37:29 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@efficios.com header.s=smtpout1 header.b=Ygbu0By2; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=efficios.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240191AbjEOOfp (ORCPT + 99 others); Mon, 15 May 2023 10:35:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241846AbjEOOfo (ORCPT ); Mon, 15 May 2023 10:35:44 -0400 Received: from smtpout.efficios.com (smtpout.efficios.com [167.114.26.122]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BFCCA11D for ; Mon, 15 May 2023 07:35:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1684161342; bh=55UiL7049d5WwXoq8Ga1ETl0Xcqd3k/79bhP8jRai1I=; h=From:To:Cc:Subject:Date:From; b=Ygbu0By2kzmJLHh1Fy4KXvfh+K9aNW5LbzhbEX5LeLatHUBNzOUEcNYDrcUw5Az5e v3SH4smxTxOMBPYSXoGiQuPqbVFeQMfc02K4UtkVhJMeHCcKxxgL795dpfbwruTact TdIrD0ixOzK3WpYzJKsh77uwmo9LwNOyNw2KZLr8q0TNVo0EGf8faPJezJsV3+JtuX zZFsLOUVQYckfd3VO9KJkCXjOA4Ixt05jknCAgSUVrIgCecJycLf3JuVsqt7twXxav 9Ma1bfkisrmpyWi+8puxAJFgbKbBvMCrn5rzbym0yGp/GAAAZNzShPunQjAlsiohQ7 NxI3QqSkgR+ig== Received: from localhost.localdomain (192-222-143-198.qc.cable.ebox.net [192.222.143.198]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4QKhk55lzxz12dV; Mon, 15 May 2023 10:35:41 -0400 (EDT) From: Mathieu Desnoyers To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , kernel test robot , Aaron Lu , Olivier Dion , michael.christie@oracle.com, Andrew Morton , Feng Tang , John Hubbard , Jason Gunthorpe , Peter Xu , linux-mm@kvack.org Subject: [PATCH] mm: Move mm_count into its own cache line Date: Mon, 15 May 2023 10:35:36 -0400 Message-Id: <20230515143536.114960-1-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS, T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1765971275893055311?= X-GMAIL-MSGID: =?utf-8?q?1765971275893055311?= The mm_struct mm_count field is frequently updated by mmgrab/mmdrop performed by context switch. This causes false-sharing for surrounding mm_struct fields which are read-mostly. This has been observed on a 2sockets/112core/224cpu Intel Sapphire Rapids server running hackbench, and by the kernel test robot will-it-scale testcase. Move the mm_count field into its own cache line to prevent false-sharing with other mm_struct fields. Move mm_count to the first field of mm_struct to minimize the amount of padding required: rather than adding padding before and after the mm_count field, padding is only added after mm_count. Note that I noticed this odd comment in mm_struct: commit 2e3025434a6b ("mm: relocate 'write_protect_seq' in struct mm_struct") /* * With some kernel config, the current mmap_lock's offset * inside 'mm_struct' is at 0x120, which is very optimal, as * its two hot fields 'count' and 'owner' sit in 2 different * cachelines, and when mmap_lock is highly contended, both * of the 2 fields will be accessed frequently, current layout * will help to reduce cache bouncing. * * So please be careful with adding new fields before * mmap_lock, which can easily push the 2 fields into one * cacheline. */ struct rw_semaphore mmap_lock; This comment is rather odd for a few reasons: - It requires addition/removal of mm_struct fields to carefully consider field alignment of _other_ fields, - It expresses the wish to keep an "optimal" alignment for a specific kernel config. I suspect that the author of this comment may want to revisit this topic and perhaps introduce a split-struct approach for struct rw_semaphore, if the need is to place various fields of this structure in different cache lines. Fixes: 223baf9d17f2 ("sched: Fix performance regression introduced by mm_cid") Fixes: af7f588d8f73 ("sched: Introduce per-memory-map concurrency ID") Link: https://lore.kernel.org/lkml/7a0c1db1-103d-d518-ed96-1584a28fbf32@efficios.com Reported-by: kernel test robot Link: https://lore.kernel.org/oe-lkp/202305151017.27581d75-yujie.liu@intel.com Signed-off-by: Mathieu Desnoyers Cc: Peter Zijlstra Cc: Aaron Lu Cc: Olivier Dion Cc: michael.christie@oracle.com Cc: Andrew Morton Cc: Feng Tang Cc: John Hubbard Cc: Jason Gunthorpe Cc: Peter Xu Cc: linux-mm@kvack.org Reviewed-by: Aaron Lu Reviewed-by: John Hubbard --- include/linux/mm_types.h | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 306a3d1a0fa6..de10fc797c8e 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -583,6 +583,21 @@ struct mm_cid { struct kioctx_table; struct mm_struct { struct { + /* + * Fields which are often written to are placed in a separate + * cache line. + */ + struct { + /** + * @mm_count: The number of references to &struct + * mm_struct (@mm_users count as 1). + * + * Use mmgrab()/mmdrop() to modify. When this drops to + * 0, the &struct mm_struct is freed. + */ + atomic_t mm_count; + } ____cacheline_aligned_in_smp; + struct maple_tree mm_mt; #ifdef CONFIG_MMU unsigned long (*get_unmapped_area) (struct file *filp, @@ -620,14 +635,6 @@ struct mm_struct { */ atomic_t mm_users; - /** - * @mm_count: The number of references to &struct mm_struct - * (@mm_users count as 1). - * - * Use mmgrab()/mmdrop() to modify. When this drops to 0, the - * &struct mm_struct is freed. - */ - atomic_t mm_count; #ifdef CONFIG_SCHED_MM_CID /** * @pcpu_cid: Per-cpu current cid.