Message ID | 20231018074553.41333-1-hu1.chen@intel.com |
---|---|
State | New |
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2908:b0:403:3b70:6f57 with SMTP id ib8csp4627230vqb; Wed, 18 Oct 2023 00:47:46 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGc5GKtDj/qQZ1MiBytZ1zTqS69xd05HaJDEEJP0lFT7S6uEkuKsW1GAED024mQmGuCbzoL X-Received: by 2002:a92:cd4d:0:b0:350:f353:4017 with SMTP id v13-20020a92cd4d000000b00350f3534017mr4674785ilq.0.1697615266545; Wed, 18 Oct 2023 00:47:46 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1697615266; cv=none; d=google.com; s=arc-20160816; b=Km8y2tKrq862Yk38GrywFtrjb4N39UcxiNS6BOEsNDVF0n+87ndngPu8oPbciHE76E hjMF43JLJPRru6q2Z/2qgu+42GjSLTaJ7JAO7F9VkwZEbOGPFbiq2b5+j95shEqwCaEV vXMA/sJ/+7SjVkpSJRE92umnf7nAR/pJWSMXX4DP4Nnv5Cin+K+VpnWz4wpf9+qbkj3C 8C27HBWjEPyeHS2sVSXTv8/IdbOfnbH7+mg++/7aZce9uiJMCXOv1XIUYM2HvcNZ96ot LLKgbWFMKjnw9q0J5rUnJ1/trDUyl+C3Iz45cvhD8uLOrVQsnICOhnM6RmWibdXnYge9 lpEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from:dkim-signature; bh=GlXNFqihSTOF5qmbQruc7onlh46iAGPJOAqOR/Qr4dc=; fh=m+j6xt0PCsyx/weGxpF4mHfplWG53lsx2uMdr0Gns3k=; b=uWd6NXruhFnI1TGr27ZjAJfU7QfML7/LnQhQkgmvqnxipyg0rdISpryekvngGH+ai+ 0rOK5qsqHDCje83wDH3bAo8B5xaIQOE/UzV0sQZfBl2zxMT7ST8DSA6u7X13+StryRQ4 7mN0wEvTZakbi8YyxyHcU6T/DYl3gHPfWoAFRT11Ag65zubro/C2Ou5NQFaSG4pBswAd Ff/tfZOdCPWXJH46/s3ujsfLmd5IZe3ycXIUi3O3cKWFbkE7/4VmH9CT33qtZkl7ZW8N GktPQGkA45Yrvj2/+pksNuCcBBGecJJau/CgF5oFWQ98tpMBZSeFIE/v6h1TktiGNBts 648Q== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=nwGhBag7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from lipwig.vger.email (lipwig.vger.email. [23.128.96.33]) by mx.google.com with ESMTPS id h1-20020a655181000000b0056546b5fef4si1649498pgq.232.2023.10.18.00.47.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Oct 2023 00:47:46 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) client-ip=23.128.96.33; Authentication-Results: mx.google.com; dkim=pass header.i=@intel.com header.s=Intel header.b=nwGhBag7; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 23.128.96.33 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=intel.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 6635A80310CE; Wed, 18 Oct 2023 00:47:44 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344596AbjJRHrd (ORCPT <rfc822;zwp10758@gmail.com> + 23 others); Wed, 18 Oct 2023 03:47:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41136 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229912AbjJRHra (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Wed, 18 Oct 2023 03:47:30 -0400 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EDCB2100; Wed, 18 Oct 2023 00:47:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697615249; x=1729151249; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=h4XVZHdvlqX9EhvYBVM6hRqpG6MuHehCUD0ZGVf9mxo=; b=nwGhBag7Umuj9lVlVHubj3mtv7m/PAUCe1g1045dsouKOV6i7Rl3jvVM nv+8yCxWRHgUlfv44L3jrCMm64f4NVjWu4QXbhDf1VMnOmnamR/x+6Bar RFxQlP9CvTrkrCrI5bGg8Zj7s2RRYY9escvec9ofvNvJBbINIF2e3OLct 8muC2TP0HwHGEsYRDdR+ZWHJ5fKXVXHcMDioYFao2Rvu+4QSz+f9LRNUU ieV22hAXO2OszHPx18NRCHM7PxE11LAzDuXkWY5QaTf9YemgNG4fW66LR 9/GJAIMoasLZ22GsDMbzJ3TXbb/JZd2CTXbAxkRB81sSLPJ/0Cn9ug0hn g==; X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="388823972" X-IronPort-AV: E=Sophos;i="6.03,234,1694761200"; d="scan'208";a="388823972" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Oct 2023 00:47:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="785795153" X-IronPort-AV: E=Sophos;i="6.03,234,1694761200"; d="scan'208";a="785795153" Received: from 001b21e7c6a0.jf.intel.com ([10.165.56.54]) by orsmga008.jf.intel.com with ESMTP; 18 Oct 2023 00:47:27 -0700 From: Chen Hu <hu1.chen@intel.com> To: miklos@szeredi.hu, amir73il@gmail.com Cc: malini.bhandaru@intel.com, tim.c.chen@intel.com, mikko.ylinen@intel.com, lizhen.you@intel.com, vinicius.gomes@intel.com, hu1.chen@intel.com, linux-unionfs@vger.kernel.org, linux-kernel@vger.kernel.org Subject: ovl: ovl_fs::creator_cred::usage scalability issues Date: Wed, 18 Oct 2023 00:45:53 -0700 Message-Id: <20231018074553.41333-1-hu1.chen@intel.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Wed, 18 Oct 2023 00:47:44 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1780078625848841549 X-GMAIL-MSGID: 1780078625848841549 |
Series |
ovl: ovl_fs::creator_cred::usage scalability issues
|
|
Commit Message
Chen Hu
Oct. 18, 2023, 7:45 a.m. UTC
*Problem* ovl_permission() checks the underlying inode with the credential of mounter. The cred, struct ovl_fs::creator_cred, is somewhat global per overlayfs superblock. Performance degrades when concurrency increases on the cred, to be specific, on ovl_fs::creator_cred::usage. This happens when there are massive file access inside container, especially on SoC with many cores. With Linux 6.6.0-rc2, we run a web workload container on Intel 4th Xeon Sapphire Rapids which has 56 cores. Perf reports that 5.7% (2.50% + 1.87% + 1.33%) CPU stall in overlayfs: Self Command Shared Object Symbol 2.50% foo [kernel.vmlinux] [k] override_creds 1.87% foo [kernel.vmlinux] [k] revert_creds 1.33% foo [kernel.vmlinux] [k] generic_permission On Soc with more than 100 cores, we can even observe ~30% CPU stalled! This scalability issue is caused by two factors: 1) Contention on creator_cred::usage creator_cred::usage is atomic_t and is inc/dec atomically during every file access. So HW acquires the corresponding cache line exclusively. This operataiton is expensive and gets worse when contention is heavy. Call chain: ovl_permission() -> ovl_override_creds() -> override_creds() -> get_new_cred() -> atomic_inc(&cred->usage); ovl_permission() -> revert_creds() -> put_cred() -> atomic_dec_and_test(&(cred)->usage)) 2) False sharing `perf c2c` shows false sharing issue between cred::usage and cred::fsuid. This is why generic_permission() stalls 1.33% CPU in above perf report. ovl_permission() updates cred::usage and it also reads cred::fsuid. Unfortunately, they locate in the same cache line and thus false sharing occurs. cred::fsuid is read at: ovl_permission() -> inode_permission() -> generic_permission() -> acl_permission_check() -> current_fsuid() *Mitigations we tried* We tried several mitigations but are not sure if it can be a fix or just workaround / hack. So we report this and want to have some discussions. Our mitigations aims to eliminate the contention on creator_cred->usage. Without contention, the false sharing will be tiny and no need to handle. The mitigations we tested are: 1) Check underlying inode once in its lifetime. OR 2) In ovl_permission(), copy global creator_cred to a local variable to avoid concurrency. With any mitigations above, CPU will not stall on overlayfs. Paste mitigation 1 below. From 472bd18eaabcde0d41e450f556691151b1bdb64e Mon Sep 17 00:00:00 2001 From: Chen Hu <hu1.chen@intel.com> Date: Fri, 1 Sep 2023 15:03:28 +0800 Subject: [RFC PATCH] ovl: check underlying upper inode once in its lifetime ovl_permission() checks the underlying inode with the credential of mounter. The cred, struct ovl_fs::creator_cred, is global per overlayfs superblock. Performance degrades when concurrency increases on the cred, to be specific, on ovl_fs::creator_cred::usage. This patch (or hack to some extent) checks underlying upper inode once in its lifetime, eliminates the cache line contention on creator_cred::usage and gets 40%+ perf improvement on a 128 cores CPU. CAUTION: this may compromise the file permission check. Need to talk with overlayfs experts. Signed-off-by: Chen Hu <hu1.chen@intel.com> --- fs/overlayfs/inode.c | 5 ++++- fs/overlayfs/overlayfs.h | 1 + 2 files changed, 5 insertions(+), 1 deletion(-)
Comments
On Wed, Oct 18, 2023 at 10:47 AM Chen Hu <hu1.chen@intel.com> wrote: > > *Problem* > ovl_permission() checks the underlying inode with the credential of mounter. > The cred, struct ovl_fs::creator_cred, is somewhat global per overlayfs > superblock. Performance degrades when concurrency increases on the cred, to be > specific, on ovl_fs::creator_cred::usage. > > This happens when there are massive file access inside container, especially > on SoC with many cores. With Linux 6.6.0-rc2, we run a web workload container > on Intel 4th Xeon Sapphire Rapids which has 56 cores. Perf reports that 5.7% > (2.50% + 1.87% + 1.33%) CPU stall in overlayfs: > Self Command Shared Object Symbol > 2.50% foo [kernel.vmlinux] [k] override_creds > 1.87% foo [kernel.vmlinux] [k] revert_creds > 1.33% foo [kernel.vmlinux] [k] generic_permission > > On Soc with more than 100 cores, we can even observe ~30% CPU stalled! > > This scalability issue is caused by two factors: > 1) Contention on creator_cred::usage > creator_cred::usage is atomic_t and is inc/dec atomically during every file > access. So HW acquires the corresponding cache line exclusively. This > operataiton is expensive and gets worse when contention is heavy. > Call chain: > ovl_permission() > -> ovl_override_creds() > -> override_creds() > -> get_new_cred() > -> atomic_inc(&cred->usage); > > ovl_permission() > -> revert_creds() > -> put_cred() > -> atomic_dec_and_test(&(cred)->usage)) > > 2) False sharing > `perf c2c` shows false sharing issue between cred::usage and cred::fsuid. > This is why generic_permission() stalls 1.33% CPU in above perf report. > ovl_permission() updates cred::usage and it also reads cred::fsuid. > Unfortunately, they locate in the same cache line and thus false sharing > occurs. cred::fsuid is read at: > ovl_permission() > -> inode_permission() > -> generic_permission() > -> acl_permission_check() > -> current_fsuid() > > *Mitigations we tried* > We tried several mitigations but are not sure if it can be a fix or just > workaround / hack. So we report this and want to have some discussions. > > Our mitigations aims to eliminate the contention on creator_cred->usage. > Without contention, the false sharing will be tiny and no need to handle. The > mitigations we tested are: > 1) Check underlying inode once in its lifetime. But the check is against a specific permission mask. Your patch caches the result of the permission check of a specific mask and uses it as the result to return for any mask. > OR > 2) In ovl_permission(), copy global creator_cred to a local variable to > avoid concurrency. > This sounds a bit risky, but maybe it can work. If you want to create a local copy of creds, I think that the fact that this is a local copy should be expressed in flags like cred->non_rcu. put_cred() should be aware of this flag and avoid calling __put_cred(). The local copy should be initialized with usage 1 by copying creator_cred and we need to have an assertion if cred->usage drops to 0. Also, ofs->creator_cred itself should be marked as a "read-only copy" of credentials and we should add assertions to make sure that no code calls get_cred() on a read-only copy. The ovl_override_cred() function should take a local cred variable to use the copy method for any access to ofs->creator_cred, not only in ovl_permissions(). ovl_override_creds() should be coupled with ovl_revert_creds() which also takes the local var as argument and also asserts that the local copy usage is 1. We can maybe take the opportunity to DEFINE_GUARD() for an ovl_cred struct and use it in many of the overlayfs methods. And maybe I am missing something and this cannot be done or there is a much easier way to solve the problem. Thanks, Amir.
diff --git a/fs/overlayfs/inode.c b/fs/overlayfs/inode.c index 83ef66644c21..62ec99316c7a 100644 --- a/fs/overlayfs/inode.c +++ b/fs/overlayfs/inode.c @@ -307,7 +307,7 @@ int ovl_permission(struct mnt_idmap *idmap, * with creds of mounter */ err = generic_permission(&nop_mnt_idmap, inode, mask); - if (err) + if (err || ovl_test_flag(OVL_FASTPERM, inode)) return err; old_cred = ovl_override_creds(inode->i_sb); @@ -318,6 +318,9 @@ int ovl_permission(struct mnt_idmap *idmap, mask |= MAY_READ; } err = inode_permission(mnt_idmap(realpath.mnt), realinode, mask); + if (err == 0 && upperinode) + /* This gets set once for the upper inode lifetime */ + ovl_set_flag(OVL_FASTPERM, inode); revert_creds(old_cred); return err; diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h index 9817b2dcb132..5b71aaa8f77c 100644 --- a/fs/overlayfs/overlayfs.h +++ b/fs/overlayfs/overlayfs.h @@ -53,6 +53,7 @@ enum ovl_inode_flag { OVL_CONST_INO, OVL_HAS_DIGEST, OVL_VERIFIED_DIGEST, + OVL_FASTPERM, }; enum ovl_entry_flag {