From patchwork Mon Feb 6 07:47:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53055 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105301wrn; Sun, 5 Feb 2023 23:50:31 -0800 (PST) X-Google-Smtp-Source: AK7set+0b/uNdYFAN4VuJLwQKKFXHe3GfaZ+dEH0WgLMnTES8UtikkUdLxp/P8sqMbARo6cUazbV X-Received: by 2002:a17:907:8e93:b0:881:f614:44ed with SMTP id tx19-20020a1709078e9300b00881f61444edmr24622138ejc.30.1675669831294; Sun, 05 Feb 2023 23:50:31 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669831; cv=pass; d=google.com; s=arc-20160816; b=sMjOIo5FU1yRRpHQj5NM6lRNQ83PNCu7WqZZpAy28uyHrYPvlpKniGqEvJENFijghL vdPV2GnfYBEba6MTexMGDqLWr8UdPGYraFkCLxa4ixxEVOvfgZfTD72jMh/jalvAaL1n buWZalbC/KCyn4POYTeOyLgBK0cfvNXFhUMukVjSCp0MIwDEsmrLducJ3S8OUCO2CTd1 j7KldfEJyW9BzdfIhXcwET0IJUduF2UkqWxSDTSG7+m1WKrlwDMqaqwe1pxTFQJ0sCd/ m8SiZxy92aVdgsjDwrpymSNctV+cAVRgJJfkSoUQhl/JvYarqABnhkJeQbscA1hQzQKh olaA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=xC0ascId9hC0FQ0mqA9gGlsD4Tec21nIaiiKW1ANuos=; b=wUYi/5OmAybvS/nFiXNDtHXfjwMalbrkO4yVHblf5qpSs6HmMOCBgl0Mo24ZAbZAnx C4TLEe6SXvrmvRTCMZWJoMOIG5sn4IdWygzo1QOs9smAgVvFtvsadA9bWcVeMw7dC0UW YtlA8KsXjnIz+yNoltSwoHdJ8boQFYYieXBVzT8AtbRU6odLl98DycKbXOGX1/VQH8dO OLfScSZ8LeGRGprTSyVI2OpDTehtGdgYMTFSmcpOZZFjurTqNFx5IZoG38YrnftoOY0H CSjsK/z09saFXyJ3/xyVe5iXWAZ6/XvH4xGPZYyVApmWcFWuSxLZP/p06rSLjFrBadGK ZuFw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=A+FFKqJY; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n13-20020a17090673cd00b0087f1daeb106si15621049ejl.855.2023.02.05.23.50.07; Sun, 05 Feb 2023 23:50:31 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=A+FFKqJY; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229897AbjBFHsc (ORCPT + 99 others); Mon, 6 Feb 2023 02:48:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229861AbjBFHs1 (ORCPT ); Mon, 6 Feb 2023 02:48:27 -0500 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2041.outbound.protection.outlook.com [40.107.236.41]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4175C18A9B; Sun, 5 Feb 2023 23:48:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aLsp8OkfnvgUX+LaCm1Dc+hdKUsiL3axpftCKUHDm1OQkZpSVWKd41QJpSxq9ko55snTctoS/hMmNjRr+itI3WbB3Q3AzmA6538HBTiXXlDO0S3AaJVzGqI3eu5++vIHZ28QtTlNa+k+dSpLAd+wLCU2EFbS1oHBSxKOk5zueaVmKcpYV50EsgXkylZrJGkkEePSOvrV9I/Bsj2oa8vxDvjsc+kGnGm5e5xycHnK4lkQOFyzYKvS14dWfW/63g0BN0S08F1XDTOjdIwd3UKL6vwTI7+T6eKerf0GqbZqLxKFY5wI1baz3a/2JfDv/WRas+jgQG8iAAdpSy/yR16sQg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=xC0ascId9hC0FQ0mqA9gGlsD4Tec21nIaiiKW1ANuos=; b=e2vefkFqyA3sW3pHTN46sHlZlBtRrXMyAf4In3Tgp7gdMSg10ziiTG5iwAIpdN+Nv9Oog6rQy/dMDY2dO6w3grxgRGp9gRCGkURF3SWuxdZ5pDPWPjWfF80aA0GfSO+a7RMpi+RETUh04airuQKTuoiT7NdVEbit7kL7o+ESiApp+a+LNO8scmfZwfOWs/wTgyVxb6qqRBkFNcpqOIFrxMM0d+LT8KWcaV3OeF5iSjB1ePBi+f8eg+llIHnurirOEwoKyueOezpUAMsA1l275ngrS9mwcE1qemLfX/zE+eYf570QvCXAfD87djcKO8wu77Flv39WV5CzVvqq2LSVCQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xC0ascId9hC0FQ0mqA9gGlsD4Tec21nIaiiKW1ANuos=; b=A+FFKqJYzlIAVESoxZN19dbMh3tOUmAXg71BZGbiuV4HgT2WwNhXvo9H+EqVd1hYY7W1iy9x2eJ4H3XwaGr2jO5d9GIETU1TZlbpbaiAHeuE5em9pnJ4MNG1/6OKDbNRjYa1RYLSWzUTRajUam0hS8av8ruJv6K2TuFY+X4QQbLJGSDzZBmJnFND8gFdvARudQbQbxKxMrYh7Rhm7JbL033fzE2OtsmZD+IYJ1sO5/fcSIk019OYzNFb21GL+KvbDlEwCq+4mkJl2w71KvdFVvoO4dgzK/vrr1yASUdmbDEzRitZTTs59gTXhfNc79Z05lPOjSUH1lVBxsi65AXCxA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by CY8PR12MB7097.namprd12.prod.outlook.com (2603:10b6:930:63::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.34; Mon, 6 Feb 2023 07:48:21 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:48:21 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , linuxppc-dev@lists.ozlabs.org, linux-fpga@vger.kernel.org, linux-rdma@vger.kernel.org, virtualization@lists.linux-foundation.org, kvm@vger.kernel.org, netdev@vger.kernel.org, io-uring@vger.kernel.org, bpf@vger.kernel.org, rds-devel@oss.oracle.com, linux-kselftest@vger.kernel.org Subject: [PATCH 01/19] mm: Introduce vm_account Date: Mon, 6 Feb 2023 18:47:38 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SY6PR01CA0010.ausprd01.prod.outlook.com (2603:10c6:10:e8::15) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|CY8PR12MB7097:EE_ X-MS-Office365-Filtering-Correlation-Id: f1f32d26-9f86-4df0-cce9-08db081684ae X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: d9R20AXDYdOn6o7fqjRKlbn88gfGEhrLB0K3l/C7lo/hVvC76CZp4upyM7QvG93ysjuMVTpq/EYzK3rKe+gwBlyHxY43+Hqd/7szzD+TETPh4YIz49dsl8Pco7UcHmIYR1jDb462M2A51jpFSyZ6+sGx3FpEttvQxlIDeCWQn6283qmfDF4QeL0MGLkLR28DA19JymNK1RtyPkmfKZ09MLSCaFGJ3h12WGdvnJ5w7VGFIKFvz5stxIwwdIt0+gYERQCSlt2L26GRq9KV2GcYX8YE/OcLGZSrecKGGipuwqQm8Z7Ynh+TybkKzd+Y+ZdZKAXFYvSW5vAWUTT6n5GHEu+1RV85ersB37hzKHCRBbBjQk2dt2lN4HS7GS390bwhNfyVDFRZjE/rWx/Sf6l81vQ0i636fN5go+YWW9m/Cs2GeJ2kUXo2o5oo3aZ6hLhWLaDc2cMg8f+dO2hQNOm31Wmf2fNIgwK112xrMKjJwP7uMzzPMRVJHnTwtouclvVkulpeSj9ZswUbyFtJMziahx/BcMaIlFXt789djlRAEZlvA5Dolz1bNDtuZdc6P5ZWA0RBVH2YzDo/Bl5+FviW2ege3AA0jp0C20IRSY3uXiO/9yQxMIEjpPmWqNjE32CPYcudf9GcWGjjbN/MTX2aSQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(39860400002)(396003)(136003)(346002)(376002)(451199018)(38100700002)(36756003)(6506007)(6666004)(6512007)(26005)(186003)(478600001)(86362001)(6486002)(8676002)(66476007)(83380400001)(66946007)(2616005)(66556008)(4326008)(316002)(54906003)(7416002)(2906002)(8936002)(5660300002)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +Q21fH/s4Vq+Fkqt0pEjMYKv/M0hPKFwEmzpwjecwZ8dEKYu6PTd6B1USsF/VIewWV1Iy7hIjEqgKdvNy9XCHb6UCEjDlG2t6luAcyANcpffEfiNXj358C55PHbMpnatLbnb82LSxL2sxcLHQRbRVzZM4h6g/3499Op4qz4eIG6rIq3sP9GRrHx6Tqds5OHj2oy53ycVE3dECU0kComR0bp0U7UTdqpK3MVIK9uKGtcSo4EkCbqps8ws+uJxEI5RpRZ5p7iBSlY1ReOPfXul3U6G9MPEBkweNR5Wo4HcAc+BjX1x0G6BMm8BJAXdvBORkAl4CFmowC8Z7E5ZKBPUT5ioGLu+OjJVqhEa8hveZC2M2WJRwoyMFTyjy2k0f0mQRnvj+PbEIX9nx0SvfTjUFABqKwU2Z/1Fm7Xv9WusQn3HTFnsyEhVU+QqZxtf77029QaAn7uZJvEiblZXE7TUUdcHfTdyjgSVOCgszweWtFLTMBCKcBoMx6yuvAiU7de3NbVmKBBV8LRfJqjM7RFqUHnRfyvtK1pKsfXe4zb2OgYq63aCbk2+0hF6r+nw/5hVzf3oLt+yiapdrUwdK204DQoAk7KTXc0CUaKI+YNGo7bKPIG5cPGui7kSYT8ZHQawvSD5trq1ErBFkeBBXADoptjHtDfi3Pg7Ff9bjSqv6u42noCshGmM1eBj496JFjjjXcJhMESM0UuKS0F2TKDR4mzK8GH/muRsjSoL0JfMn014g6VRrKHh+a22/xsmKrh3u4HljrqaMeDB4He5JHwM8s6+kyeU3CkKgOVeyN4xgfwHQ6JVUOEPzp4X3pKkge5V1WIX3TkAs/DwB8n2QfUb6ruFRychXQVgfdrAn7N5YcN5N2pPqmatRWTthyyJoTWcZlP35J8yKRhYIQwQbIQ2MaYqf5KLWk134Bzhxzo196snfUec2RW4rFit/7Kw9A1emZZT6nrbdKxVMb2x4rLRIxM6jc3snl4WQrocd8r+Jex/vzsh4zSwhhQK/ZeitOfxrewOqeLpG4hAG6rV+1VYTwlTQSKpvx/dthFLHcaIWOWtpB7h22PdCdxl4ctObXm+w4mkFr6WPaoITf8lxJbTwuYTIvy/Mq7HbwmfN07KQI8tkrtcRQ3iewm7r0HFtWQlYXGN5XBXsek/Aegzb5gF66fpZ5YIsbXbHelBwOnZo3v5sLifuZO8AKQCMFaIOK1Twc761o7A+q72PinuISoun30BXs8bsrh8ysRQpqLPQZ+xf48dz4zs6W4s9UdVoaLqG6poioJSXQUKEZX9xL5SLtn6VQrm9k6GtYtOTt9BqPFEyEwU4aQb7bAr9tjtFNFQyul65a2UajRWmTk8oOjmPQjG+0Na8AIoKRBXDTYolD25NtClc2GWUtxdF/r0g6WMambOrmoQN6PDvnNKiH2Y3LfxvCe/hoG96VD9Gr2SfaI7ut/DvIDGvS6cWJREA9GsI7fQpkHJD5us1nkC88mmR73jv99T+Zys5Jd8GStZTQDKhn9QFylG+VJvDSWAbh1nlVL/o3r8ujdUkBp+ttAnY5R83qnC7BMvidrwhW6C5R1H2kqzZKGGZ+24X44EGeMu X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: f1f32d26-9f86-4df0-cce9-08db081684ae X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:48:21.2970 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VKLsBrolhjVwe4eRftYA1HkxxQ/SlJh07YGyjddLzMudWGCcOJdVx+UWim/wuIqJ+Bp6tmOMG+wJxI7NtMvsPg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7097 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067169006334848?= X-GMAIL-MSGID: =?utf-8?q?1757067169006334848?= Kernel drivers that pin pages should account these pages against either user->locked_vm and/or mm->pinned_vm and fail the pinning if RLIMIT_MEMLOCK is exceeded and CAP_IPC_LOCK isn't held. Currently drivers open-code this accounting and use various methods to update the atomic variables and check against the limits leading to various bugs and inconsistencies. To fix this introduce a standard interface for charging pinned and locked memory. As this involves taking references on kernel objects such as mm_struct or user_struct we introduce a new vm_account struct to hold these references. Several helper functions are then introduced to grab references and check limits. As the way these limits are charged and enforced is visible to userspace we need to be careful not to break existing applications by charging to different counters. As a result the vm_account functions support accounting to different counters as required. A future change will extend this to also account against a cgroup for pinned pages. Signed-off-by: Alistair Popple Cc: linux-kernel@vger.kernel.org Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-fpga@vger.kernel.org Cc: linux-rdma@vger.kernel.org Cc: virtualization@lists.linux-foundation.org Cc: kvm@vger.kernel.org Cc: netdev@vger.kernel.org Cc: cgroups@vger.kernel.org Cc: io-uring@vger.kernel.org Cc: linux-mm@kvack.org Cc: bpf@vger.kernel.org Cc: rds-devel@oss.oracle.com Cc: linux-kselftest@vger.kernel.org --- include/linux/vm_account.h | 56 +++++++++++++++++- mm/util.c | 127 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 183 insertions(+) create mode 100644 include/linux/vm_account.h diff --git a/include/linux/vm_account.h b/include/linux/vm_account.h new file mode 100644 index 0000000..b4b2e90 --- /dev/null +++ b/include/linux/vm_account.h @@ -0,0 +1,56 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_VM_ACCOUNT_H +#define _LINUX_VM_ACCOUNT_H + +/** + * enum vm_account_flags - Determine how pinned/locked memory is accounted. + * @VM_ACCOUNT_TASK: Account pinned memory to mm->pinned_vm. + * @VM_ACCOUNT_BYPASS: Don't enforce rlimit on any charges. + * @VM_ACCOUNT_USER: Account locked memory to user->locked_vm. + * + * Determines which statistic pinned/locked memory is accounted + * against. All limits will be enforced against RLIMIT_MEMLOCK and the + * pins cgroup if CONFIG_CGROUP_PINS is enabled. + * + * New drivers should use VM_ACCOUNT_USER. VM_ACCOUNT_TASK is used by + * pre-existing drivers to maintain existing accounting against + * mm->pinned_mm rather than user->locked_mm. + * + * VM_ACCOUNT_BYPASS may also be specified to bypass rlimit + * checks. Typically this is used to cache CAP_IPC_LOCK from when a + * driver is first initialised. Note that this does not bypass cgroup + * limit checks. + */ +enum vm_account_flags { + VM_ACCOUNT_USER = 0, + VM_ACCOUNT_BYPASS = 1, + VM_ACCOUNT_TASK = 1, +}; + +struct vm_account { + struct task_struct *task; + struct mm_struct *mm; + struct user_struct *user; + enum vm_account_flags flags; +}; + +void vm_account_init(struct vm_account *vm_account, struct task_struct *task, + struct user_struct *user, enum vm_account_flags flags); + +/** + * vm_account_init_current - Initialise a new struct vm_account. + * @vm_account: pointer to uninitialised vm_account. + * + * Helper to initialise a vm_account for the common case of charging + * with VM_ACCOUNT_TASK against current. + */ +static inline void vm_account_init_current(struct vm_account *vm_account) +{ + vm_account_init(vm_account, current, NULL, VM_ACCOUNT_TASK); +} + +void vm_account_release(struct vm_account *vm_account); +int vm_account_pinned(struct vm_account *vm_account, unsigned long npages); +void vm_unaccount_pinned(struct vm_account *vm_account, unsigned long npages); + +#endif /* _LINUX_VM_ACCOUNT_H */ diff --git a/mm/util.c b/mm/util.c index b56c92f..d8c19f8 100644 --- a/mm/util.c +++ b/mm/util.c @@ -23,6 +23,7 @@ #include #include #include +#include #include @@ -431,6 +432,132 @@ void arch_pick_mmap_layout(struct mm_struct *mm, struct rlimit *rlim_stack) #endif /** + * vm_account_init - Initialise a new struct vm_account. + * @vm_account: pointer to uninitialised vm_account. + * @task: task to charge against. + * @user: user to charge against. Must be non-NULL for VM_ACCOUNT_USER. + * @flags: flags to use when charging to vm_account. + * + * Initialise a new uninitialised struct vm_account. Takes references + * on the task/mm/user/cgroup as required although callers must ensure + * any references passed in remain valid for the duration of this + * call. + */ +void vm_account_init(struct vm_account *vm_account, struct task_struct *task, + struct user_struct *user, enum vm_account_flags flags) +{ + vm_account->task = get_task_struct(task); + + if (flags & VM_ACCOUNT_USER) + vm_account->user = get_uid(user); + + mmgrab(task->mm); + vm_account->mm = task->mm; + vm_account->flags = flags; +} +EXPORT_SYMBOL_GPL(vm_account_init); + +/** + * vm_account_release - Initialise a new struct vm_account. + * @vm_account: pointer to initialised vm_account. + * + * Drop any object references obtained by vm_account_init(). The + * vm_account must not be used after calling this unless reinitialised + * with vm_account_init(). + */ +void vm_account_release(struct vm_account *vm_account) +{ + put_task_struct(vm_account->task); + if (vm_account->flags & VM_ACCOUNT_USER) + free_uid(vm_account->user); + + mmdrop(vm_account->mm); +} +EXPORT_SYMBOL_GPL(vm_account_release); + +/* + * Charge pages with an atomic compare and swap. Returns -ENOMEM on + * failure, 1 on success and 0 for retry. + */ +static int vm_account_cmpxchg(struct vm_account *vm_account, + unsigned long npages, unsigned long lock_limit) +{ + u64 cur_pages, new_pages; + + if (vm_account->flags & VM_ACCOUNT_USER) + cur_pages = atomic_long_read(&vm_account->user->locked_vm); + else + cur_pages = atomic64_read(&vm_account->mm->pinned_vm); + + new_pages = cur_pages + npages; + if (lock_limit != RLIM_INFINITY && new_pages > lock_limit) + return -ENOMEM; + + if (vm_account->flags & VM_ACCOUNT_USER) { + return atomic_long_cmpxchg(&vm_account->user->locked_vm, + cur_pages, new_pages) == cur_pages; + } else { + return atomic64_cmpxchg(&vm_account->mm->pinned_vm, + cur_pages, new_pages) == cur_pages; + } +} + +/** + * vm_account_pinned - Charge pinned or locked memory to the vm_account. + * @vm_account: pointer to an initialised vm_account. + * @npages: number of pages to charge. + * + * Return: 0 on success, -ENOMEM if a limit would be exceeded. + * + * Note: All pages must be explicitly uncharged with + * vm_unaccount_pinned() prior to releasing the vm_account with + * vm_account_release(). + */ +int vm_account_pinned(struct vm_account *vm_account, unsigned long npages) +{ + unsigned long lock_limit = RLIM_INFINITY; + int ret; + + if (!(vm_account->flags & VM_ACCOUNT_BYPASS) && !capable(CAP_IPC_LOCK)) + lock_limit = task_rlimit(vm_account->task, + RLIMIT_MEMLOCK) >> PAGE_SHIFT; + + while (true) { + ret = vm_account_cmpxchg(vm_account, npages, lock_limit); + if (ret > 0) + break; + else if (ret < 0) + return ret; + } + + /* + * Always add pinned pages to mm->pinned_vm even when we're + * not enforcing the limit against that. + */ + if (vm_account->flags & VM_ACCOUNT_USER) + atomic64_add(npages, &vm_account->mm->pinned_vm); + + return 0; +} +EXPORT_SYMBOL_GPL(vm_account_pinned); + +/** + * vm_unaccount_pinned - Uncharge pinned or locked memory to the vm_account. + * @vm_account: pointer to an initialised vm_account. + * @npages: number of pages to uncharge. + */ +void vm_unaccount_pinned(struct vm_account *vm_account, unsigned long npages) +{ + if (vm_account->flags & VM_ACCOUNT_USER) { + atomic_long_sub(npages, &vm_account->user->locked_vm); + atomic64_sub(npages, &vm_account->mm->pinned_vm); + } else { + atomic64_sub(npages, &vm_account->mm->pinned_vm); + } +} +EXPORT_SYMBOL_GPL(vm_unaccount_pinned); + +/** * __account_locked_vm - account locked pages to an mm's locked_vm * @mm: mm to account against * @pages: number of pages to account From patchwork Mon Feb 6 07:47:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53061 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105654wrn; Sun, 5 Feb 2023 23:51:56 -0800 (PST) X-Google-Smtp-Source: AK7set86eQSE7s7+AHih/thx0GAl2GUUT24vrOT98mzWgj6u0KSElk6QJTe+nBN2KI3iX8bCmg6i X-Received: by 2002:a05:6402:2407:b0:49e:4786:a0e2 with SMTP id t7-20020a056402240700b0049e4786a0e2mr13833333eda.14.1675669916429; Sun, 05 Feb 2023 23:51:56 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669916; cv=pass; d=google.com; s=arc-20160816; b=IGlwLp4Pyh3rVRCG2Sn6jV3gY/DhhwUwONJmu3wDrPqwqvFQm2LKq/wvj8t7ks4QC9 jtt0qb2waEN4JS9ZAHkx/m9aaBDLmYNvkAgaD27CDxlTOkYSGCi8LYEKt8gAcv6VKUa8 0lCFEaFpFIcuv50p7/Ev1pRl7pGw2LbwIIo4ILsemVG//bfvC4sMrghnocTMcVi7bhXy 0TLcvxSUuj7jKO8Kebw/wdqOrUl4OxL2QiyKdvTxQdpiDYCFVKGt8B56eGKyhRH1r0ah BcMVmTlX6w8HeFlWdheIaZ85hYw6ueS+8sdAMe8OzkMqj70brnyYQLz67ikyZBa2c6eY sk/A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=acjFfLRivjpU9Lg3kvi0rHomIWvWZtcIDwMkhY2Ky00=; b=K3GT0P8NEsgvIMjTna60AuJxzoxT1HOW3bLD3G0dqoJU6mTtS8Q0I3kiRuDVGzSnk8 eUbFEMhweRs5/Y2UbWsdw8YjKMGXGjRXnEDL0j1oGhzNZee8EUjEZsUcVEtu3+a/x5Yp mmYM6aN2WAAM4ieVuI11S6b4d1KldvHyksUuhaIhkWArXPsTfYFdt3u6lKDsRofUrwSU aXlH/PDzS2NjMIJJnWLOoh2ZBSjQ0oTPB6cQ7TRdL0oFLlayzi2GI32DTwco9VoPUfqE giMP12fuNV2XOcLPp3h/U1IqDOtLh8arXfUJsDAplrnUjkCnUbJ40jbsbxcpAeYTTE3c 63Dg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=fHCzouLa; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u18-20020a05640207d200b004aaa4fdb7e7si5985846edy.15.2023.02.05.23.51.30; Sun, 05 Feb 2023 23:51:56 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=fHCzouLa; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229903AbjBFHtG (ORCPT + 99 others); Mon, 6 Feb 2023 02:49:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55498 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229879AbjBFHsh (ORCPT ); Mon, 6 Feb 2023 02:48:37 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2051.outbound.protection.outlook.com [40.107.220.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2F6618AA0; Sun, 5 Feb 2023 23:48:30 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=g1HGJAAb6x79B/2uO5UNl7Wjc2N7fO6FI2Dh1hbWDDlv+NO5uTH6mAdcm+er4k8nzu8P0Kaskg3FjHIudIEyRFfVc9Xm0Vk4e0HCDKF0MnkAmGYeoOz1TxF9jUHoP5rWgtbNqzdKjT/nhBnQVpGpsOmo11SmN+OhWMVQc8a3pxFwCVuSBKUFpmuYEYL9tW9J4OqExG8v9wf3KR1LoFdsLRxTuFo9mrXN5sxOPEk9l5jiCTs2QJNp66zeZI9ZeDpGsJCib950WH5sOGgsBpMI6sdq/39XLEmNcRQEo0wrXgYinIoFrAcUd1SZ9Tkb0u7ydaqjIuPUMKsr0WAVQ/xlwQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=acjFfLRivjpU9Lg3kvi0rHomIWvWZtcIDwMkhY2Ky00=; b=WvQHQQY6ns3n0KefoFCSMggyX/x7tUCFBPrbp9NdXBsKIpZia9IjWTqt3eSONUAX+yLUVf/f7jSzoAsBeMeFbcbxdyGcuCNe8fhii7LnJQwbEVhoQP9ez9OILgeKvxr7XE6uG1UX4f3wtfuyx7uueogMthclboyNaqd3qFFXp9gdFuU9z5OQmVn7sX41V3f9gtlgjEpjlE4qtYykdA3BQ0qzRDi9puFRt2fttZpYdTSlTHSwl2yUwb/g78fOqsXXPtwz0BfZPXlM7XilI1VnZOxEpgjccCLr2WPMYZSyjgS55En9q1eQ/dhi5wL5ZcrgPrKpLkHNz+1GhTQZnn1PQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=acjFfLRivjpU9Lg3kvi0rHomIWvWZtcIDwMkhY2Ky00=; b=fHCzouLaE5kk62Ua+XJqCu+1MTMXe9QNNft5c35YdS0i50bvUAWqQwwDrn6WVqaAYfc5Aidg/xzaQ5SFBBhZ4gA+1ACieJwCaudY6l1BzZjCaxKVgn4qTaBx9hGKlo3/y6J/vkzzjo2G9GpCRfcCOkH2sQmIVJC57BwZg+W0DjiRsab/SpX6BqJr5kYCky0e9uZliBjJYNKp4n5PYxO1SsvltINtqi+iaoiKq/hCDojR1uoXMTeQmZEY3+t1wjV97EZ0mdNWs5YOSNhOa1mvSuEUpJttEznQpB3KvsQ9L9ynFNXmo9SFjZFg14jddI/QyfyTAlVenv/bKHF6vnVsrw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by CY8PR12MB7097.namprd12.prod.outlook.com (2603:10b6:930:63::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.34; Mon, 6 Feb 2023 07:48:29 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:48:28 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , "Michael S. Tsirkin" , Jason Wang , kvm@vger.kernel.org, virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Subject: [PATCH 02/19] drivers/vhost: Convert to use vm_account Date: Mon, 6 Feb 2023 18:47:39 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYYP282CA0007.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:b4::17) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|CY8PR12MB7097:EE_ X-MS-Office365-Filtering-Correlation-Id: d3c58a87-391d-4ca5-0a89-08db08168933 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qrNBIKLnIaAVk8pRj+5fZdPHiWQLbiCyBMI2zKg2cn/Fb5SIPPTq4lnCgIO9odXsb8ca5CT7XeYfek6V6/WonuEcy5JdMn6sLVlCiVnJ8+Ov1rIIBf9gLZjZuBCjnvH/Wd6xnfauUrOnS+56ZmvqmsGiKMU6TZbflY5PyeUFiyx3IRDXixQZZIT5ym6ZQ7MKh8raEJ/LH1o+UP7NJoPzzuQ48Jb0ifrzKOsxekho/unsBaIl2VV/xLiygpn8AGWQxju5/jRKJnca49WShesGkRCwuOwffP4oE2yAHv1f8cozt4/Yv0H5Uz2ztFJoql8uXWgUwQRX6IUgRTwCE7tQQuLG+Qz4yuo5eO9UsiTTacmJ8Km9Z6QVPwNo0KCrtfKt5095/INsZiz3W/Z6LobWgqDJ54JjdrO5q7VngPFH4X8T9mvvUz8lrVAsBxtlxyWg+EpHxRy7IUxEZP5qoAKo6gPJP0O2ia04BzC700tnSx/zt72yOijmqvpn2vGzxW2oWQqmePxjQduwtPn9uh9Dk2DH6zAoNnmfSHc6IIHnLxiJZaS8qc/la4R7HFKCn5HbPPtBif8eAJAqs5LEH2cvuBbe9xD5r6x0dls19uIBKIJ8FQRsdEQfuF6JagQheOknCfPj5fxaLNnCmWp0X7T88g== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(39860400002)(396003)(136003)(346002)(376002)(451199018)(38100700002)(36756003)(6506007)(6666004)(6512007)(26005)(186003)(478600001)(86362001)(6486002)(8676002)(66476007)(83380400001)(66946007)(2616005)(66556008)(4326008)(316002)(54906003)(7416002)(2906002)(8936002)(5660300002)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: BcZpvnNwT9Gb780Y2kPdCwKodPcm4oSD6aoJb8it9MH7scy9VMdJ8eQmHUo/tYk3av3s2dxZOcg3ogvWJjgYJU4CmqrjLA6eULGYWsluLBw3iUNDcrusaQMgU/JBCYKX4CFq4sbi4garbnQxS/dBlAc5UWaBAEJv37P0BnLLCYvW/Dwx9ye4gQxLpwkc2n5jCYJeEc6WoZby6mWJO568dMYKg2KqFAI8FtEb6Bo9JoSvaYrJXHnMENDbbxrwkfTQtV8qXbCNBsDq1bG4y4mUzAcpT6ltL5rHPqvzBcfp9a0GvjXvPnCh1LX1l8HQUHGtYt64+hq7tVHXRKdpIcExFvOwAmgADX31MwxqDpZ1EsigM4KMtzPfDVx/vfmJvaGRTwbpGx0EbqDCfRpQ/NiwpLIUmNj2TLoK9Fgqx52KVlNmvkfN/l+nVudtM3k3zgo7LK0zFvA4Opm+lkj5nJUUHw5HsDR07JdM7STkTeJ6+JFp8GUKNp2S3lSzvg+LD0UHH+9NTqzGYQR1OwQmv9X9qGGWuHfFfRshcD81zL7CSri0B6r0PZn2Skm99ITXb4U7QwWYgs8Mvnu0nltCKavJuxcKRHUid7Kc7yBRK6mvMSgyp1Vs1mvQ+0wMs/inEdtR24LXqGfQ2aWUSH4B4rhEUuKmK5VFvCbXiIbnoW07w8zy+rE9MQ4V2KxbUqGETOP/tnKo7LAqgyPnAZ86Evukt1YLLGqhWC9r7V7i+mv4n/P7mgmDR9Jk1VLcWKqTWA7TOBQHOcEGIUkPABm1MrZhjL02MNgERX18/btiG1nGe6WpG4Hb/+zY3+a16GI9j88p3JSFcs5xMYdMUgaDQElzTqvfLZRqq8dtaxudsmCkpMPccM85x3bsyKBzyUhOP5MgnZ7/qkaCt9tFdWlBIzyV6B0DN/8vtjiFBwpFdpR4rj0IU8zBuus0sVu51D/E4eGnljmwG46lyuzyj3hbjrpiJxL9cB+tZiZKndr5GDKLeOr5sV1FGNUfePKzE7Zb3cLL2zqzvs68vmMFuSwXCjEGF0hiTUqmUeAceR2buVQ29OrYUgQL5hgOPrjdgQnhr0AOdQyUVu+ZxlHmB9rtkkghCCZvOqN+HZQ5L5Qfu1tumWUtGXKur+We/kQfOlcujaDorNtRdjDJ57GxX1e5QaE153VWR9tbzgD3JE9w4nww7MTRGrthmSwRkuN2D6gN6680FZRM0Px/D0U8S1b+pXvjU36m3/F7jAHDOiewSiudjzHu2kmF/NEChDPlDSFg2JjWnmtwXC4I2NBznXKXqG09yDjvXfhsuspBJRQA5AXnbiEbStPOeCzR5Cmb9xR7X17C9fQEW3gmKHacB9XA1q9yijtgrp3HSRHGuDWlC8M7HjTVWQR8WM/jbVCBXk7zyCwt3eX/Ij0h6a/HZhYTBUCfUWznODflv9lgUyMH4VvEt9Ic7Ks70kksherMhreAowx2RXIO+QQfJcDbFi90lmLWQnUx5vZMdmzU90pOXc8Mjtz2U3x5Ed8urI5Ni3+MqGIBprHYtdNtEDAXVR+dp3tWVhl0URpCAcBafVezXDC9KtVY3LizG/14x+HYNs6/KqUA X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: d3c58a87-391d-4ca5-0a89-08db08168933 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:48:28.8589 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XPSgdGbBaHcZDUXMyU+Av5M6xE1fuCzjJ6iTymGC1b4edw5OZwM9c0xmYlltTHFtDUBV871cdAahWmlacV+qeg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7097 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067258245595018?= X-GMAIL-MSGID: =?utf-8?q?1757067258245595018?= Convert vhost to use the new vm_account structure and associated account_pinned_vm() functions. This means vhost will start enforcing RLIMIT_MEMLOCK when a user does not have CAP_IPC_LOCK and fail the mapping request. Signed-off-by: Alistair Popple Cc: "Michael S. Tsirkin" Cc: Jason Wang Cc: kvm@vger.kernel.org Cc: virtualization@lists.linux-foundation.org Cc: netdev@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- drivers/vhost/vdpa.c | 17 ++++++++++------- drivers/vhost/vhost.c | 2 ++ drivers/vhost/vhost.h | 2 ++ 3 files changed, 14 insertions(+), 7 deletions(-) diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index ec32f78..d970fcc 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -716,7 +716,7 @@ static void vhost_vdpa_pa_unmap(struct vhost_vdpa *v, struct vhost_iotlb *iotlb, set_page_dirty_lock(page); unpin_user_page(page); } - atomic64_sub(PFN_DOWN(map->size), &dev->mm->pinned_vm); + vm_unaccount_pinned(&dev->vm_account, PFN_DOWN(map->size)); vhost_vdpa_general_unmap(v, map, asid); vhost_iotlb_map_free(iotlb, map); } @@ -780,10 +780,14 @@ static int vhost_vdpa_map(struct vhost_vdpa *v, struct vhost_iotlb *iotlb, u32 asid = iotlb_to_asid(iotlb); int r = 0; + if (!vdpa->use_va) + if (vm_account_pinned(&dev->vm_account, PFN_DOWN(size))) + return -ENOMEM; + r = vhost_iotlb_add_range_ctx(iotlb, iova, iova + size - 1, pa, perm, opaque); if (r) - return r; + goto out_unaccount; if (ops->dma_map) { r = ops->dma_map(vdpa, asid, iova, size, pa, perm, opaque); @@ -794,15 +798,14 @@ static int vhost_vdpa_map(struct vhost_vdpa *v, struct vhost_iotlb *iotlb, r = iommu_map(v->domain, iova, pa, size, perm_to_iommu_flags(perm)); } - if (r) { + if (r) vhost_iotlb_del_range(iotlb, iova, iova + size - 1); - return r; - } +out_unaccount: if (!vdpa->use_va) - atomic64_add(PFN_DOWN(size), &dev->mm->pinned_vm); + vm_unaccount_pinned(&dev->vm_account, PFN_DOWN(size)); - return 0; + return r; } static void vhost_vdpa_unmap(struct vhost_vdpa *v, diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index cbe72bf..5645c26 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -556,6 +556,7 @@ static void vhost_attach_mm(struct vhost_dev *dev) dev->mm = current->mm; mmgrab(dev->mm); } + vm_account_init_current(&dev->vm_account); } static void vhost_detach_mm(struct vhost_dev *dev) @@ -569,6 +570,7 @@ static void vhost_detach_mm(struct vhost_dev *dev) mmdrop(dev->mm); dev->mm = NULL; + vm_account_release(&dev->vm_account); } /* Caller should have device mutex */ diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index d910910..b2434dd 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -14,6 +14,7 @@ #include #include #include +#include struct vhost_work; typedef void (*vhost_work_fn_t)(struct vhost_work *work); @@ -144,6 +145,7 @@ struct vhost_msg_node { struct vhost_dev { struct mm_struct *mm; struct mutex mutex; + struct vm_account vm_account; struct vhost_virtqueue **vqs; int nvqs; struct eventfd_ctx *log_ctx; From patchwork Mon Feb 6 07:47:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53053 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105257wrn; Sun, 5 Feb 2023 23:50:21 -0800 (PST) X-Google-Smtp-Source: AK7set/1H0pVgx6Z6kwBWjji+3nXH+YLHRHIlRGPffQyVmdRxfIZX16jrUMChbWvetPbK0JaU7qU X-Received: by 2002:a05:6402:d4a:b0:4aa:c337:7eb2 with SMTP id ec10-20020a0564020d4a00b004aac3377eb2mr192353edb.8.1675669821696; Sun, 05 Feb 2023 23:50:21 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669821; cv=pass; d=google.com; s=arc-20160816; b=sSMzplqFkbozn+yh5DkfPxF2OlE/XCaoKGNn1Q2qu34Lgaq4o08xni1jgnifLPcrYd 0POr64TQ47G+32e2AS/uwYe+8xzpTxXecyFkpSW5YW01LMb65M8nB8WVgEr/hIqBVk5J FF7f+n3X84ZG28kwGJ4S+Sx4/9WaHQljEImcrH8lS8iu3C0H/hRIamJIdmmvaftu874d Mm5i1rOvXDh8RZSjoejcINiDnv2SrYB18Vx2OXzsgqOJ/IYAkzrdKVFxnDgpMr8uZ8jv GGp5PnJ8k580V7SDyLNJBc8j6XCryZfHxZlfOSwJR9efPo1vF9Yn+vMNLhOxYleXSm5g KYLA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=LhgVGBn1pGz/qFGyTs5jsxzxhZjuaYEqMKugHfht1g8=; b=nlULsG5wNqNNQE7nwzY9QZfh4Eyici0127CN4s8+CKIm0y6KBQxN/Lue1Jp/1WJMyq P34a6gkXAlO2H9tNdr4GcIS4yzZ9EfAj3QnFDEAov5dquBUlflhF6werYhKNUNo0iGtE KpadoSRmNSf+5sYaXtbQEJeWz7BSadLNDYAB9hjNOVILOQ3omsm5VTd5YyEIkTv8HKpy DVmYJFfFUE/veTejDvNCpqsCx+7bJDMXtcN8BT1JBJrpo9IPXGDunsZEIOyrtuL/V33x seTEqnLQJs/6KNnE97GiNbfNronXQPubfz9gQWhEY0bKXDyMlBeG8SUOsNEK1vCaI4dF srrw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=EG2h74uw; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s22-20020a056402165600b0049e21096f4csi11262662edx.177.2023.02.05.23.49.53; Sun, 05 Feb 2023 23:50:21 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=EG2h74uw; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229925AbjBFHtQ (ORCPT + 99 others); Mon, 6 Feb 2023 02:49:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229924AbjBFHs7 (ORCPT ); Mon, 6 Feb 2023 02:48:59 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2051.outbound.protection.outlook.com [40.107.220.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 534C91CF64; Sun, 5 Feb 2023 23:48:37 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gUcES+pqo8/gaEV3PIFIib52lHn/VOrsx8jpSMRqFVAbxhEWmN4n/jz1lmkOFPC+v/ona85JZMVKWGyE+dP7g9xQNODasZtLIrlPAxi9UppgdNPz3UgQph7F+qteVnFm4pHB5EAw5myrCcmPZnTm7Q1gPlGF05jgS5uwBghj9HRBvtHwIXf013ytQQX1ej6gO6PFu0BpVmnIc2ExWlXlKfcdzFRpf2ap6GQcXS2mebZ2FlhDYPwZG2Bk+zPf2I/6S1UjpvwuEkJJKyVRrirYE5jXT9HF36yC38RAijgrDQSbe8CpfSbVLAIhc3Ue79/nwmaL+hBOe9GS5KSorNhOlg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LhgVGBn1pGz/qFGyTs5jsxzxhZjuaYEqMKugHfht1g8=; b=RTie0AH6Ja7bdpQzSh5zdBasN0/5ojxTsM9jwnwtNcwBVdEvFEXIKVV7+g+Wwcs8lVE76MB92ITe+WQ9Ruovaai+fX1EoQiJ24irLiDm+P+m9ShUuraEX7H5S/k0pGl0RhE82Ndwy9i/hdLXTX06EVQyPBOV9DD82Cu2cLYoKe6xHe0exusqZnzW0iWHqbZUnTqtE1e8Du2GBJNwuTRi6zh9bBwXu1o06TKN9fh4DhqNGenUMGwfT9ve7ljc+eFgo9dlvhzp4F3m/CsC/MtVKVxQ2qT+QN8a1AcZfZw/TQsBP2AuJOAemzXZ0gR0AZoPWUIXPEKs5AcrFqjghg2aqw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LhgVGBn1pGz/qFGyTs5jsxzxhZjuaYEqMKugHfht1g8=; b=EG2h74uw6Cq6Qi5G5zGTzvDShXAREPg8KHKhk+2GPaAzK+tFLY3MT14mM5g2K1/ZG3mh3Ft15z2j+U5iMcWXbGYi3uI4kckvs0M73U983ggfRMQFz/jQfcXKvt1JqMDvPkz8AxcYBo2atzcnlbTTj2JjXJtlSlz1FJot3i/WV9NKUsNBSU4jw17F/8pdWkkWKzc4NWnvBCZfWPVUBOXFN+Qp7Q67fVBe2bSs+Ig0BP+las8ZCbRLCsBGCTUUQQAF8l6GzjYxONrywhz24NlvhW5pLF87RqKW7dWXRpDpeVywpCGz9DUTlMD0qb+yfysENSEH1rJ6SM8TOV6+RHbbzA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by CY8PR12MB7097.namprd12.prod.outlook.com (2603:10b6:930:63::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.34; Mon, 6 Feb 2023 07:48:35 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:48:35 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , "Michael S. Tsirkin" , Jason Wang , virtualization@lists.linux-foundation.org Subject: [PATCH 03/19] drivers/vdpa: Convert vdpa to use the new vm_structure Date: Mon, 6 Feb 2023 18:47:40 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SY6PR01CA0096.ausprd01.prod.outlook.com (2603:10c6:10:111::11) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|CY8PR12MB7097:EE_ X-MS-Office365-Filtering-Correlation-Id: 4f96889b-f944-4864-2769-08db08168d53 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: hQiwSvhTcfTWvrs7VXaaan5BN+DICeV4GPI0BBRWGC+rNyrbf9kpsKW3RLEeol7Zm81u8gvOfgIp9+UorH5S+tW77GrVIrWFc5xBmSvA66wy6zcYzGrjES+LLW6f/S8RoV4Q16obP7fudBNRHnzp3dQpAp2xEb9RFlMnEEJg46tzo/Jkga2phFSgq408NR46v0J5xbXQRl2Y4IH86wZHKA7M0X5vhsE1GREpdg5VJ+mz4P4Oeg6Ne496cn8OGdXhPF0c3nTFZdIAI1j4uDvg0bxuZke878M0+nwMPR8W1qlqqDAGphbGQ5w/64hTEHa/o2TNbNhhQJbN8jeZGgugVNV8ahzKYSzAvTF6DDLANRk7vFCUzWte9OjNa2T+h1gfeNYlbGulSS1bRH9taEbN3xiRB4nEw7aKy7droXDOw3xASWCHx8CS77JwTQEZqv4T77Zt3uVlpJEvK9Bcpr0LLlmav2S6sulz2EVhlnG/tIJsFxduGYg/IUy5VcX+xz6fcg1HkHc7uTDkWCLliHZZ2jFsNiHIEH2hNmFi7cMkO/ZMz00K6qmsV8JdTCVfp2yitw6v9NSyk3BhwVXPjz4aLoMk0bPu8QRxwcRdgLnNiEkQrCLNy3DJyns5wdebyCjZpCQwJX310T78qQPNLTtrpg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(39860400002)(396003)(136003)(346002)(376002)(451199018)(38100700002)(36756003)(6506007)(6666004)(6512007)(26005)(186003)(478600001)(86362001)(6486002)(8676002)(66476007)(83380400001)(66946007)(2616005)(66556008)(4326008)(316002)(54906003)(7416002)(2906002)(8936002)(5660300002)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: k0mLF4eRyOue/HvSpJitpT6GorGUlHNMDaBeTwzcOkP+IHge4vG0XvM/3k1Kb112t12MKY8alY7sgvTb0cVZ6JGyRBoEINfXVWXFIYCEb8wzoZMeOOClB/gusKDqpGl2b57ZClnF5WLKbElPmkg9oVApmhw+IpF6a864u4Zu2dGSuTLj7JdHwdafz+/saMGh8tM8ux/zj9KZ1lvakrUfhMSPYTDos48Foae+flFBFaljn3LF1U+tzbjkJB9mB0wnOvbRrGKgntE85yz4xacN8W6bYt7aYKmRu3CfT5LEsJbUox6Mbh/Z6+S2y2Kl0sTLhODmz/sShp4Wdx5FbuKo/Y4hJQfKL05z0yu04Y+lQuXIO61LwcsXKGWHjIrnvfss0He96SCwZ6tzSef0bBSWV6vAQQvXeo54moxZEQss9M27H4xA9lmsLapxTJUA1fcsIWk1+sbdKN6ldn58B/yAGRB7pDeKb4kfeYcfM1Ucfs0u8gmX3S8Q2lGThhxrqrGfe6mvG09Fc+mDpjafH/4xH2a+4cvzvOxeIPrf22qlAReo8MRbDjiWWzZpKtbZ0OrpuZE0QuU5WzgToM2YWFJiuUj20Vcw2cpi9k0/DM5FZTxtzlfAUptDhmqFF7zngut81gF7Xd3v7ibIhcmPjrIF0NLG5Eh7LVJYs0ZpQvGgjih2M1DcU8Go3zJD2oU7xqLyqtG7wls/6/2xQIZ/egtNv5mr6HTVLLjnWIFyzexYFRLRH1mi1Wo3KCy7raftkqlWqFTTHan4Cup/3FHKmlIyDMjV5W+RfppMqo4j7/NltH/uBuvfKWH/Wk6yysL1QfsDzaq9zYW+F4apMygtkVpYfCRGlrtt5XS6Nfy4l6hVnrfxMXLED88MoEeEFHhL6a4hyGZrp+IFXkyDWD5Az2UFiXpGepgzb+slM1IU0vmGtN6okc8hO4PdpNlIKj4Eg+CH0SLDRZdBO2yUpFb6Y7RkQzbYUpLuwcwH93QSDKVRsjmnaBA0dHK3zLGH7t2j3luWzG1YTnS0CRppsmxRzxWAX9nbkI0Lg1luHKCEBzXhZPsh6pNkMYzcoqG4pOm7LO4HvcMHEh8/G9bYZiuFfeB+Y5DvafNiDGNBrIRA7OcVjDaOIfcjQovda21+7+VNj07fOgNpqbg84TGmXdK/KacuSZnIbqn/pXnq07MYLn3T8cSqL20+muD83tapi+knWpozP4qusz+d1XcVSpnhZBrytBEMUPp/Dq021szGeOunQvDUPOa48nfNRz9h0Jb+9j8fC+b3iO10iFB5Mk5mWqMJf6ToMMx9iznNs7S40TOYsMkvITuxvlJYgKqx42N3bx0adFVt7pGhRfYe7KmGuInx1b1qwbx3u3hz/IkT68Tx+bT9LeTCwWTrtd24dlCY7nvkEagZaw4Bhhwy3tt4oqTYFL0J0dQDMDq/v7rDzkZAoodMLYPhfkUW7pHWKn9apDB5Hf+32yeHs6PXcKytAzyjLovbAdOzSdhqAAAEfW+jxYFIRg3pkA4t5S20yL6tNfxT18yzbUqaMAV/9kTyVLFFj/3nG7HWb9dMtoL/eaXXb3IrHlY50oNe1sECjTLg68V4 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4f96889b-f944-4864-2769-08db08168d53 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:48:35.7803 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Zukif6znujczilGtEBPn7wj4+KQRdgve8X5+fOdAxWvqS2ez0x9z6k+im9OFh3SMgIuhyXqLexI0/d6efbD8Rg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7097 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067158753361203?= X-GMAIL-MSGID: =?utf-8?q?1757067158753361203?= Convert vdpa to use the new vm_structure and associated account_pinned_vm() functions. This also fixes a bug where vduse_dev_reg_umem() could exceed the rlimit due to non-atomically checking and updating mm->pinned_vm which could lead to a race. Signed-off-by: Alistair Popple Cc: "Michael S. Tsirkin" Cc: Jason Wang Cc: virtualization@lists.linux-foundation.org Cc: linux-kernel@vger.kernel.org --- drivers/vdpa/vdpa_user/vduse_dev.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c index 0c3b486..bc300e2 100644 --- a/drivers/vdpa/vdpa_user/vduse_dev.c +++ b/drivers/vdpa/vdpa_user/vduse_dev.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -70,7 +71,7 @@ struct vduse_umem { unsigned long iova; unsigned long npages; struct page **pages; - struct mm_struct *mm; + struct vm_account vm_account; }; struct vduse_dev { @@ -950,8 +951,7 @@ static int vduse_dev_dereg_umem(struct vduse_dev *dev, vduse_domain_remove_user_bounce_pages(dev->domain); unpin_user_pages_dirty_lock(dev->umem->pages, dev->umem->npages, true); - atomic64_sub(dev->umem->npages, &dev->umem->mm->pinned_vm); - mmdrop(dev->umem->mm); + vm_unaccount_pinned(&dev->umem->vm_account, dev->umem->npages); vfree(dev->umem->pages); kfree(dev->umem); dev->umem = NULL; @@ -967,7 +967,7 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev, struct page **page_list = NULL; struct vduse_umem *umem = NULL; long pinned = 0; - unsigned long npages, lock_limit; + unsigned long npages; int ret; if (!dev->domain->bounce_map || @@ -990,8 +990,8 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev, mmap_read_lock(current->mm); - lock_limit = PFN_DOWN(rlimit(RLIMIT_MEMLOCK)); - if (npages + atomic64_read(¤t->mm->pinned_vm) > lock_limit) + vm_account_init_current(&umem->vm_account); + if (vm_account_pinned(&umem->vm_account, npages)) goto out; pinned = pin_user_pages(uaddr, npages, FOLL_LONGTERM | FOLL_WRITE, @@ -1006,22 +1006,21 @@ static int vduse_dev_reg_umem(struct vduse_dev *dev, if (ret) goto out; - atomic64_add(npages, ¤t->mm->pinned_vm); - umem->pages = page_list; umem->npages = pinned; umem->iova = iova; - umem->mm = current->mm; - mmgrab(current->mm); dev->umem = umem; out: - if (ret && pinned > 0) + if (ret && pinned > 0) { unpin_user_pages(page_list, pinned); + vm_unaccount_pinned(&umem->vm_account, npages); + } mmap_read_unlock(current->mm); unlock: if (ret) { + vm_account_release(&umem->vm_account); vfree(page_list); kfree(umem); } From patchwork Mon Feb 6 07:47:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53054 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105294wrn; Sun, 5 Feb 2023 23:50:29 -0800 (PST) X-Google-Smtp-Source: AK7set9ODYnGtT41DYmowCpYeX11K37Nhfl6PSHlWPVjpNzETPcNzwC79a7k9i+ZFCHDv2vQ+eoY X-Received: by 2002:a17:906:e109:b0:889:5ca0:146e with SMTP id gj9-20020a170906e10900b008895ca0146emr18184143ejb.16.1675669829016; Sun, 05 Feb 2023 23:50:29 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669829; cv=pass; d=google.com; s=arc-20160816; b=PGG0w7UpjIQWTlUuiVtyCpxAe/JK45tfJUWaiE5C+Vw7FpO14r7Fn0JY9vR9DlDsA3 7rYX3SOMo/bjWIHLyYB4Oho+QB3QuFvnIdgmL1ubHu3i25JwWnXemOzNdOCWvVW8GGhb 3I+jCApAYlfc0TYV7gYSG2LI2qwrFhcP+y8z0lRZhUgL2HOZgLFrx8qbbXvhVc6+J6YB avMfgQfJK7bLpdwcCA3tKRRuc+9tKdgLLPIrl10wTr7jqt5Rp7YMwwZplwlREAv1gRA/ 3p7vgGMTrMDL7CDcrubU4KQEn9kfM9kSxXZtvYUPn1l9Bh3ck8NtvN9rRR+OngKFqHNJ 7QGA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=Ggww8xGqVkEn1m6fFSqIeGeKfBiyQH3gSOfKyWdB+MU=; b=yDWKYmLnwNiIMhIQQ60U3Xmg7Toy0tspKcC9PeI4OjD0GBnZQzIUtuckpgDLq1OOTF SP04hZ92ZnJGDSN3JYw+6/U/GgbqY4hiA8/Es6XRKyPBmDCZzBTLWFYs2K5CxqPm9pEt E00Ll0MMUzsJDXWwC6d/VRIiTQXAm9eudXE5Ww8HRqaZ5yvcoXuHJ0wJxq1uZo1vOsMg jYg6DzfxLiRbNNWwO8RqrvVySOKqYjAFBYSfBFUzQ7q+1THQQwcvSolbSxgbjGqPlBho BMgTnaSn0/DkR+/HV1O3C8MzVTO73XeVQcRsLoLkMWcen5R7SYz4YgNHsM65ydyhBQ6I 2dTw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Y4HQgHQb; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id et3-20020a170907294300b00891c5c87dedsi8218072ejc.475.2023.02.05.23.50.04; Sun, 05 Feb 2023 23:50:28 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Y4HQgHQb; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229916AbjBFHtg (ORCPT + 99 others); Mon, 6 Feb 2023 02:49:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229921AbjBFHtP (ORCPT ); Mon, 6 Feb 2023 02:49:15 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2051.outbound.protection.outlook.com [40.107.220.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08F8B1E1FE; Sun, 5 Feb 2023 23:48:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=E3zRX50/Lp7tD2TtC29nYwzq6A4eIeezs2ezZLJbj5liFzIktI7OS/65Hdh0xdp/PbjYV4dFv+Sfu+uAVbcPXM8eyXcRocoD2zYb2z/CjuVbgm8wovpcT3YDm8Xc6nM8tbGiKCvwSE23Wky+fOWMCzGDoPVmQRD3fA1WLL3UwycdPIC5/p+uybOlCh1MzBl+B75jF/G774DNZqcRUJ+0Op60U7lfa4JnFF0616GDdrUQxuI69isqjPKJaTg25VkAYWAM4tk6zwg9aJvrbGYMRcMXK9TLfMTmDixIohIc98CIWnJmGhPE8kG3ZO3sYzL8rIXc/P5PRKjgvYqAmfNTMg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Ggww8xGqVkEn1m6fFSqIeGeKfBiyQH3gSOfKyWdB+MU=; b=nRqcU8SlI5ozKvuRTjwIW7R1YRuy+MsMXgCydX2k1NVuRPIGWcYO8v5Xv4jnYToHxkvADsrPTDLwrUvNDhxWy8WF0yPBNQPUG+jlpNQTy7572cdqNHgV+oP/R2sRd05uPR3VcyUY/5LKmnMIkDEz0ta3etjkfVH4U8FOVAI/uD8OqMf6iaK8edHggSwZSUXatrSS6TxXu3XNiH8Z/RTEtSkKO7FZBpkm/7kMghp8mEqm0sECRqs5QO4kYWsHR3s1zM61ZQCwbZKmhjgXiYWJm2i6jVGaV4H+Gyxw2V9KsoEGgTDt1E4tWgvsWNXfQ5u9CVTPHm9aVfyXpwIVtnLWpA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Ggww8xGqVkEn1m6fFSqIeGeKfBiyQH3gSOfKyWdB+MU=; b=Y4HQgHQbpd91DuQaDucY1BFD5ChpJLapbErCJK2J+Qdg5Zx3stAKxZfg6VuRoL8s4vyXjQXCzFra6r6nA+gOs06zJbufQur7eGvFXmHJbzoSFxkGL6DF+KhHE3q3v1ciT5JfcQtrpuAa7n1zHzaXkTUZEZfvuMMfQKE71BIrdvzHusNUsFlE0TrMLO/PV0c0lXrVjw4CBw7P+B5+SVnn04YuhMPSKptsOCYFvaEaexRvZykf3inWlxlrpw8GKCfR1+2MVlVy2sYAd831w0X9oO2VtIruCF1GfWwqfQyAItcM+2e0HwP31U+iWy2oh1Y/G8H42/B1RHbxRGtzlWnuuA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by CY8PR12MB7097.namprd12.prod.outlook.com (2603:10b6:930:63::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.34; Mon, 6 Feb 2023 07:48:42 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:48:42 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [PATCH 04/19] infiniband/umem: Convert to use vm_account Date: Mon, 6 Feb 2023 18:47:41 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYXPR01CA0103.ausprd01.prod.outlook.com (2603:10c6:0:2d::12) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|CY8PR12MB7097:EE_ X-MS-Office365-Filtering-Correlation-Id: 5a73dbe3-3319-4ce2-5ac6-08db08169148 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: F/B80LQTVpL3jSLjBEjJ9QzNuXDWASskW1lHet9GroRZuF0YWlRLQJ2l6+dKcvBNT4uIdj3tgA50O6uDks7H3QyeQQ2mmNBfrQwGddxplhOX+lzizG0xwlfnhw733kRTbNW5m9EF7VmCDQU/ksswQ0Xvfq8Zq5s/1UC4s79qXLLvWzyRqJah4PaY4VukpAf2mK2sTLhRsGaV6p8fdYqI3lR4POWM93c/dKkAxlQRI1TzhldgsdwWxlXfew/fjG+QalrH9eRuTwPNLzgF/9s4+R239SOVYyzPdOq//F8y2RIxBal2+R78/9ZQkj7F/8dFZ0z8ET88cIyClSZIqNkOYA8/1YeyVWPGUG84L8qdhCssAy54zvJBGXlpzMkl4XjBp0yo9xqHAqwmu84/gql6ORMSdWpkFbleqf0pxocB31fOAliRLE26qLtbqQ5/1Lca0tKOMITqd7aBQJe8IrKMOSwTmj4A+z8CJ1yt7E9cQo19MW0LjJsqd5DhISCVs0UdExjs44vaMeUT/xLHO1DHOzY/Sbk/juEiillJDTHT6z++t33Sss25mBfe3AwjY4SPgZ2HzujYCccPx1VNfUfWbHNgowcQCR+Y6K0iLAeGv3EktxQpf8nahk4jzTTFBrN2 X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(39860400002)(396003)(136003)(346002)(376002)(451199018)(38100700002)(36756003)(6506007)(6512007)(26005)(186003)(478600001)(86362001)(6486002)(8676002)(66476007)(83380400001)(66946007)(2616005)(66556008)(4326008)(316002)(54906003)(7416002)(2906002)(8936002)(5660300002)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ubW3xrU0eGbFRq+G6RJcwlsfu6Ao+kgomSilgOyEGWZ8GnBsXzXYNDlWM32S9LpWG3caLY+45OVebDiKsG9RjYQeSCNLseBRz9Q/nsngnSZ44QPcejoiJJjHzbe+9ASDL3KZ8Vd0FbNgelO3asfiSNs0ykO57Hd+OOqppUqmwpg7qGk4FHDR1rZAQZHVxbyBswPbqdDLmXVPJRxyCYz4g/dIWPObS6YibcM90G1kPud6Gd5blcwePU/DQnD4ZbnfqTi9NAxTLz826/zRE4QvOemkr6fxeGr3PGVoSADxhqbu3CVMSY0Gq+0slPx+k9H6FuCEXqCS+wCVlcLsb3rNn4KJzc9RR6qwUx1IAXOLDhhxzkHWeRWm7RpWkM353mwqX5pqx7c2zklTbKFIvUzos4hva0ZzNjGOZJKkXRguAFsTKMyK8Tt26D/Ns1IxMBNu0ogQD9zsGTvBcvoq2mdnjJkH+x8SU8+r2xdUC2IC48OwBPfd9O68iZybjp5hi0m69EU3qrtcrReabHCrO8UdlOL7243ACaNmlyABkfU7nkpKepp9R/Y+ScXtI7UMtyNAuVB5UluoNjLx5SATikTtw3PtXPj7herx2KK1/o4QHS4ytiLWQRauR0Ws5IsuY34yrO46U72rdx1OhHXfLQ5FfPdDIYm+ygPb5/6WSDtwnf6pYl/41nKB46r8CqJpsGKRxl+fXlc6aLTc0HUfX++4VycAwoCRmWeM3cQXQehFj22YVnMpSUZ72bvYKsBKZvBwAS2e6VpDieFx3qm2e7di/Q7qfeNmeY3uMX91421Vq9H+jPqS6i2NbI84Z8PkJPevUu+1jyhZF6+MZ2fee+SoP0ej9TZOPhjjP6k0bUJHvHRgNXNrfe5doszBcgA8QxcXajOIPoQDnzbS/aADmmNO0lmPTRh3sfBtZo+Y/9OywcgJS2X/JKVA+KWID+sg/Mj8JQT6Mf0B+7Yc9IrP76s7IdRSyVbwCBhqFbSnSRy6NbEGh8EPkIQs/coWPsjzqdN77uueeJ19G3sj6OZTe4kcqMm/bmGB4ckjmBJ919mPyo55zKyQWnDcjsPxCzWlkzdNhB3K5qPbi7G3kAaz8qYId16bwGZPrtOwpf5EelwU35o2B7ndMRLqlEyGSpWH7fNS6fW9hlNlG+jcFpO6rRe79QlXH8I9m4iEbxiySI3St6d0lzO+cUX8H+0kcAMswl5HYwzMJpYKvRsQMEnzR255R2ocS4daaQwzJsP1RBaLJ52cus8zYbOpvYvQSjy2a5LDeu73N4hMIbrEhUQT5ChLS1NLaVdjr8gbt1akfdL6bpbbMnnWV8bv0aC54TcvNKS+Ea8b5udNZfZWD1J4odCyKsX2QXxwIs4UP5+CwYxbY4AGVtxB5nc1i9UQRCLplKteBtBcmzKOF6ugM99Dyo8d1JzOejv5dkdXCneCfDsamuTe/PKBihj2UHA0YSZdSzOY3vjXsyfib9hdwrNQmS+uL+u6RxerWAXuDRWa7M8M0C/HudBP80dHFEgxiVEn7NnRTfXODB3v3WRylzD02OaypRCs1sYhWDuc649+9Wfhf4wF6+wFKVpWik+fdBDInjfZ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5a73dbe3-3319-4ce2-5ac6-08db08169148 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:48:42.3891 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 7xkdaR23n6b0+bhaWr+Avl1uc+gTnm7SJDHdHr2AR6iQKfTKWkLr3YjTvCIVM1p3eNklwHEvskiS3YS7BrKIEA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7097 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067166334485368?= X-GMAIL-MSGID: =?utf-8?q?1757067166334485368?= Converts the infiniband core umem code to use the vm_account structure so that pinned pages can be charged to the correct cgroup with account_pinned_vm(). Signed-off-by: Alistair Popple Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- drivers/infiniband/core/umem.c | 16 ++++++---------- drivers/infiniband/core/umem_odp.c | 6 ++++++ include/rdma/ib_umem.h | 2 ++ 3 files changed, 14 insertions(+), 10 deletions(-) diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c index 755a9c5..479b7f0 100644 --- a/drivers/infiniband/core/umem.c +++ b/drivers/infiniband/core/umem.c @@ -149,8 +149,6 @@ struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, { struct ib_umem *umem; struct page **page_list; - unsigned long lock_limit; - unsigned long new_pinned; unsigned long cur_base; unsigned long dma_attr = 0; struct mm_struct *mm; @@ -186,6 +184,7 @@ struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, umem->writable = ib_access_writable(access); umem->owning_mm = mm = current->mm; mmgrab(mm); + vm_account_init_current(&umem->vm_account); page_list = (struct page **) __get_free_page(GFP_KERNEL); if (!page_list) { @@ -199,11 +198,7 @@ struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, goto out; } - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - - new_pinned = atomic64_add_return(npages, &mm->pinned_vm); - if (new_pinned > lock_limit && !capable(CAP_IPC_LOCK)) { - atomic64_sub(npages, &mm->pinned_vm); + if (vm_account_pinned(&umem->vm_account, npages)) { ret = -ENOMEM; goto out; } @@ -248,12 +243,13 @@ struct ib_umem *ib_umem_get(struct ib_device *device, unsigned long addr, umem_release: __ib_umem_release(device, umem, 0); - atomic64_sub(ib_umem_num_pages(umem), &mm->pinned_vm); + vm_unaccount_pinned(&umem->vm_account, ib_umem_num_pages(umem)); out: free_page((unsigned long) page_list); umem_kfree: if (ret) { mmdrop(umem->owning_mm); + vm_account_release(&umem->vm_account); kfree(umem); } return ret ? ERR_PTR(ret) : umem; @@ -275,8 +271,8 @@ void ib_umem_release(struct ib_umem *umem) __ib_umem_release(umem->ibdev, umem, 1); - atomic64_sub(ib_umem_num_pages(umem), &umem->owning_mm->pinned_vm); - mmdrop(umem->owning_mm); + vm_unaccount_pinned(&umem->vm_account, ib_umem_num_pages(umem)); + vm_account_release(&umem->vm_account); kfree(umem); } EXPORT_SYMBOL(ib_umem_release); diff --git a/drivers/infiniband/core/umem_odp.c b/drivers/infiniband/core/umem_odp.c index e9fa22d..4fbca3e 100644 --- a/drivers/infiniband/core/umem_odp.c +++ b/drivers/infiniband/core/umem_odp.c @@ -130,6 +130,7 @@ struct ib_umem_odp *ib_umem_odp_alloc_implicit(struct ib_device *device, umem->ibdev = device; umem->writable = ib_access_writable(access); umem->owning_mm = current->mm; + vm_account_init_current(&umem->vm_account); umem_odp->is_implicit_odp = 1; umem_odp->page_shift = PAGE_SHIFT; @@ -137,6 +138,7 @@ struct ib_umem_odp *ib_umem_odp_alloc_implicit(struct ib_device *device, ret = ib_init_umem_odp(umem_odp, NULL); if (ret) { put_pid(umem_odp->tgid); + vm_account_release(&umem->vm_account); kfree(umem_odp); return ERR_PTR(ret); } @@ -179,6 +181,7 @@ ib_umem_odp_alloc_child(struct ib_umem_odp *root, unsigned long addr, umem->address = addr; umem->writable = root->umem.writable; umem->owning_mm = root->umem.owning_mm; + umem->vm_account = root->umem.vm_account; odp_data->page_shift = PAGE_SHIFT; odp_data->notifier.ops = ops; @@ -239,6 +242,7 @@ struct ib_umem_odp *ib_umem_odp_get(struct ib_device *device, umem_odp->umem.address = addr; umem_odp->umem.writable = ib_access_writable(access); umem_odp->umem.owning_mm = current->mm; + vm_account_init_current(&umem_odp->umem.vm_account); umem_odp->notifier.ops = ops; umem_odp->page_shift = PAGE_SHIFT; @@ -255,6 +259,7 @@ struct ib_umem_odp *ib_umem_odp_get(struct ib_device *device, err_put_pid: put_pid(umem_odp->tgid); + vm_account_release(&umem_odp->umem.vm_account); kfree(umem_odp); return ERR_PTR(ret); } @@ -278,6 +283,7 @@ void ib_umem_odp_release(struct ib_umem_odp *umem_odp) kvfree(umem_odp->pfn_list); } put_pid(umem_odp->tgid); + vm_account_release(&umem_odp->umem.vm_account); kfree(umem_odp); } EXPORT_SYMBOL(ib_umem_odp_release); diff --git a/include/rdma/ib_umem.h b/include/rdma/ib_umem.h index 92a673c..065cd2c 100644 --- a/include/rdma/ib_umem.h +++ b/include/rdma/ib_umem.h @@ -10,6 +10,7 @@ #include #include #include +#include #include struct ib_ucontext; @@ -19,6 +20,7 @@ struct dma_buf_attach_ops; struct ib_umem { struct ib_device *ibdev; struct mm_struct *owning_mm; + struct vm_account vm_account; u64 iova; size_t length; unsigned long address; From patchwork Mon Feb 6 07:47:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53057 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105399wrn; Sun, 5 Feb 2023 23:50:55 -0800 (PST) X-Google-Smtp-Source: AK7set95/JXnbXAJ60NCeegEUmvaRjExrUP3KHTtL8tuwjqTJOIPO1ckmOszg6ZMSSp+6pIGo5Tl X-Received: by 2002:a17:906:c0d2:b0:86d:67b0:6292 with SMTP id bn18-20020a170906c0d200b0086d67b06292mr19434946ejb.73.1675669855131; Sun, 05 Feb 2023 23:50:55 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669855; cv=pass; d=google.com; s=arc-20160816; b=HEQW1AAlk65q7xAwmtmQwE0BuqqHe91Bl3skLGVKLjrFYij8xHoC1ve66jYLdERFI0 GxyE8tUvgld2oYUQRwlzbXzjikLDuYqGTPrG8PwmoYdzS23HKl9VAe7hc8VGUx5PCXWV veboYVNpFlMT8NthkErXcOYo6CHtWZSfCbcfJUaP6tqbdWhcOCxpDJ9rM0Yhf9h0SMA9 4nRTRQVJ0cJXFz0dfbSvE2MrwJ+TtGkNkgzzjQ8lrK8Xuty1AVIdiE08nl4sTSkzFKxD cEsg8xWetAIUW+x6zux0L0zcze2bCVKHB5u8qIWG5H7kPYbCRAcVJHOVD1ICU4YY0hEC 5FIQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=dbbfhEFK8Y5g+MZUUcF50R2sJRtJy6ACgT2r4mlmBvQ=; b=iTn8PQgsdQfffshZ9+j87yChCzVGLPy8UkVaNJeqQwV4PDe4oZvleSQFtQxSLY/zdB EZeuLRz0uMB+e6rd24aSKmqrrufJn8/YhQccN0XfFMch28S3cgMzpUI6sKhr+PCc8ShX 0o8hdeEpicWLR2pARvzKVV9fikf5Y8qli7ocq0Kr1ympSKPDAP1mdbD0dYAjKJU8IX1t c9RjpdM8F2Gw6Ycvogv3kH4qnhns22zfoeGk8enGcyLHzFbAAqFka/OWrPgi4xHq4t1r cstp6QwWSioJeNlYjDzOWi9vIx9/IA6oC9QdLueG0QE7uaCC/eDz+qNDQRiV9AQfHBMB dZDQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=VGMmwdNV; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ft31-20020a170907801f00b0089807bf417asi5679692ejc.50.2023.02.05.23.50.31; Sun, 05 Feb 2023 23:50:55 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=VGMmwdNV; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229880AbjBFHtx (ORCPT + 99 others); Mon, 6 Feb 2023 02:49:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56788 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229940AbjBFHtj (ORCPT ); Mon, 6 Feb 2023 02:49:39 -0500 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2051.outbound.protection.outlook.com [40.107.220.51]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 720E01CF58; Sun, 5 Feb 2023 23:49:14 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XN2jl/HppWe2WbAIMAP0hRU+gmLJ8m8azfsx4T2fHkNR61TRgEGZ44NeyHuSbxda6NryeHkfrWbq63Vxtp1zXYaIkorIk8m+cIHfVF9x+iK5W6jnd2ZbN6theKf2u8keLZvndkprY04XDl5viaWga8GkM/1JzZqbofc7wLr599/RxToXo8xHr6NDWknk9LomxzBcEFh+cZVmE+1Be2lwAAVUTxYMJ1bLDhlFFOHNhgV+87aFoNzJVqcta71Ay3wcfPnP0binqM8sLAYd6Jo4Wf9mZTVype8SzRHoY3kAiujeU+eQ0qnwTOUnpQMsEro88ABOrH6Ez5PXTwe2psLoGA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=dbbfhEFK8Y5g+MZUUcF50R2sJRtJy6ACgT2r4mlmBvQ=; b=mnTUGofFQWPhl3fcsDP3n2cRv3qzr1bRl0HVJ02Lxc8TzF93TRVBXbMXUU68lt4Tx648Gdq1bubNI8fKhBrukIS+8b4U5kyE+hCXuNUo2i0A/KSCacXQ2mbfIvpCFeplwLyBjmYs2CovgCqVnhgyzUUS+PXCHc95FcN0yXfM5WMh7IZPFXkxRsN0WCLgqRLn1F+PK3ALsDdLsqyMyAJDdIfUx3IFmGVSgnlEi0rcwPKFBrOoVCYn79FfvwcLrxEB3dmXdpSvi3sYHFG7tf7q/pVLJ3An8d+GxeuUV0VRKMVEWrhmRfPQmkN64dH2yug2qGlQZNdg9xzBtQ4jLPq3dQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dbbfhEFK8Y5g+MZUUcF50R2sJRtJy6ACgT2r4mlmBvQ=; b=VGMmwdNVNSN3VYOHfasKm4H2Vx/Wsl7VT43FSwx/C/VBjckm7Z0Ur3KvfO19Ogny3/QHdBpZR2TxH1cyjqViL8X1LvbN7064TIHNBeh5EjfdDvEfA9S3YswQnr4H1rsB9do+VmEPmu6pR+CKJrKAesYOh7UiZhgT6Xw1jeOcZwImkHzmAKb+GtfQH0O95kt1BZek0Szut/nml4gDoO4uPrTyJpYp8S8EaeFo1NPKt7hQA+eomC/TPpsGxux4huucOmRA290Iagg9Fe09yDrmQVpkKxf8qRnjTtgW/tFPs/ooArhZWI/x21/myiFVMKdhOB7I3OyX/xtuxucWY24OvQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by CY8PR12MB7097.namprd12.prod.outlook.com (2603:10b6:930:63::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.34; Mon, 6 Feb 2023 07:48:48 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:48:48 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , Bernard Metzler , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [PATCH 05/19] RMDA/siw: Convert to use vm_account Date: Mon, 6 Feb 2023 18:47:42 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYBPR01CA0180.ausprd01.prod.outlook.com (2603:10c6:10:52::24) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|CY8PR12MB7097:EE_ X-MS-Office365-Filtering-Correlation-Id: d1719398-e6af-44b3-bd37-08db081694c8 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mEgnJ2byJYYRp5TDKz4QiINbbC95btTBNqenhr792EFCNVZ5vubMIiUg9pa5i8+JU4s6qxBvrlhVt62u/Zl0xs1+bEMXnhrKqnvck8WgN666G9HSZaXadccoWUbVd/7M5ctXxwfasjKap9ptYFuOcMnI8YWN7oJ9/3//X5c5lNJXiG02aE4Kb9/DLSlFMKNINOUdY71D44WUnVevEHmNItB9w0LfSU4UEynLrCz4RHyQyCupVnofIhNqzbqySBF4Px441wauBPi0JNCdYF/88QHoquXb+bfHXE+zJhh53sxPAtT6G3Rnjb3OY9Cy91M8Im96+qROqIa4V8TSSL7ZBEoFpw14soRTthjWO6wFthqoDpsDjzaXwsMScJkeOCwpnyOKvjj/I9D/Se1J1X/il9oow1t1+HtvqbvTtb6m5jZgJh+mKl96Or9M5Y9VQGnFTKAiLM5cMQr/h6B7Xex0+lGlc7+7ziT2i09YSsdW5CTYPtkUsLwIQ6/oBjJM0vQC3ZvG8QvkcNzbbw4Y6YLW2CbcfHtJ/Jh9jmkkGcIMc93pXD2LrFY4qtS4GYO+JuwRXmFJQuiHkUNI9oxN5kewX2Ez7C9QmUzbRRMy38siwC5FFdLW7tl0QR9JjSXHcQAyO4f+QOF/1jys2tDj8jYDeg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(366004)(39860400002)(396003)(136003)(346002)(376002)(451199018)(38100700002)(36756003)(6506007)(6666004)(6512007)(26005)(186003)(478600001)(86362001)(6486002)(8676002)(66476007)(83380400001)(66946007)(2616005)(66556008)(4326008)(316002)(54906003)(7416002)(2906002)(8936002)(5660300002)(41300700001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Kr2qioBvqyrNtr6voJ4NIWmH7hfXk5CZrJHSm+qHv3srTYOWKjzA1CpgwZCgqcJVQ4YkXGff1cPcWGA37PI2bs2wtH7njipEtd6uKWj1jGmbSbjS0BDjK5rV5YbzHXvHtpui72aQ1hrrI07TWwP7XA5fMLzs5GniW6Rc/dBNkp5gtgDHQCMMDPfHSRK8/R53VuVQjPKMSwTs9mrUAzUI/xQG7PavAhIG+bIg4iSwSKLQj7MVaNZ3WMiEDt8sHxLADR/Opb5zOxgMXOGTCExol0A0Dvu8Kj6DgJMs7QeL+8pfT9jB0YkUvqzD4unMyaXyrbLEfJgKF3SLwbI0oRGAZE1Z8mRBLKl7X1VjtGr/iJu+G8qN5teztyw3lobuTIHY/hoLX1LZFkWYgw94RjvPHE4kDdc4HuaR5nnGSCGovylAiDdUPhxLWZSm/AwEkfTO4iCByHjOJPlU3ZT6d8JJzriZTBNCbSweqkLhW5D8m8hj1rcqIbdSynWpkp8KVMSdHP3dVr+Thb8mPemNz+IVfyKyW/EcpUzvkEk5rUhBLcILG5SnTP3TK6V2yEw0PaldIfrWNcC6jv017vh6KYz3dL841DsEGSfZ42AfhH8ry9oGXCgK/GIG4sgxMebqbIU27Vbj9xajK7lDu1NVtP9a7ZKqg0pYccepGekpoL1t6i4V+78mWdTwgBzJEoud/OBNRXCKYeyvCC8Y/Fs8F61pzLfE9Nj58os52ijNdSK44nUFMdP0Zk6/lbD76rx4U0ewygMRlryukzDU1aFSsiGRcVxeZd/f+okJWuJSau/vsxGL0TzXqefEKr+kyF+Z2SmlUbfJYE1vgTdDSOJ3FcbNkTRt94j2JUJtTW/jV5FTrBRiJ0KO53X9VjhAl7KJ9Wfp5e+yXVWy2DeatsgGkqDL4LMSdWL978est4D8SCl7I3+p4w/Wxg4ndosb3+nKoiFakMfM3Hnv6Unb9D5Rtf3LtJsfZ+vTNldIcKGY11RgdDVgsT1E8qw3QB3SRRvbzbr4ysQ2FCXxE+SeJuQMPyE+B5XykMfwVyPLCAqypLzNwHL5BD/htEZg2Ea5XApVVFCgNlsWnYQ4xOfA4DEB5VDMjfrF4Lt96O8VZukD03QHEXxk6dXd8wN0YsgVpUsqCjZ48uYbpNFjGyUfffGy1YPE5l+vj7rW8C7Uno20VN+1ou1JIv/sBYR1gyxc7ZTm9KyPQ/rs7HcIaDkhQ7zms8yQ8VsD2Kil+OAftlXJsypAw/wYZvZAirWFGaOjaHmCsN/bdfw7mHn7VI11bFi88VhUklQFUkQrNAO3wE2xKKtBMuHxXyBA9lFXsQErduHwoHy3eRbW3gWQkchIDeHPEhEiHjPOosZvzogl+SLpx9Da5rHvqo73UsF6sUTYk43A9O/hfrur8OTrExrN3M0aFu6o9Ly+kXtRZi86zzMdLsH0Xnea1al5RwYUZ4vV7p5ypIGwmu30mR4zzu8kderGOKFxHzFycWeC9pp7KPnFInUaGgg9Qp4VcigyW/5yN2LOisgkIzHfeO0E+cLWwKuJ3YiGdRgLMDovtfbzmIS01cDomgzsAo0btV+7j5qIGD3lGr4L X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: d1719398-e6af-44b3-bd37-08db081694c8 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:48:48.2637 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: sPTS0ifI+wgKwzc8U+7q4QMN/tEOXb02+weR/NeNFNKlKfv+KJERpFtqaAe3R5M1A2XI8D6T5oQAMoc8J3A+vg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY8PR12MB7097 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067194031510820?= X-GMAIL-MSGID: =?utf-8?q?1757067194031510820?= Convert to using a vm_account structure to account pinned memory to both the mm and the pins cgroup. Signed-off-by: Alistair Popple Cc: Bernard Metzler Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org Reviewed-by: Bernard Metzler --- drivers/infiniband/sw/siw/siw.h | 3 ++- drivers/infiniband/sw/siw/siw_mem.c | 21 +++++++-------------- drivers/infiniband/sw/siw/siw_verbs.c | 15 --------------- 3 files changed, 9 insertions(+), 30 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h index 2f3a9cd..6d4aabd 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -13,6 +13,7 @@ #include #include #include +#include #include #include "iwarp.h" @@ -124,7 +125,7 @@ struct siw_umem { int num_pages; bool writable; u64 fp_addr; /* First page base address */ - struct mm_struct *owning_mm; + struct vm_account vm_account; }; struct siw_pble { diff --git a/drivers/infiniband/sw/siw/siw_mem.c b/drivers/infiniband/sw/siw/siw_mem.c index f51ab2c..be90121 100644 --- a/drivers/infiniband/sw/siw/siw_mem.c +++ b/drivers/infiniband/sw/siw/siw_mem.c @@ -68,7 +68,6 @@ static void siw_free_plist(struct siw_page_chunk *chunk, int num_pages, void siw_umem_release(struct siw_umem *umem, bool dirty) { - struct mm_struct *mm_s = umem->owning_mm; int i, num_pages = umem->num_pages; for (i = 0; num_pages; i++) { @@ -79,9 +78,9 @@ void siw_umem_release(struct siw_umem *umem, bool dirty) kfree(umem->page_chunk[i].plist); num_pages -= to_free; } - atomic64_sub(umem->num_pages, &mm_s->pinned_vm); + vm_unaccount_pinned(&umem->vm_account, umem->num_pages); + vm_account_release(&umem->vm_account); - mmdrop(mm_s); kfree(umem->page_chunk); kfree(umem); } @@ -365,9 +364,7 @@ struct siw_pbl *siw_pbl_alloc(u32 num_buf) struct siw_umem *siw_umem_get(u64 start, u64 len, bool writable) { struct siw_umem *umem; - struct mm_struct *mm_s; u64 first_page_va; - unsigned long mlock_limit; unsigned int foll_flags = FOLL_LONGTERM; int num_pages, num_chunks, i, rv = 0; @@ -385,20 +382,16 @@ struct siw_umem *siw_umem_get(u64 start, u64 len, bool writable) if (!umem) return ERR_PTR(-ENOMEM); - mm_s = current->mm; - umem->owning_mm = mm_s; umem->writable = writable; - mmgrab(mm_s); + vm_account_init_current(&umem->vm_account); if (writable) foll_flags |= FOLL_WRITE; - mmap_read_lock(mm_s); + mmap_read_lock(current->mm); - mlock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - - if (atomic64_add_return(num_pages, &mm_s->pinned_vm) > mlock_limit) { + if (vm_account_pinned(&umem->vm_account, num_pages)) { rv = -ENOMEM; goto out_sem_up; } @@ -434,14 +427,14 @@ struct siw_umem *siw_umem_get(u64 start, u64 len, bool writable) } } out_sem_up: - mmap_read_unlock(mm_s); + mmap_read_unlock(current->mm); if (rv > 0) return umem; /* Adjust accounting for pages not pinned */ if (num_pages) - atomic64_sub(num_pages, &mm_s->pinned_vm); + vm_unaccount_pinned(&umem->vm_account, num_pages); siw_umem_release(umem, false); diff --git a/drivers/infiniband/sw/siw/siw_verbs.c b/drivers/infiniband/sw/siw/siw_verbs.c index 906fde1..8fab009 100644 --- a/drivers/infiniband/sw/siw/siw_verbs.c +++ b/drivers/infiniband/sw/siw/siw_verbs.c @@ -1321,8 +1321,6 @@ struct ib_mr *siw_reg_user_mr(struct ib_pd *pd, u64 start, u64 len, struct siw_umem *umem = NULL; struct siw_ureq_reg_mr ureq; struct siw_device *sdev = to_siw_dev(pd->device); - - unsigned long mem_limit = rlimit(RLIMIT_MEMLOCK); int rv; siw_dbg_pd(pd, "start: 0x%pK, va: 0x%pK, len: %llu\n", @@ -1338,19 +1336,6 @@ struct ib_mr *siw_reg_user_mr(struct ib_pd *pd, u64 start, u64 len, rv = -EINVAL; goto err_out; } - if (mem_limit != RLIM_INFINITY) { - unsigned long num_pages = - (PAGE_ALIGN(len + (start & ~PAGE_MASK))) >> PAGE_SHIFT; - mem_limit >>= PAGE_SHIFT; - - if (num_pages > mem_limit - current->mm->locked_vm) { - siw_dbg_pd(pd, "pages req %lu, max %lu, lock %lu\n", - num_pages, mem_limit, - current->mm->locked_vm); - rv = -ENOMEM; - goto err_out; - } - } umem = siw_umem_get(start, len, ib_access_writable(rights)); if (IS_ERR(umem)) { rv = PTR_ERR(umem); From patchwork Mon Feb 6 07:47:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53056 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105322wrn; Sun, 5 Feb 2023 23:50:34 -0800 (PST) X-Google-Smtp-Source: AK7set8cAUDXbXl7u6CE7MvQX1ckDAHY1EB5N+CRbAyEnD2LFtOggLgKDiPs+ylbqIoKc/ycHIVC X-Received: by 2002:a05:6402:1950:b0:4a2:260f:3bbe with SMTP id f16-20020a056402195000b004a2260f3bbemr21142996edz.23.1675669834596; Sun, 05 Feb 2023 23:50:34 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669834; cv=pass; d=google.com; s=arc-20160816; b=kPwHdlIhNJu9qufk31cpzUcbFy2FRIw0XYyaSJmYrYFYM5j/8wROgl/UetMGjUhe3a rQxCD23aOTcd9EshR2oc9Zhtrl5WkLgWc+tng9J6sYXu+SaeIgh4o+3zAx13vRfFi4OJ 6F+FJtNoOEwMFx9cjXBjje0GNJbbX1ZYXIKD9fyuT2KZpJHu8j+JpmieYLQyU0j7tMm1 NGPKH74SC3NHJsmTbj7/YGymHfCUbWdl5oZU9lXtKSDTUnB9yS1WnIF9QegIvhFJ4n9g w24rPp30ioJ7V+9f3z2IXudtcpoXiK1hOd/nMQtDjW3QcIo5885ATCEq2uH3abH+b8wv jxxQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=i/0d5FeS1Bthw7AbdBW2nD+WSiMSiecdaEvA9kRz1hA=; b=FoK81TN+LG8ZBA70JuQyij+M3hkgbTC+gbEIJAxwn2ufeBImxdBDsXFTERpg16IBRc 4b8AjU1VD7ZrP/zLC5nP6GIsFAvtCLAPlckHL/4NjNtdnZssYksTyvj1PzJMNVEqwVqB qU/YZspU9whNpW0rYCzC+hGsVqZFmBRzLLSi4+h0gVXZbS4UJlpkBCjifvVYG6novEOv FctMqIeowmW3/GhsKzTdu0o/HxzvkHaI/skLiURUqQIaXodq5om0yfyvgOMeO7mfZftI fVq9GJ7lLrq8nL/8+MgKdrnyva2al+S+RczUbnhrmwUDGZYKCTFZs1Ntf5+MFnLrUGDs em8A== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="aw/rHJr2"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id b13-20020a05640202cd00b004a207728808si10214677edx.496.2023.02.05.23.50.10; Sun, 05 Feb 2023 23:50:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="aw/rHJr2"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229964AbjBFHtn (ORCPT + 99 others); Mon, 6 Feb 2023 02:49:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229958AbjBFHtZ (ORCPT ); Mon, 6 Feb 2023 02:49:25 -0500 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2083.outbound.protection.outlook.com [40.107.95.83]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6DCBF1E1FC; Sun, 5 Feb 2023 23:48:56 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GKJrhrOon4wlA5wPX4OgjWNUpE9E+c1jOh93c/zCGBbeLn07FIVt4YX4sE9IqemJ8c1yi4CZPKt0xXD7GRr6bnwCbvQWGE+romi83aNMzkn7FIwS+6DIIP0gHwBnU9MwQ8bkLRWWA4nQYrLmRKRG+pfpB+4QqxHU775Gyy7ET3SyaZAFYL92xfkMtWpb6jcU3lK08/z17bV/BE1gESDWEn3n+MGEhwrCo1e6WRyetwzNmxLTL4TgFZ+6pb4TA3uYMh5xj/X7wjH3ywcPiAfAN5pkKKef7c8jxhYV074amLxtrWlO8Oo7lBZTDIncmGl//0COup14QuwThztkNPhh0w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=i/0d5FeS1Bthw7AbdBW2nD+WSiMSiecdaEvA9kRz1hA=; b=kkoj2JaYpXi8X54ed9xTPQErejMk9izMe09As2gSABGWKF2gbe0vkJ0/Yf8hdBVRkMRvK4ihURt2xIJ28eH3vPgVFvnrR2UdcNpkVwaz5M3P01HK8iPs5/AoBawMFuxWHVlArSOFCurC0Fxnl79VXpKr0khtqxQ7P569snIGkstxE8cpdXfQEPjJH75ydxqcM0bpy9Zi9IzhkralT+QLsYB0hRvdxE8n1+0qHCGiLAcWYa0uEpwgArGZqHVLGF6mYOTba31eVPmS5Q7SyyRFG1OPwWMwawXjHJTio1hBoh15fySLC+JbauswPUcvpifky5+T+wFUQpR2VMJF9oyJuw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=i/0d5FeS1Bthw7AbdBW2nD+WSiMSiecdaEvA9kRz1hA=; b=aw/rHJr2ROPaFuGDV679uXnmMAVYxwJKSnQkyoLcw8StHbziYVi6NRsPvM7sHHS79m9ckuv3d6jk6AyVcrVplCXNH2/eOIUyD1MHXph6KJdLz3w+i6r1thkA4iEvEh/pOzTSma4LyfpC5NvgHsNmEyJ4/SJHbq+gRw4rcuRP1l7UPCrbfN9Puxz7MpQE66lA5G+iGGp5+qml0eX5+gb0lKtSyvK9lgOjo2gm/o/4Utv+ZrX/BBBV7wNZvVMxRJPzU+2mEOXR43REOdP+pgdN1f+k4vl/C/zk4yDCxxgb78jXMhY66NgnALKtjhQg2MCeNVwe978L/CV7THEfMW2Ibw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DM4PR12MB5118.namprd12.prod.outlook.com (2603:10b6:5:391::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.31; Mon, 6 Feb 2023 07:48:54 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:48:53 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , Christian Benvenuti , Nelson Escobar , Jason Gunthorpe , Leon Romanovsky , linux-rdma@vger.kernel.org Subject: [PATCH 06/19] RDMA/usnic: convert to use vm_account Date: Mon, 6 Feb 2023 18:47:43 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYYP282CA0010.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:b4::20) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DM4PR12MB5118:EE_ X-MS-Office365-Filtering-Correlation-Id: e7cfeabc-8a1a-4d0c-50c3-08db081697b5 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3Em9qIZeJCfdgX0Lzj0VhOFmFq2EkCQRhWl4EGeJW5wv581NiTeB3qWDj/2KkrcOuqtnmFNpvGMPFFyDafJpKhr+yt/lAAuBJTJ/0EzEpVBbFXaEKjBPdT8BU15HuRumyMEkfx0BbYoM13wDthnNA6deYNF03KPugeUqFR/I0H/AkMobrm96K5PVwrC9EMyq3fWTwYdoaiC5isDXTd+NxsDCazGyi2fhfraxkB1aBbOL3n/bla6jM91LeGZ1TBDL49o9MzdNaU1Jk3xGfusYQzrtK2Ich94VqN9la43JjIqT7gGPSFwJZreNCddsabmeeOgh/CDMOwN5NWo6GocFPptuxXA/Y4dXehbxWHiDOFfKWaTiOXLaceClgOu+74ccrL7CuMW7EPomjUDZ0Wo1YO6Wcp3Xn1l66TW7tZ+PuPx/Zmj0jdjbPhsyIa2Jshmgi1/ED3XyYzgPOOc11ED/Od6E5UhrWw6Eb3yKNrFiSrRsIg/bTzQfg4DYVtF2G6Y/+eaXfatkPWJCVHbvNABN0QB2Wmz1KVr3xq+sAjruExCxNLGNfxo8qG4bAh+feJBni5/mNiWzqM2sHFV+LC0elXXDi2YMCYt42jrBcVAnIayEzChmu5f4Qr1uBNlx+a5o0wDkbisjqLX1cmwX5jrDrw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(39860400002)(366004)(396003)(346002)(136003)(451199018)(66946007)(66476007)(4326008)(41300700001)(8676002)(38100700002)(8936002)(6506007)(6512007)(26005)(316002)(54906003)(7416002)(66556008)(6666004)(5660300002)(478600001)(6486002)(83380400001)(186003)(2616005)(86362001)(36756003)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: gu/4IBsNj6nwZ9YzlHu6d9380Bnoq2pXo6uSCdgiZzrOqndXqixi020w2Sj8cE73xa6O2iU1HKpzwDh7IPVhsmrfRVv9scORa8lbM05alj5/eGk6hqAA4ZXvFTyZZQMzmJ443F58pP6X/ksLqkOGBmofIqLQui79wP0fHuFMPFsitEYe5rEV4UU3q0pNDCxHIH9Vdox5/A7GhucomumQ0UTZxAsMVK2o/uPMY2F8EFm4BzAa11Eq86IK/V3qatq/7m6oWNgu8HTl7K+f4+XT+SrHR3WTA2DUKzc7m/mBtu0yKBUTMao7cXxYyN7zvOjeAunli9mohjjqbm+ZK0wFNEGPwWUprLWxDcPLyBDLdASwBf89GFqJmlAcb/OXVQMVc0eszVaPnagi2/KES6bOrw7JfdM2wS9ClD6uywGFNTKR/cUt3IAGmMIY4F7ksZWsQvThrIi8Mufu7EXylLXMHZy7kGe9W+yOaMfmIxW0plIRntqLQrYLf6Zsk3oZ7j/kZiqQs8jN/Gsk86xCXoptqmjnikwIWKtcfg5a2to/VWNRxrNFi8pH0sE+XtgsWuUZz2DYZU4HaEP8HiLEUzo+huZB9aJYR93BWdlGTd0LQ81nb4vdEQwHrTmhegnZ5YTCD7H6Cdb3rvMIrM1XZ5IFbbjzdvl8cD9PS7++e1BwXXSavFJw59HfV0xcTRJQL+E9lR2DC5Uz54/Wll1BaA3WFx8f6j4WSohCdUDtGg6sbkxZ5t9KiWb/LcYrNDF+qQlJMYDatxJZuvpU1FX5stzve/QDAxxnLTl6hqHAK2y+EJ1C++A+iZ21ar9+rnn4PTZk4WX5G61YVM9rArSqvkZvv7p/Z8ac3BdeWx04yZMXZPREVV6rtR4ufkIzD47j2PehoS4aIMW9pd7NkyFSjy1WYMJ89bT1YKAtETZ4aki+Ptyld8hY38dQRupiZqYPHupWJnAWsSqgfZegpTd8KI1Q5gZ1F65I939hxvDz9ze81xTeieVyeca/aULBeDVCEaU86lWNzPjWG+ui0c0NJneqiNWKutHdnBeaJlQrrdkILvRCJ4gBH1M643iyqUqYUtmQKdefHahsOAwjC8GWZDVs3A06wourOGendz91To4N0psczC2gEU8BZGbURHU2vMw2IkkoHlBOJ/qxWsfuAPA0P9Pj8QwlfcjoMHConJBJ2PvfMN/22KY01FPsc1T1u4Qf55dhIgKBjGhmfDsJ8QJ+ei6dJfKU5c13AuUYl/imwrTfXsgrF0PYYbwYEDrdsXjEOvgP4ZG6za+/Aj4AcPBtELI9TWZUjqnMluJWpiGa1CgngFO0V7XQNfSZSOiJmXwir430Hgti9Qqx0O/bMFXh/BP7iYGkrRvFHZwnVjc8/Rw5CkTjydK+ViI/Eq9uOKXHN9JVwy1sDvhWNozutF1xH7aij8vS2OubOPV2i27wRLE3iRPClIQRlMD8+SXng7ECiMuAy4LQHDzIaDaJsjRo+QihH0UhwiEacDtR9n4maS6wB57XlSwBGX25NBNqhWgZS0X8+09xky6KnQAH+A26Uxt6nR+L7sCb79CYN8i2ZJFS2XyA2+/hIbKVFqZvv0Om X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: e7cfeabc-8a1a-4d0c-50c3-08db081697b5 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:48:53.1852 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: iAeyFeiJEkHUn1ZeyVxeCBngdX4GNzhLl6PF7Zc0nYgz31KSrcPpqt+6HXJvWuZhTRt7P2ZNwHdFXo6uVF7lsg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5118 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067172604329438?= X-GMAIL-MSGID: =?utf-8?q?1757067172604329438?= Convert to using a vm_account structure to account pinned memory to both the mm and the pins cgroup. Signed-off-by: Alistair Popple Cc: Christian Benvenuti Cc: Nelson Escobar Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: linux-rdma@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- drivers/infiniband/hw/usnic/usnic_uiom.c | 14 ++++++-------- drivers/infiniband/hw/usnic/usnic_uiom.h | 2 ++ 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/hw/usnic/usnic_uiom.c b/drivers/infiniband/hw/usnic/usnic_uiom.c index c301b3b..8952ee5 100644 --- a/drivers/infiniband/hw/usnic/usnic_uiom.c +++ b/drivers/infiniband/hw/usnic/usnic_uiom.c @@ -89,8 +89,6 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, struct page **page_list; struct scatterlist *sg; struct usnic_uiom_chunk *chunk; - unsigned long locked; - unsigned long lock_limit; unsigned long cur_base; unsigned long npages; int ret; @@ -123,10 +121,9 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, uiomr->owning_mm = mm = current->mm; mmap_read_lock(mm); - locked = atomic64_add_return(npages, ¤t->mm->pinned_vm); - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - - if ((locked > lock_limit) && !capable(CAP_IPC_LOCK)) { + vm_account_init_current(&uiomr->vm_account); + if (vm_account_pinned(&uiomr->vm_account, npages)) { + npages = 0; ret = -ENOMEM; goto out; } @@ -178,7 +175,8 @@ static int usnic_uiom_get_pages(unsigned long addr, size_t size, int writable, out: if (ret < 0) { usnic_uiom_put_pages(chunk_list, 0); - atomic64_sub(npages, ¤t->mm->pinned_vm); + vm_unaccount_pinned(&uiomr->vm_account, npages); + vm_account_release(&uiomr->vm_account); } else mmgrab(uiomr->owning_mm); @@ -430,7 +428,7 @@ void usnic_uiom_reg_release(struct usnic_uiom_reg *uiomr) { __usnic_uiom_reg_release(uiomr->pd, uiomr, 1); - atomic64_sub(usnic_uiom_num_pages(uiomr), &uiomr->owning_mm->pinned_vm); + vm_unaccount_pinned(&uiomr->vm_account, usnic_uiom_num_pages(uiomr)); __usnic_uiom_release_tail(uiomr); } diff --git a/drivers/infiniband/hw/usnic/usnic_uiom.h b/drivers/infiniband/hw/usnic/usnic_uiom.h index 5a9acf9..5238d06 100644 --- a/drivers/infiniband/hw/usnic/usnic_uiom.h +++ b/drivers/infiniband/hw/usnic/usnic_uiom.h @@ -36,6 +36,7 @@ #include #include +#include #include "usnic_uiom_interval_tree.h" @@ -72,6 +73,7 @@ struct usnic_uiom_reg { struct list_head chunk_list; struct work_struct work; struct mm_struct *owning_mm; + struct vm_account vm_account; }; struct usnic_uiom_chunk { From patchwork Mon Feb 6 07:47:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53058 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105421wrn; Sun, 5 Feb 2023 23:51:00 -0800 (PST) X-Google-Smtp-Source: AK7set/PNTBhfHwLn7I6fWNHl5btozX+KmM0/juTsuSPkLofSZBvVvdIudU7lnGH0KLS0GpLGaU6 X-Received: by 2002:a17:907:7887:b0:877:573d:e919 with SMTP id ku7-20020a170907788700b00877573de919mr19496879ejc.20.1675669860801; Sun, 05 Feb 2023 23:51:00 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669860; cv=pass; d=google.com; s=arc-20160816; b=JhmiFyBQzU9cjpgsDBRkhvGOPMJa9oGl3AZ3tKyvwkshN7WYu7oQvWKFF82vFHx2av /J0wcEHU4SZx3fNHg40yV4S0c0SlczEGJ95m/FqRdCS1wbl9LBIewf5phQJHmLgS1gk3 TPEPAmTVLthbSqjse1AtEB0fXE3FhPPGgHMIsQzZYXCzKBwKGc2wh62WkgQr9oJzk77E nAcYyaqKaGiU5SX3SsApjkEbzLK4VmItxmh4FqB4ASNBuNt53sY3+6NpZyb33uEhyaxG obcBtUFXOZc92Fw3eYPo4M5ucbJz0mqqjhrxncEjE8vV8E2VWxPDl4DrA6MytqK8ytrd da0w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=9EJyBunn0zT5d/t0YMoiL/ukwvRArAw0I+071Y/Pm2k=; b=Y5EfOUQn5ZjemTYAR9zmTLHueMSSRf4YZGq88nmhG7oSv94tJc27j/WNz9JGiVUc5w bVtT9cTRqGp5sTE3oe0baS1H1uBLRgjoK4l7p3+hb7IJDaVT+FCsa2VcL/GERSQ6z8i6 qK92Wk7ZBeTPeNLj/fi3F0Mbs3jEmY0A6gLwkB4UL+YvLuo701cWvgcb4pvP9AnmfT/q lu5QiCwyQuEArb2h22tsVB14Eg407FXIHDwCjIGxlPkXkqqvfGoyU/xNsrLJBBRnYvPh LMwQROHRIIgDl6CU55qaFdQy5CFBOcGeIKX5kmc9J9a5+ZMxaVW6xJgKjSz/5Nnpmo7v 3S1g== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=HdDvwTln; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id vl9-20020a17090730c900b00887f7d97f5bsi10876009ejb.29.2023.02.05.23.50.37; Sun, 05 Feb 2023 23:51:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=HdDvwTln; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229892AbjBFHuC (ORCPT + 99 others); Mon, 6 Feb 2023 02:50:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229921AbjBFHty (ORCPT ); Mon, 6 Feb 2023 02:49:54 -0500 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2083.outbound.protection.outlook.com [40.107.95.83]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B96CE1CF75; Sun, 5 Feb 2023 23:49:25 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FTNiPCF3AmyP3MYgZuTvz3MSn9Uq9zP3GgybfDDeC3e3lCUyL0UGqIWzr+L5C71E7bgB6hMmGqYga3WpRF56apR1HE3eRZlWxASaqHKXKdrBokTb7Ozg/gO+l2MC4Ld5RljeihEUYBK9IM+kWk+geqtPeyJJkTgdOKsQKcPlNU2CJzCmfZD3HRuO5byjQ3bAfKjWgC2LTdbxt81fe8oUrE5q0F7fKIWmJ4GxjfDOgl2M5gLeYH+QNuhWmmTTyxMbTmy4wRn+SpAGi/WUyBWrFnLE3DX+YUX78/SNDZ+BGjfT8chJUx110ciY+Y5Djwrz6rWHr6YPWf6uMI4gQbl1Cw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=9EJyBunn0zT5d/t0YMoiL/ukwvRArAw0I+071Y/Pm2k=; b=VDyB2Hv2n45OEPJ8dQ9xlV4KOt3ZfMypAsGfaul0is5oM+cezUOq6+OHZ0fUJovFJF1q4cWJJgOKxMxDGFhGamJggn1ODOBfp0mEGCjfNHnbqtTIbsc94O42ORC7WJJ7pMeTSnqikVAvJ7uTW+vnSfpNgvtObwjqcPrr7oR96GErwF89fjgxsXW3FI0ibBHxFTl55wvrb8NvlK1yP1sgcni4B+6jPZDnPel+17kjARloof10xce5QhXn5ojJwHKo2qYOUHXUIGykKRujWbmldQPbSl2WPmS87CwCt2Yj9P0m2AuSYfTMBu9F2iZ2T8mQ5lZxxGJTdwndM+Tm4KVQJQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=9EJyBunn0zT5d/t0YMoiL/ukwvRArAw0I+071Y/Pm2k=; b=HdDvwTlno/2TJFtEwBIiS5c+AD1oEMvYqSxurzNyHD01OoIabNHwyzh/mIkTTgM9RtXxdonmgc5LImTBneN3gImrJwxvCtO3prL0TKaAWcP2MQIz0mjULA2GezhsbkxUfHouo9nN2zMGi79arJIK2cKCP0k29UU3oY1Vmesh6XMNMyXzk6ZdTnewXJuiGT0l1P6WEaPvoOtgok2ByKpKE+ZvFtpqZ8PxHNcfPKVqlAxX/8758AmEMs0krkoG0pqz0I6A0XwdxrMuKnjldSim9waR2MPiOJ0ZDre6eK3KkIk9o3QjRm4x6CRM0eqLykse91XKNIZp1mbef0MmDS5IEA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DM4PR12MB5118.namprd12.prod.outlook.com (2603:10b6:5:391::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.31; Mon, 6 Feb 2023 07:49:00 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:49:00 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , Cornelia Huck , kvm@vger.kernel.org Subject: [PATCH 07/19] vfio/type1: Charge pinned pages to pinned_vm instead of locked_vm Date: Mon, 6 Feb 2023 18:47:44 +1100 Message-Id: <60d8812272042f497b8d661f031f2cb593370b84.1675669136.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYBPR01CA0191.ausprd01.prod.outlook.com (2603:10c6:10:52::35) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DM4PR12MB5118:EE_ X-MS-Office365-Filtering-Correlation-Id: a122e17a-0e75-495d-a900-08db08169bd0 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: D19c20F/3H5aBZUJtRuOisDUPLPCb3g6LkxaoePLUkkit594ylhxEln8lwG/d82+XWI05rsnfmCDpC1vsCuFjbdq0T+KCv+KHNH3Z1aAqkjU33XcFrBUQZbMEeFQgtvkBiGTeIJSKIQ2MddZlxG3YfOmSew5kvepMZq48suLfmTQwBEwo33I70s+Ja7PUUn3QNSE2FUThE0XFwbxgUaMIVe9GUF7NxVwc0qy2wj7pQXxrzDJgUHC0Q/JBTDXEsmthnznbrRHDiwJWaA+aWWrSoIXSx0VRQWGcVYheLkGBLu+xXeyIC2Gw18dBemMc1lxOq9jPC/DEAWZgqE7doc79lVQdyjo3KWLVWjm0JFbRAuXGheBNJAPf+5889LNCR5KX85ktUrTfOcg9QmjlEGFM6ixS39AqYugLMq7WkqF9RollRnLkbbzSyyHvE7kVodm3SJcWO9ptdE4zFYUWLipDugvSD+eqj9yI9v3B3/ox962XFxUFNzLjsjNCmQByB6eqtA1ZuSgg2tVdvGMkg9/RJlmsGzydh7de7yXSi9TUo5rfcq4YSgkGmaVEU3CzfVio1l9S/WGrRARhBlmNFsOVmZEcnOu6RLIM+iasIdbBDJLU4TfVxUJv/FRYQK+cf6Yk+LM9OBZtSt76bhD3O/kNw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(39860400002)(366004)(396003)(346002)(136003)(451199018)(66946007)(66476007)(4326008)(41300700001)(8676002)(38100700002)(8936002)(6506007)(6512007)(26005)(316002)(54906003)(7416002)(66556008)(6666004)(5660300002)(478600001)(6486002)(83380400001)(186003)(2616005)(86362001)(36756003)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 7zc1D3qcVz4pASVKaNOAYrrUtV515FbAbdEHWZkkbDHYjdqtKw0XPemXYJSYwcfzLz2N9qbtXnZJfs0tBfDB3GSEH1wVwNPZ1X6UY1BJ5KrPEplmF0HpxUZPsWX/gkHfhPPhl/FvT4t/8RXEzjCW9utzJPt1FSXPpNFy0wJvExmYLYGZliTc1vfC87bBfJ8P/upzU6ZwIuW0LWEBZ0TU7hNJQUK2ux5fpkRiux5QqH1EpPELod6+8+qHoXHKoWqjCAOCNFCIiM5F3sMg3f35sg62LsokIe5xTlMQWHSYlIOtv5HNIHWZaLJrqiPqD5GHCUJMUoruk4gxNmPkdWy4YTA91t2WFJajEHXQchyNePxdNSsLESXHZEPRmIfHIWUa/FXfhK2k5oLJ7EUi/Hz/vg37L2ip9MzKCGP1f1MtrZem2u0dS/oMmRY8wcCUgiedpGDfPp6TAzB4l3RAwnGq+VoSbkO/OEB1qAO/IY6mkMicIxQsOK1U7b/1SMMw1k/31EwMPMRVYNpKy3Qi1si8fFkTaNjcfpUMamA2bQl8HojR4dbZvu+kj2aXstteEXKk7+phR0UYv7Y4isQsqxem2f9QEtruYJGvO9qghNvs1VaZ/0iBS+N0UEzwEN663iGblYAGHlYVrnOU8sUd6YsOSQuba1ohT3+BzDMfhKym0u36eIbfwLYO6tjopztiaoJg77KqGbYGTr1ClDpDZH6yI6h/bfpBWef3lfgUoXTqskTgc+xq90tObDDdTXwsIOy6avYGH7xDcfFJrWFQgsNt3x1CObnIpEO2yHE+n1kw7WXR/wR8svDv2R8SJ/ycXZTmlZ3WogEY3gW/hBiaso/oZdTSl76qJGT1pwqVQCFjZsuJyxXjned/0NkrGnTQy9RGjWMWa3Y7HoamZbppfx3D+cZNoVUH4A7mdhLi/of3bKSbp4gnekkEQAhLOi7cGLz2UbtrH0Ji1fAe984WujANhSml4Tl4OQjAGSUPKQwWBScmb0mebOu2Ze2Lc6wR1cbriRon0gWwJNCqCWiyLu+yxeCWaanKbvUwSI3xWAClIRq+F6Hel6j33cT617ZUEojAfuS5quvUQH+xd7teq9ytucZb4M52lvZAzsiCjuLODlYHuwwfsEKMhu7PsKg1viWrQGxHxBax5NhSPmSsFb73FBaBWlSOq4ulWZQo1M0Qs28g3IjtoENbr7Wdb2t9E3AFJoaeQYNUzwrSDwOE7ARLQrq7us7gDNOGQR62g+f88SHwPq/D/D/hbmBVJYGY0JqE44wM0Zshh2j5g4t1znhkLLxDJuFvfefxHF6Z94MZGwci8zh+bwxhHueDNR/jB1YC5/krCaVWho7fV4J4PEPyfHN9/HhCvdBhhdA9DbjnXenyy1WwDwj0rojUpYu/JmgUwY578KclsGklzVMkBXHquwuys7EGWL+DAgJYfIIWMop+ns4bOfJ1/nduNR+t2xGPB8cOeXF132uSMdDue+oaGLwGRzbK4ltbp+RExjC37GcxTHoXk5J2BXSYY7x4hlNeGz425lHvovqNcU2nJ4s8/HNl3flko8QKgwAAa8P2g/ug2lOCRA/95k2JHR2Kid1U X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: a122e17a-0e75-495d-a900-08db08169bd0 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:49:00.0753 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wp+R7my8ypqqhhSOfLO+TVLqtpDWGatzGK9uJJC/Lw68ZvcGUlzUXp6UMcD/A7fO22cL5YxY86EhX2g6PtdhlQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5118 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067199680500516?= X-GMAIL-MSGID: =?utf-8?q?1757067199680500516?= This switches the charge to pinned_vm to be consistent with other drivers that pin pages with FOLL_LONGTERM. It also allows the use of the vm_account helper struct which makes a future change to implement cgroup accounting of pinned pages easier to implement as that requires a reference to the cgroup to be maintained. Signed-off-by: Alistair Popple Cc: Alex Williamson Cc: Cornelia Huck Cc: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- drivers/vfio/vfio_iommu_type1.c | 60 +++++++++------------------------- 1 file changed, 17 insertions(+), 43 deletions(-) diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c index 23c24fe..a3957b8 100644 --- a/drivers/vfio/vfio_iommu_type1.c +++ b/drivers/vfio/vfio_iommu_type1.c @@ -38,6 +38,7 @@ #include #include #include +#include #include "vfio.h" #define DRIVER_VERSION "0.2" @@ -95,11 +96,11 @@ struct vfio_dma { size_t size; /* Map size (bytes) */ int prot; /* IOMMU_READ/WRITE */ bool iommu_mapped; - bool lock_cap; /* capable(CAP_IPC_LOCK) */ bool vaddr_invalid; struct task_struct *task; struct rb_root pfn_list; /* Ex-user pinned pfn list */ unsigned long *bitmap; + struct vm_account vm_account; }; struct vfio_batch { @@ -412,31 +413,6 @@ static int vfio_iova_put_vfio_pfn(struct vfio_dma *dma, struct vfio_pfn *vpfn) return ret; } -static int vfio_lock_acct(struct vfio_dma *dma, long npage, bool async) -{ - struct mm_struct *mm; - int ret; - - if (!npage) - return 0; - - mm = async ? get_task_mm(dma->task) : dma->task->mm; - if (!mm) - return -ESRCH; /* process exited */ - - ret = mmap_write_lock_killable(mm); - if (!ret) { - ret = __account_locked_vm(mm, abs(npage), npage > 0, dma->task, - dma->lock_cap); - mmap_write_unlock(mm); - } - - if (async) - mmput(mm); - - return ret; -} - /* * Some mappings aren't backed by a struct page, for example an mmap'd * MMIO range for our own or another device. These use a different @@ -715,16 +691,8 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, * externally pinned pages are already counted against * the user. */ - if (!rsvd && !vfio_find_vpfn(dma, iova)) { - if (!dma->lock_cap && - mm->locked_vm + lock_acct + 1 > limit) { - pr_warn("%s: RLIMIT_MEMLOCK (%ld) exceeded\n", - __func__, limit << PAGE_SHIFT); - ret = -ENOMEM; - goto unpin_out; - } + if (!rsvd && !vfio_find_vpfn(dma, iova)) lock_acct++; - } pinned++; npage--; @@ -744,7 +712,11 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, } out: - ret = vfio_lock_acct(dma, lock_acct, false); + if (vm_account_pinned(&dma->vm_account, lock_acct)) { + ret = -ENOMEM; + lock_acct = 0; + pr_warn("%s: RLIMIT_MEMLOCK exceeded\n", __func__); + } unpin_out: if (batch->size == 1 && !batch->offset) { @@ -759,6 +731,7 @@ static long vfio_pin_pages_remote(struct vfio_dma *dma, unsigned long vaddr, put_pfn(pfn, dma->prot); } vfio_batch_unpin(batch, dma); + vm_unaccount_pinned(&dma->vm_account, lock_acct); return ret; } @@ -782,7 +755,7 @@ static long vfio_unpin_pages_remote(struct vfio_dma *dma, dma_addr_t iova, } if (do_accounting) - vfio_lock_acct(dma, locked - unlocked, true); + vm_unaccount_pinned(&dma->vm_account, locked - unlocked); return unlocked; } @@ -805,7 +778,7 @@ static int vfio_pin_page_external(struct vfio_dma *dma, unsigned long vaddr, ret = 0; if (do_accounting && !is_invalid_reserved_pfn(*pfn_base)) { - ret = vfio_lock_acct(dma, 1, true); + ret = vm_account_pinned(&dma->vm_account, 1); if (ret) { put_pfn(*pfn_base, dma->prot); if (ret == -ENOMEM) @@ -833,7 +806,7 @@ static int vfio_unpin_page_external(struct vfio_dma *dma, dma_addr_t iova, unlocked = vfio_iova_put_vfio_pfn(dma, vpfn); if (do_accounting) - vfio_lock_acct(dma, -unlocked, true); + vm_unaccount_pinned(&dma->vm_account, unlocked); return unlocked; } @@ -921,7 +894,7 @@ static int vfio_iommu_type1_pin_pages(void *iommu_data, ret = vfio_add_to_pfn_list(dma, iova, phys_pfn); if (ret) { if (put_pfn(phys_pfn, dma->prot) && do_accounting) - vfio_lock_acct(dma, -1, true); + vm_unaccount_pinned(&dma->vm_account, 1); goto pin_unwind; } @@ -1162,7 +1135,7 @@ static long vfio_unmap_unpin(struct vfio_iommu *iommu, struct vfio_dma *dma, } if (do_accounting) { - vfio_lock_acct(dma, -unlocked, true); + vm_unaccount_pinned(&dma->vm_account, unlocked); return 0; } return unlocked; @@ -1674,7 +1647,8 @@ static int vfio_dma_do_map(struct vfio_iommu *iommu, */ get_task_struct(current->group_leader); dma->task = current->group_leader; - dma->lock_cap = capable(CAP_IPC_LOCK); + vm_account_init(&dma->vm_account, dma->task, NULL, VM_ACCOUNT_TASK | + (capable(CAP_IPC_LOCK) ? VM_ACCOUNT_BYPASS : 0)); dma->pfn_list = RB_ROOT; @@ -2398,7 +2372,7 @@ static void vfio_iommu_unmap_unpin_reaccount(struct vfio_iommu *iommu) if (!is_invalid_reserved_pfn(vpfn->pfn)) locked++; } - vfio_lock_acct(dma, locked - unlocked, true); + vm_unaccount_pinned(&dma->vm_account, locked - unlocked); } } From patchwork Mon Feb 6 07:47:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53060 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105548wrn; Sun, 5 Feb 2023 23:51:30 -0800 (PST) X-Google-Smtp-Source: AK7set+207YrBNUECkEfYX4XbqnKNXp1G5MO3CrPU7xK/JX0R5Y42Byz7JgHpF5bvb3Row5HuB6x X-Received: by 2002:a17:907:6d17:b0:88d:ba89:1843 with SMTP id sa23-20020a1709076d1700b0088dba891843mr17222406ejc.20.1675669890678; Sun, 05 Feb 2023 23:51:30 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669890; cv=pass; d=google.com; s=arc-20160816; b=H+5Phfj9KgifjsguJvWSlHOxnbM9ZTIBw9rVtYDoWha0J5C801K9cLY8f92mYwaIir DmNfP2TbrZxwcGYuRw4O+u0vpf5jm1fH1xDb+vWAZxMfz0v5XfO9cPlnNmzCX4nq36Np RCAJD4+kypzBNohrt5CoVUDuSNQTXg5itrSjs0mDQiHH2pBKXUlyWmBDH/nxDv+PFUxv MU67BuQQopiwY9++X2FtMSMffPpRnW3oWqSwQGwDv9ZpTauH9DrstP05ReZ65aWHZ9e8 pM1WOdbQJxFgvSZHHTGZe8qU/QcCdHmAQtLbzdfIuUndaOByyfAwD9vwCHd0H+cSK2uy GcBQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=4aNGQ6CEFF/Iwo4DWZWCLfPBaTkYJe6nei2IWX2bVLk=; b=qKLXMYamYVHrIAaAMB06paBYPiVzZcnbLBMJy3/k+Z0avacD2G1NJKvm9EaQ/rjFQ2 MvMnOg99HprVc6OdX1L7KV8rVCFuwPCfZqQsgtAyJD5DEOfZ6fa+HpQc75kt1j9JsErM XbaOwgP17+QH8KmtPUHQSHqHivyVewC3tB1DaHEeIUym+yvW/MskQfB/YZ5ZZhFfI7VK Mria/yBKjTeMXJUdmIyaj7+GYolgZKndsyyxWXwo5DJD8xMMtYGCUxVs4sulqTFqJVc3 EhkocDbgzYN0NE4Ir0kSvfogwu1DeBXATd460+DIHmQuYtEQTmeGGChZQUXxRYohV9BH sQjQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=VT3h3Buu; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n5-20020a17090673c500b0087bd1423fdasi15147519ejl.910.2023.02.05.23.51.06; Sun, 05 Feb 2023 23:51:30 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=VT3h3Buu; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229844AbjBFHua (ORCPT + 99 others); Mon, 6 Feb 2023 02:50:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229788AbjBFHu0 (ORCPT ); Mon, 6 Feb 2023 02:50:26 -0500 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2083.outbound.protection.outlook.com [40.107.95.83]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F67518B2C; Sun, 5 Feb 2023 23:49:54 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ScbFAXZwDSHDQYWfo+yBKa5EfS66fkePVN0KUmOunWodLhHCV4ULVvfwNauI0zQ2eBvnik0jEi90CMzWz5DR1c6Cz2v8NQaBE+tWBt7nr4z0CWa10xiznBdgp1Nm+k1z4xDUJ7ye12qJ7T+J4cEtfuMMrxPdgZ3XbBqw6GRxD/ZwtvmU5PvrZGd1TDzht3yybqzIY3WAfqqINM5HegVLwJ+xrvke7Cb//NFdGo4yyK1cll0HqzKuL5jQ3bDL7w+1MLnYchBhYmlkXkwOAb1VEKqP6k55wRdeIn1vd7jSMr0Otg5wbt/EgYcaeFtLjUUsCOnTXcifHQvh3wCKcJH/mA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=4aNGQ6CEFF/Iwo4DWZWCLfPBaTkYJe6nei2IWX2bVLk=; b=LqvQqbW1VJRiXVEeYE2GmDfVFnPTJiQFDnWiyaE2fKtllJZNL8ZrjRiGcS+1GgtroJtApXou7U1szAKFuscrS8aTGqqBt1H1R4fte3aXwMG2Fm8ZpDmj7WmM3I542Qd5Itnh10o4X0h/BC6C2Dke/IswXSCFqovNS6UHd3FJwaDpvMHShncjv/gUFDEsj+ZCtlv29xxAX+8YqNctu7D42Y8wWdhoKx2wFvKSnwhCzUNWDZgMSg5ZTDnhECuel5n0D2Z6CYfCro/c3uCQnvcaAwRFbg2s7k6sgVTccfZQlbemI34922wSM58l57WYsgNnbpIEbY1eC4nSpEToU8vadA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4aNGQ6CEFF/Iwo4DWZWCLfPBaTkYJe6nei2IWX2bVLk=; b=VT3h3BuuFakyBa1tv3zMLWgdy0gFVM4xRbhSGmYmbGmqzVcCtyci/LlF1QBXJUbYcfqQfRAGEzvCGyS97l5/O8gteE7GX0CdtPbs52p6BURnxvnjHJspqYOjLDaThTCu4IvpaqqjDxhkgNVXSsp9lkSnDOTmiGW5EtiOkLLfcHwq6/fZSwaW5ITif9x+5mFkec0S+WcCxKzehbDCIKbbn/BUXbT/NNndSJ4xwCA2g+piUzTPeaOveLcnKjI/epZ6yyBYT86+NrBi/q8u9ZzZfezDPbPUSCaxiY05/P+coJEKaaS0S864sYfyKsSPOyxFPjBpmlmG4Jslgofey51zyQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DM4PR12MB5118.namprd12.prod.outlook.com (2603:10b6:5:391::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.31; Mon, 6 Feb 2023 07:49:07 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:49:07 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Cornelia Huck , Alexey Kardashevskiy , linuxppc-dev@lists.ozlabs.org, kvm@vger.kernel.org Subject: [PATCH 08/19] vfio/spapr_tce: Convert accounting to pinned_vm Date: Mon, 6 Feb 2023 18:47:45 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYXPR01CA0088.ausprd01.prod.outlook.com (2603:10c6:0:2e::21) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DM4PR12MB5118:EE_ X-MS-Office365-Filtering-Correlation-Id: d4a9ea4d-62d6-4d24-e89f-08db0816a035 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iWUpf7QiSNpZ/BcD+AET9d7eIgcFareDzNOWhk99Dr/Ree6FI7pWqkP+D6TDXFKgY8TZ5XQ+RaPznI/kT55B3NJLbNkZrC6XyIUQ+uS+KzEPaNXGt0SPCwRVLF/egr6P5SexkJX9JwUJcKB6DKsLauOBEqRBUrgsWkSdBZRL0xisfwyTYcfgT4aUxOD6PveYHjh05ByLg2KqTXLbEZegTendZQWWSRguwakP8Pu4pSc7oSAKtf47yRVRj7sZvUKxHjmIEpwt1UXuONjAYH/+WTs8eDQTKj0+Oc96d5UOKXKIDxeYnAgNSeyKs3qWuwVmGdPoEdfHE8Bc0z2JrhYs44Z+51tYalcIMwPp2JWpLdeLtopMhcyR7CxIoEn/6FV/6Jp1+MQ5ymTSs91yipbEPNSwq00AGa1/MCqsZ+UT+zC3FmhhkdG6TjvRt9Q5e3Uj6xOskk3CW2y20+iqlSdM8liDU7ATBa+RDMqKOuczBmU5xlRR+4UIF1MlbNhlq2d2zlO9AF0Xat+u3C++9xus7by4OHqgYNtrtBtz723yxLGYW/72hska2T12DgReKGk+zTVqtepdD92mhiMdFY6VpIiUoEQN5D6+OhZR/0iCP0csDkfh2HsLjhte2EcaA2JjFmeXrzRdyu3g3649c31Alw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(39860400002)(366004)(396003)(346002)(136003)(451199018)(66946007)(66476007)(4326008)(41300700001)(8676002)(38100700002)(8936002)(6506007)(6512007)(26005)(316002)(54906003)(7416002)(66556008)(6666004)(5660300002)(478600001)(6486002)(83380400001)(15650500001)(186003)(2616005)(86362001)(36756003)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: J1lKquI9iAaemJegDUiKMox/qBsp+Xu2+DY++QRpmFu2QoWhovc9ib2zZo1G0eBx99eI1o60+2qoVPxEZsYAur9+TbUbor5EWDFYAuP2q7YyF9B6C64YH/301qDJdftQOmSC9O/wYd2RBUvPyPAtIjIj+4zEkJoFQcRlDv2hyMunJFFSTcWWm2TSqAbQoCTBKJ4kD0c+LSHsF9HYlZfQlkh/8X1snQVDsd/zBgU7rZ3R1zTmWVnm6jTDddQm7I4gRRnTHtyTGNe6NCMsB6eMHU6qgPaSF/A5iwsn3NnN6qUwU0tRwF1twT38zBAO2NmATvUJHDJkW8PixdBPoO+gbsDzPdxnaZRdo+zpnOZhvVx6SBtGJlT4Y2dySOAyVWVsbsqWpcs/tl1cd/zRfsC53ujuYWZ+q++5HwybfvVAqsiviPHv7z1K1x2Vdda6TEeH6S2rMjBjkvZHTVJSeVuwfs4+tcrLpECxV3HeiAy7oTrEv7IGbFVzF+C7KcRSaAM2SUv6gsQ0a+iGwqBPcsLyqxwZ3dCH7CuC57njt7PrfshTdkcNSU5G2vDOW2buKTZxe6F9NA8BhDn+ALD379YY08kmNpy4kpS1ZSQcvorw1Kk38j2qw55ljmZ0NTHktplOQN0f4sKdOQ6Qudf8nkM4Ndjr9+CHNdCNQl/eN5Pqc6J76tiB9XKevtclW8DcWajvicHkxDvmfAV9oQ6/WykblpQINY8vH87IkUbowuOC+RVsWg2P75oMch5v2sQKse4G5uWvEgHg5oHoALR0nlWwzLO21bn9UK1xSQipB//zA8WKE4DL0JsNmHlR8KYRB34ytvK2+ls4w8ZZtNKSR0uSnvuG3VsPsiN0p4Wi/L56Lqt/GYJ6Bl58SO8A2OV8gL4hfQxFcTFbreeXUQ160Nt9x4dbwL+4QckFmppuSqIsZ3xtQuc63FiQ+NpwvWpIgw/np0Al11hPivOjhWvEFAYq6AcMDRaqLEaKN+FOREWiQ6CCHlCZ6HrY5flgoIQdqsjJAOm/9Q8JeueN2rctmsBL4GkuebsbN4eXwBijfAbXafq44Q4x9GyPdBxLHorrrYJ/tn5I0Tv8rf7gmFnF1Gwgoa9Z6U1qsBQN/Ys9PH7YQBWH4k0wLqEREtpcSiPuGCwTi/GkR6g69bcmlTyOOzzLUnkc9KXHwQImA2mNPANldwc3PIAs1ytmZjp0RQeEwDUswEAULDJudNK/+SvQEHw5pT1en25CnrWqJwf/RUfu5z0UNIbJWPXyIIP7CmHy6+EEtc2EwZCOIvNZU1Sw8bYkESR21uXdb6dbYY3bW9cVZmmyVgDY2Rz2mjCsDjTMf1LFth/trwHqzeyDGpj46YlT44Tm1EIl+x/8/b+yT4bvEi1owVKwMAgyHXUrIJSEWMReQAr/YxRNRwlPMQ+d/5kYPfdqtvRAPQQYkWmWrEhzYYWR9Zf8I/2LwYnjCWbIy+234kzcnayjyKBTeqaQSvoIrUEu5N/xxoYTRZC2RUNO56usZwjOJ3Xg6iwgmPyci3b7yOmY0q02UDrefLlnVT97xyfmbwrDsxQTRGpDnGXMdzRSo1mAb+1+GPViNtvykK01 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: d4a9ea4d-62d6-4d24-e89f-08db0816a035 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:49:07.4497 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 52T72HI+k1c4VKZrgbWoZZ7MGyhRNB9OFQRaWOU/PKhx3Et+QT3kFLjIAGrHRAdgJ7sUVRlsW6ZUl94FqMJZnA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5118 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067231226075122?= X-GMAIL-MSGID: =?utf-8?q?1757067231226075122?= Convert from accounting pages against locked_vm to accounting them to pinned_vm. This allows struct vm_account to be used to track the mm_struct used to charge the pages. A future change also uses this to track a cgroup for controlling pinned pages. Signed-off-by: Alistair Popple Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Alex Williamson Cc: Cornelia Huck Cc: Alexey Kardashevskiy Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org --- arch/powerpc/mm/book3s64/iommu_api.c | 30 ++++++++++++++++++----------- drivers/vfio/vfio_iommu_spapr_tce.c | 16 ++++++++++----- 2 files changed, 30 insertions(+), 16 deletions(-) diff --git a/arch/powerpc/mm/book3s64/iommu_api.c b/arch/powerpc/mm/book3s64/iommu_api.c index 7fcfba1..338b111 100644 --- a/arch/powerpc/mm/book3s64/iommu_api.c +++ b/arch/powerpc/mm/book3s64/iommu_api.c @@ -18,6 +18,7 @@ #include #include #include +#include static DEFINE_MUTEX(mem_list_mutex); @@ -30,6 +31,7 @@ struct mm_iommu_table_group_mem_t { unsigned long used; atomic64_t mapped; unsigned int pageshift; + struct vm_account vm_account; u64 ua; /* userspace address */ u64 entries; /* number of entries in hpas/hpages[] */ /* @@ -62,20 +64,24 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua, unsigned int pageshift; unsigned long entry, chunk; - if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) { - ret = account_locked_vm(mm, entries, true); - if (ret) - return ret; - - locked_entries = entries; - } - mem = kzalloc(sizeof(*mem), GFP_KERNEL); if (!mem) { ret = -ENOMEM; goto unlock_exit; } + vm_account_init_current(&mem->vm_account); + if (dev_hpa == MM_IOMMU_TABLE_INVALID_HPA) { + ret = vm_account_pinned(&mem->vm_account, entries); + if (ret) { + vm_account_release(&mem->vm_account); + kfree(mem); + return ret; + } + + locked_entries = entries; + } + if (dev_hpa != MM_IOMMU_TABLE_INVALID_HPA) { mem->pageshift = __ffs(dev_hpa | (entries << PAGE_SHIFT)); mem->dev_hpa = dev_hpa; @@ -175,10 +181,11 @@ static long mm_iommu_do_alloc(struct mm_struct *mm, unsigned long ua, unpin_user_pages(mem->hpages, pinned); vfree(mem->hpas); - kfree(mem); unlock_exit: - account_locked_vm(mm, locked_entries, false); + vm_unaccount_pinned(&mem->vm_account, locked_entries); + vm_account_release(&mem->vm_account); + kfree(mem); return ret; } @@ -229,6 +236,7 @@ static void mm_iommu_do_free(struct mm_iommu_table_group_mem_t *mem) mm_iommu_unpin(mem); vfree(mem->hpas); + vm_account_release(&mem->vm_account); kfree(mem); } @@ -279,7 +287,7 @@ long mm_iommu_put(struct mm_struct *mm, struct mm_iommu_table_group_mem_t *mem) unlock_exit: mutex_unlock(&mem_list_mutex); - account_locked_vm(mm, unlock_entries, false); + vm_unaccount_pinned(&mem->vm_account, unlock_entries); return ret; } diff --git a/drivers/vfio/vfio_iommu_spapr_tce.c b/drivers/vfio/vfio_iommu_spapr_tce.c index 60a50ce..454ccc4 100644 --- a/drivers/vfio/vfio_iommu_spapr_tce.c +++ b/drivers/vfio/vfio_iommu_spapr_tce.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "vfio.h" #include @@ -67,6 +68,7 @@ struct tce_container { bool def_window_pending; unsigned long locked_pages; struct mm_struct *mm; + struct vm_account vm_account; struct iommu_table *tables[IOMMU_TABLE_GROUP_MAX_TABLES]; struct list_head group_list; struct list_head prereg_list; @@ -82,6 +84,7 @@ static long tce_iommu_mm_set(struct tce_container *container) BUG_ON(!current->mm); container->mm = current->mm; mmgrab(container->mm); + vm_account_init_current(&container->vm_account); return 0; } @@ -291,7 +294,7 @@ static int tce_iommu_enable(struct tce_container *container) return ret; locked = table_group->tce32_size >> PAGE_SHIFT; - ret = account_locked_vm(container->mm, locked, true); + ret = vm_account_pinned(&container->vm_accounnt, locked); if (ret) return ret; @@ -310,7 +313,7 @@ static void tce_iommu_disable(struct tce_container *container) container->enabled = false; BUG_ON(!container->mm); - account_locked_vm(container->mm, container->locked_pages, false); + vm_account_pinned(&container->vm_account, container->locked_pages); } static void *tce_iommu_open(unsigned long arg) @@ -372,8 +375,10 @@ static void tce_iommu_release(void *iommu_data) WARN_ON(tce_iommu_prereg_free(container, tcemem)); tce_iommu_disable(container); - if (container->mm) + if (container->mm) { mmdrop(container->mm); + vm_account_release(&container->vm_account); + } mutex_destroy(&container->lock); kfree(container); @@ -619,7 +624,8 @@ static long tce_iommu_create_table(struct tce_container *container, if (!table_size) return -EINVAL; - ret = account_locked_vm(container->mm, table_size >> PAGE_SHIFT, true); + ret = vm_account_pinned(&container->vm_account, + table_size >> PAGE_SHIFT); if (ret) return ret; @@ -638,7 +644,7 @@ static void tce_iommu_free_table(struct tce_container *container, unsigned long pages = tbl->it_allocated_size >> PAGE_SHIFT; iommu_tce_table_put(tbl); - account_locked_vm(container->mm, pages, false); + vm_unaccount_pinned(&container->vm_account, pages); } static long tce_iommu_create_window(struct tce_container *container, From patchwork Mon Feb 6 07:47:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53063 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105672wrn; Sun, 5 Feb 2023 23:52:01 -0800 (PST) X-Google-Smtp-Source: AK7set9SnvAIzVVqlqulbjstC8+wARmvJVYot3SX9zREo4dfUzBwyJCLvjPIcOUQFCxHZcEpFJbP X-Received: by 2002:a17:906:24c1:b0:87f:5d0a:c610 with SMTP id f1-20020a17090624c100b0087f5d0ac610mr17035345ejb.32.1675669921300; Sun, 05 Feb 2023 23:52:01 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669921; cv=pass; d=google.com; s=arc-20160816; b=mU/D53ElqDwGSp7YAi5Sio41KSI8gyDIaGVWTbaEC2cR/WSzTVEUBAhFDas4JsUzGw ztCSvW85ouo6NocEZttvEpi90CG2JFYA4Ym7kiIjDsvltjRPY113BvgLyQogf94otVIX j6IVNnhYgfV+npnE/AC6PsVTr/sDMy5nRmO2CVGVqzh8/tqTehLBh5K4riBM01ZkbdfD jleClrpB7wxTZODvAJMb4qa8BAHfAtNfnvOnyL74/TQUvZucKj2FMXbKY2Y0vhd/yvUn ju3HjdDErbkamzVQVM7IYAn7k6NNni4AK/lDErTCBm+Fcb3NTATahWKUzheR0kXofTyO f+gQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=r9iLlWenkXS+qWZNyDxmBNuM9bHSYN74X2PJc4f6L4s=; b=s4RPPcf07IyJ/IMz9g9DeUjUT7/UDi9XYQF7mj6t+JQ/oS+N14F2R1HPgA/jlGoFtb A9aFlUdTsSMtsx/76WLA8qORcMD2YvPCtx91JpfIUrJQszS8xJ0WOA7E++J49HVH4YhO TKhGrnRnpzaRb5wzjeG4PMj8hs/E2YI5YORxuYsk6dy9xP9tLJ4F6IOEPQFjJuif56ZG /jiB3EIxBp12OICts9QuEjLXiB2XLWtQRS5iIjWXZqcDJx0Z/HCzHRtlQ6LBePNNnIgl sv2w+oNJMkifQ2a474Fy9Z2g3JD/ZRp9IuaQf4WfqHbJCPsGzmbMotn7vFCX2f91sWdk M0hw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Gf8LWJbG; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id f8-20020a1709063f4800b0088c0e9677b5si14348009ejj.552.2023.02.05.23.51.36; Sun, 05 Feb 2023 23:52:01 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=Gf8LWJbG; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230019AbjBFHuz (ORCPT + 99 others); Mon, 6 Feb 2023 02:50:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230040AbjBFHup (ORCPT ); Mon, 6 Feb 2023 02:50:45 -0500 Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam02on2083.outbound.protection.outlook.com [40.107.95.83]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C97D11D917; Sun, 5 Feb 2023 23:50:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Tto6HDLH7veTHgG9SgcxScWU8/ZVYhlCZNpKm0CL8b/NkVFJ7mxrz2j2ozWxv8bhr5B2K6LM7xOgWRgWNgE83eRhzhAwY+TCPDrzbUVLSEAKcZuvxj7+muE8+6a73MJ5f72vKTmzAXnYDfX4Gm7ui0L3mC5QXX/WSpF1xjRqmNU5GG9pLeQU8eZIaiWWgtfjRGfnzpSD/oOe4Ou8wQ3cDlJT0xyb1I/0MWEMRK7WsD82+ClKnPP4EzxVuUoE1pq2m/4oA/4G+DX+3LifWs8opySTBxkUGgc97DVN2yKb0mi8D6OQubT2YkjKL6lBhGVbkZ+9jsKE2YgwKQFtcx9cUw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=r9iLlWenkXS+qWZNyDxmBNuM9bHSYN74X2PJc4f6L4s=; b=US4drmNjS2dMnPKb1UZzmNp7bdWqcuZGJy7ui+bE9RCN2vl9KjlZKq5PzGdzZyPUgw0iX3XjFKFryyzCxHuevZUltMSWyAxyyYctKsi3v91ftLDg5euJbgsQjdVhtf+O87qMSe+qq2Fd2ebTjg7bq6/eBKfAzLpcepPFcNU41TnF8SNc7MjeEqKleRSHxH3/+u4LH/7ixlp4gIvuiSu3Dt4oEyxv2xX54c0zAXpk5was2XHvWeY+05DxeH+5Jmg0OIs7ziHxKAypoD02DJTAwnBOD26i4hejAGPSWGSisg5OFIZBqoWFVrISLlFSUTZnHAkZXMEYxuJQK47fbWwudQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=r9iLlWenkXS+qWZNyDxmBNuM9bHSYN74X2PJc4f6L4s=; b=Gf8LWJbGXqVUQFIga30K7D5qcFdSDCuAnHvDMezj8AWMzPHRlHJmHv/EgBw3+Pek+2ot4lkY6DzrLks7zPkBQc0mKU7N/HXV7xHUHNFXe/FjnT4ImGnQ26NfbptOrwfbPeylxKJquBK2fUlZUBJFgqjueq6jhj9Ndn42XPKFEL2403dAn3q+H8JGNR3Rc+nAG+Y5bOrcbt7BXB7TdIJhzQDUiESy9IEcx2aKaP9ZQGlk6F2V1Bdp1RfcVhmntk2GLmBchLbVhbgS63AFirVVKjYGHbKotlZw1pKWYxqYi/KkYXUd4gYpnPeE0n5GIN+7ze1yk7KsDLIBPsE4ccatcw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DM4PR12MB5118.namprd12.prod.outlook.com (2603:10b6:5:391::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.31; Mon, 6 Feb 2023 07:49:14 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:49:14 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , Jens Axboe , Pavel Begunkov , io-uring@vger.kernel.org Subject: [PATCH 09/19] io_uring: convert to use vm_account Date: Mon, 6 Feb 2023 18:47:46 +1100 Message-Id: <44e6ead48bc53789191b22b0e140aeb82459e75f.1675669136.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYCPR01CA0027.ausprd01.prod.outlook.com (2603:10c6:10:e::15) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DM4PR12MB5118:EE_ X-MS-Office365-Filtering-Correlation-Id: e6c72ef4-683f-4fab-e4ad-08db0816a3f1 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: W9IdeFq7Vrg8CD9yMMSSYTG9kGqws1YS8W2dUbffEu98uQ1Gi2Z3B6t9juAX1RdjSfZyBFpNzopuwa7jw6/x+SrXzlgcGw8isGBU+2pohoLvR01U9OX+Ml94XuHAHlblXkXIDxOJNoR4EMs4dgN1/tU21Ec4rpCw6qItcq1F4oHigpXlANsPg3/5hihtPXhGCr87JDpK6aUE6t27QIB6WMo+DqLVnxgKoRRBnYpkxcHQzMqmTEyfhuQC7Bm62o8mNGBx1I4JK9Gk1KKbPWKn/yXEaQ1VsO/4kH0WyJiWAAed/zs7hMDmKwjPjBfyOxtgmadSUSKsmjZoLjAgX/t/gbyDEEY4fYHP+okBJlHgb8vJ9Yog/ZuBJOBZ0kx/llNFtslu7PYpIpxrEeLjUaP8OmRe+jecSjMb/fhJUp2NVPG9jON6L7+V00IQdalw6NF21h5uc09+CJCtTBl47kGidM3yohwuB59O61B7TXrDZpEjZmuTgFSzA0IVwnZv1nju7x59YaTOW+XOv3mMSdBN2IBdDVFP2/a7B+h1aIyqnd8nSQggTIsEC7KqiStReFR8h+WtvIgOM/CRTzDJ13AsMoqJNt56/eDzxzmLTnTBRbbmaOldDHcWfsbYpBUNy10hjQQ6rVexiYY6IdMcVHdO7w== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(39860400002)(366004)(396003)(346002)(136003)(451199018)(66946007)(66476007)(4326008)(41300700001)(8676002)(38100700002)(8936002)(6506007)(6512007)(26005)(316002)(54906003)(7416002)(66556008)(6666004)(5660300002)(478600001)(6486002)(83380400001)(186003)(2616005)(86362001)(36756003)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sJB/lu8SsYSUOQ3U33wUhkBHtA5BcV+QZ1uLtPiwnJiVJHTZQVsIF6mIvextTHE/AK9JGcVBH01l9E0igxUfiaJ1F9CooEZA/oxVGED5x7I2uI+w3gvHdaE0J5CPvKJ49e5AmIXuMJucfkCzoemYHXKJ7YtogPikDfMAVySwYs+4Iuwp8EunOGxhScOVZgz98aToCyeANKipNjLexBg8RoAOjIs6IU93uLI8r3G1lkIx+SS1A6wUOS5pC+1quwX0EtcS0lE2CoeRV1RWNAz2+Higf8Co3Y904daEpX9B9oaVua0T0HBASCZzamhLEmCuTR+wRbk080IjRkeaAsGUU//Rmf5LMZTOW+lBsVYuRAd5KEtYxmdSDfftbu7zwId76wzC3wszaXdBDUjqvWJys46gJGIl23LWR150HD+WLtpd4iStikgsKl3sUBIlkgONau7dvNbiCbyUwDYLSrv+aOSy9HTSMVbhQyKnbW0fA45JwuBZB+G8wiVtFlwJ4K6mIk06mvAfq3HEHYKhQXdu66gBEkYCUueVVcwv/daULbcZOSYE3di9UCclnhpf3az3qXX1bYD/tblPtltnz1NIZSza9qStOUhEKyb88Mb0iqFT0+cROULMRDc1X8B5Lt7LqTs12CPYWGNHk4gmuAn+DdQdKS+IwZIC5ZhoTM/NDmpTmi/0+ghuIcUqjkFB4v4o2c8tKExlbnce3caWIlwxmz0MmPKs2p9Bkucyx0tYJPUmjjeJRMdixdYybMrtITDR8amSkmpAWLq+8+VG82Rn4Xxd6LzZ++Zq8vtiaF1ZHxbuFuHfJD9QbEOsQhiWQNwkdoMai3g5uxcEQl/pyYmzSR6/05IwgIZWbL7RP+H/sWzqFftzBefkZKgM9znPhU/rlp9nJ+Jjk2zwgqrFBJRYPB7VQnZGELtveM4YZQGe365Gtd9nLLszNWXudTzPKkjMFzVt6p01+YafFolfyynU1Y3z4oAy9eQv+T7cHb/izOxsG3B6ygKHxooP0OGlGPQssUGVIrlPZ7434g7rpOn6/RubZoQdbP61QRc6P1mmMNMgEvRTj72nidTCyES9CyHkEmhd3m3p5h+5MHHZOBNIdTnynbBpY0upXHOf36MLhbdiAe2k9k/3J+NPSm+UfEaRWERCFs3fSkivY2l4YfjAMSYzexKbpGlNMv7aPhJOWtZe6uFNY+X0ktTB5RPrx+h40MjrF+S++7HzmkNoY76YxITq2J+yI5HOkx5Pg7gRjCSYX0Vq6UOt+1FTHwg2A7XDBR70/zplgj7nS5aPZHxBoXsZM2soI4plGyNgJox4Yd34JGVDmsfZrc/YLVl8RthC+o/kkLPGl0TPCK4WfGKY64Dd06GewbV/UGwZKHN+hopKXw07i7KXiNyI7xDy4OhkvU/02Ga2kPahSq9pBIo5lCIlDMD24I3DHg+ahuAdDGC+B80zYArW1qUR87+23ccG8j7yR/Rl1qBzr92LBFjYneJMdBYJfpKf9wu2UJYBkHLgLp7/Tgbf7tE8QHf1ExoR3vPzs8QC5QhhK+sHiycjFIpkctXOxQaJ/SoK4GIRqPrkoA+G1VXwuXoQC509DCGh X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: e6c72ef4-683f-4fab-e4ad-08db0816a3f1 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:49:13.7304 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nCGlYGM5jkLy5MtbClu9G5nINFV9C9n+5uWcbYP52urB5fWT5ZaxjYfStseduUlIEppg6KcFL2431p/aXlfS9A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM4PR12MB5118 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067263596014062?= X-GMAIL-MSGID: =?utf-8?q?1757067263596014062?= Convert io_uring to use vm_account instead of directly charging pages against the user/mm. Rather than charge pages to both user->locked_vm and mm->pinned_vm this will only charge pages to user->locked_vm. Signed-off-by: Alistair Popple Cc: Jens Axboe Cc: Pavel Begunkov Cc: io-uring@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- include/linux/io_uring_types.h | 4 ++-- io_uring/io_uring.c | 20 +++--------------- io_uring/notif.c | 4 ++-- io_uring/notif.h | 10 +++------ io_uring/rsrc.c | 38 +++-------------------------------- io_uring/rsrc.h | 9 +-------- 6 files changed, 17 insertions(+), 68 deletions(-) diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index 128a67a..45ac75d 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -5,6 +5,7 @@ #include #include #include +#include #include struct io_wq_work_node { @@ -343,8 +344,7 @@ struct io_ring_ctx { struct io_wq_hash *hash_map; /* Only used for accounting purposes */ - struct user_struct *user; - struct mm_struct *mm_account; + struct vm_account vm_account; /* ctx exit and cancelation */ struct llist_head fallback_llist; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 0a4efad..912da4f 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -2744,15 +2744,11 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) #endif WARN_ON_ONCE(!list_empty(&ctx->ltimeout_list)); - if (ctx->mm_account) { - mmdrop(ctx->mm_account); - ctx->mm_account = NULL; - } + vm_account_release(&ctx->vm_account); io_mem_free(ctx->rings); io_mem_free(ctx->sq_sqes); percpu_ref_exit(&ctx->refs); - free_uid(ctx->user); io_req_caches_free(ctx); if (ctx->hash_map) io_wq_put_hash(ctx->hash_map); @@ -3585,8 +3581,9 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, ctx->syscall_iopoll = 1; ctx->compat = in_compat_syscall(); - if (!capable(CAP_IPC_LOCK)) - ctx->user = get_uid(current_user()); + vm_account_init(&ctx->vm_account, current, current_user(), + VM_ACCOUNT_USER | + (capable(CAP_IPC_LOCK) ? VM_ACCOUNT_BYPASS : 0)); /* * For SQPOLL, we just need a wakeup, always. For !SQPOLL, if @@ -3619,15 +3616,6 @@ static __cold int io_uring_create(unsigned entries, struct io_uring_params *p, goto err; } - /* - * This is just grabbed for accounting purposes. When a process exits, - * the mm is exited and dropped before the files, hence we need to hang - * on to this mm purely for the purposes of being able to unaccount - * memory (locked/pinned vm). It's not used for anything else. - */ - mmgrab(current->mm); - ctx->mm_account = current->mm; - ret = io_allocate_scq_urings(ctx, p); if (ret) goto err; diff --git a/io_uring/notif.c b/io_uring/notif.c index c4bb793..0f589fa 100644 --- a/io_uring/notif.c +++ b/io_uring/notif.c @@ -17,8 +17,8 @@ static void io_notif_complete_tw_ext(struct io_kiocb *notif, bool *locked) if (nd->zc_report && (nd->zc_copied || !nd->zc_used)) notif->cqe.res |= IORING_NOTIF_USAGE_ZC_COPIED; - if (nd->account_pages && ctx->user) { - __io_unaccount_mem(ctx->user, nd->account_pages); + if (nd->account_pages) { + vm_unaccount_pinned(&ctx->vm_account, nd->account_pages); nd->account_pages = 0; } io_req_task_complete(notif, locked); diff --git a/io_uring/notif.h b/io_uring/notif.h index c88c800..e2cb44a 100644 --- a/io_uring/notif.h +++ b/io_uring/notif.h @@ -43,11 +43,9 @@ static inline int io_notif_account_mem(struct io_kiocb *notif, unsigned len) unsigned nr_pages = (len >> PAGE_SHIFT) + 2; int ret; - if (ctx->user) { - ret = __io_account_mem(ctx->user, nr_pages); - if (ret) - return ret; - nd->account_pages += nr_pages; - } + ret = __io_account_mem(&ctx->vm_account, nr_pages); + if (ret) + return ret; + nd->account_pages += nr_pages; return 0; } diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 18de10c..aa44528 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -42,49 +42,19 @@ void io_rsrc_refs_drop(struct io_ring_ctx *ctx) } } -int __io_account_mem(struct user_struct *user, unsigned long nr_pages) +int __io_account_mem(struct vm_account *vm_account, unsigned long nr_pages) { - unsigned long page_limit, cur_pages, new_pages; - - if (!nr_pages) - return 0; - - /* Don't allow more pages than we can safely lock */ - page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - - cur_pages = atomic_long_read(&user->locked_vm); - do { - new_pages = cur_pages + nr_pages; - if (new_pages > page_limit) - return -ENOMEM; - } while (!atomic_long_try_cmpxchg(&user->locked_vm, - &cur_pages, new_pages)); - return 0; + return vm_account_pinned(vm_account, nr_pages); } static void io_unaccount_mem(struct io_ring_ctx *ctx, unsigned long nr_pages) { - if (ctx->user) - __io_unaccount_mem(ctx->user, nr_pages); - - if (ctx->mm_account) - atomic64_sub(nr_pages, &ctx->mm_account->pinned_vm); + vm_unaccount_pinned(&ctx->vm_account, nr_pages); } static int io_account_mem(struct io_ring_ctx *ctx, unsigned long nr_pages) { - int ret; - - if (ctx->user) { - ret = __io_account_mem(ctx->user, nr_pages); - if (ret) - return ret; - } - - if (ctx->mm_account) - atomic64_add(nr_pages, &ctx->mm_account->pinned_vm); - - return 0; + return vm_account_pinned(&ctx->vm_account, nr_pages); } static int io_copy_iov(struct io_ring_ctx *ctx, struct iovec *dst, diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h index 2b87436..d8833d0 100644 --- a/io_uring/rsrc.h +++ b/io_uring/rsrc.h @@ -167,12 +167,5 @@ static inline u64 *io_get_tag_slot(struct io_rsrc_data *data, unsigned int idx) int io_files_update(struct io_kiocb *req, unsigned int issue_flags); int io_files_update_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); -int __io_account_mem(struct user_struct *user, unsigned long nr_pages); - -static inline void __io_unaccount_mem(struct user_struct *user, - unsigned long nr_pages) -{ - atomic_long_sub(nr_pages, &user->locked_vm); -} - +int __io_account_mem(struct vm_account *vm_account, unsigned long nr_pages); #endif From patchwork Mon Feb 6 07:47:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53064 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105689wrn; Sun, 5 Feb 2023 23:52:04 -0800 (PST) X-Google-Smtp-Source: AK7set+96+2+jX/woDl72Y50aJGlN0WPRq+eCxUNfRGvwqWFRqGS7Hd9GVTpSVkpiyrZpr5YRIDp X-Received: by 2002:a50:d490:0:b0:46b:34a:3945 with SMTP id s16-20020a50d490000000b0046b034a3945mr7695141edi.31.1675669924416; Sun, 05 Feb 2023 23:52:04 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669924; cv=pass; d=google.com; s=arc-20160816; b=KPSE+skDei22T29O4qcFEMdev6q5fFWY/GeXx4Aokfu5iPZg9LqaU4inkdyVIj4Xlw rMmrN1sQWWmrL+BVV7blt+rV+znmBgnsyB8Sg1l7pjgP9vj829mKznl/6WSSXtylWXDY hTQKrjP51Gnqv+gBsZzPzeKBnyoTfew5AGzfVT+cw+oHGFAbkOsQB1I0cWs8b/CZdrTC decGPi7k33ku5HZvtlRf78iEMQaxrfzP9WzmjmC6HlDShk1fZhTE2ayBQF0lGVrNSUZq WYOKWZBg+F0WoQJEwe5/Ku+t430fehagcj2NuIoJzWEhiVb+FjaH84jnrWrJ5ca6Lg4p d8dA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CYW4u4ayE2LZHX01iQfnpEcejDU0yOEhVudNlOUAn3Q=; b=joQ0AYap0L/mABqRDtVTBrwfMYTqYlqUDYfZka3f/4LhVmrQPhv0ZJbkSbVVucfSyA WulfaX36osBWL+ro5ipsPEOlHvQ+Lw8mx0Fyw9ZtKlWei074pP7/wM8W/bgdoEV3B2JB vYZxQgRVI8+oOhGwsyqQNHrj2X6/rkB6NGKCMtCbq5M/utfnBqu7bQFv4ruuAORfkWlL or0a+y4VcdytnmmCote8Sl9QN4GAkKrnaBgp1gDYkrsW5UxQVd7Lmt+4lyKHBVz6m/nl qDdRw/HgNsdYgl5aHJ51TJpxJS6947JfJJVI9BCkVkgfo9Rg1pZH9RUTMYH50Sc0kzbw zhsQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=LCGOkAUc; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id c8-20020a0564021f8800b004a24574dc2esi13320912edc.557.2023.02.05.23.51.40; Sun, 05 Feb 2023 23:52:04 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=LCGOkAUc; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230025AbjBFHvA (ORCPT + 99 others); Mon, 6 Feb 2023 02:51:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230006AbjBFHuv (ORCPT ); Mon, 6 Feb 2023 02:50:51 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2054.outbound.protection.outlook.com [40.107.223.54]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2DC921D90C; Sun, 5 Feb 2023 23:50:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=N9bc9Rni+JvKnGuBzzNjA3HFQdH8BUD73poGipHQPVvXZk8lqhD5m5xluB82xR1ZYPlLFMMcA/XJIRBbC8IXGIDSKNVea+O+Fzii0g30tNJ3O2S4AmzxOEqW3D2Qvuon+MQVlLn7GyTXVSEfFbYsTHlDJuK1Hh6wJkvfzT3p562S43cw8vxf8oasBS4ixhxYXtmK59OEPclkXB8Mi/sY/ZofOuTGX3w5Obb8kh8yf7/SzYGIcxkB4vPovi+3a7nO8oyAqS6w8L7ZCB9708bLY1RCvTO+b/TwDEY7p5mrK76AkDmB4Tv6uTExcL3gYbtjyaIsaeoD4aj9Gzx8TKXt3g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CYW4u4ayE2LZHX01iQfnpEcejDU0yOEhVudNlOUAn3Q=; b=KxpDdjzItMs7oMHnWxbFIbAiMDe//rfEgzw8Xx4uyE41tMhzwhGMc9WU4LSAvslw0x8f0uk0OFbxX2nv/Sw06BUWw1caobMo/yoVYiRhlH6HGYkzUwLy8g/qM6N5PcvasVmOrSp9ZPccZ9o0vI2mOxly+cvdsvq7uhA5PVZKkIEYzof55ofQxwt412MqxWjBKo3LnWmWsKK+phwGkyLhLdO9KT0UEjmGBGS0Rms7C+fURopd0z3Yi/4TjAmsWJX87X2CdWD8XUJJMxH0EkLPHeqOpe7bwIx5mfbz2gf0xZD1Pr8FLFxprpMCRBMBvIRkxVLTCKLm5rlGTgs8jbSMtg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CYW4u4ayE2LZHX01iQfnpEcejDU0yOEhVudNlOUAn3Q=; b=LCGOkAUcbw0vOxMEIqkDhbrUyi+Q/QrFLfm8WDWZwttEqb6SGC6ytVojDYGQ3W8mMwzlLiMgJMgSnx7+zNvr2LQEVYWRea3af/eSAGZaYBm8XzYgewKMCKiq4zxplUG3zXxw671Q60xeqaEtCiKb8tpwKOhM5UdGv6OXJ9hczT+M3xm4a21xSkGcU8nGENwG9zuRr0h/SAFp2kCyXV9veXiWao7bXHNqUZXFR9Wzh29VahK6lyIG+GmV2adHPhQQ+mei1crxC3ZXczm/kChnd/igJYJxz3yWKrpKcjzz4EPYAw9JvWxNelIov5X8e2T6DgBJVaiRUgFG7UANPuk7dw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DS0PR12MB8573.namprd12.prod.outlook.com (2603:10b6:8:162::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 07:49:21 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:49:20 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , netdev@vger.kernel.org, linux-rdma@vger.kernel.org, rds-devel@oss.oracle.com Subject: [PATCH 10/19] net: skb: Switch to using vm_account Date: Mon, 6 Feb 2023 18:47:47 +1100 Message-Id: <60a51160809d14e3e4a1c681cd71e37ab303adb8.1675669136.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SY5PR01CA0054.ausprd01.prod.outlook.com (2603:10c6:10:1fc::20) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DS0PR12MB8573:EE_ X-MS-Office365-Filtering-Correlation-Id: 691012f9-8863-4506-9d29-08db0816a82e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xFfTjdXiD80AhU3nEngcc8g/lI3dcbeLVrSsnPMuN+fKt+EWkUDZSnio4VD5lBMIAuch6KJGUCWhOCGe97jmA4HFkA3G84zc7kOnJNo+4MLzmGCFKWZ58ld4u3ICGWJQz7E1OldMsI+Qkp4M0PUdPdY0XQctSnshwjc4Jh+DUFBvgFH+PybQOVEfBxHsKNJeWXOAK/c+8PqVrdvLNe4fb+rKjQWGawqPelgzvC8OcK88/6ZHU/RuMY5A6sb0Y3NTIxR2BENvqGg45YO4MyJgUR5pM5KB2DjSMMGNEh9GkBigxsWeRB/bequ4IIQx/LqXDxn33Oep8MGwlayGJKUN6I2Y+XmHh4tav/jy5WHFLO6H9ONAUem8VosBR63H1yAAZ/gTM+l8YEWVGnkMEN5xFaSklhZuzCwvGCE1hX1xBFWKBTMSWT8xhEji+40hdE7deebHd8hIAx9UGKslTsLOD/ropcmE6fU1gTdHIdjP54TH+bzIs+HNp29TlV+IUKyDgyjs1sld85lCV7/oy9AGHl1i651Vv2m0r3khm+xBwtzUCkLTG+sSEu/7vsvRSDbThcqMQTxrKkNw7uF//O+sdboeot1AKmC2gDR2aGH8JIw0TBdRKPXyoyfU+LbDFM5QjfCZPMny+mzgBEOiEUg7xw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(54906003)(316002)(86362001)(6666004)(6486002)(38100700002)(186003)(2616005)(6506007)(26005)(478600001)(6512007)(7416002)(5660300002)(83380400001)(2906002)(36756003)(66946007)(8936002)(4326008)(41300700001)(66476007)(8676002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: HfySn4qGwrHdAGcR9NI3HHGKHYCbb8+FZraSoTtJ41bryVQNnyttKxXVHU+32ojDted1A2fFyVPNEGqoAhxdnAB+J0ZjKD2qn042Bt60Mg4QqsxbSQLaEcj2LoNbEJHvpvOzCXiuNZ17/vUQQkoudgNqoypgvnHU7Jfm+ZKm5YfU7xmaCTkHelHzSnHYcJ/3n2x0SySAStQrusifTU3GZ0mxsYgyCaPn4CDQI/hvZligYWwohJexh0AjvZ8u/VAOjKburqaoZclhK+QoC/FdYHk9ApNob4C5RPJz21mx/r12JAd7dwc+Ks1QWECAcGxnNSASPbO4JsKFD9ZFwHBOddRWq6FZliqGkqoUYPHUlDNJDrAzr5S/8H0JNp4RuHp4e3plu+7dH5+yAUAIKkiMdyyM1n7f2cDi/iLf+1sTFu/g+71qZ2aWBRFZQFZNlLXqfplApfT/QwF3FzLcKKtMeuflej8h/Pm2wIkMXLFhwbcu/Vj4CeL9wFfUcVmu9JtnnbWI9JkYNglMvP403q+JB8AFkMPpTjb+YP6Z4c05x1B0fSdz+JvPGIm8kDHfXCbuRqZBdcPQm7joEcghy8YWUbHVA5kHCi+Zhf5l6OJC3J4InvaGpoAHbboDmvMBgxt2Unz8Tqkzbv4FjWDm2qAajyAM0atSvgpx5nI9Lqv8RI0gsdqyJIVe9Gp290GXZasucrYdOQgEb3WikjEwS+4r2OhyT3L5lU592zbk+kf3H7J1rpojfwZC78FBtDo2kmkVkS8ZgguQBgpYY0JYTheuAGeO+knzPQBHdHR04gDI+bKsKBaTixeUh7+6D3cdgtxxO0mbZQMPHMpAqXHcNjNYcXk0rFlABhDJ20rD/YSdQ0IPAc7C5iKr9LuLKrKAoKXBNBdaa/Z1FgC7PHpWPvCRnd2FudRBU/tAKN768c8Sx9D2+9fLTJukfahYnq0w8kbqwzBAqMIfkNyTMOD+qad00i1nK+FXXa+ZgendkomXTAaVnDVb5NIT3TiA59U6nAFGCtmZeiECIHXVwrz53higNp3ExZo/LD91upJyyru3h5il3StfoSfL9Uf3QbWoMVoMvcK/WvQu93nymozXUrh5Y87oenl1sxFO5JpYGZ4eao2yzaC9lCR3hbaVtwi9EZ6WVvPcE6mkLC/E4xQ8tdTtYKOOnG0feJwuV8FfmWyl/ztSBy/tnOZS0+6Nk2guzAswhZyzCKZKu6rp3/msB3GCceYnlKuxUODxmzf5yyABMDNJOF6Jsh8fM7d8n7Mo9dWcoyC3dvIatLEYziKUN0oI+7cHcYOPHCPwVdR4GctlQGQ0XA4Jd4JMInxFGRTXH8gWeo3U1Is8HWkniMwMLTyfVDp/B4puew5UKe5NAgDa4GIoBJHM5mdm8RpoAlnepw3lENFQQpVUCe/92S7MJdj9nkZ/g86a3YbiVXbzZl1wyioWtBBOHSolbOEpgAs+AOvelqCk+ysmyEeVxY0EuO0hfIYWT1GJRVt0JVVT+qsqNlxGU0zCCt3d/3SVqQif1nGZo22fUDjJf6eSLJ5lRvVlteDhd6q3gmcaPQrKOTvMdfsP55vduCCcBo5M72Zfy1Hb X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 691012f9-8863-4506-9d29-08db0816a82e X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:49:20.8549 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: dpb1f26IarNDW8rGdBUOwWkregUlG126Mwd8oOA6dUNXCiRaJC2iJ1ljjSQMDKqz02PWIBAGu0RngiLg8yUHaA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067266955569231?= X-GMAIL-MSGID: =?utf-8?q?1757067266955569231?= Switch to using vm_account to charge pinned pages. This will allow a future change to charge the pinned pages to a cgroup to limit the overall number of pinned pages in the system. Signed-off-by: Alistair Popple Cc: linux-kernel@vger.kernel.org Cc: netdev@vger.kernel.org Cc: linux-rdma@vger.kernel.org Cc: rds-devel@oss.oracle.com --- include/linux/skbuff.h | 7 +++--- include/net/sock.h | 3 +++- net/core/skbuff.c | 47 +++++++++++++++---------------------------- net/rds/message.c | 10 ++++++--- 4 files changed, 31 insertions(+), 36 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 4c84924..14f29a0 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -37,6 +37,7 @@ #include #include #include +#include #include #include #if IS_ENABLED(CONFIG_NF_CONNTRACK) @@ -554,7 +555,6 @@ struct ubuf_info_msgzc { }; struct mmpin { - struct user_struct *user; unsigned int num_pg; } mmp; }; @@ -563,8 +563,9 @@ struct ubuf_info_msgzc { #define uarg_to_msgzc(ubuf_ptr) container_of((ubuf_ptr), struct ubuf_info_msgzc, \ ubuf) -int mm_account_pinned_pages(struct mmpin *mmp, size_t size); -void mm_unaccount_pinned_pages(struct mmpin *mmp); +int mm_account_pinned_pages(struct vm_account *vm_account, struct mmpin *mmp, + size_t size); +void mm_unaccount_pinned_pages(struct vm_account *vm_account, struct mmpin *mmp); /* This data is invariant across clones and lives at * the end of the header data, ie. at skb->end. diff --git a/include/net/sock.h b/include/net/sock.h index dcd72e6..0e756d3 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -63,6 +63,7 @@ #include #include #include +#include #include #include #include @@ -334,6 +335,7 @@ struct sk_filter; * @sk_security: used by security modules * @sk_mark: generic packet mark * @sk_cgrp_data: cgroup data for this cgroup + * @sk_vm_account: data for pinned memory accounting * @sk_memcg: this socket's memory cgroup association * @sk_write_pending: a write to stream socket waits to start * @sk_state_change: callback to indicate change in the state of the sock @@ -523,6 +525,7 @@ struct sock { void *sk_security; #endif struct sock_cgroup_data sk_cgrp_data; + struct vm_account sk_vm_account; struct mem_cgroup *sk_memcg; void (*sk_state_change)(struct sock *sk); void (*sk_data_ready)(struct sock *sk); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 4a0eb55..bed3fc9 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -1309,42 +1309,25 @@ struct sk_buff *skb_morph(struct sk_buff *dst, struct sk_buff *src) } EXPORT_SYMBOL_GPL(skb_morph); -int mm_account_pinned_pages(struct mmpin *mmp, size_t size) +int mm_account_pinned_pages(struct vm_account *vm_account, struct mmpin *mmp, + size_t size) { - unsigned long max_pg, num_pg, new_pg, old_pg; - struct user_struct *user; - - if (capable(CAP_IPC_LOCK) || !size) - return 0; + unsigned int num_pg; num_pg = (size >> PAGE_SHIFT) + 2; /* worst case */ - max_pg = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - user = mmp->user ? : current_user(); + if (vm_account_pinned(vm_account, num_pg)) + return -ENOBUFS; - old_pg = atomic_long_read(&user->locked_vm); - do { - new_pg = old_pg + num_pg; - if (new_pg > max_pg) - return -ENOBUFS; - } while (!atomic_long_try_cmpxchg(&user->locked_vm, &old_pg, new_pg)); - - if (!mmp->user) { - mmp->user = get_uid(user); - mmp->num_pg = num_pg; - } else { - mmp->num_pg += num_pg; - } + mmp->num_pg += num_pg; return 0; } EXPORT_SYMBOL_GPL(mm_account_pinned_pages); -void mm_unaccount_pinned_pages(struct mmpin *mmp) +void mm_unaccount_pinned_pages(struct vm_account *vm_account, struct mmpin *mmp) { - if (mmp->user) { - atomic_long_sub(mmp->num_pg, &mmp->user->locked_vm); - free_uid(mmp->user); - } + vm_unaccount_pinned(vm_account, mmp->num_pg); + vm_account_release(vm_account); } EXPORT_SYMBOL_GPL(mm_unaccount_pinned_pages); @@ -1361,9 +1344,12 @@ static struct ubuf_info *msg_zerocopy_alloc(struct sock *sk, size_t size) BUILD_BUG_ON(sizeof(*uarg) > sizeof(skb->cb)); uarg = (void *)skb->cb; - uarg->mmp.user = NULL; + uarg->mmp.num_pg = 0; + vm_account_init(&sk->sk_vm_account, current, + current_user(), VM_ACCOUNT_USER); - if (mm_account_pinned_pages(&uarg->mmp, size)) { + if (mm_account_pinned_pages(&sk->sk_vm_account, &uarg->mmp, size)) { + vm_account_release(&sk->sk_vm_account); kfree_skb(skb); return NULL; } @@ -1416,7 +1402,8 @@ struct ubuf_info *msg_zerocopy_realloc(struct sock *sk, size_t size, next = (u32)atomic_read(&sk->sk_zckey); if ((u32)(uarg_zc->id + uarg_zc->len) == next) { - if (mm_account_pinned_pages(&uarg_zc->mmp, size)) + if (mm_account_pinned_pages(&sk->sk_vm_account, + &uarg_zc->mmp, size)) return NULL; uarg_zc->len++; uarg_zc->bytelen = bytelen; @@ -1466,7 +1453,7 @@ static void __msg_zerocopy_callback(struct ubuf_info_msgzc *uarg) u32 lo, hi; u16 len; - mm_unaccount_pinned_pages(&uarg->mmp); + mm_unaccount_pinned_pages(&sk->sk_vm_account, &uarg->mmp); /* if !len, there was only 1 call, and it was aborted * so do not queue a completion notification diff --git a/net/rds/message.c b/net/rds/message.c index b47e4f0..4595540 100644 --- a/net/rds/message.c +++ b/net/rds/message.c @@ -35,6 +35,7 @@ #include #include #include +#include #include #include "rds.h" @@ -99,7 +100,7 @@ static void rds_rm_zerocopy_callback(struct rds_sock *rs, struct list_head *head; unsigned long flags; - mm_unaccount_pinned_pages(&znotif->z_mmp); + mm_unaccount_pinned_pages(&rs->rs_sk.sk_vm_account, &znotif->z_mmp); q = &rs->rs_zcookie_queue; spin_lock_irqsave(&q->lock, flags); head = &q->zcookie_head; @@ -367,6 +368,7 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter * int ret = 0; int length = iov_iter_count(from); struct rds_msg_zcopy_info *info; + struct vm_account *vm_account = &rm->m_rs->rs_sk.sk_vm_account; rm->m_inc.i_hdr.h_len = cpu_to_be32(iov_iter_count(from)); @@ -380,7 +382,9 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter * return -ENOMEM; INIT_LIST_HEAD(&info->rs_zcookie_next); rm->data.op_mmp_znotifier = &info->znotif; - if (mm_account_pinned_pages(&rm->data.op_mmp_znotifier->z_mmp, + vm_account_init(vm_account, current, current_user(), VM_ACCOUNT_USER); + if (mm_account_pinned_pages(vm_account, + &rm->data.op_mmp_znotifier->z_mmp, length)) { ret = -ENOMEM; goto err; @@ -399,7 +403,7 @@ static int rds_message_zcopy_from_user(struct rds_message *rm, struct iov_iter * for (i = 0; i < rm->data.op_nents; i++) put_page(sg_page(&rm->data.op_sg[i])); mmp = &rm->data.op_mmp_znotifier->z_mmp; - mm_unaccount_pinned_pages(mmp); + mm_unaccount_pinned_pages(vm_account, mmp); ret = -EFAULT; goto err; } From patchwork Mon Feb 6 07:47:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53059 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105532wrn; Sun, 5 Feb 2023 23:51:27 -0800 (PST) X-Google-Smtp-Source: AK7set+LhKDDZ2NaFOiCqEmKF0sqlQpJqxDItV14KjkMU0a4NTEkTN6qQu7zLeak69+kE+LbVGNh X-Received: by 2002:a50:9987:0:b0:4aa:bb71:423d with SMTP id m7-20020a509987000000b004aabb71423dmr596681edb.1.1675669887220; Sun, 05 Feb 2023 23:51:27 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669887; cv=pass; d=google.com; s=arc-20160816; b=rg3mAlkJX/thww6/R7kczqFMdYdGhT0Gq8zMjkDEVzxm2hUJtY1BWYn/EyeD0ZZUcT dhPvUcXKHmWQ81hRGcjXiAx52KW9vpzHtUNT17q7wQDBsuIX9LHd9CJvy2MXezOaMQAf iH5bdK5fCMg7YVC8Ak2skaxM1fpPnf2atr5YWE8ULDBqVaPIAlNffdNII8hsBlQS3OfT aTieWq0/UKmDsMR2iHhlGBp/zgYt3ORrkqfM67cFNxvQLoR71kdx8wZnOfdOPSZZ8ACW 9x9l/LWASyhahC6j8FDqtAXKswNbOQTjPSuDV5+Z6Ngu7sY3nHYBz2NICxB8Zk0iP7IZ Z4lQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=ekaPX/00S+h70lFxojvYEzzxngZqMCIdq9dGbDFXgv0=; b=DWgoZDemsgeADbhu8taBTeJUD3/nM9+/DS9dDiNshc0tgVQoseFYg49co1/vTCett4 qgw7YXbC+5zUVKOcB5QdY2VHVDJFOrWCRe8Hvq758SUzuqD1F4Axq3rZAkvxx2yRls6F fFdyfy+HzMKtC9XV8K/QZJTsqXGl+mSdHY3SBOBoTlADeLSoa7VabkcE0km6QjHhqpxL kzRReOL9B5cJlFq/HzhfZ58OVxZhmK4uScrCgxPpHND60bFc+l6f/S15eN4+KhIn9DLm E8wPLaqwW0GuqtDLVBOgJxURtdhKHGKxHaSk4f0dqSxh2avbOUTLZHxmxs8Pv1rOx4R+ +dFw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=dNyGq21X; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id d9-20020a05640208c900b0046b3055748csi11070295edz.75.2023.02.05.23.50.57; Sun, 05 Feb 2023 23:51:27 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=dNyGq21X; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229939AbjBFHuH (ORCPT + 99 others); Mon, 6 Feb 2023 02:50:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56180 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229873AbjBFHuB (ORCPT ); Mon, 6 Feb 2023 02:50:01 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20603.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eaa::603]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7ED691DB8C; Sun, 5 Feb 2023 23:49:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fl0haf409z2GVuUVMTZhi74sVRNd/3eb3P+XKkXnOrz/OZay6AtJ+tlwc1aZhLVpArLa9aEv6XGmL2kHSNcMwyt+TG95tPddfsQYKjHPpki8hvX5nJHMI0SRWQYk0HYIDl5tVq2yKNTTzFGyKmXlNODVjuVz3+ObltL36ZyQhmXQczcaOjDCKjQVydnMHlzEj2nvm3uI5RnIBPEt2D0K+h5xC2bentbNI4CCphhWtQ7112FIsdxQ46MSgEEm3o0iWmd686aof8ngHzfeS8AZGhrqRY1O6c6/a3xfqhNSFJmqJc0mDM9wOw7LxQl+tiuVNqm+r+LdC4dWCQvdxm2fgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ekaPX/00S+h70lFxojvYEzzxngZqMCIdq9dGbDFXgv0=; b=BK3OS8gOFEMksJ5VUhjDVC+0VX9SjWKxeKdwBquaqec5ZUx+y7/efgn3BS28Mi6oLOk2KIaEIVrUPXZAhhHJfgOfVbX/hD/pRcWU2zYSIJ5QPgy5Xxd/z1cErKhbtrsLj1glJD+iXd3A0qzkdg9O3XP6yuwMSXCjG7aex+pvBBmYoN8uNFy7e7E0uiOOiUXyoXF08vTnxNOOOopxH5JjpmMk/iHeAdUjuMOW59ncl57U7wTY488D0sGHJx/I5CiPjTWyV1tdIg16qzENQOyS0HVAGHSZS10ISckpLfVCWUQrNDMIs9OXYIJ2htnCq4xqmpC9Axlm8bs93hRI5gblsQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ekaPX/00S+h70lFxojvYEzzxngZqMCIdq9dGbDFXgv0=; b=dNyGq21XS2lvzTJSJQWV2Dx2hCd50LZlN6ntem5wh+6FJ/wf25TDeGen5qzV/Ru+sSrrRfFfbT2vX0LXlq9OCiuFpBI7KMGEUOyWR6fOXlmu1oVxm+MKYfKxOSs17oG9LlGjoGhF9kxJWb4vSXo/L+2bzdrveRZ7QGe/9BJ6D1ZOXD71dpYeFa9kyJfAUNPBXvBBiuKAoTkCB9TNSundH10vtOke9XV5FGVFHq9TfBBSkuNyXy595aiL5Go9zAK6DAFviaAhBZGTVTWhHUbjkI3iI5WmEX/SU1onOp1pJvG/7mW5f5AdV9KcXE4gbAW5baCMauVktp/tKI5dQecLUQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DS0PR12MB8573.namprd12.prod.outlook.com (2603:10b6:8:162::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 07:49:27 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:49:27 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , =?utf-8?b?QmrDtnJuIFTDtnBlbA==?= , Magnus Karlsson , Maciej Fijalkowski , Jonathan Lemon , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 11/19] xdp: convert to use vm_account Date: Mon, 6 Feb 2023 18:47:48 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYAPR01CA0006.ausprd01.prod.outlook.com (2603:10c6:1::18) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DS0PR12MB8573:EE_ X-MS-Office365-Filtering-Correlation-Id: 4e499f2e-8b0b-4fff-206e-08db0816ac13 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: U/Hb5bUvPgq/8Bo/zhdDbA4PUQM4ddPIcEFNo7CVj8XX4kbiyc/wkH3Kp1WtHKJMiUG5tR1bYnYSI76gmFHStENAOhqNWW3CXT7bQp6RTqgIEtBpq2hsclqS4KNds1lKhU9c+UnZfOlsJMqDte4FbavG2LRznoGOMIVDtpg9TUojH0btY4kRvyAKcl8enew/vHUNWuEPfUx700I1hHO6teXAPLBXayByqXmuvl1AfcSNaVhLiqbOM+vREwqQEFDAEkqQfTZ7/7U/i6CzV/zDWLKrSZOnahP/PmyinY/kap4pNBnXyi+BCkzGfDfIdDW0XlFcqhSrSubdBDMw5kwQoOhB57LDCDevnsz1+B/xtgk56PDdP4Mxs6Q/o4Zkodw6uThm6TaB6wsN06uBi68F3vUs3Drq40ujckInxTegARQVlbHQ/LegvaWh43lulz+YE7Wy9n9lT6FvIWL7OoYadIzAPPnORUJJRRo1IGAcmzWVfYOQsLyELfGfdSlToN6X7uM46Z9LFTtLL9PFLYnYg2nvm9VJ2p0M/q69vpOc0O6qq4KhQxyvT71NJIC6t9VAQi4aFIfaqIkwVxqiOLYEEhRBpVDR3TkDb7hEJLkVzruSCx69/o0TIZzAbciD9oMN39mkfXEOyrH7vdOChdS+hQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(54906003)(316002)(86362001)(6666004)(6486002)(38100700002)(186003)(2616005)(6506007)(26005)(478600001)(6512007)(7416002)(5660300002)(83380400001)(66574015)(2906002)(36756003)(66946007)(8936002)(4326008)(41300700001)(66476007)(8676002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?D7uxsrNSd4gdjxjXR39Ev+fBUv88?= =?utf-8?q?xIdG+/vw6hQF4DdiL5i0N1SWaPStyyHjrljxkxxN5PhVwiarIhwn1PYfhNsk4Es6v?= =?utf-8?q?xb9qvz08NBRAibPAgoH5oYtNLg5/U8Dpwq6Mqj/9cP3Ati6K2KwUB19x8eCfINY9u?= =?utf-8?q?DDJ5Qj79Tcq3UTdV3V0Y0yi/wGH+Q02O38d6UywtNb7qXaH0K2/iZ83su6hwOWiFa?= =?utf-8?q?Nl8FHRjAr/i6GocDXnTO1c2z3PHz4H6mcaNp73rlFjcVsbDjTqaeqI/tvHcL1EO4m?= =?utf-8?q?2/QyncpA/4m5/+tzxtEoooMVNrOSbT+V+8iUGq7P/+ssw3Z5DOPxApgyI5vuwTRnC?= =?utf-8?q?AZhOIDlJfHocP2vlKTm8EKkUtpQoTRAmcLOb2/1mZeQ4c3mW6ueeS0S0Yzjr7ER1F?= =?utf-8?q?wFD4cpQPzB+0F/23gaSjxukg2SP5tGG11rv952yHZEG88YMdMvzNPvL+HDXKUDK26?= =?utf-8?q?unpp6Bo/ZbFnmYF+GWNkhVtQRT0EOZtfC6ojqpmwDDc2fX1aQXIwPZTd1VegU0ivd?= =?utf-8?q?HydwqINnjl3aTI+vUQZefxdViuY9IGKKRAfYncHdrKXcMTTbkrFQOnOZ5b7/2G2oy?= =?utf-8?q?Herzo6E+EFFUm7S6fm0F6HNLct1ffAWbOOBWbRyPh/JtXgIk3SXRtArIzq4f9jsK5?= =?utf-8?q?GCZNhAYJ3BkB/s0XWQeWcKD25vNiyL91PuBUViJqQWtmX5kFR0yJUpUZAWNw+kaKR?= =?utf-8?q?1Aq06fp+SYF2l2H9GRzj39maOasQRKnwRODzRJln+VzIzEqTMdp/p7CwG6HSVljFu?= =?utf-8?q?1CI9SlloJg8zZmUyAbhJxskhbQb5sa4Mtar2u6G1q7Or/6TANgx9rJWix59QfsvEX?= =?utf-8?q?mCHCmlwsWaxzOfexwzJq6nl+cEEHJ1bAmVcax2FJs8HPgXg3r9sktS1NVBX3L6Mk7?= =?utf-8?q?FNEbRa73ZSHWL8eQOAEUNmMYMr7Iok4Zm1ja9RMj0UoRfkPV0P43DGRS+VUYe0ud2?= =?utf-8?q?82/tT+EO0UyIEpg8OTGXP5ak9x5DIDh5qO7l2R0z5Z1wo3TzMqPtHX7uJ6P3ke2K0?= =?utf-8?q?GU9dx5mea66YDJKb0WC2cWEvVlHD2xSB3e192N/9DX1ML8bIW4EtmC7+fb93KNynY?= =?utf-8?q?RWc2XRRJnbyt33zedFYSy4euYcYADBrNpIqreRt584gsL9fVhxAYmAk2F3If38qH8?= =?utf-8?q?36wvvWGn6XhldBAGBofFCRHJSaMLzTDhKMyaJw8g4pZhNa7xz4JAxRmysaqoRgVLl?= =?utf-8?q?JdZHH0J/Oc3hwl2Aw2oHlo8WYgqtzGJ8oPSbDsoEOTzDR99RIVrg0SbVa5R6UdWJf?= =?utf-8?q?dvSnPjaptlALjHkJach30IF504m3yWq7AzI6tPxXoi4OvoWfFWkqSTHtsno43erN/?= =?utf-8?q?/VSulbHq+vz/Sohs3EtVMoFTJFtJDYU2qnOFgp0jbpcLWfN27qRBMLWkDOnSlAPkc?= =?utf-8?q?6S9DM5NaSy60NrWixOH67guHnmAFHARfVfApqD73DCFmO7OOoaDHVbGKxdbB4e/qY?= =?utf-8?q?4z4HkhdyzNWoR/xpWkxHC7m9fpPqFFqerQnhPUeZw80g/hf6wgGI6cCAygo0RZv6i?= =?utf-8?q?cGB6AIAhNoyY?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4e499f2e-8b0b-4fff-206e-08db0816ac13 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:49:27.3700 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XJm4eb5u09y5J03JJHgvgQg5AfLScdpIxX+UezKBxSPaIYMXVvTqX7hlUQm5cMjyyZJZmClUqbaQCKoadT1diw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067227560136982?= X-GMAIL-MSGID: =?utf-8?q?1757067227560136982?= Switch to using the new vm_account struct to charge pinned pages and enforce the rlimit. This will allow a future change to also charge a cgroup for limiting the number of pinned pages. Signed-off-by: Alistair Popple Cc: "Björn Töpel" Cc: Magnus Karlsson Cc: Maciej Fijalkowski Cc: Jonathan Lemon Cc: Alexei Starovoitov Cc: Daniel Borkmann Cc: Jesper Dangaard Brouer Cc: John Fastabend Cc: netdev@vger.kernel.org Cc: bpf@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- include/net/xdp_sock.h | 3 ++- net/xdp/xdp_umem.c | 38 +++++++++++++------------------------- 2 files changed, 15 insertions(+), 26 deletions(-) diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h index 3057e1a..9a21054 100644 --- a/include/net/xdp_sock.h +++ b/include/net/xdp_sock.h @@ -12,6 +12,7 @@ #include #include #include +#include #include struct net_device; @@ -25,7 +26,7 @@ struct xdp_umem { u32 chunk_size; u32 chunks; u32 npgs; - struct user_struct *user; + struct vm_account vm_account; refcount_t users; u8 flags; bool zc; diff --git a/net/xdp/xdp_umem.c b/net/xdp/xdp_umem.c index 4681e8e..4b5fb2f 100644 --- a/net/xdp/xdp_umem.c +++ b/net/xdp/xdp_umem.c @@ -29,12 +29,10 @@ static void xdp_umem_unpin_pages(struct xdp_umem *umem) umem->pgs = NULL; } -static void xdp_umem_unaccount_pages(struct xdp_umem *umem) +static void xdp_umem_unaccount_pages(struct xdp_umem *umem, u32 npgs) { - if (umem->user) { - atomic_long_sub(umem->npgs, &umem->user->locked_vm); - free_uid(umem->user); - } + vm_unaccount_pinned(&umem->vm_account, npgs); + vm_account_release(&umem->vm_account); } static void xdp_umem_addr_unmap(struct xdp_umem *umem) @@ -54,13 +52,15 @@ static int xdp_umem_addr_map(struct xdp_umem *umem, struct page **pages, static void xdp_umem_release(struct xdp_umem *umem) { + u32 npgs = umem->npgs; + umem->zc = false; ida_free(&umem_ida, umem->id); xdp_umem_addr_unmap(umem); xdp_umem_unpin_pages(umem); - xdp_umem_unaccount_pages(umem); + xdp_umem_unaccount_pages(umem, npgs); kfree(umem); } @@ -127,24 +127,13 @@ static int xdp_umem_pin_pages(struct xdp_umem *umem, unsigned long address) static int xdp_umem_account_pages(struct xdp_umem *umem) { - unsigned long lock_limit, new_npgs, old_npgs; - - if (capable(CAP_IPC_LOCK)) - return 0; - - lock_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; - umem->user = get_uid(current_user()); + vm_account_init(&umem->vm_account, current, + current_user(), VM_ACCOUNT_USER); + if (vm_account_pinned(&umem->vm_account, umem->npgs)) { + vm_account_release(&umem->vm_account); + return -ENOBUFS; + } - do { - old_npgs = atomic_long_read(&umem->user->locked_vm); - new_npgs = old_npgs + umem->npgs; - if (new_npgs > lock_limit) { - free_uid(umem->user); - umem->user = NULL; - return -ENOBUFS; - } - } while (atomic_long_cmpxchg(&umem->user->locked_vm, old_npgs, - new_npgs) != old_npgs); return 0; } @@ -204,7 +193,6 @@ static int xdp_umem_reg(struct xdp_umem *umem, struct xdp_umem_reg *mr) umem->chunks = chunks; umem->npgs = (u32)npgs; umem->pgs = NULL; - umem->user = NULL; umem->flags = mr->flags; INIT_LIST_HEAD(&umem->xsk_dma_list); @@ -227,7 +215,7 @@ static int xdp_umem_reg(struct xdp_umem *umem, struct xdp_umem_reg *mr) out_unpin: xdp_umem_unpin_pages(umem); out_account: - xdp_umem_unaccount_pages(umem); + xdp_umem_unaccount_pages(umem, npgs); return err; } From patchwork Mon Feb 6 07:47:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53062 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105665wrn; Sun, 5 Feb 2023 23:51:58 -0800 (PST) X-Google-Smtp-Source: AK7set+q5dkVbdGHIbctbWXIgVnWMiUsHXrja4K5d5zpxc3ZcArsXBUK4SSPSzzfSE2QaJvXUHXr X-Received: by 2002:a50:8e1b:0:b0:4aa:c14b:59e4 with SMTP id 27-20020a508e1b000000b004aac14b59e4mr422836edw.11.1675669918728; Sun, 05 Feb 2023 23:51:58 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669918; cv=pass; d=google.com; s=arc-20160816; b=alx0eom+7vJDLkhxxr57L+bem6JzxzDprQqMtp1SuzNs9LUwIIZXrm1oXLJ/NAXFqN s3VeUPIbZDqxOFQheikDxQEwbqSKfgI9SLOmxjy2RQ9sOJzPNfWfwZIkUkAz341caO1a gxaGLh3vowblEu1p3p7dd2q/GAB77UAx25UUz7tRzZ/1hfOD39pKWHhW87wQm1iswta6 xxz0o3Q7Rnc9bZq2yFuOkosgd7O4lxDEfYA/pfbUUnvk8m3c7yE7cHRfNmlkDUJZieWX e+NXYVVP/CODXVjdU6OWlk2A2V6TQj/NW/451xI7k6xcMoCe2NN3nRprE8IViedfqniA fltA== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=sABmSCNhqUaki37ROAvn5I2o7zlTWP9iNz0mlKPN/jY=; b=uwNRoJgDqKKzy5GUb81t3HKwonWmAN13JEesYtLEpH4+NKUzXxceBuJiPUbiCnsNn/ QYqq3hlbcwcT8IJ9Gbxdd0j6sNnbIERCTflPM24SA2DQp6JFAvAcS/UCr5ESj2w3Q/6m Uh7WAg3dtUQ94HAvQk2aEC1nPeKX9M22ooZe+8hidZod/S3CnYPLpnvOQXaDRN3qDyuP bSJxhnlKu4xpneuS+vUnkid0ZLmKA/T5mQeprG6vBxM9+pY7kab4RpjZCOfTGhi/dyiC fV76teAo1mcdg2AnQQnj2U38TVSnlLalcfYtcPP0yU6NyyEDZ/RUN2HPUxbRX9wcjhvL VgXQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=CHQwI+iS; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e5-20020a056402148500b004aaa5054e88si5810290edv.361.2023.02.05.23.51.35; Sun, 05 Feb 2023 23:51:58 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=CHQwI+iS; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229981AbjBFHuh (ORCPT + 99 others); Mon, 6 Feb 2023 02:50:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229921AbjBFHuc (ORCPT ); Mon, 6 Feb 2023 02:50:32 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20603.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eaa::603]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E950B18AA1; Sun, 5 Feb 2023 23:50:00 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=obie/PwJcOe2hDpi6+/p0mbSZE+iDhqWzJUUigICaut5IiNVmMZcMK0s1zPImn5mryTULm8oG6FX05Q+fIoIJ48WxWZLG8j0S6AYWXDjPrYtZmdOlO7ztoKDhgjInSqWe+pDAxICHsYnluMjoSz3U6wTTjmbmpBLuTOW7bnT7WwcXK8BaXp+ZAoAmghhz/eZDuTVhoSgq+Yub+eQmBp00p/B16DOhWTzesK9GBmTheTGJjSgpHFdMgQbvi3dH7PyuFOrwI3gdkOuY0Q4e8Z/k3JksCiPrU2s71luD/qYO2rlokmq8mLCMFwg7ekZxSXAu3c85YcujhyBylouCQBJ6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=sABmSCNhqUaki37ROAvn5I2o7zlTWP9iNz0mlKPN/jY=; b=CFPPMeAcsBiiJ46UEfy5VNfnk4mAIN6Dr9ckIFEop+AVXMXunqipKYFibH3NfUpVVLybvP4oE7GZRlJH7uE57qY7G3+DUvMQ+X3WbITsattT5PXhPfZ5CrVtPRHeOJNDLO2oTZDuawODWrGJDJcOIIcHRpgi6MXri/KS6PEB8CmBZPXonbKPMwEBFBcQ+wfiEwYtKcGA99gATAvqib8pez3DEKJbGIJea+7+BrO5kC0YcQF+hd5Q7uDl65rGhKN4jFj1jwP1gtSF0kA6BOp+xuoSJas3Oqyi4kkDQsSmdccxxrq5XtsJdnf3MxW349vZAQmWTJzukswV4GNS2HNZVQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=sABmSCNhqUaki37ROAvn5I2o7zlTWP9iNz0mlKPN/jY=; b=CHQwI+iS0hEE2NUjjhh1a8duKr1QEY6gPXkrUEGnhAhiYZh+EwVnEAFuu6/3arCe0qEFPyeE2mBJS0Udgi5k66OavT3AMmplZcz+uqJ83WF5eUQ5sS442WqtwUFWuLpFxK/zyKakGOwypeQsFr1EktqHWHS5HGj9MTzIj/AKquxhLnOuUR71GyVwKhW56iQeZoCko6AgUbTs5rLuExRipjp4QI8mrKGzItovJHdZJzO2Fpz/Ba0JCxnBKnH4N5jmB2+0c3yQah1YL+DcG/bEQext5pLO3aDQcjy6sqSW9YyWytpFSiGFz0j/RXveAB7XaOc8DdrOQSGve00kTSX1UA== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DS0PR12MB8573.namprd12.prod.outlook.com (2603:10b6:8:162::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 07:49:34 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:49:34 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paolo Bonzini , Alexey Kardashevskiy , linuxppc-dev@lists.ozlabs.org, kvm@vger.kernel.org Subject: [PATCH 12/19] kvm/book3s_64_vio: Convert account_locked_vm() to vm_account_pinned() Date: Mon, 6 Feb 2023 18:47:49 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SY0PR01CA0008.ausprd01.prod.outlook.com (2603:10c6:10:1bb::7) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DS0PR12MB8573:EE_ X-MS-Office365-Filtering-Correlation-Id: 1711d04f-1228-49b9-6835-08db0816affc X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ivvth6W4nqFwCtLaRy31c0XU0ftAT7d0a+ncwklm/9hY8CDYIaLq8S9Jt2xQjUHxCuE39zwggwD/XO53FRHcYW1qtRs2tTLOW7vRhqIrBHYMJ9323/945oJADFzxZKHDVs6lUFv50zoYf0bfEusrFpmVmPVJQLmXVgYDaqhXSlybJ54Nb+MoKlEYynZapcQ13LWsg2+98QqVzcvI9NiZH4u12UOdiXeL29R/tul5Gx5oHmlnrsYMHLVbYioDcqqURbUci5Mwsw5cdFsI2/jjQdQ1LGAlr4ca4tdToJy6ks9DEVIb0CoVjPrJ3mmvqOtileYU5KI091ItQnZZdNTO9G+Ew9R+MxfNlFWij0HejOfkyr+GwPn/bJvNDs0QZGHbV5zwtu/7CvLHGuEsBLQbtuPSyAyOaNtdkwsBgX4ILtycqzAUXa2SuJNL3KL+iQU8SK/635mU4gavA2nari5HHpXFjPKm3fPLJsBM6IGHUdYfGDyF9/tMbN1Wfm+HbYu2hhi1aZuach6sacGBh0z7TgaMd0iDiWPRD5BLqE8YTEFT/D16746WxEkx0Ok1H6xxQ245Udui6PaqaB+SHubEythSM2HiRTk3B8ltzXn/PygycjHCQxhMrJpallA2QZIeQxTKJiB/X78qMo62cUKse58eA40a69NMAtKUimwAxyrRhv7TbnRiI5nrrqZKWN/2vbvQs0zCJkwryFodHzS3dw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(54906003)(316002)(86362001)(6666004)(6486002)(38100700002)(186003)(2616005)(6506007)(26005)(478600001)(6512007)(7416002)(5660300002)(83380400001)(15650500001)(2906002)(36756003)(66946007)(8936002)(4326008)(41300700001)(66476007)(8676002)(66556008)(473944003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ay8XosogCpkzfmHNeC+DwsG6+2Mjp57f5iWgheg8h1dEFMr2NETA+gBVc0bYAPJv9JWXE263bQhiw4FP0PqKc4qUtuFNRZZ59XV7v4yHQU1T4a56TNbxaxHMFwG7LGQTMNg/jT0xVXpb8rSm7/z7dV4NrDckvYimWl9aodag6vY3l2LHvTs4wNS/La/dsWq5nOsK6QeI0rJmbpDpFngl5D986CNO2Vi3y3d1YQPIrkEQw8q2B0XUo8ExB3gHezUReau3OlMkQ8U6f6yU5EvKvX7+wCtY+UvXwCL2DwmIBMu8TTQ2vYtdJuqD/v0puA44iR8zcR8xYqjhp79cFFAwKZHDRPxpCw7vl06vRlITj6+B+Kn3yEW4xKVRFUQvGgZEoY6C6wKFwswXF9+f6zLkWftBmPibS9cBFE37y2c9FxUgoZFqaWdFy1bSP0VE7FiNhE4ccnivmk48iA6C3wbl7Qmf2gdG35dTUHApoTY74Bn0Nl7vVjc34616QxCdbBPWZrE7YPBoDxircSESUzSXRw0odxAGtFL4G2y5AavytvlgQavkOiSQwUTNHQwlYZQkzRDGo/NUgalbMI1+U73/xiQ+s7hnTA9EEs5vF6jTORbBotq5iE3gbufrqf9FQr80pzeT8urtpuX6xnuC9gyt7p1cVUBgZymWci/gN/z/Yorhpy8f1AhrIl45kLi9Fg0bdaGdGc8+xbuNc4LYyPkOUC/rCCBBV05mygMS9I06XydHCtXv2qMkw0M9XR6w+jljsWYGN5WVFaY+QSMUPl/tz368jzEJsCMLf3K8OxGMhvwnRD9Axf3NLz2JPWUHxtQ/Z59Z6ohf/fCPFE3XHQqTxMwzXzRn/E1Vj/oQddE6shZNenfuh2H5K9CurOaIXwvNrW8U+ZKskJfZn1p1l0Xgs8F/9UNeZUYX+f4llgO1FBTx9rwTImOm4+ezUZGYiNU8XVOhsxX/ko44nf9j7/niHDddmKcoxXO1ce41cj1wl1HOAT+xzBOCGxyqVsp49WhnHdKJvUgWduOylyhXr5r/0lF3ZAx4qH3Qu3N1NEszkeytXWG9YQKW6zNTm3H2ebdLOHxd8XQP61f/fKG38eC9B4IOh4GsXCylh6gRszJ3WIZjDG35JNcQkoP1sfG/xAwFRniir2Ipt3GbDiPjS34t7ca5A4XOk9bmxL5hJBpNF3us5+InTE9rH7qgH86T4ojOv4GLqIu/DlX3+JSDYDDFNLDGUQ0ZM0+QPi4ZO3zLsuPmOkK9CH03s+TLaKhzmlzvq/xU6W57wj7uF4s5o91nJwe9mYjKpLa6BrVzProPwFqtJoNi7y2wgI/j9353o1+8FLEY/veLPTutass9w4mSPjMDX0kHwfJp3ew6+VYOfimqvzYs/4X4NMXUJR4u6qhEJOarofjVSmAWiD5oGmjxS2RbCujzBeCgsO0y/ixQfmGbY5RR5a+/JLYdWq1WQV3jZqcbxo+6AwmXmxVyNdKZeNq7vU6xc53fke44EmeHgk5LjnTiEKr5CyKIrY2ReJSxnjFxfcIUbO6W+E0U6fdlJB5GmcPDcggpLbUqX2wDPn9PQDGXxt6HTikcSBMkvLb+ X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1711d04f-1228-49b9-6835-08db0816affc X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:49:33.9164 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: c9J1tJaY4pyotV0qCSTqLu5myKDaf6kn5nhV5WxMGeFphKFkr68yrsw6s6yLnhpSy6sjXSjUlVCrpf09HaGxPA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067260271912940?= X-GMAIL-MSGID: =?utf-8?q?1757067260271912940?= book3s_64_vio currently accounts for pinned pages with account_locked_vm() which charges the pages to mm->locked_vm. To make this consistent with other drivers switch to using vm_account_pinned(). Signed-off-by: Alistair Popple Cc: Michael Ellerman Cc: Nicholas Piggin Cc: Christophe Leroy Cc: Paolo Bonzini Cc: Alexey Kardashevskiy Cc: linuxppc-dev@lists.ozlabs.org Cc: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org --- arch/powerpc/kvm/book3s_64_vio.c | 10 +++++----- include/linux/kvm_host.h | 2 ++ virt/kvm/kvm_main.c | 3 +++ 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/arch/powerpc/kvm/book3s_64_vio.c b/arch/powerpc/kvm/book3s_64_vio.c index 95e738e..ecd1deb 100644 --- a/arch/powerpc/kvm/book3s_64_vio.c +++ b/arch/powerpc/kvm/book3s_64_vio.c @@ -273,8 +273,8 @@ static int kvm_spapr_tce_release(struct inode *inode, struct file *filp) } } - account_locked_vm(kvm->mm, - kvmppc_stt_pages(kvmppc_tce_pages(stt->size)), false); + vm_unaccount_pinned(&kvm->vm_account, + kvmppc_stt_pages(kvmppc_tce_pages(stt->size))); kvm_put_kvm(stt->kvm); @@ -301,8 +301,8 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm, (args->offset + args->size > (ULLONG_MAX >> args->page_shift))) return -EINVAL; - npages = kvmppc_tce_pages(args->size); - ret = account_locked_vm(mm, kvmppc_stt_pages(npages), true); + npages = kvmppc_tce_pages(size); + ret = vm_account_pinned(&kvm->vm_account, kvmppc_stt_pages(npages)); if (ret) return ret; @@ -347,7 +347,7 @@ long kvm_vm_ioctl_create_spapr_tce(struct kvm *kvm, kfree(stt); fail_acct: - account_locked_vm(mm, kvmppc_stt_pages(npages), false); + vm_unaccount_pinned(&kvm->vm_account, kvmppc_stt_pages(npages)); return ret; } diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 4f26b24..bd7a7be 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -35,6 +35,7 @@ #include #include #include +#include #include #include @@ -717,6 +718,7 @@ struct kvm { */ struct mutex slots_arch_lock; struct mm_struct *mm; /* userspace tied to this vm */ + struct vm_account vm_account; unsigned long nr_memslot_pages; /* The two memslot sets - active and inactive (per address space) */ struct kvm_memslots __memslots[KVM_ADDRESS_SPACE_NUM][2]; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 9c60384..770d037 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1142,6 +1142,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname) KVM_MMU_LOCK_INIT(kvm); mmgrab(current->mm); kvm->mm = current->mm; + vm_account_init_current(&kvm->vm_account); kvm_eventfd_init(kvm); mutex_init(&kvm->lock); mutex_init(&kvm->irq_lock); @@ -1258,6 +1259,7 @@ static struct kvm *kvm_create_vm(unsigned long type, const char *fdname) out_err_no_srcu: kvm_arch_free_vm(kvm); mmdrop(current->mm); + vm_account_release(&kvm->vm_account); module_put(kvm_chardev_ops.owner); return ERR_PTR(r); } @@ -1327,6 +1329,7 @@ static void kvm_destroy_vm(struct kvm *kvm) preempt_notifier_dec(); hardware_disable_all(); mmdrop(mm); + vm_account_release(&kvm->vm_account); module_put(kvm_chardev_ops.owner); } From patchwork Mon Feb 6 07:47:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53065 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105723wrn; Sun, 5 Feb 2023 23:52:13 -0800 (PST) X-Google-Smtp-Source: AK7set9UlWRvBLpePHMT2qp7XrGd8LuDs66IYd92w0+RdSy6zpbCMPuZjpic4MZ9rGGBpG1ikuwx X-Received: by 2002:a17:906:cc8f:b0:889:d998:1576 with SMTP id oq15-20020a170906cc8f00b00889d9981576mr19112415ejb.66.1675669933675; Sun, 05 Feb 2023 23:52:13 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669933; cv=pass; d=google.com; s=arc-20160816; b=R1X9ct+Av4HYPOpYU/AEr9TyX+9LX4mvOZguoamPm6JYVaLzt8n/O6Mg+UO4jL4k6j c+3aoiyNmcpi7vNjkAFVAnc+Tul1fCi2+wL2jwOC0YTtJipbAG0m1PAhpWK4z4yzviUY SakXRZmyeyXQE8eS7qqBtzi3z+LY35HTpthkXFGqJV96w6Q/CI2VJveztIdNmABkKEoE xVcSx8L/0JL5+BB5cxjQzwfVqVoKgEwdHYlXXOsWL5HZlbh/zZqZyXKUKQ5/QDvh+VZg 6WnVvJmj0jQ39ifodKhWzYjkFacmLGzc50kYXN83YXvbIGKY5z/ReG3H2gTGs+TzpdTD Bqtg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=BwomzvkBEgGdYf53g0ZL4eXFuNuoN5c1QyS6vm/9qcg=; b=RzQFpiWeR8n9OYuIhOJkdsBKH97F5YSFGA2dX45DMHCpn8c6SzzEaN0GYCQK2a3SoJ J+L0N+HnRFvIe3sR6bQag9PZVpJFYMHBIA6BDJxqDOmYeR55+IpKyIF0DfDsVvIntSr3 EKUWI16K4UJZ05RfIxUy73h8Q7IeJi+IwDBv7Tja3SVsuPVJyY2YKBOMy6gQUaL25Oys YRNylv3XWuQcWCT87PSF9mk7uHnXID5+4EdWVY4CjMiA6GssKbhTIWOeyxczoN67eEU/ wCdGO8CHsANQA41FtpUJrGxZ/4Kb5wjkWSw1+j4PH1nnTwvsXWQmxv1hxQm8ukgokm/c N/Qw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=szKlcEMH; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id gh35-20020a1709073c2300b008a60f2bf2a8si1340143ejc.235.2023.02.05.23.51.50; Sun, 05 Feb 2023 23:52:13 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=szKlcEMH; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230059AbjBFHvG (ORCPT + 99 others); Mon, 6 Feb 2023 02:51:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230007AbjBFHuv (ORCPT ); Mon, 6 Feb 2023 02:50:51 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20603.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eaa::603]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D67EA18AA4; Sun, 5 Feb 2023 23:50:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LPvQxGiTkoiWrZ5zej42gaaZmCy0r2LFi2JJxCW5pegeXGXROCuhL+wljc/9MFIA8/zc3hF/gtTiy/fD0EDoeNIuKceTpu2APp8dCPhkK6GvkW5KQO6loRUhLjXAOL5AGClnyvFLB29kX49a5fe3wJrHH9RgYEDJOWucAgtRxXCKbIMxkhD7Fran/EhydcMnhw0mb5hAl7Te+S3ohdcQNIXeATXKBVZKU2lg8MTz17SCQeZeIEP7GQ0HMAeUSd+O249vEpkhcJdOAXtYhNvMy1qMOWBlZS+JM+q92jv3wFB/aOcqzc1QtIseO+8RVHXGt9FRcohfWEzqiIduevaT5A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BwomzvkBEgGdYf53g0ZL4eXFuNuoN5c1QyS6vm/9qcg=; b=HvGq/9Db+VX9JBQK9/VtN4fT1wA1SjRB/GMrkFnhITV9x2rN+I8CzIo6M9CeAes612bAqsi+LqiRgWnp1NNw6wD1ViQR/SN2BHl3XfKOeUEHR8Vua1djfXbISmMVO6wT4V43KysUbHvwhSMqzUOHeZVrZeJDXLXrl1uaOnPL0BI5s1vLxHvMHdGNwv5r6atA1RfOA83ieNVWlr1cWJ/myDWxVdBsIYwQzX3Ycze7WkKi2rpit+EUG7eWk3Z4kbMMPqg0TzbIPwwHMo8YpgqT8QaIB+qogAur3xBiw4uHW+F+bqIJ75vN8HpMpbzYa9g64iA1qaivvcEyfp4UeWD7QQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BwomzvkBEgGdYf53g0ZL4eXFuNuoN5c1QyS6vm/9qcg=; b=szKlcEMHt1BgLN5SGRfE+C6PmQoXkp0dTCEshpWFI/Smn6e5C+3VYLY2QZzhs3jW5Zr7Wb/+8xkUe/R4GNRV+cmPWyFKJL8Rtr9kBvKwp3wwldrosMOZQ3UIpx5OzAWgfJTO92qN5mGGd9qIo11rdP7k19YYqVqCHD9Jwlu5k6n83GJqUsitF+DtgNz/H+KWLy8+f5on9rsznstPZHuGI34hmt2ynwQbuw/phRbFl5Ax34Z1QisK+TUxM+R2451e3RmAO8/TsTWUYggivf2fiJuilRN4NluWPVknhV7EPnUE8NEfTowCMIVRHcPY5sjsFOxFtNnxAdD7+VElgb+BPg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DS0PR12MB8573.namprd12.prod.outlook.com (2603:10b6:8:162::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 07:49:40 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:49:40 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , Wu Hao , Tom Rix , Moritz Fischer , Xu Yilun , linux-fpga@vger.kernel.org Subject: [PATCH 13/19] fpga: dfl: afu: convert to use vm_account Date: Mon, 6 Feb 2023 18:47:50 +1100 Message-Id: <4c3fcdfd0617ca7e21bb826c8749adcf00bfe956.1675669136.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SY6PR01CA0093.ausprd01.prod.outlook.com (2603:10c6:10:111::8) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DS0PR12MB8573:EE_ X-MS-Office365-Filtering-Correlation-Id: 6822060c-def8-45db-664e-08db0816b399 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CDx0ysmFC40w7m91S+8U0DYJ94WMQ65n7J0cRFi/UtBy+LhuTz1paH0LNMooPPPAwwoUiWDvrBRLq6zdNVn3EgJnZ3/WM+gt/leFaI1d8EJLYnBapMlpCaKp0JsEoLLxw5Js6bRcx5q0d0q1PrvIpltXVuzWguf+Z3HP3cFF7A0AgKY7vJkFRRAiOFon/UFq7VC2+gAZVr0iFCk8cqRV4AziMgAJx+qPcKOj4qrHeaK6AEOLeAQ7SsEsopTKcU8mRvs8f7E72FKGmVUadgCFf4LGKb6RKep4Do/GxPfcp0XFfh0YoUYbzj1mRNdOpwDXDJePg5juc1Q8Ja4E4BKxOeZHny8+oaYfVAiZ9HdKjdgCZNhpk8FNYrMuN0Ac7e+DoGV7jXm2m/8zJvYzD/8QokrSWJcmktaKERy43LJSTorXKNLcjHSQm9NU4/Y0G8o/Ayb+0Qflj4ad3fvypFFATZJnCjATWTBg4zpCNmD+MRHVaexhn172/4x7+3aKBxJV8nRYyVISxSgQywE3l+zsi+bOQXTApmge0Eu8ZdoDpCN0pDuzmMLqpT9D1OxAWuae/Qi1/SBgotzY/hbwQtB9RvjSUJ2ZQvvpPBndsbz05Ohs0cGfiTCVVDV78pzDgNLv21gTnsFJoZNyEV8dae6TTQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(54906003)(316002)(86362001)(6666004)(6486002)(38100700002)(186003)(2616005)(6506007)(26005)(478600001)(6512007)(7416002)(5660300002)(83380400001)(2906002)(36756003)(66946007)(8936002)(4326008)(41300700001)(66476007)(8676002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: NUtzb5X5UNLvv4FIgh9XzomWui5nSY1LhzIKgG5erK9DlDJ2L/PSsivDrcH0cHzGAlnAYoXc5X7cdLoCEcShz3j2GEdKTPozqdOxrqfoVCEbAUl73y16R7IKBFEBrb2RU/SrJnLmxLzyex5ex3hzR425ULvO1bxKFtC6RIjqqvzIrYcVchUZOjC31sqHbLGzUZx8tBOAFDagaoUPKa79Q9RUdjJzsqIRNXG+kojmlRmYgKQqcNlE2voSqhZ6npKQt+zE/+EdwjHvYw+MoLR/O0/aHteZ38qlkQPRqsEs099J0nxRTZNpV9Nk4m9luE7fx3DimDybanGTnOoOkLf5WgY7mfkVec0ZmyIqx6Wl+QOsWuJ0bhM8vmk2wOYgu+sxjhosYCXzhyfmBE2OtdnPcjHH591A94RZ6wX64k+CSW0Bcsr5kxY+b8A/4FNvBa79Ea3gI+alzRlVOeH1pnyyEvQCu0uBSzCt9ra9uJDZd/HTrEt892XabY9oqxdqKvJXrWYl0FaViLtsZMbuuyY34kmSXHwz7tx+2ke+ewFK3GxZCmMRQevOA8ZS8FGKP3ShhHYOI5nMpHEvR1eaBKa1oyuuVh4rUxWACeXzBBExcz4tnU24gOjlRvRXOrVM6V13DktiQ9GDDKBNcCSy1PVziLToEVduz3c7T8liZYjE1lJw0uy/ifOcf0y3MKWLqGBUoA7zzMoO35UwWTbqYuFD45Bhy2X18OIWIRVP/d6rsn3N8MvWlwIzghJhP1lHaKGY/IzfN5wtoQiOZ+4Ry932Yk9h2fP7DWpSTQHCDUpp6fc4c3NQnbtWtZud64goBnp26McW9/mu/pFXXq1HsSa+FE2lb5JOz8svDn2aS9UHsuJ5MT4e7idfU5KmT0adRrsHHbz1QMiAvG1d639nXZmwJ251V/rhT2JNC2yS4BvxbsbvNgCr6uAxpFGGK9VC/+nm7SPK5G7V3z2oDCKrEBwre/lXGRWg+c780adQSNmFS+JmUngkQwLBUFPMbXQY4gVjzsrZy4rMageeG/Y5ZX5sZ5nomG0H1BXGtM8gy5WyftyeHCEpLO5bPfSzdV3+LEuuuItsLc2bT1WXgZppQMrIntUkmLk/jiiEdf+PjciuIorbOuOjvWkoUSE5AtXVFvLa5DWl49dcT+gtbmaSDi7wcWIn96RezwuZ7pjDVZ9hpUgvY0dmTF3AR+x1RyA4aY7hAgLuaCesRFZkTquRwfnNAsAG2SWCCxv9l8vs3dfoHrdRt6YCRsybyMzG4ZVmP4YgWIcUBF7YzRoUwHn16BJ1VfBwSb/1XHdn5s68Pa0w7zN2A3afidhBy3FqUKas6utRFSSPvLsian1jnJ8TnHrJJiodu5mY9qya1j53KKs77rlOw9RhkTTAkEUmTMx0GvF/98S3p/C2rm+cC+HSbPkR1up32LhW7WNtc0ckcjYXza4YLoenZe41C3qw+y8KAFsBG6V5tD5Nj3BqVZndmE3K/4NC8McScDBnUCYfrnxeRHumNpOKCHjgn7wOAcTwMS+Nnx7aig2S6pnEtLWNshmWNeCiqV+d3htScxdnes0VaAJMeYxKlpF2KAVDPl9y6uu2 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6822060c-def8-45db-664e-08db0816b399 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:49:40.0254 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CuSL9rlvFjl/HyfoUGZRCwT31ulEpvCM2H5SkvAPmqtVS9b6f+w2qAPpJbKBV/EgUo2/N114qMiod9Bw2sS2cg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067276515669349?= X-GMAIL-MSGID: =?utf-8?q?1757067276515669349?= To charge pinned pages against the pins cgroup drivers must use the vm_account_pinned() functions which requires initialisation of a struct vm_account. Convert the dfl-afu-region code to do this and charge any pins to the pins cgroup. Signed-off-by: Alistair Popple Cc: Wu Hao Cc: Tom Rix Cc: Moritz Fischer Cc: Xu Yilun Cc: linux-fpga@vger.kernel.org Cc: linux-kernel@vger.kernel.org --- drivers/fpga/dfl-afu-dma-region.c | 11 ++++++++--- drivers/fpga/dfl-afu.h | 2 ++ 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/drivers/fpga/dfl-afu-dma-region.c b/drivers/fpga/dfl-afu-dma-region.c index 02b60fd..3b99784 100644 --- a/drivers/fpga/dfl-afu-dma-region.c +++ b/drivers/fpga/dfl-afu-dma-region.c @@ -38,7 +38,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata, struct device *dev = &pdata->dev->dev; int ret, pinned; - ret = account_locked_vm(current->mm, npages, true); + ret = vm_account_pinned(®ion->vm_account, npages); if (ret) return ret; @@ -67,7 +67,7 @@ static int afu_dma_pin_pages(struct dfl_feature_platform_data *pdata, free_pages: kfree(region->pages); unlock_vm: - account_locked_vm(current->mm, npages, false); + vm_unaccount_pinned(®ion->vm_account, npages); return ret; } @@ -87,7 +87,7 @@ static void afu_dma_unpin_pages(struct dfl_feature_platform_data *pdata, unpin_user_pages(region->pages, npages); kfree(region->pages); - account_locked_vm(current->mm, npages, false); + vm_unaccount_pinned(®ion->vm_account, npages); dev_dbg(dev, "%ld pages unpinned\n", npages); } @@ -223,6 +223,7 @@ void afu_dma_region_destroy(struct dfl_feature_platform_data *pdata) afu_dma_unpin_pages(pdata, region); node = rb_next(node); + vm_account_release(®ion->vm_account); kfree(region); } } @@ -322,6 +323,8 @@ int afu_dma_map_region(struct dfl_feature_platform_data *pdata, region->user_addr = user_addr; region->length = length; + vm_account_init_current(®ion->vm_account); + /* Pin the user memory region */ ret = afu_dma_pin_pages(pdata, region); if (ret) { @@ -365,6 +368,7 @@ int afu_dma_map_region(struct dfl_feature_platform_data *pdata, unpin_pages: afu_dma_unpin_pages(pdata, region); free_region: + vm_account_release(®ion->vm_account); kfree(region); return ret; } @@ -399,6 +403,7 @@ int afu_dma_unmap_region(struct dfl_feature_platform_data *pdata, u64 iova) dma_unmap_page(dfl_fpga_pdata_to_parent(pdata), region->iova, region->length, DMA_BIDIRECTIONAL); afu_dma_unpin_pages(pdata, region); + vm_account_release(®ion->vm_account); kfree(region); return 0; diff --git a/drivers/fpga/dfl-afu.h b/drivers/fpga/dfl-afu.h index e5020e2..2ca6117 100644 --- a/drivers/fpga/dfl-afu.h +++ b/drivers/fpga/dfl-afu.h @@ -18,6 +18,7 @@ #define __DFL_AFU_H #include +#include #include "dfl.h" @@ -51,6 +52,7 @@ struct dfl_afu_mmio_region { * @in_use: flag to indicate if this region is in_use. */ struct dfl_afu_dma_region { + struct vm_account vm_account; u64 user_addr; u64 length; u64 iova; From patchwork Mon Feb 6 07:47:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53067 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105814wrn; Sun, 5 Feb 2023 23:52:41 -0800 (PST) X-Google-Smtp-Source: AK7set9gOoCY7NY58Bq6wHo18sJVjuWfr18aE335F+1pkE3VoWK0T25UhR2hic9ghzVVBlWgFYhn X-Received: by 2002:a17:906:184a:b0:889:ed81:dff7 with SMTP id w10-20020a170906184a00b00889ed81dff7mr18689582eje.9.1675669961336; Sun, 05 Feb 2023 23:52:41 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669961; cv=pass; d=google.com; s=arc-20160816; b=vJwdH1CepIDBboDikb1pUxsVpxayCwtvPwhFxn5LhJparpEECwh4qXHiIk9AssG1JI tpMvtNMykl21h6cst9z6/nBMgCZXTO5vYUmAoGJMfEw7n92lWTkm8fawXFyrbrmn12gq g0LDTcrt1x30AzF1Qmx0ks4DV8OXalWsMqR2noiujfc/J3Cq3K0XLAR7l+s6RB/Wymbl PhFFYNclapCmih8yli//bi57nAS7+9CSNhkZqWByXf/wePFBtHI3W++sARja7Oi0y73l Spjc8Kw20pcyeEn2ysAxoWgaxnx9UTjaZ2zUe8Z2KeIy5NdEp4aTE5tP50D5zBhDarFi jVRw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=67W7G5XqVQ/WEqZY3BEK3/0UYthlxkah8cbfNm84T68=; b=aw4Log0CY6oKTjN2rDCxE3u4nsEx65ovMrIqHxNQbp2XhdBBhCn9A7FAvFBS6Y1Xxm bXs40WXjreITmjDLFq9oCVgWwoKtWllbufWpUvBhC2VP+E+jOfKIvkSjNy7S1mQeFjxE 6LlnTA+N/z7kPJFkkgNxZI41Lkwn2jbIfzprrDiXwi6gYk/TQriPDg/gW9Yvsc5BDKBq ADxb6FgyyuNigcrolRgtyAfRW8MmjEeuJRAcCkug99wflyfF1vWhaVk2e7HOTOFR6xg7 RCEWnmVc8IuAk3WM+AF4S0XBZKnH30QV8hIw//MKb5JHTkggz9ooNoi3xBh/EcNuAmtf C+WQ== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="nbwt/S3U"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id w8-20020a170906968800b008785c366471si11185706ejx.510.2023.02.05.23.52.18; Sun, 05 Feb 2023 23:52:41 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b="nbwt/S3U"; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230072AbjBFHvj (ORCPT + 99 others); Mon, 6 Feb 2023 02:51:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230028AbjBFHvR (ORCPT ); Mon, 6 Feb 2023 02:51:17 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2054.outbound.protection.outlook.com [40.107.223.54]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81FCA1E1EB; Sun, 5 Feb 2023 23:50:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WPMugDb3K7VLwP+qPD3IW09XSncxBAySSWi9HNMG3ZMxVRbB0cT2+2UGt8TO8pA4T+GtX+4v21zMIboTuxT2BpGRIjBWKot9cAM0m4h+q3a08A7is7pOtYZ9k5FeXqCLH4y7ChHRYEArS2mYFwmwM3Lp+HZcbSeaF2ffyUDSjQ/7eTB1U0gz+HFWjxWCirAUKlmZOB4b696k6WbMGawayw7i11sVtdKPXg1er6w/MMQ9b2yhmcx2u86vgOGwEd6A0be7Xfb0sBKEF3X42dOXmzl3A2HfgurTS2f2WV28rPOCXq46zb3EI3Oe+awdfGgW0QnUtljkvB5FqO3+hSAWbg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=67W7G5XqVQ/WEqZY3BEK3/0UYthlxkah8cbfNm84T68=; b=iJfJt434uaasZGJCT4ix91v7/hDgybDtscIUYe7yZ/UhmIrrI8GsUytKMXkDhX+iDwtAzm+bVKgkGLJWAPXJaE28H6rvovANPz+nB0oLVo1qATP+OM8gjVbua+q6+QBNpQskwW1LvJ+yjh8NwMkeOvsVeC4V0tkzBaw9zvfLyu//e6HrbFuWdqb6u6r5aEKFYMPtiNv+eMAlFjKHiyHzWnKzDCSm/8gNpUWaf+iZWqeTh3Ntye54BrjHmi3DH6HPw4x5+rtFKG9t4YRPjk4B02BYI/3tnhS+rqCExumUz2ALGAQrr4lAU+7f9EwZQt/RRIyFY/n68dxS/WIMM3m/gQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=67W7G5XqVQ/WEqZY3BEK3/0UYthlxkah8cbfNm84T68=; b=nbwt/S3UZ8XqtXWM0lOLCqHrD7InkYkc90zJhknYiD8orctYZvNKQDN82tCkVSJRF95vZJvbfPvqn4KeJVzW9PEN6NJ5u40Sk7cPDVoA56ULRpeG1klRlzqgr7Q22o8Ry8f5EBj1PziIK4VuT/TDO8qdvhynADmtxR6Ejr7oCVTU5EOiz+ZgqqyFPUjt5cnMRPBIjj7jOLEAoyTOmdtUaD+HpmhW6lpb2Vi59DRsGlF2/gAahgX0cGnky+LmIhINeU2xgfqJhD0QIj0vRR5fdJnkZQNmTfkF0Fsg5q/SNAXnvoz4TmoJQ3AC7rK0dpaJmbCxUVeTLBwWaNxGIklMBg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DS0PR12MB8573.namprd12.prod.outlook.com (2603:10b6:8:162::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 07:49:47 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:49:46 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , Tejun Heo , Zefan Li , Andrew Morton Subject: [PATCH 14/19] mm: Introduce a cgroup for pinned memory Date: Mon, 6 Feb 2023 18:47:51 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYBPR01CA0092.ausprd01.prod.outlook.com (2603:10c6:10:3::32) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DS0PR12MB8573:EE_ X-MS-Office365-Filtering-Correlation-Id: b4e4a522-d38f-420b-49e5-08db0816b79f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: dzr5rw2KFFgMukaQWx8cTx2cDDmEyKDDI8Db41e3wffekUhWiQ0KxQPG967Ny1rCKcsIF3HVTcdjjbE/PxltjyLQc/hoVAD4rEH5XjPboqRy12efrJyo+D2oRRAhMVDbta+ZeM8nYs9dj94DORztZzQ73nO/JSslxm3FySEtLjlsJCC+ToHZUaJD6hxQs0iUyIWOXsG1xU8LrL4J+BWkePxLzp0ab9nlkw52mbN453b3UfOkprggTiGo6PjewO4u3jxRJl+V7sPS8zMOH2jlrSwcgi10QTTz+hz58nWBXkVSOi04euM5PBDByVKNkouEKCvsPJY6aw4fCxBcby6+lrqaFEzeUctNt9JkIbJC7yfx9VdRpJjKgN/M+f6yZzJiTqMGUAcY6Asg4Kdy7RENW9+W9YaioKABJTH6E9kmyMTutXi1bzY4IR9pbiEBRrh4JTl2cmcTF2m223vuIPLCcMFAN4aBZR86mSN28m08LUEOU5e6G48arSv4dgHTj4LRNGCLQqU9FKqGqmvGddzn0yh1RTalKGM4Ai8ht3yfVYMwkZ4ReOM6+s8qQSkOuMB2v0mjqPij5AyHY0plXg95wTjiUBzNYsresxHGs6fZfj8wjEbKyKG7aQmwBEqFZEdX6pASnqKyiwCSVYGGvXUOLQ== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(54906003)(316002)(86362001)(6666004)(6486002)(38100700002)(186003)(2616005)(6506007)(26005)(478600001)(6512007)(7416002)(5660300002)(83380400001)(30864003)(2906002)(36756003)(66946007)(8936002)(4326008)(41300700001)(66476007)(8676002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: jskS2os3I4atajsRYYf6bHzyGFPYQQsZWmjNoxEVxmjCTpPufXCrGIrz0lBPXQRqpjjNJQ6zXuOzSr8X1Z7x/nlQmpNfVQMXg2hWovWTw39sjr/fNi+KVRTM4ygLPHUa6Bx2H4CcCImgifxMNlzJaDJq1eFPkbqefQn5v73RJgYBqM3fZ0FuREe8/48abH5UMot19UtSQ58oKLqSghtIwvf4DCs3FpNbB+W9Y6+SG8WXsRVYyG+YQbbofcH+5ijujmOZpm2uTzVGgEF/wP0ITLrAna4R4E0V7UHzAOxV9XyVBKvSbMlsUTsWttJ9vnc/ON/m9aJR+PcJqicHqPFvyUOJM3Pd77Or2kWPmx+kS4e88FJeMxddxXi9VIHL7bLBdYbMi+loTvJhYF8Pi4AndLT9mFGjIejJ3puyThJL3xpdciJotKBj1LkfdG3wuUHgLq41/ArCZ5I3Q/IKLrZwpOQtdJC7v4sfllAJBka7OPrTzPXOuTEdHWeEoPXXjdI64nQ/1ahz6JtnJhO1MtlBdJwshJY7F4CDqREtuZgTRyXyYBv2CftL3umhymPhA1kVvrSTofu7Wpn8EFRk6Wxama+NxD3YWsk3N7grFVfrQLqw6uECd4AYgDXX2WxFxhvSPE9ai3kBrQVpQRW8FuGtdbHRs8ui4Je01XNBzR/TRzIrQ8NgOUgR5Pu7jOMmvIXiyXOldbicXOqwf/pk0xbcxnr6hxd+y2XnYk5KRMvMukLH5W0DZq9JZ9q4WIcwoiuUxTCuPDauIrbOlq2wEQvwWLU30c2NTIa+XuxLut1bDuE0Sz0Diehqb8hF7zXo26bQe2g6ieYoxIwt7s/RzGAAwzmj3Im9Qxu6vMmLn0khKxd9MbmbfxSDdHNotdxWgB5jLiCOwXolO0wnwhfgHj7vc0bt4/rMMBj9vXpX/DZ5Jfi4qONYptEEtkIU5+bwE9YjOE8Cu38uBSedi0kuC8YjzsSD4gQusqPLV0aDkNWYbDuEY5PtstwZYqCqR4wZKOp2qYX934BLDcJOlSr4PHbJqcclbtfP7QPnaFNQFWHSYaZ7mN8BtA0P2vr6j+NzyUlenBqEav7MhU1FYhPEjaniYxZ3GbiGYPsr7DE5HSrpVyYTOJxPnB3yUcM3apOHHSK02U/QGhtKY26ZITxUK9r/YXNC77fOt1iLFB9xB66aJM1/zwtU+p3W7vW8P0JiJRfcI2U3sHtQq4w9rvbxmy/9n0D9SfRQO9KQcA9zN01mFCdSXQ1PvdZDf1ofuC2IwMCmrz4lKVm2W1dPa2ZR3XiA1zDZ9j3mJui5vVtZNyyRdnpOjgQi4WFUcJIUjN7f4pp6XDsgXWmxo4HKIl18UXnfErBqynfhvveBKFipaVwmIVGezk44shcvMoLTH+QnB69iltd7BUgmF9Ib4k72U5I0MtTsrZahtolSTFAZvId4LXQXnc+lAYPRa1a/Cc0RobnXNoYoK61BHOhXsl7081Nbb69KUKM9CVw8Y3hlJhKJfZfNcetWYCKPSjmrKlPhLI8wLshCs0Qn02n1WkpkpfIdQbYi+zr/gzsDjmXFpe9CGBOZtYTdDpS5rwcJ/MJ0fIFN X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: b4e4a522-d38f-420b-49e5-08db0816b79f X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:49:46.8841 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Uw7shkhp0nozyKHfhCyagx37WKAuCW97CcAMHQEQlN5RJCzM4EjouQI4LsjKCvBkTM6QDSpdTe51pNaCtdCbbQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067305446554923?= X-GMAIL-MSGID: =?utf-8?q?1757067305446554923?= If too much memory in a system is pinned or locked it can lead to problems such as performance degradation or in the worst case out-of-memory errors as such memory cannot be moved or paged out. In order to prevent users without CAP_IPC_LOCK from causing these issues the amount of memory that can be pinned is typically limited by RLIMIT_MEMLOCK. However this is inflexible as limits can't be shared between tasks and the enforcement of these limits is inconsistent between in-kernel users of pinned memory such as mlock() and device drivers which may also pin pages with pin_user_pages(). To allow for a single limit to be set introduce a cgroup controller which can be used to limit the number of pages being pinned by all tasks in the cgroup. Signed-off-by: Alistair Popple Cc: Tejun Heo Cc: Zefan Li Cc: Johannes Weiner Cc: Andrew Morton Cc: linux-kernel@vger.kernel.org Cc: cgroups@vger.kernel.org Cc: linux-mm@kvack.org --- MAINTAINERS | 7 +- include/linux/cgroup.h | 20 +++- include/linux/cgroup_subsys.h | 4 +- mm/Kconfig | 11 +- mm/Makefile | 1 +- mm/pins_cgroup.c | 273 +++++++++++++++++++++++++++++++++++- 6 files changed, 316 insertions(+) create mode 100644 mm/pins_cgroup.c diff --git a/MAINTAINERS b/MAINTAINERS index f781f93..f8526e2 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5381,6 +5381,13 @@ F: tools/testing/selftests/cgroup/memcg_protection.m F: tools/testing/selftests/cgroup/test_kmem.c F: tools/testing/selftests/cgroup/test_memcontrol.c +CONTROL GROUP - PINNED AND LOCKED MEMORY +M: Alistair Popple +L: cgroups@vger.kernel.org +L: linux-mm@kvack.org +S: Maintained +F: mm/pins_cgroup.c + CORETEMP HARDWARE MONITORING DRIVER M: Fenghua Yu L: linux-hwmon@vger.kernel.org diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 3410aec..d98de25 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -857,4 +857,24 @@ static inline void cgroup_bpf_put(struct cgroup *cgrp) {} #endif /* CONFIG_CGROUP_BPF */ +#ifdef CONFIG_CGROUP_PINS + +struct pins_cgroup *get_pins_cg(struct task_struct *task); +void put_pins_cg(struct pins_cgroup *cg); +void pins_uncharge(struct pins_cgroup *pins, int num); +bool pins_try_charge(struct pins_cgroup *pins, int num); + +#else /* CONFIG_CGROUP_PINS */ + +static inline struct pins_cgroup *get_pins_cg(struct task_struct *task) +{ + return NULL; +} + +static inline void put_pins_cg(struct pins_cgroup *cg) {} +static inline void pins_uncharge(struct pins_cgroup *pins, int num) {} +static inline bool pins_try_charge(struct pins_cgroup *pins, int num) { return 1; } + +#endif /* CONFIG_CGROUP_PINS */ + #endif /* _LINUX_CGROUP_H */ diff --git a/include/linux/cgroup_subsys.h b/include/linux/cgroup_subsys.h index 4452354..c1b4aab 100644 --- a/include/linux/cgroup_subsys.h +++ b/include/linux/cgroup_subsys.h @@ -65,6 +65,10 @@ SUBSYS(rdma) SUBSYS(misc) #endif +#if IS_ENABLED(CONFIG_CGROUP_PINS) +SUBSYS(pins) +#endif + /* * The following subsystems are not supported on the default hierarchy. */ diff --git a/mm/Kconfig b/mm/Kconfig index ff7b209..4472043 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -1183,6 +1183,17 @@ config LRU_GEN_STATS This option has a per-memcg and per-node memory overhead. # } +config CGROUP_PINS + bool "Cgroup controller for pinned and locked memory" + depends on CGROUPS + default y + help + Having too much memory pinned or locked can lead to system + instability due to increased likelihood of encountering + out-of-memory conditions. Select this option to enable a cgroup + controller which can be used to limit the overall number of + pages procceses in a cgroup may lock or have pinned by drivers. + source "mm/damon/Kconfig" endmenu diff --git a/mm/Makefile b/mm/Makefile index 8e105e5..81db189 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -138,3 +138,4 @@ obj-$(CONFIG_IO_MAPPING) += io-mapping.o obj-$(CONFIG_HAVE_BOOTMEM_INFO_NODE) += bootmem_info.o obj-$(CONFIG_GENERIC_IOREMAP) += ioremap.o obj-$(CONFIG_SHRINKER_DEBUG) += shrinker_debug.o +obj-$(CONFIG_CGROUP_PINS) += pins_cgroup.o diff --git a/mm/pins_cgroup.c b/mm/pins_cgroup.c new file mode 100644 index 0000000..2d8c6c7 --- /dev/null +++ b/mm/pins_cgroup.c @@ -0,0 +1,273 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Controller for cgroups limiting number of pages pinned for FOLL_LONGETERM. + * + * Copyright (C) 2022 Alistair Popple + */ + +#include +#include +#include +#include +#include +#include + +#define PINS_MAX (-1ULL) +#define PINS_MAX_STR "max" + +struct pins_cgroup { + struct cgroup_subsys_state css; + + atomic64_t counter; + atomic64_t limit; + + struct cgroup_file events_file; + atomic64_t events_limit; +}; + +static struct pins_cgroup *css_pins(struct cgroup_subsys_state *css) +{ + return container_of(css, struct pins_cgroup, css); +} + +static struct pins_cgroup *parent_pins(struct pins_cgroup *pins) +{ + return css_pins(pins->css.parent); +} + +struct pins_cgroup *get_pins_cg(struct task_struct *task) +{ + return css_pins(task_get_css(task, pins_cgrp_id)); +} + +void put_pins_cg(struct pins_cgroup *cg) +{ + css_put(&cg->css); +} + +static struct cgroup_subsys_state * +pins_css_alloc(struct cgroup_subsys_state *parent) +{ + struct pins_cgroup *pins; + + pins = kzalloc(sizeof(struct pins_cgroup), GFP_KERNEL); + if (!pins) + return ERR_PTR(-ENOMEM); + + atomic64_set(&pins->counter, 0); + atomic64_set(&pins->limit, PINS_MAX); + atomic64_set(&pins->events_limit, 0); + return &pins->css; +} + +static void pins_css_free(struct cgroup_subsys_state *css) +{ + kfree(css_pins(css)); +} + +/** + * pins_cancel - uncharge the local pin count + * @pins: the pin cgroup state + * @num: the number of pins to cancel + * + * This function will WARN if the pin count goes under 0, because such a case is + * a bug in the pins controller proper. + */ +void pins_cancel(struct pins_cgroup *pins, int num) +{ + /* + * A negative count (or overflow for that matter) is invalid, + * and indicates a bug in the `pins` controller proper. + */ + WARN_ON_ONCE(atomic64_add_negative(-num, &pins->counter)); +} + +/** + * pins_uncharge - hierarchically uncharge the pin count + * @pins: the pin cgroup state + * @num: the number of pins to uncharge + */ +void pins_uncharge(struct pins_cgroup *pins, int num) +{ + struct pins_cgroup *p; + + for (p = pins; parent_pins(p); p = parent_pins(p)) + pins_cancel(p, num); +} + +/** + * pins_charge - hierarchically charge the pin count + * @pins: the pin cgroup state + * @num: the number of pins to charge + * + * This function does *not* follow the pin limit set. It cannot fail and the new + * pin count may exceed the limit. This is only used for reverting failed + * attaches, where there is no other way out than violating the limit. + */ +static void pins_charge(struct pins_cgroup *pins, int num) +{ + struct pins_cgroup *p; + + for (p = pins; parent_pins(p); p = parent_pins(p)) + atomic64_add(num, &p->counter); +} + +/** + * pins_try_charge - hierarchically try to charge the pin count + * @pins: the pin cgroup state + * @num: the number of pins to charge + * + * This function follows the set limit. It will fail if the charge would cause + * the new value to exceed the hierarchical limit. Returns true if the charge + * succeeded, false otherwise. + */ +bool pins_try_charge(struct pins_cgroup *pins, int num) +{ + struct pins_cgroup *p, *q; + + for (p = pins; parent_pins(p); p = parent_pins(p)) { + uint64_t new = atomic64_add_return(num, &p->counter); + uint64_t limit = atomic64_read(&p->limit); + + if (limit != PINS_MAX && new > limit) + goto revert; + } + + return true; + +revert: + for (q = pins; q != p; q = parent_pins(q)) + pins_cancel(q, num); + pins_cancel(p, num); + + return false; +} + +static int pins_can_attach(struct cgroup_taskset *tset) +{ + struct cgroup_subsys_state *dst_css; + struct task_struct *task; + + cgroup_taskset_for_each(task, dst_css, tset) { + struct pins_cgroup *pins = css_pins(dst_css); + struct cgroup_subsys_state *old_css; + struct pins_cgroup *old_pins; + + old_css = task_css(task, pins_cgrp_id); + old_pins = css_pins(old_css); + + pins_charge(pins, task->mm->locked_vm); + pins_uncharge(old_pins, task->mm->locked_vm); + } + + return 0; +} + +static void pins_cancel_attach(struct cgroup_taskset *tset) +{ + struct cgroup_subsys_state *dst_css; + struct task_struct *task; + + cgroup_taskset_for_each(task, dst_css, tset) { + struct pins_cgroup *pins = css_pins(dst_css); + struct cgroup_subsys_state *old_css; + struct pins_cgroup *old_pins; + + old_css = task_css(task, pins_cgrp_id); + old_pins = css_pins(old_css); + + pins_charge(old_pins, task->mm->locked_vm); + pins_uncharge(pins, task->mm->locked_vm); + } +} + + +static ssize_t pins_max_write(struct kernfs_open_file *of, char *buf, + size_t nbytes, loff_t off) +{ + struct cgroup_subsys_state *css = of_css(of); + struct pins_cgroup *pins = css_pins(css); + uint64_t limit; + int err; + + buf = strstrip(buf); + if (!strcmp(buf, PINS_MAX_STR)) { + limit = PINS_MAX; + goto set_limit; + } + + err = kstrtoll(buf, 0, &limit); + if (err) + return err; + + if (limit < 0 || limit >= PINS_MAX) + return -EINVAL; + +set_limit: + /* + * Limit updates don't need to be mutex'd, since it isn't + * critical that any racing fork()s follow the new limit. + */ + atomic64_set(&pins->limit, limit); + return nbytes; +} + +static int pins_max_show(struct seq_file *sf, void *v) +{ + struct cgroup_subsys_state *css = seq_css(sf); + struct pins_cgroup *pins = css_pins(css); + uint64_t limit = atomic64_read(&pins->limit); + + if (limit >= PINS_MAX) + seq_printf(sf, "%s\n", PINS_MAX_STR); + else + seq_printf(sf, "%lld\n", limit); + + return 0; +} + +static s64 pins_current_read(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + struct pins_cgroup *pins = css_pins(css); + + return atomic64_read(&pins->counter); +} + +static int pins_events_show(struct seq_file *sf, void *v) +{ + struct pins_cgroup *pins = css_pins(seq_css(sf)); + + seq_printf(sf, "max %lld\n", (s64)atomic64_read(&pins->events_limit)); + return 0; +} + +static struct cftype pins_files[] = { + { + .name = "max", + .write = pins_max_write, + .seq_show = pins_max_show, + .flags = CFTYPE_NOT_ON_ROOT, + }, + { + .name = "current", + .read_s64 = pins_current_read, + .flags = CFTYPE_NOT_ON_ROOT, + }, + { + .name = "events", + .seq_show = pins_events_show, + .file_offset = offsetof(struct pins_cgroup, events_file), + .flags = CFTYPE_NOT_ON_ROOT, + }, + { } /* terminate */ +}; + +struct cgroup_subsys pins_cgrp_subsys = { + .css_alloc = pins_css_alloc, + .css_free = pins_css_free, + .legacy_cftypes = pins_files, + .dfl_cftypes = pins_files, + .can_attach = pins_can_attach, + .cancel_attach = pins_cancel_attach, +}; From patchwork Mon Feb 6 07:47:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53066 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105787wrn; Sun, 5 Feb 2023 23:52:34 -0800 (PST) X-Google-Smtp-Source: AK7set/AQxHbC8/5k0K2NwgHX5P0KeE8F44LGv58I6bLuV/mpqrberlLA6skwmz4AbqbZVDUu2n7 X-Received: by 2002:a50:d657:0:b0:4aa:a0ed:e373 with SMTP id c23-20020a50d657000000b004aaa0ede373mr5769612edj.7.1675669954056; Sun, 05 Feb 2023 23:52:34 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675669954; cv=pass; d=google.com; s=arc-20160816; b=CnngXVsqg7wBcCLp4VJHW6xHPM+j9KBg30+oTtcHXmB2Vy4eFEWJ6QQZ2BsNymbXnI aCk3iRoBZ9Izf2oMcoRM42EVCjHoAg2wo6DV6TpoEtaalFzwicatvLKJWdTTA2Fm2lQS PwKnM9B7vvrKFtB+O2gkqr4Bxhple4Balkfip2R4I2fECTp+nFWxCvF51++DRYrmINPt eRONPicu+3JatWJZyrCVBbkso0GooegUari1Id3WyUjmyXFo7fvj5AWvKfv7J9FB/d8Q nUf/cb3URFAg/lcslhcvnWnPgO4Igyc5soK8mxYYHB0DS50wSQy1cTdGUDKXu/8IUCGF Csdg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=/U8SJoSKDmLPXz4EfnAjwHkQK/6X73YSJ1HtmZ4UlyY=; b=oUiGbnRYdTt0k+OJ1lwHAm7CASsCzY4gD0swVZ1rYXETP5N/bxZNRtlMBOcLqbJHlg J/m3kn1VXJzaRm0/Zg1ri/tu923lVQohiBREdANGSD8UNwdlPa9csqI/CjCJUKv2UCA1 Yi+TyZH/msmJZxIV2xK+8ID3effq7xopTXLKLJiRkWsI5a/8Sb8pbw5ME9Xy8TSACUqc VMfjOpvr9yIBXAZ3uAGxZw6SJ4UWDT0vw4edp/jVG51JuuOBLpOJzPVka7HN81hR9CCe /9RwRfvXi2UdmSou3Ttbew4qUArs5+qDnsEIWG7s3Ac/M+EStme61xNQigvpl2myqqwP e8xg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=dAAIMDW5; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id eq22-20020a056402299600b004a0a7f10836si11824186edb.60.2023.02.05.23.52.09; Sun, 05 Feb 2023 23:52:34 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=dAAIMDW5; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230027AbjBFHvg (ORCPT + 99 others); Mon, 6 Feb 2023 02:51:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58518 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230026AbjBFHvQ (ORCPT ); Mon, 6 Feb 2023 02:51:16 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20603.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eaa::603]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C69D81E2BE; Sun, 5 Feb 2023 23:50:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hiUOKde3JnExUrE//qUgKpl+6GpkO5kv1pyyyUfYMB8hhDLlGfutXJEAyycwzqwiZ/OYFN/7xneLVvSyxvxk0AIaIlWeGqFJwX5Qm5eWt8gUMaCep9dVNXoQtjYJOWAnM+g+YAxJaE+KvD2pLgtbpvdGagB8c5uh4lDb1skkiQc4+2dnN6pOluHlTSVt+6172q3uGTgUn45vgYPrc01+KyAgOY5UT0mc1knvOSvjXBqykHYGF20p6nqQxOhr2zYuJKMUbjIiuukEwAZrHOFWlpWs2+/ibNYf39Di1opEu306xWNQUIoGWyS4zFAVxX7SSEWwXCAdnB8Otgr5+jsQ/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/U8SJoSKDmLPXz4EfnAjwHkQK/6X73YSJ1HtmZ4UlyY=; b=XeuX1XoPJs1REchFnlDl+XV43L79W8E30JxFO2gD8QhCqTgoDRCwBCKp1YrMRCmrwWGhg4zeoiOT3oacHc6YMe625/RTLAjA+vrl2WNUXdbbIwRsQ7ygyIgkZQ1fc1sjM6/ljGGnTdjzKg6KwZ3B50ltsyCY7L5Es49XX8MQ71zfJnsI4zjMRpySqJ4RVj4k0BLtLFKPmzIyTqdS60M/OR5MXXPBeLdiXyqxmwtChy/44/NbJ6oomCWPLOhgJDh++8pxLSWD4a9Eingrcw9kIj2HtFjA0iwJSwjnWuaHU2lW8v6c0pnNBJ+aLNpJNTjZ3UuluK1wqc4zF6KctVun5A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/U8SJoSKDmLPXz4EfnAjwHkQK/6X73YSJ1HtmZ4UlyY=; b=dAAIMDW5Q0uOtcJk21Jl8UlVFM2xF6/6jf8SZTj7poyAZJEhAz+4IK3Gnjh8Kj7m5zHwcQdem7WNx94PYk768sb4b4ryluMsF9kSJBa1QHx+T99XapGI6gXMN69mZ9UomyDR8tA4WQ3g0l5YnammbpVFW0GI1ZDr3XV2Uo3QniO5T369TPCuPTi/lz+2EN0q9QOwLMvlxb9Wdy6yauSHCDpHAnqtzzT5mU3f7/JeuDXLFZgdyhevQ1y7OMHumnUNyzOF1xM/lj7cbestDuosqy7s4w+emoriYKZFsRnd00XTdrTsAVGkIjpw+w67IgyNslCJM4ohhfqvpJauUvxZ2Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DS0PR12MB8573.namprd12.prod.outlook.com (2603:10b6:8:162::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 07:49:53 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:49:53 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple Subject: [PATCH 15/19] mm/util: Extend vm_account to charge pages against the pin cgroup Date: Mon, 6 Feb 2023 18:47:52 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SY2PR01CA0019.ausprd01.prod.outlook.com (2603:10c6:1:14::31) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DS0PR12MB8573:EE_ X-MS-Office365-Filtering-Correlation-Id: 5304682a-003c-4ba0-7927-08db0816bbd2 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9pqCY7aiHhVea1fBbDGml4HA2yjB9ffOiBCQiepLv3dI25PrVbY2Sc+mcQFlIQVWoTFS/HviX7kk1Dro9fbRIaOUm+n66DuTHAQZT3KoIYB+biaKJL5B1XTnPUIWTUoDqsg9Qc5kTrv82gvn31KM2YDmCA+tw6iFoKIjZ7zWt9dmMXDUvwryfRkDhlcJZ6D6+1aYSt4A9dw40Au2Lc8fTGnzCkjVspEC60iamkE+VvwFmvb7/g70gunuDZ9oNas97AQQLjpPDgWBUUK++89NkVBaW99yqUkIO41E9YPBGWOpmesmEZEjtS8zYgfTpgCvkBC4AZ1sfXxAFNizARkj3yvPlZ15YtDSsPFi37JaGnpz02cFh8Eb8nFu853VYLRjSBvFoLjjeI2gR/RqccVJjPrZGrJ5QlnkOWhY0TkLz4BMe+lAhKfuk2EOsWnueaF8oV/dAZw9AJ0X9kc+rJx4uk76trHBBoed5W7gmoHaHjDIoTfwR45Mm3uvavzbN9lxffomGK0R4CuxkGjkLaf0JPCempnwuJuTfI8qv4DZZQliTpvCa0AyS26JO6X9V0SG/45Eu7g46oK3SknhGQk6KAjASg/UGQLhCRG4b+GjIbsnf/gtcCNs7xE+lKDUJy+FNdxeKcuWqL5RFS0DqmKpXg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(54906003)(316002)(86362001)(107886003)(6486002)(38100700002)(186003)(2616005)(6506007)(26005)(478600001)(6512007)(7416002)(5660300002)(83380400001)(2906002)(36756003)(66946007)(8936002)(4326008)(41300700001)(66476007)(8676002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: mnSJNOyeedE/vXRDi3G0lpuoTZbj75HMvgFQ9We9gTKg6PSYlqO74q6IDUFfjFr04XmgDJczXXyKkSlm1WHJD9QJT87bVdC0vvGkqbdq8tlObilm55DTIPdT7T56rBMmACPOdedBYkMjwEUf1D0kHHTK7vXMBgxXa9zKS8t5Rr5RyakdwiNtnQXvcCGRQUYU/gYq3e+5v42CrLho6tKU1XjoXf8Mp0JEUv0tU1dO1szKt6bxy43tduA92mxJcmOhGq6+59R3hVvdD5Ldhry54s0O6NY5+GzkbR2oFcN5ZQq6nt5YVI2yAhneQRgYM2WJutbzOvKHGFQB3ZT3MNSUHX0Y1GBK7Ii8tEafl/lp6TGOE2j7l18L51nXmH6+6o72Q0PeRecG18y/qqc8yyIQT7Po0xJeOm8RH6ZJ05JjHH0madAq4Kyw9SDBZ+a/1A2/l5MSBoy6i2hscgznUYD9Yz6UWEGSUYwKrryzITI5erB240FMiqdTBPy7orchmDrMTgK4EQ5HjJviKugamlGG2b1hfODMiRsdmD1IvGiGtcbpRff5MOiVIimZeuDTCtBrplTZIVJ48LfSXDYGpp6r8P99hFXy/Jn0lJ4837h7DhMNbS4Ddj99ZFOCzGMmiF6dUUx4+wk+U0UGSuBOfjw0Z5TZ+z2vvcgYWyis8lL8qUMiErJFKJWwEQ0nRYi2EXp4+qdWcEH7f9pr+oKokqDnLeh8BD9nRmPK/TRKS9yclxVIJCIC/naf4HEQlW8EgT7Xg9TPUj+/LNmNSzyD/zO8LrvsMQ2Be1RbQ+QlHpqdJqdQQr7bMPLS6kw+wLCyEIoPqvQwwBBIgEg45Ys+qdV15Pj2hrPs3OCokR0pcJntdKvhEl/5hd99nMgkb1Dxg4Ay3av4dvDwskkKKLH2hLgDkAeL8hC/8nE+YusfoYZ4L+3A2sZVvMdzpBhAH3mX82+/LXctoI2lE/bf3pEhn2BMvRcIolnlhuYY+GX3+9c29lM+mdT6bsdHyTToDW/beidp8/ApntkXGpqDyNkTM1mnwGPjrKMWk0nOT4zafgmWTGI2webd/i6WoOG6BNaDatc3uIg4SLuPfv5na3wT6GOVk8O82dlKrSy+hsodnMfN1uIGWIBnSkSyfTSsOnPTl6CdjH/U2qShbHIPKXOBRWDo/0BkflZpb9mZqzXmPffzEpDYU4VtlapiZwJjdbs7obqLtDhVqqmkoJxBhVC1XGJlZaDkbga9JRyQvyLI9Yw4JWc6uzM50GlzfDg4ZcQ92slsJCc4SY4NIWfvcgw06RBrQlgFdVHnZ3HgR3XWWsDT0NfvK66AnYXmjjml2xusIn8iZY+6bePYmUSq66Hj63OSsfcooLX9v5jkwWiAX2DY4woM+ASKUCOgdESB8aLiUPu2IBnlmlgT8A31xEWipizMrYxTZccj1c3s4amn1G4ENuq6IhogmpQaVOwptN03wBicPN56TAoVy2zXxi/kum5R2MErlUinwdU2pSH2ibmgeCoPYSbysohSURARiCQ2VMETOYLi1LvXzi8PEIJVDIkWc+AziJygpanCmlOiQnMEwRoZCsKAzwGw0re8gXWuYY5P X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5304682a-003c-4ba0-7927-08db0816bbd2 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:49:53.7898 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: pHiH4yNzHOKl8lB1/tqrFYEcZ5GZ7zB4hXjQDFhgsIZjYiBDDvs+db6oKKpdttdPxHrv69MlSjOmpFFByMTStQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067297708650664?= X-GMAIL-MSGID: =?utf-8?q?1757067297708650664?= The vm_account_pinned() functions currently only account pages against pinned_vm/locked_vm and enforce limits against RLIMIT_MEMLOCK. Extend these to account pages and enforce limits using the pin count cgroup. Accounting of pages will fail if either RLIMIT_MEMLOCK or the cgroup limit is exceeded. Unlike rlimit enforcement which can be bypassed if the user has CAP_IPC_LOCK cgroup limits can not be bypassed. Signed-off-by: Alistair Popple Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/vm_account.h | 1 + mm/util.c | 26 ++++++++++++++++++++------ 2 files changed, 21 insertions(+), 6 deletions(-) diff --git a/include/linux/vm_account.h b/include/linux/vm_account.h index b4b2e90..4fd5d3a 100644 --- a/include/linux/vm_account.h +++ b/include/linux/vm_account.h @@ -31,6 +31,7 @@ struct vm_account { struct task_struct *task; struct mm_struct *mm; struct user_struct *user; + struct pins_cgroup *pins_cg; enum vm_account_flags flags; }; diff --git a/mm/util.c b/mm/util.c index d8c19f8..0e93625 100644 --- a/mm/util.c +++ b/mm/util.c @@ -453,6 +453,7 @@ void vm_account_init(struct vm_account *vm_account, struct task_struct *task, mmgrab(task->mm); vm_account->mm = task->mm; + vm_account->pins_cg = get_pins_cg(task); vm_account->flags = flags; } EXPORT_SYMBOL_GPL(vm_account_init); @@ -472,6 +473,7 @@ void vm_account_release(struct vm_account *vm_account) free_uid(vm_account->user); mmdrop(vm_account->mm); + put_pins_cg(vm_account->pins_cg); } EXPORT_SYMBOL_GPL(vm_account_release); @@ -502,6 +504,17 @@ static int vm_account_cmpxchg(struct vm_account *vm_account, } } +static void vm_unaccount_legacy(struct vm_account *vm_account, + unsigned long npages) +{ + if (vm_account->flags & VM_ACCOUNT_USER) { + atomic_long_sub(npages, &vm_account->user->locked_vm); + atomic64_sub(npages, &vm_account->mm->pinned_vm); + } else { + atomic64_sub(npages, &vm_account->mm->pinned_vm); + } +} + /** * vm_account_pinned - Charge pinned or locked memory to the vm_account. * @vm_account: pointer to an initialised vm_account. @@ -537,6 +550,11 @@ int vm_account_pinned(struct vm_account *vm_account, unsigned long npages) if (vm_account->flags & VM_ACCOUNT_USER) atomic64_add(npages, &vm_account->mm->pinned_vm); + if (!pins_try_charge(vm_account->pins_cg, npages)) { + vm_unaccount_legacy(vm_account, npages); + return -ENOMEM; + } + return 0; } EXPORT_SYMBOL_GPL(vm_account_pinned); @@ -548,12 +566,8 @@ EXPORT_SYMBOL_GPL(vm_account_pinned); */ void vm_unaccount_pinned(struct vm_account *vm_account, unsigned long npages) { - if (vm_account->flags & VM_ACCOUNT_USER) { - atomic_long_sub(npages, &vm_account->user->locked_vm); - atomic64_sub(npages, &vm_account->mm->pinned_vm); - } else { - atomic64_sub(npages, &vm_account->mm->pinned_vm); - } + vm_unaccount_legacy(vm_account, npages); + pins_uncharge(vm_account->pins_cg, npages); } EXPORT_SYMBOL_GPL(vm_unaccount_pinned); From patchwork Mon Feb 6 07:47:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53068 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2105978wrn; Sun, 5 Feb 2023 23:53:23 -0800 (PST) X-Google-Smtp-Source: AK7set9pW7FplFRr36bMnph8gGxSVGNGtnCEtqqpdbSaa/TXkYaMg7edPkewz4UBYll4bLRpTfqg X-Received: by 2002:a17:907:6ea6:b0:8a6:1020:8799 with SMTP id sh38-20020a1709076ea600b008a610208799mr1411754ejc.56.1675670002930; Sun, 05 Feb 2023 23:53:22 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675670002; cv=pass; d=google.com; s=arc-20160816; b=oTCcgezHjZKvPF6WSkI7PSs0ygDaHtMB9fS6InBuXomLuF4lhSlPQzPc32wbw+hAt8 J/JVkv+z7DihJBYYwrP8K3AN8ZDy2VeN0+gSVFvYaEdbw3SCYohnzTbWGOZpuIqYz6al olr4h0VfzIFYhyhHfNfC6SqPmVy+6dxMePS8D2VfLefsg/Np7WyQYa6Mtj8kPn1SSmwQ BTk7EfxScgu+ahnP9rzRNNSuJxNgiftuUs8BQGZZxMlzyJmIsnVPZAnClAaHawjFxjeE cCxd0YjHaYZqXWV1S+Oo0v6IdcLSN3b9cre61vyK+BpMXLVwiPBGwwsjyH0hEXTUBoyM vI9w== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=QEHW8jtg1/hfxonS1lsmN1J+cQAP43J/WL1D3f0LfQQ=; b=y44zwOKyfakHEvi5sxP9+9bRxgWrlcGO26sDfyyfoqgzLWGunBYhz3vwcbhEXWXq0d wt0ERfi8/YSVIqx8ql6zF7ccJW9D0aJMq4DXm/vd2aNId5zWNqR8aELbN32fdLZ34/UY 9gkf6dmL5AGqEs7T34w1JxmU5/gYluLoxAHp0YUjKEh9faVnCPnrjo8KkpYXyH6Rpcr5 njBQO4oe0nmCAxc3F0Jcziq08eiUws6iqtdtDCaIdwGLrDcvnd40JfCDOqHjvz0S2vfn xTh1fPMXUyFnUwHIKivXxJ5x3IH5vO+pxE0OdNv1i4AKidmfVkVrqYosrnlwtojxmr75 V3Bw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=XW8hnGQn; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id ex2-20020a170907954200b0086cb918f2b2si6590732ejc.581.2023.02.05.23.52.59; Sun, 05 Feb 2023 23:53:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=XW8hnGQn; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230094AbjBFHwH (ORCPT + 99 others); Mon, 6 Feb 2023 02:52:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58526 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229994AbjBFHvu (ORCPT ); Mon, 6 Feb 2023 02:51:50 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20603.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eaa::603]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CC6521CF57; Sun, 5 Feb 2023 23:51:16 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kWjQetCOL1IOxz7vpvyfK+0Cd33y/o/NEcpXxKh5P1+ZrQf+YsEjwinD1l9fg3ISZgzzYrDVQCLlPzl4l/i3OS5EoiAY/Gp3M7fqv6ksk2Em5l/xQpK9mvehByBXmbAqCEzGq/K2Hds8SeTYx/4vVaZ64XaSrrBnY34644XeCGSL7l/tsMZ50jKAAeBmZ/l6D6gQO+6gsndnaqgSdKA2V7Ap6vW4LjMQWqLD4Ho4TboPACcBG4PTJDoVp7hHWAG9lK3oS3OxkGyml2v62VRcByTpix8HzAlXv+aGhj5McyBO7lMLue7xjCPByVpT3lHRYeyxRx1UpAJHF7ValgfJWg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QEHW8jtg1/hfxonS1lsmN1J+cQAP43J/WL1D3f0LfQQ=; b=jJvbWqLxgbYxhQptiGJXZQiYcU6kVPKGSIr5duSAFPMglMnxQvO3lj0Wt2kq646uuAaNQXn60s2CAYap/nh28jmW7CZxp20MCAuIEhs2HZOBMxbftTVfFT4Pw8KZk7nHXTqyOE8Tp6sDfUa+VAY3Q03AHtWdlijr37Lg+JF9Z5OcqrdIuaUYrLi3t4VosaRO3fyu/i9isf9y+SlhfEQ4r7N8eoUKsQE3yrBe779dFfl3xIVuFSwucYn7ocHk/SeBWYKz0phl6HuARJzbxrzH8Pp5lnAOQj/K77VCSzdqTYqxlqbf8cpQp0StmFpOF5pPRTjGMQCgsKr+BLfxtnsNZg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QEHW8jtg1/hfxonS1lsmN1J+cQAP43J/WL1D3f0LfQQ=; b=XW8hnGQniuHn9VXIbENHf4WtjGwfzYug4MygncqFpMEtxF1AB+JDfJQ1Tkx2Z1NKd9y1GTZ51iOmErTZM6w6prtjsRFCVBSZg1hsJDgXKUb1Th6VDzUvEg2T3VP1bcZ3Lo3GLKGvYUh8ohsGixuvfGHKyUnQaZIpI+9O3muKXUBYJUmhq/4iyR95wj9mSBDig0OJy5h6iUf/27sY4hyBibl4jcQQVcf4W7F6y83+FPNdEuihY0/h/lsaIkmNJzeeR+Ag2vKMhiSBL9+BlpZ9iUbwG5fozIsD8hnjq5ILOrfmpSxFb8QwFkky4+FBBWx/GLbNeKUNFqVI6IBwKkrY5Q== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DS0PR12MB8573.namprd12.prod.outlook.com (2603:10b6:8:162::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 07:50:00 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:50:00 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple Subject: [PATCH 16/19] mm/util: Refactor account_locked_vm Date: Mon, 6 Feb 2023 18:47:53 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYBPR01CA0093.ausprd01.prod.outlook.com (2603:10c6:10:3::33) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DS0PR12MB8573:EE_ X-MS-Office365-Filtering-Correlation-Id: 9db330a4-1aa2-4d64-980f-08db0816bfd3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5kM4XDhr1D/mKBdbDY2lMn7dtF5X4TVQ1MEhmUz9G4uhDzwM/fO9qVuffLHUmSXdIVbM4BCFQe648p+NeM3zfRwZ2Ra42/Dajim+M31nKdkWgb6v0pF3KuhTzGPfwlf6H75gVGI1T9SslnFNnGwQFbo/r9IZs12whuQKvpyzadBANlhPdYvzTiE6xzXT6LuGt6lT5ILcwtCYGNe5hhC4kpCwS0bS1jrggvM8zC17mU5xPpLfbpVdfaPgc+JCTnSRDsPN3rPlNX4/FQM+kdn/Cmy3zfuSQGd7XJXTaSN2VatDvI2/x+urvba8ndJroHF10ZAoo2f//rN8QMBwZKna7AgGGR5dQJN6SzeuLUjKvrCGDkvR6BSHLlK2JDfin7xVkeKAzQBlVouYhx3ARnvQSOgjMitv2LsXJ34qTPs9FHXtR/V0jfGFyd5q+bUz9X7LloOq1lb/iArV72f6ZI13t7NGPI2rGlC+coHWE95Ts6wUNAgVqe0mHbSwpXVFbqvnS+lW/cwJMMeXF4jYlj48nq7KrL8hwJw4ZlXi/fe5DTVEmiKXDUrWURnfdwd95SWoIbaDSfqJ/V2QvOhtpLErYVmTE4ASfMdmE00mlgk4cs+mlpNWZblJKfmHykx7F5/klB3UIl2K65XkI47f02oUhg== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(54906003)(316002)(86362001)(6666004)(107886003)(6486002)(38100700002)(186003)(2616005)(6506007)(26005)(478600001)(6512007)(7416002)(5660300002)(83380400001)(15650500001)(2906002)(36756003)(66946007)(8936002)(4326008)(41300700001)(66476007)(8676002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 0DxbcFt2BlVgcJiQ7dPYPczJRG/zZP8NGAJeoBO9QntBrkcy1PqALwI05MnOhxHIy9VWSg1yEEvQMT/Rp5yOsnyG9IRhSj1UaptFhWXJ6zETbDr+aR4L/FXJyRqX02HVYIg41Dr6nyeXp1IdEyG0xypxg6s7tfHJYVv+mq4a9o6TVyfqfJQak4HhmllIeVmZvmAijsSClBS9b4ogCE3CiKtfEPi23WbUJUKvU55GuQ8rJHdw49w571oX2FCpetlIj9Pl+oOb5yFljCSYPDJzgNfY93Ni0gjqyIlxzWCperGBjNHAlfmiaBTGxY56CigvHkoYWNtH9y6P5a9ntKminn6n+Xx0qbI/8Sd+L2p04YEuH+BCPxUWeO+hYMtPV+uereZYjQRDfmgBnyPCTO9QR9Oy02TaDLUkQTY4njj9geAhkm0gkzFND/Cf+x9O4eNci46b+E4WTKAeMVtQbMZHNClrg810xK+CNCmaYYPQ/4WI2hDxQFuP5n/l6v+XhIXF6cb5RSOKYSNEdquVrpEeBtrEW4kNpZPRNQjamUGIO1zxM5FC71W4XvxfF68S+WRvAY8MdgWSVE4Allrt9sL5qA5ds8b7g5QOafpUip5W+B4TzU/UoXrNhC3SvcggmffchMZvpqnxF6Ff3rXYJLoFP+toKnwrbwS2PiOd6REMsZt+sIS1LD/bF0OFNqZl9iRMsHf0APDv5IVt1wo01ikskyYC+SoNnenUxQb+XRyW4goZ2O71Rj5gkWlk9p/t7abRctWElOwK/o6WEajls3xHaU+iZW0zXrcaYpHlIsFRLKGqtpGzs43O5Xu9zNAessVUGWK9cVJAV47L+viwktpKz9JCo+68kANabwYMGDF6zg8u4igHTH4iggYkTIwA3wPpZGawLAjdPIGtHZfuLOB5Fs/oDKtYtDCKxwWflFbBRQAMVawhAUaeTTdB+saZT69iHrwMos7g1ZTTLlsRzTWKjZFWQIfPl6YDT5kpdWlu1QI9cfwG5MVlQIpFEIZ4Cx4EmlGfY78rbFbXT9x29Ac92VAcfHxZbwkslg9YADFZTmxtGiIJeNu8GVWtlpyxypX0wkV3yZCVZOKJzHJVt1BBknAdAllhPgRdWhPcwz04bcpgizG+v+imLUG0aGQcWpXMjUBVLqEuk4nqOpEvzph4FxjUqOr0t68tgIroqdMxTvUnXTHqK9/LISomVry0bssStRdqnPOeUB5zN0WRleQrqSvlibf5cBije3Hv7K7HHtMTNxlzR1ZNtEFAu5/r1uRsIrPNdsQRAuorfJy9Medkx4tvYr5VUc/JXgk4ddIMS9y3ujxgnCX3navWiSs0MyVxYftgXlXJPnhLGthXaCI7ZkHSuJr/5SNfV+ac4NEcYYbo9mVNQl0LGuj/ZEosqHmsyMm5Xi/zZ8Hj1jG6GygZnYLFRZKV+doYb372Amzc1DSRGHE6FPQA8yUxD2SuqqNXTSEP6/QKRdP+m4tBky0t4vshezrRANjGhpQl0JWn9dSSUoQHaQDPrh7rkeVyEaF7SfsSTcONN76wJcI+9mvcNDMIuZAQnHzPpwMYKwWC2Sy/B8A58MbvnbWGYnUnH74W X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9db330a4-1aa2-4d64-980f-08db0816bfd3 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:50:00.5081 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AI9Z7Fltk+94VB3iChMgHrGBmz4J5uTx5CyN5GljI4YVl933AZgacw6g7m5Ot/i+vbp/LPl8YNNGjrar9RWcrw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067349186455824?= X-GMAIL-MSGID: =?utf-8?q?1757067349186455824?= account_locked_vm() takes a flag to indicate if pages are being accounted or unaccounted for. A flag is also provided to bypass rlimits. However unaccounting of pages always succeeds and the flag to ignore the limits is ignored. The flags make calling code harder to understand so refactor the accounting and unaccounting paths into separate functions. Signed-off-by: Alistair Popple Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- include/linux/mm.h | 5 +-- mm/util.c | 73 +++++++++++++++++++++++++++++++++-------------- 2 files changed, 55 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 8f85716..126b756 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2090,9 +2090,10 @@ int get_user_pages_fast(unsigned long start, int nr_pages, int pin_user_pages_fast(unsigned long start, int nr_pages, unsigned int gup_flags, struct page **pages); -int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc); -int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc, +int account_locked_vm(struct mm_struct *mm, unsigned long pages); +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, struct task_struct *task, bool bypass_rlim); +void __unaccount_locked_vm(struct mm_struct *mm, unsigned long pages); struct kvec; int get_kernel_pages(const struct kvec *iov, int nr_pages, int write, diff --git a/mm/util.c b/mm/util.c index 0e93625..1ca0dfe 100644 --- a/mm/util.c +++ b/mm/util.c @@ -575,7 +575,6 @@ EXPORT_SYMBOL_GPL(vm_unaccount_pinned); * __account_locked_vm - account locked pages to an mm's locked_vm * @mm: mm to account against * @pages: number of pages to account - * @inc: %true if @pages should be considered positive, %false if not * @task: task used to check RLIMIT_MEMLOCK * @bypass_rlim: %true if checking RLIMIT_MEMLOCK should be skipped * @@ -586,7 +585,7 @@ EXPORT_SYMBOL_GPL(vm_unaccount_pinned); * * 0 on success * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded. */ -int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc, +int __account_locked_vm(struct mm_struct *mm, unsigned long pages, struct task_struct *task, bool bypass_rlim) { unsigned long locked_vm, limit; @@ -595,33 +594,44 @@ int __account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc, mmap_assert_write_locked(mm); locked_vm = mm->locked_vm; - if (inc) { - if (!bypass_rlim) { - limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT; - if (locked_vm + pages > limit) - ret = -ENOMEM; - } - if (!ret) - mm->locked_vm = locked_vm + pages; - } else { - WARN_ON_ONCE(pages > locked_vm); - mm->locked_vm = locked_vm - pages; + if (!bypass_rlim) { + limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT; + if (locked_vm + pages > limit) + ret = -ENOMEM; } - pr_debug("%s: [%d] caller %ps %c%lu %lu/%lu%s\n", __func__, task->pid, - (void *)_RET_IP_, (inc) ? '+' : '-', pages << PAGE_SHIFT, - locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK), - ret ? " - exceeded" : ""); + if (!ret) + mm->locked_vm = locked_vm + pages; + + pr_debug("%s: [%d] caller %ps %lu %lu/%lu%s\n", __func__, task->pid, + (void *)_RET_IP_, pages << PAGE_SHIFT, locked_vm << PAGE_SHIFT, + task_rlimit(task, RLIMIT_MEMLOCK), ret ? " - exceeded" : ""); return ret; } EXPORT_SYMBOL_GPL(__account_locked_vm); /** + * __unaccount_locked_vm - unaccount locked pages to an mm's locked_vm + * @mm: mm to account against + * @pages: number of pages to account + * + * Assumes @mm are valid and that mmap_lock is held as writer. + */ +void __unaccount_locked_vm(struct mm_struct *mm, unsigned long pages) +{ + unsigned long locked_vm = mm->locked_vm; + + mmap_assert_write_locked(mm); + WARN_ON_ONCE(pages > locked_vm); + mm->locked_vm = locked_vm - pages; +} +EXPORT_SYMBOL_GPL(__unaccount_locked_vm); + +/** * account_locked_vm - account locked pages to an mm's locked_vm * @mm: mm to account against, may be NULL * @pages: number of pages to account - * @inc: %true if @pages should be considered positive, %false if not * * Assumes a non-NULL @mm is valid (i.e. at least one reference on it). * @@ -629,7 +639,7 @@ EXPORT_SYMBOL_GPL(__account_locked_vm); * * 0 on success, or if mm is NULL * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded. */ -int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc) +int account_locked_vm(struct mm_struct *mm, unsigned long pages) { int ret; @@ -637,14 +647,35 @@ int account_locked_vm(struct mm_struct *mm, unsigned long pages, bool inc) return 0; mmap_write_lock(mm); - ret = __account_locked_vm(mm, pages, inc, current, - capable(CAP_IPC_LOCK)); + ret = __account_locked_vm(mm, pages, current, capable(CAP_IPC_LOCK)); mmap_write_unlock(mm); return ret; } EXPORT_SYMBOL_GPL(account_locked_vm); +/** + * unaccount_locked_vm - account locked pages to an mm's locked_vm + * @mm: mm to account against, may be NULL + * @pages: number of pages to account + * + * Assumes a non-NULL @mm is valid (i.e. at least one reference on it). + * + * Return: + * * 0 on success, or if mm is NULL + * * -ENOMEM if RLIMIT_MEMLOCK would be exceeded. + */ +void unaccount_locked_vm(struct mm_struct *mm, unsigned long pages) +{ + if (pages == 0 || !mm) + return; + + mmap_write_lock(mm); + __unaccount_locked_vm(mm, pages); + mmap_write_unlock(mm); +} +EXPORT_SYMBOL_GPL(unaccount_locked_vm); + unsigned long vm_mmap_pgoff(struct file *file, unsigned long addr, unsigned long len, unsigned long prot, unsigned long flag, unsigned long pgoff) From patchwork Mon Feb 6 07:47:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53069 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2106041wrn; Sun, 5 Feb 2023 23:53:40 -0800 (PST) X-Google-Smtp-Source: AK7set+/DYujCBHA9nACoMWKYWWamxXuA/PYksD+Gq1WG9u3Vw2ahytrhR8SRvid8smArKFuuVov X-Received: by 2002:a17:907:6e87:b0:88e:e6ee:fea6 with SMTP id sh7-20020a1709076e8700b0088ee6eefea6mr18902340ejc.69.1675670019952; Sun, 05 Feb 2023 23:53:39 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675670019; cv=pass; d=google.com; s=arc-20160816; b=yjkfp9+bOjOZQt57zyur/BRYa1vdbyW7fpZmKZUAyVVMC6bxGcN6MmtnkhUMiWdcuP DgCFN2SBT+kDCWc5mSkSsQeM6iqJzDY3vO9GQToYq8ilYyrctLgNcRLdoldzQ5xnSjYN BDI0jv8lTe4/PyCegQ0fjXNyUsNaf14vdadP7mdp6WeFnCS34LzwEEsr13V0ZZCnXyrs tTrvJVippGJ2cFApCnVfO+oWLgQAz/GSPuTmHGbeYqBMpO5ocEqpYWqmtDai9H6d555S kdcYw8xqsl/AOmRePlPSzQJTVReHhhQ32gxHKAjTcyPuVSHv711NV4YfvXG5jiY4e1lP eplg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=UULxjfjg3cKi8O85puJF65UHMaoo5iv3EW5HbTo5aKQ=; b=TOxxmnWxTB2sCf++1zEgtOYD2qA14MViRsFVD+EJcBZa2z6KNOjHw5tRi7MZoLoVzx 82GGse2SolU9Zt9H+ytXA80IlAGmmy6S/ZCl44yL4MqNtfVO7TtzVFsgyn4qzLzijH3M SPNA6LSZkpV1pK5kSGKbb4WfFnGyrXRh7kLCsg0+JVgZMitSCJ0Yiq7RNBjj0Mg5E7UE WbRUwj4Gr2arxjAuzwdX9vkjENuanCOrOHB7VvgZllgsaoHdQpJK40qxU3Wm8aO3pAhO DIrNz71HOyqxy4AD5KqITpcwbRjNfj2b0hvSG02moPtN2CxdqVj6xsCCbeqndccZvU4z r5Ig== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=HiTkcGel; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 30-20020a170906209e00b0087a56fcad93si13595254ejq.262.2023.02.05.23.53.16; Sun, 05 Feb 2023 23:53:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=HiTkcGel; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230049AbjBFHwI (ORCPT + 99 others); Mon, 6 Feb 2023 02:52:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230013AbjBFHvv (ORCPT ); Mon, 6 Feb 2023 02:51:51 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on2054.outbound.protection.outlook.com [40.107.223.54]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49AA31E1CC; Sun, 5 Feb 2023 23:51:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MPAS6xf1LAX8nbqvyAPjry7Ok3FZJMWJFBf9Fbo8M7i7sIyOpVEhUTfhlZUz0lTROVxTyxkyNX8XSxpc0ToKn9Bcl+wTUp/nPlodlxGD762wQAjKIs5ciz4+XUdeAIPJrMlzl/BH13sjo0ChWjOn2CfuhXw4ihTGVUibE+4fxTDaI9w+6qDzp+c848l1tpG6cdS0VhZ6HQil/1p0fcSmCGYgswbXfnvHwP855kDAV2Cj7ikvO2z12It+p5IHbf538E31u82Awpf4ywAXDu+PJN74fYdT0wupoE7Ln2rm6Fw3Gp+Jsxvrr0bjAZrkRmrvk4EyEInejJQMQg2wLa3DxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UULxjfjg3cKi8O85puJF65UHMaoo5iv3EW5HbTo5aKQ=; b=nrw/buuxNkvje+ziK0hFoBOp6Zue6IEW526Bxf9/QKOe7o4hK7U84Nf9DHdlXw/6J6v2k8ZfgpBQUlRjzNvt/ojioy6TPEGszu5u9n1xOx+O0XOwwSIVJvwev+yzBijqEHhJRyfTeARTMEZtTt4Rgr5Tbc7dKtU+jVbEZcyzmTnO8thAyT0YtAmnHB3JPc4lDf6Z7HS0Rf8vP3/MaO0B/nIeQnF7403I1KKYPtWhSMYb508Fe1lnRwCIR7ArwdVpjoFBwl6ABv37nk6GMn4I3VKMs0q4mKBZH9gt0YVXoXNWlRBYkMVEgAzJZLjZjrU9PMZ1h+PPyVkT5YindexsQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UULxjfjg3cKi8O85puJF65UHMaoo5iv3EW5HbTo5aKQ=; b=HiTkcGelB2wmweWAVNOmNkdU0rKyyXrnj7D3F7u2iDvA97S4KIV+yn7o+6KkADoYBokw3zgP5YOldpcyfFY2QcbYkmNEUUdS9DCvsGa0toTCaBaG1exG0y/r9nWMQB3TzCd53GzIfEGiArBODGSaUuj7pNrqRR1H4iCkvuKqiUiinGbdAT4MmQQyCNmm/wSmt4Cfr7oLi0Z32OOzFN8Jd6Ht859/mwAYJhNSTTgAlPvgCTFef94HceTm07YxCtjiSQdhpcW/BsrhCVWxlJ6Jdxyuwl1TKZpgQza6x5k+x+ce/GI+PeI1UhfdDhvIwevcuE186gG6pEUIM+ufeb94bQ== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DS0PR12MB8573.namprd12.prod.outlook.com (2603:10b6:8:162::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 07:50:07 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:50:07 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple Subject: [PATCH 17/19] mm: Convert mmap and mlock to use account_locked_vm Date: Mon, 6 Feb 2023 18:47:54 +1100 Message-Id: <83b0638b0aa88ac57656111ea9e1e771f2b755e3.1675669136.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYYP282CA0001.AUSP282.PROD.OUTLOOK.COM (2603:10c6:10:b4::11) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DS0PR12MB8573:EE_ X-MS-Office365-Filtering-Correlation-Id: c719672c-a25e-4472-baf3-08db0816c396 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: TzvseV88qNLSsZY++QftlI/fk/YJ15Zq6eR8ZNKN5dXaNb8Lh7bOQjT5MHKOUHHMS87hBP/D4NNeho4WMdU1Sto3z+mY995YURjRPpRmqBgQ4OYyZVALEG6zP+z0PJ/B2nnFtRcw/2uT3TPXDLq8H/z4BBLRZFbJTV+GpGF5HDh6vmRbkX29fP3IOA9ky89b9i9dyMJUw6XKRcjOpauYyP63/31+oooepryfo1hXul4886/U/uHRKm5ClV/W2XmvIfZ1LDo8Km5vSQiE/rfD645H7jDN15IT1SddAB6uK4fMuQPl+vomKNjmyNnMykGQK/DQBo1Czc2mXkmhXYQNsKxufbMRawcHX8LZDPreYE1GqmVD79AyJ59YRqjucmUWAPsIbImdtDhY9Gh8TqK7W6cH8wELYsAL0c5hxNwv+d2195KsriKVjwkhh4eQmcuppsc0w718QOkleqgNYIfEkdoVpNyI9S6BdNKlQQAn7LXbM35V1Dskkb/MBFy65iCl+RUCs9Qn7dcDdzmKGIT6b1YSAjJkBV/hcNT33tPDXBV2pdlitg0ImMyV3OcJXSYzYp2PXaakD+GM4nFQK0Z4WVswVij6RUOfjksG2XHlpdnVrEC3qD2anI3R4uCjqsGvo8i3vxSOwdb110VsFCTulA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(54906003)(316002)(86362001)(6666004)(107886003)(6486002)(38100700002)(186003)(2616005)(6506007)(26005)(478600001)(6512007)(7416002)(5660300002)(83380400001)(15650500001)(30864003)(2906002)(36756003)(66946007)(8936002)(4326008)(41300700001)(66476007)(8676002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IJLIgaaFcbO0fRdT4i/Qk3vSH8n7ATsGeJRSU1NpCfmbtbM2AKURep5S4/XtHV/2NRU6+glQHUUv0NWElTpuV8XhHm9swVUPrpuKZJ09iWW9hJsTwn4RQkDX0NjqoWjne3BbKhua95gEr2w0PL63TSR7zCPv5oMdFAci1EuIjcVT/CdMZmJvxHiPEnfOc+HTBYz1MYQKEkbesJHTOjvRR3IXxFwh1epLk//X7NzdFGYXun/eVo/NZ6blm2C9jHQJJ2zA6WR5aX94CNtSSeLLka5MZ5/jJG+COkH7DSdwPdJSuu92bJEpRJaQjgt87rNb9B2O76lpWswMEMnv10thkfP+TQoFpIFlPrJihsFzgfFF5R7Hh0AEkSHx9IgGIBNY+w6xSxmbCHB1+QLpoBo/39gmzxpnSYBniNpVGliUPEvacB1PX0r63dNWQtmzFiJGvffim9ElgLHNZePAeChw91CZfWUuA5F0Jp3fARivbjY7RaTjKRElWrkPqGbnPRHmL11k0Qg5zJtzNOSV+GELlLmLv7t3dlw+sRKuL/cdOfEp69+yvwDyzlQRypSDdZLh2DvtE3gYu1Ux/awAjDbF8Fx+8837mdk0DbYCCYDKSkYXeaoAK4mM/J8lz9gERisdsrodYKlxe6JF8DYuhULc+Bm05aPFM4UYaDEq4j/sNPbVjipXzGtsdsodbX1lqOknkFwSGmZDY8oPgUfCoM1xHpVpjsYoTjSHR9rVugH35x6NqSPg64RXqni0WX0UkSMjsT9P4nlZja9nHBOTQQ2mDnGgSyZ89k1GIeWX64FZpRUvo1E9sDNN8E31LOjNsUG4x8Ix4EUUu1rH7myefCSH7eWVyiHdoHpeWfiS3aexAKjofemcTbRrFHApMVS4zxRUAqd5CGOTNWvZmMo1+/6OPT1IvHFAcS2RlpLimWb0lhkacaMcVxLhVLucCU/ll97xSZ3vfgxbmV72DAmS1ai1sYDeGCz96LwiurMUxAWUx1vHVG2fv/NJvenJ2DK0Ixs6fOt4BdIl/unKPOP9kl62bDQCvJSDQWBa3m9w6S/m49rMoGqhpl1qpIOGOg4GTxR4+8lVVw9CFzLDrIJJKArzJ6isBsLjthUgynq7+axVbTsNXwNKNpJIqZ6oU+A1qWEHEGYNj8iYZs2sFNPCrxblrmWV9RiuX8t1fStaNVmb7slV1i8G0N291yVnleZFvPowgOPtSva2b/ag706vlV6gHhaC3fxip5RiOsUkUX6Po5UJvpgcQoaYywoWt5ZX//fWkqaIWdkhpAlFNpd9NcQByP8WQroPkHqQ/n8jdfY/gGOCtHeWjnAUJO4u7r2suiyKWrYGjD+CeSEAjfsGh4oQdNqKsTPMbo97MVttir1WHiiBT6xtdJqrhafWAEOdzRKtsVs9IdFMGGZJeWtnWjPh2cUVrQZVptM+TcCrCN+iXgXnywIFoBKVgFRiqSu5i0dXnZyAFg5x2XEK1jTPN/p5nZenjWcsIA1nlwjiGMZIrTmbm5P3DNrMxE4bBls0bIabweDP+ykMda64BJqQV5t/xBJS0g3BNqO+rzA+vK3ycsLQMuZGShetGPxi4E0sFkQa X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: c719672c-a25e-4472-baf3-08db0816c396 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:50:06.9451 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mwpity7BF1x59t16Wj0FBo9LE+yZ69ukmJnQWN1euVwEo38i77kLduaxC/oj+AhQ3BGF+elulwNSAobgy9fsxQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, RCVD_IN_DNSWL_NONE,RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067366906741078?= X-GMAIL-MSGID: =?utf-8?q?1757067366906741078?= A future change introduces a cgroup to control the amount of locked/pinned memory on the system. To ensure memory pinned via mlock and mmap is accounted for use the common account_locked_vm() function. As cgroups can outlive individual processes also unaccount for the locked memory during process teardown. This patch should introduce no user visible change. Signed-off-by: Alistair Popple Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/internal.h | 2 +- mm/mlock.c | 76 ++++++++++----------------------------------------- mm/mmap.c | 76 +++++++++++++++++++++++++-------------------------- mm/mremap.c | 54 ++++++++++++++++++++++++++---------- mm/secretmem.c | 6 +--- 5 files changed, 95 insertions(+), 119 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index bcf75a8..7c8c3f2 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -489,8 +489,6 @@ extern long populate_vma_page_range(struct vm_area_struct *vma, extern long faultin_vma_page_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, bool write, int *locked); -extern int mlock_future_check(struct mm_struct *mm, unsigned long flags, - unsigned long len); /* * mlock_vma_page() and munlock_vma_page(): * should be called with vma's mmap_lock held for read or write, diff --git a/mm/mlock.c b/mm/mlock.c index 7032f6d..a97c8c5 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -416,6 +416,20 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, /* don't set VM_LOCKED or VM_LOCKONFAULT and don't count */ goto out; + /* + * Keep track of amount of locked VM. + */ + nr_pages = (end - start) >> PAGE_SHIFT; + if (!(newflags & VM_LOCKED)) { + __unaccount_locked_vm(mm, nr_pages); + } else if (!(oldflags & VM_LOCKED)) { + if (__account_locked_vm(mm, nr_pages, current, + capable(CAP_IPC_LOCK))) { + ret = -ENOMEM; + goto out; + } + } + pgoff = vma->vm_pgoff + ((start - vma->vm_start) >> PAGE_SHIFT); *prev = vma_merge(mm, *prev, start, end, newflags, vma->anon_vma, vma->vm_file, pgoff, vma_policy(vma), @@ -439,16 +453,6 @@ static int mlock_fixup(struct vm_area_struct *vma, struct vm_area_struct **prev, success: /* - * Keep track of amount of locked VM. - */ - nr_pages = (end - start) >> PAGE_SHIFT; - if (!(newflags & VM_LOCKED)) - nr_pages = -nr_pages; - else if (oldflags & VM_LOCKED) - nr_pages = 0; - mm->locked_vm += nr_pages; - - /* * vm_flags is protected by the mmap_lock held in write mode. * It's okay if try_to_unmap_one unmaps a page just after we * set VM_LOCKED, populate_vma_page_range will bring it back. @@ -517,42 +521,6 @@ static int apply_vma_lock_flags(unsigned long start, size_t len, } /* - * Go through vma areas and sum size of mlocked - * vma pages, as return value. - * Note deferred memory locking case(mlock2(,,MLOCK_ONFAULT) - * is also counted. - * Return value: previously mlocked page counts - */ -static unsigned long count_mm_mlocked_page_nr(struct mm_struct *mm, - unsigned long start, size_t len) -{ - struct vm_area_struct *vma; - unsigned long count = 0; - unsigned long end; - VMA_ITERATOR(vmi, mm, start); - - /* Don't overflow past ULONG_MAX */ - if (unlikely(ULONG_MAX - len < start)) - end = ULONG_MAX; - else - end = start + len; - - for_each_vma_range(vmi, vma, end) { - if (vma->vm_flags & VM_LOCKED) { - if (start > vma->vm_start) - count -= (start - vma->vm_start); - if (end < vma->vm_end) { - count += end - vma->vm_start; - break; - } - count += vma->vm_end - vma->vm_start; - } - } - - return count >> PAGE_SHIFT; -} - -/* * convert get_user_pages() return value to posix mlock() error */ static int __mlock_posix_error_return(long retval) @@ -585,21 +553,7 @@ static __must_check int do_mlock(unsigned long start, size_t len, vm_flags_t fla if (mmap_write_lock_killable(current->mm)) return -EINTR; - locked += current->mm->locked_vm; - if ((locked > lock_limit) && (!capable(CAP_IPC_LOCK))) { - /* - * It is possible that the regions requested intersect with - * previously mlocked areas, that part area in "mm->locked_vm" - * should not be counted to new mlock increment count. So check - * and adjust locked count if necessary. - */ - locked -= count_mm_mlocked_page_nr(current->mm, - start, len); - } - - /* check against resource limits */ - if ((locked <= lock_limit) || capable(CAP_IPC_LOCK)) - error = apply_vma_lock_flags(start, len, flags); + error = apply_vma_lock_flags(start, len, flags); mmap_write_unlock(current->mm); if (error) diff --git a/mm/mmap.c b/mm/mmap.c index 425a934..2c05582 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -160,7 +160,7 @@ static int check_brk_limits(unsigned long addr, unsigned long len) if (IS_ERR_VALUE(mapped_addr)) return mapped_addr; - return mlock_future_check(current->mm, current->mm->def_flags, len); + return 0; } static int do_brk_munmap(struct ma_state *mas, struct vm_area_struct *vma, unsigned long newbrk, unsigned long oldbrk, @@ -1184,23 +1184,6 @@ static inline unsigned long round_hint_to_min(unsigned long hint) return hint; } -int mlock_future_check(struct mm_struct *mm, unsigned long flags, - unsigned long len) -{ - unsigned long locked, lock_limit; - - /* mlock MCL_FUTURE? */ - if (flags & VM_LOCKED) { - locked = len >> PAGE_SHIFT; - locked += mm->locked_vm; - lock_limit = rlimit(RLIMIT_MEMLOCK); - lock_limit >>= PAGE_SHIFT; - if (locked > lock_limit && !capable(CAP_IPC_LOCK)) - return -EAGAIN; - } - return 0; -} - static inline u64 file_mmap_size_max(struct file *file, struct inode *inode) { if (S_ISREG(inode->i_mode)) @@ -1310,9 +1293,6 @@ unsigned long do_mmap(struct file *file, unsigned long addr, if (!can_do_mlock()) return -EPERM; - if (mlock_future_check(mm, vm_flags, len)) - return -EAGAIN; - if (file) { struct inode *inode = file_inode(file); unsigned long flags_mask; @@ -1882,22 +1862,27 @@ static int acct_stack_growth(struct vm_area_struct *vma, if (size > rlimit(RLIMIT_STACK)) return -ENOMEM; - /* mlock limit tests */ - if (mlock_future_check(mm, vma->vm_flags, grow << PAGE_SHIFT)) - return -ENOMEM; - /* Check to ensure the stack will not grow into a hugetlb-only region */ new_start = (vma->vm_flags & VM_GROWSUP) ? vma->vm_start : vma->vm_end - size; if (is_hugepage_only_range(vma->vm_mm, new_start, size)) return -EFAULT; + /* mlock limit tests */ + if (vma->vm_flags & VM_LOCKED) + if (__account_locked_vm(mm, grow << PAGE_SHIFT, current, + capable(CAP_IPC_LOCK))) + return -ENOMEM; + /* * Overcommit.. This must be the final test, as it will * update security statistics. */ - if (security_vm_enough_memory_mm(mm, grow)) + if (security_vm_enough_memory_mm(mm, grow)) { + if (vma->vm_flags & VM_LOCKED) + __unaccount_locked_vm(mm, grow << PAGE_SHIFT); return -ENOMEM; + } return 0; } @@ -1975,8 +1960,6 @@ int expand_upwards(struct vm_area_struct *vma, unsigned long address) * to guard against concurrent vma expansions. */ spin_lock(&mm->page_table_lock); - if (vma->vm_flags & VM_LOCKED) - mm->locked_vm += grow; vm_stat_account(mm, vma->vm_flags, grow); anon_vma_interval_tree_pre_update_vma(vma); vma->vm_end = address; @@ -2056,8 +2039,6 @@ int expand_downwards(struct vm_area_struct *vma, unsigned long address) * to guard against concurrent vma expansions. */ spin_lock(&mm->page_table_lock); - if (vma->vm_flags & VM_LOCKED) - mm->locked_vm += grow; vm_stat_account(mm, vma->vm_flags, grow); anon_vma_interval_tree_pre_update_vma(vma); vma->vm_start = address; @@ -2281,7 +2262,7 @@ static inline int munmap_sidetree(struct vm_area_struct *vma, return -ENOMEM; if (vma->vm_flags & VM_LOCKED) - vma->vm_mm->locked_vm -= vma_pages(vma); + __unaccount_locked_vm(vma->vm_mm, vma_pages(vma)); return 0; } @@ -2525,6 +2506,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, struct vm_area_struct *next, *prev, *merge; pgoff_t pglen = len >> PAGE_SHIFT; unsigned long charged = 0; + unsigned long locked = 0; unsigned long end = addr + len; unsigned long merge_start = addr, merge_end = end; pgoff_t vm_pgoff; @@ -2560,6 +2542,15 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vm_flags |= VM_ACCOUNT; } + if (vm_flags & VM_LOCKED) { + locked = len >> PAGE_SHIFT; + if (__account_locked_vm(mm, locked, current, + capable(CAP_IPC_LOCK))) { + error = -ENOMEM; + goto unacct_error; + } + } + next = mas_next(&mas, ULONG_MAX); prev = mas_prev(&mas, 0); if (vm_flags & VM_SPECIAL) @@ -2605,7 +2596,7 @@ unsigned long mmap_region(struct file *file, unsigned long addr, vma = vm_area_alloc(mm); if (!vma) { error = -ENOMEM; - goto unacct_error; + goto unlock_error; } vma->vm_start = addr; @@ -2725,8 +2716,6 @@ unsigned long mmap_region(struct file *file, unsigned long addr, is_vm_hugetlb_page(vma) || vma == get_gate_vma(current->mm)) vma->vm_flags &= VM_LOCKED_CLEAR_MASK; - else - mm->locked_vm += (len >> PAGE_SHIFT); } if (file) @@ -2759,6 +2748,9 @@ unsigned long mmap_region(struct file *file, unsigned long addr, mapping_unmap_writable(file->f_mapping); free_vma: vm_area_free(vma); +unlock_error: + if (locked) + __unaccount_locked_vm(mm, locked); unacct_error: if (charged) vm_unacct_memory(charged); @@ -2942,8 +2934,13 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, if (mm->map_count > sysctl_max_map_count) return -ENOMEM; + if (flags & VM_LOCKED) + if (__account_locked_vm(mm, len >> PAGE_SHIFT, current, + capable(CAP_IPC_LOCK))) + return -ENOMEM; + if (security_vm_enough_memory_mm(mm, len >> PAGE_SHIFT)) - return -ENOMEM; + goto unacct_locked; /* * Expand the existing vma if possible; Note that singular lists do not @@ -2993,8 +2990,6 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, perf_event_mmap(vma); mm->total_vm += len >> PAGE_SHIFT; mm->data_vm += len >> PAGE_SHIFT; - if (flags & VM_LOCKED) - mm->locked_vm += (len >> PAGE_SHIFT); vma->vm_flags |= VM_SOFTDIRTY; validate_mm(mm); return 0; @@ -3003,6 +2998,8 @@ static int do_brk_flags(struct ma_state *mas, struct vm_area_struct *vma, vm_area_free(vma); unacct_fail: vm_unacct_memory(len >> PAGE_SHIFT); +unacct_locked: + __unaccount_locked_vm(mm, len >> PAGE_SHIFT); return -ENOMEM; } @@ -3064,7 +3061,7 @@ void exit_mmap(struct mm_struct *mm) { struct mmu_gather tlb; struct vm_area_struct *vma; - unsigned long nr_accounted = 0; + unsigned long nr_accounted = 0, nr_locked = 0; MA_STATE(mas, &mm->mm_mt, 0, 0); int count = 0; @@ -3107,6 +3104,8 @@ void exit_mmap(struct mm_struct *mm) do { if (vma->vm_flags & VM_ACCOUNT) nr_accounted += vma_pages(vma); + if (vma->vm_flags & VM_LOCKED) + nr_locked += vma_pages(vma); remove_vma(vma); count++; cond_resched(); @@ -3116,6 +3115,7 @@ void exit_mmap(struct mm_struct *mm) trace_exit_mmap(mm); __mt_destroy(&mm->mm_mt); + __unaccount_locked_vm(mm, nr_locked); mmap_write_unlock(mm); vm_unacct_memory(nr_accounted); } diff --git a/mm/mremap.c b/mm/mremap.c index fe587c5..67cc4f3 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -574,7 +574,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, bool *locked, unsigned long flags, struct vm_userfaultfd_ctx *uf, struct list_head *uf_unmap) { - long to_account = new_len - old_len; + long to_account = (new_len - old_len) >> PAGE_SHIFT; struct mm_struct *mm = vma->vm_mm; struct vm_area_struct *new_vma; unsigned long vm_flags = vma->vm_flags; @@ -594,7 +594,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, return -ENOMEM; if (unlikely(flags & MREMAP_DONTUNMAP)) - to_account = new_len; + to_account = new_len >> PAGE_SHIFT; if (vma->vm_ops && vma->vm_ops->may_split) { if (vma->vm_start != old_addr) @@ -618,16 +618,36 @@ static unsigned long move_vma(struct vm_area_struct *vma, return err; if (vm_flags & VM_ACCOUNT) { - if (security_vm_enough_memory_mm(mm, to_account >> PAGE_SHIFT)) + if (security_vm_enough_memory_mm(mm, to_account)) return -ENOMEM; } + /* + * MREMAP_DONTUNMAP clears VM_LOCKED on the old vma and + * implies new_len == old_len so no need to account locked + * pages. + */ + if ((vm_flags & VM_LOCKED) && likely(!(flags & MREMAP_DONTUNMAP))) { + if (__account_locked_vm(mm, to_account, current, + capable(CAP_IPC_LOCK))) { + if (vm_flags & VM_ACCOUNT) + vm_unacct_memory(to_account); + return -ENOMEM; + } + *locked = true; + } + new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, &need_rmap_locks); if (!new_vma) { if (vm_flags & VM_ACCOUNT) - vm_unacct_memory(to_account >> PAGE_SHIFT); + vm_unacct_memory(to_account); + if ((vm_flags & VM_LOCKED) && + likely(!(flags & MREMAP_DONTUNMAP))) { + __unaccount_locked_vm(mm, to_account); + *locked = false; + } return -ENOMEM; } @@ -696,10 +716,11 @@ static unsigned long move_vma(struct vm_area_struct *vma, vma->vm_end == (old_addr + old_len)) unlink_anon_vmas(vma); - /* Because we won't unmap we don't need to touch locked_vm */ return new_addr; } + /* Make sure do_munmap() doesn't unaccount locked pages */ + vma->vm_flags &= VM_LOCKED_CLEAR_MASK; if (do_munmap(mm, old_addr, old_len, uf_unmap) < 0) { /* OOM: unable to split vma, just get accounts right */ if (vm_flags & VM_ACCOUNT && !(flags & MREMAP_DONTUNMAP)) @@ -707,15 +728,11 @@ static unsigned long move_vma(struct vm_area_struct *vma, excess = 0; } - if (vm_flags & VM_LOCKED) { - mm->locked_vm += new_len >> PAGE_SHIFT; - *locked = true; - } - mm->hiwater_vm = hiwater_vm; /* Restore VM_ACCOUNT if one or two pieces of vma left */ if (excess) { + vma->vm_flags = vm_flags; vma->vm_flags |= VM_ACCOUNT; if (split) find_vma(mm, vma->vm_end)->vm_flags |= VM_ACCOUNT; @@ -768,9 +785,6 @@ static struct vm_area_struct *vma_to_resize(unsigned long addr, if (vma->vm_flags & (VM_DONTEXPAND | VM_PFNMAP)) return ERR_PTR(-EFAULT); - if (mlock_future_check(mm, vma->vm_flags, new_len - old_len)) - return ERR_PTR(-EAGAIN); - if (!may_expand_vm(mm, vma->vm_flags, (new_len - old_len) >> PAGE_SHIFT)) return ERR_PTR(-ENOMEM); @@ -1026,6 +1040,16 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, } } + if (vma->vm_flags & VM_LOCKED) { + if (__account_locked_vm(mm, pages, current, + capable(CAP_IPC_LOCK))) { + if (vma->vm_flags & VM_ACCOUNT) + vm_unacct_memory(pages); + ret = -ENOMEM; + goto out; + } + } + /* * Function vma_merge() is called on the extension we are adding to * the already existing vma, vma_merge() will merge this extension with @@ -1038,14 +1062,16 @@ SYSCALL_DEFINE5(mremap, unsigned long, addr, unsigned long, old_len, extension_pgoff, vma_policy(vma), vma->vm_userfaultfd_ctx, anon_vma_name(vma)); if (!vma) { + // TODO: We always unacct memory + // regardless of VM_ACCOUNT flags? vm_unacct_memory(pages); + __unaccount_locked_vm(mm, pages); ret = -ENOMEM; goto out; } vm_stat_account(mm, vma->vm_flags, pages); if (vma->vm_flags & VM_LOCKED) { - mm->locked_vm += pages; locked = true; new_addr = addr; } diff --git a/mm/secretmem.c b/mm/secretmem.c index 04c3ac9..4515eb4 100644 --- a/mm/secretmem.c +++ b/mm/secretmem.c @@ -120,13 +120,11 @@ static int secretmem_release(struct inode *inode, struct file *file) static int secretmem_mmap(struct file *file, struct vm_area_struct *vma) { - unsigned long len = vma->vm_end - vma->vm_start; - if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0) return -EINVAL; - if (mlock_future_check(vma->vm_mm, vma->vm_flags | VM_LOCKED, len)) - return -EAGAIN; + if (account_locked_vm(vma->vm_mm, vma->vm_end - vma->vm_start)) + return -ENOMEM; vma->vm_flags |= VM_LOCKED | VM_DONTDUMP; vma->vm_ops = &secretmem_vm_ops; From patchwork Mon Feb 6 07:47:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53071 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2106238wrn; Sun, 5 Feb 2023 23:54:34 -0800 (PST) X-Google-Smtp-Source: AK7set8dpEZwOGk0KTwMq5L8SJcX4n58IvgIr2tARoPFezhk0plZ+Sm6C63cyyHANPNgLAqzyBzS X-Received: by 2002:a50:9998:0:b0:4aa:b409:77aa with SMTP id m24-20020a509998000000b004aab40977aamr1932306edb.34.1675670073861; Sun, 05 Feb 2023 23:54:33 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675670073; cv=pass; d=google.com; s=arc-20160816; b=myyG/yV2h1fBVZOP897EK1Fc7UoyfIQ+m5UZ6peYaObhZc6itJ4ekPwNmyo186K/oU xUgqzm29SzdT7OPeNu2rIH+lECrKiVJUkVSlhquu7iYTgZjky18mSZskaARrhZvcDdim 1MYdi3w7Hbhxl2yunEKKS8SuojskrTouqmCUNqTU6RpwXzxl8jutdN06ZLHI28DwLQu5 yoPbiNgxBvJ+gEPh/AzwmfKtjQ4KFZANPUlav8m5k79Jcbf6BsfrSmNUqmKdPO6oH+2B qkFgGS+lMB0X9U+B6Oj3QHZOAbNl/q3Vu9brFInkQSC+ppmYzGibidKnjXfTZmL2H1Ir Nrfg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=kJx4d6L32xsb7lM+TLJ9IAT+Zep1O/ydPt8JKIy3HJs=; b=K7qgNB0NfTRaobqcCatxMriFoLBVA6kS9BFElt16lmj3HrKGU0xpIDPhaOOzzaXJ6o 1aOEewvEqGHwA9xkp++f8h7eRyOJq8WgYOWWOn285oGp1ssPH8Z2WGg0q7PyMRjc/bze KosRWt9PsBxKttfPyp8B4RUnaK+qoMQWVUb1b3Q3zOFJ3nmyGfMYLUX26y8iPL4hBv81 nMnLLWSl+INj2D9c/W1clpOmXrvMyZIFO1bl7i1S/VPyj+9KOkl/LS75b1WB6xfCHHFN QbX6pn5tuD1E+EJfCUITDYJwGaEPtw0y8Wz2ecJTGNDif6qJakjW8AZfQdv23gbKYRp5 goSg== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=ZtlV3xqk; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id v7-20020a056402348700b0048ec063ea63si13375837edc.396.2023.02.05.23.54.07; Sun, 05 Feb 2023 23:54:33 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=ZtlV3xqk; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229878AbjBFHw7 (ORCPT + 99 others); Mon, 6 Feb 2023 02:52:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230063AbjBFHwy (ORCPT ); Mon, 6 Feb 2023 02:52:54 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20631.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eaa::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6091E1557F; Sun, 5 Feb 2023 23:52:20 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RyP2XQnz0N73PcQxjcUT4uAOGaufarRgV3t4vLzSzzaurNS9YOcieNt9HHUpRyhwuM0ST/JTfrnniUiAz2oiucF4XvGxWynkY75nZ+/jKnYF5jyJfTFHJLzLl+inSQwhMUp8KF/KTV6R0uHjgfrLSiQzdcb1w/PvSdL/whUjfUrVgoT0XHwiTaYRCpIpkuHalxJIBNxvQkJZ+M34gmal3k7aTaYlxnf1FZs2FlRsifhjhSiiUTTYgpp7TyZLepR00cZWTRFO5Vl65rxRkGfwPYmaCuzRjYw17K5klZ03gT+QESmo5i4AZg+VzA4BP9dJ3jf92mFMPCEYcQtJMJBfzg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kJx4d6L32xsb7lM+TLJ9IAT+Zep1O/ydPt8JKIy3HJs=; b=ffHCQDtLu++R1E5ReQ7NDzjd2FnKcI31hET478cSznvJlv8G0gIdRGmRi5ovquC71bWDlL3D6+QbMGZacXA/lCwmCpeTFM13yhgSlUzhUUj9KSftTbmdDfMNVcPrh4lGp8gYIzzVm9MlilIb7XqLKIR+WCii0mBFs9TLYxqUTlGDY5ZlMzx3rW6jbN5nbnTOgzU78KBoqWZVQijbW2YIpB1a2PAihRChWoHiG6qBbl8fZ7Gtjnz7uWkLhJBs3AucxGtk52dUSy3xz+Kbyvj/mGn4YsqIkOTRx+jmFqkX+P8PUHkR5ft9LW0YSUvvdsqfYgbZBBhii8UPBuVSo4TIXg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kJx4d6L32xsb7lM+TLJ9IAT+Zep1O/ydPt8JKIy3HJs=; b=ZtlV3xqkVCdEoJMfxTySEMtXYlBcu3RxzlI8GNdhznj8a3/UJ6829hjPsh6TZDRu2tpHt00m6JeVT9mO19xTSFl14OLTKMl2qMVxWKCQ08L7ww//5xTfGaLbzbp2s/s+uLONIvlAYGNugDKhqxzwxxpHpX1Q05Ec8ZZUZEusqbBXkWlLLK1Eb7YUEv1W9yawV01c44sPu3WlKPNxoO1W2hfj+pC9GmCitT/gDCXV23DrMjJW056Q12HoqU6w/6LxfLme95bxIla/oE9b5Tf8DZO97bHQwNjl3phF0Ws5x7pCH4BQc+7zmpLqeADSEpnxfCVrYTxZfdoWv2as/21C2A== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DS0PR12MB8573.namprd12.prod.outlook.com (2603:10b6:8:162::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 07:50:13 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:50:13 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple Subject: [PATCH 18/19] mm/mmap: Charge locked memory to pins cgroup Date: Mon, 6 Feb 2023 18:47:55 +1100 Message-Id: X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYBPR01CA0114.ausprd01.prod.outlook.com (2603:10c6:10:1::30) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DS0PR12MB8573:EE_ X-MS-Office365-Filtering-Correlation-Id: 427e0896-7913-4612-52a1-08db0816c76f X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: N8UuAVBxTWt5/pJf6aD+fYRN9CGKIzqzua+ZekDKIw+u7Cw+ngRBnvwoB9EW06W5OR0HXypCbToXB/cTcd5hiyTJ4YX4oO9HDgWQ64BG1KiuYLkpOYZfB9slmftj8NkWdQMDICe4WBXEAudCmcOSQpCOVpdMi/UhZdvuH1pZtbZrSg0jh32quXs3bIsZh/I+WAET/3XXCpMr4+ZnA1P14qXZR1GGx+UvgPTLunxC0V3bx9EMasETCjCThtsfLbPoQlyi/xXMFqF4N5AMD6+Ggcx/PYVi46aL5nnpIQS1piY4WVYUz6r/yGju67Hw2klWH6/UgXbVn6p2oPQOrUVL/DkCe/WilMc60kJsFbu/ASymE+K8BUoQBoqefccmV2J5IQab1Nm5oQ6n6h4dYvPMCzFxe/uZ6DscTTIgCKgE3W/bhaglxgsPWqQG5+aCypZ4hqCrRcjKkIJ5hu2vhB1yUq4iQsDz1zDM6T/dgbV26YW+Jrusm+AVcjKbr59XR4/09BclrJ0OjBoo7tyF8XARfOT4an4oPbYDJvUzT34OHJaTUSauIUkw6aBmCNAH5iLUWR5hsrxp3vf3lAG59oP4GKI5Z6z2vAHJDaOLw+36QOOi0Ayqvl0QKAnlu1SkoOoCznQ1YH5U4TKxwNkqESttoA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(54906003)(316002)(86362001)(6666004)(107886003)(6486002)(38100700002)(186003)(2616005)(6506007)(26005)(478600001)(6512007)(7416002)(5660300002)(83380400001)(2906002)(36756003)(66946007)(8936002)(4326008)(41300700001)(66476007)(8676002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 8l7kC3OaK8BVTwVGt5goAI8wanQmAamsLKwN4NQpQIf4Omil0phmccbSqFeG1Zj6ueoJx6xMDXquKDcQe5Er2cYvwVL4XX+2VHao5jZOwFkVnR28N1Sn4fn0tSRX1+dzFIipP5IPwtwykErOR0e3SkceMaAVzAQtlErlGG3oyjrFLk8oqRyfSp6gJpZZw+8PQWV9V1LO6JuAIUrR/CTzXD79JZnPtEBmhASDkMQHjJoUQ5TOxO5ONGXsnwI+n6lyNQm5W9G/fVdR3JBwGneg3qghzZf8j50sqW5ZOOiTAa4kxg8orZXUEuTgdmYGeAgWNpCtE2HQSKk1vvxJEJYsHzROorPqUshOjTR6tdE+QrGL5cuytzC+/2qr+X1Pit8TjfmyvELisUPWATl7QwbXwk4yOOBRCnCbW2FNSIzcYCDmt39TTs4w6RPni5BgIjHahkFqf5q4NFxlmctuxuHZ5eFe4T11pSkBRpqdK1cVGKu0J9ulk2cBYqJ9BNm8uW8eLEYWTXUG+C8fZ89hUQxsappHro1KbasRoaW5vum5BgCfWUBHNcU5aMcGJVQX8rrPQ5MwtgFllVkVtAj7RWLpfe0F60pGmjJyN6g+97czYDhflfKCZW7gbDXBJP8l8MFpLc6fNTJNNtm6jhsK9ZYyyS3KRgKwRF+gRJk2BBcjoi0Djtpx2dOJlX+F4Ioo8q4GvL0cKkLrXbH5g6Q5KNBtmmNjf3YpDhxpekbGluthLLUc2v5CCpKr9/ArAtsUb089j0p6582e7zMikLUgOQhOJ79dy9LNIY3hqH1wKTO5ieCBb/VsE5LpJpP4+WmZ0bKchrCLR2lEQShlUg4cSvKBX36Zo33V0/f1MtH/lr8xbj+BrwvlrpHbSHHelW3vZf9S7bVlqXuCzjKH/ulZdQDcAw+MvUsTD4vddgzeEBIE5+BoB4NxoR+FJbzLNPBGF1xv/OC2WXsNaOXRWGO24ASTDFKdpJBRWDSAs51qCJ2LqdhVJqwQZ6HOr5zz8VG3PTKVvQUKbflh2hf0LzJ0UDeGJ/+5xt81MysKO6VIcEElzItaP+lrrwX9ooJ+sTQWdI8Htq9Lxus9hnccpTFqmaVqQfxIhDIfSyqxv2qiaAUmWYENztI8W/pdTYAvSX+6Dmj4U3QQguzMoHuY/A9xU0ZXsadHFEhGh0KF2W8A5LPt4WSB6UolJ8T1uaNfmqZHKrMjZ9P004V0HSzhoV3uo6ZD0Pa8I73QgtzYTsKMjTacKcbXZgbPhGioZvLA4EZTdjWrS2lzZ5HJPAT/FO8Gm2C5UWy/2Y4ndsjX+Haq9k5c07U8wL0Le846YqGhqna+JnRUPv+xoJNaKPqpiutkX9qoF7mRn31zGwTrz8YYXtQL+by/WxJZOa6XPJSR9TAIG0g9xFEIVNuPlXyCkNJv2CptGfhmkYmsH+3Gr4cF1eOWtFDCPDAkkQPXJV+ubu4+1Q4mQMhH6v5DN3cF2YZ6EV5MphqpPQOlr9u3GhFBBwt0x4RCVlUeFqdEoU88XsrsQxekmUkMCAP54hc8+2jswdSuEFxkvPyie5g73YQgJQXm/7CKzyMT0q01G3Bv2NS8CUew X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 427e0896-7913-4612-52a1-08db0816c76f X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:50:13.2727 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tD8S4PSSN2NXQP7cLnr9ez07RFo1dGSONbQ2/RD76uWwk7J/TTGeB0H+KUhyxz4kuaksMFWXf+J2eqKvEedMsA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067423025824588?= X-GMAIL-MSGID: =?utf-8?q?1757067423025824588?= account_locked_vm() is used to account memory to mm->locked_vm. This adds accounting to the pins cgorup as it behaves similarly and should be accounted against the same global limit if set. This means memory must now be unaccounted for correctly, as the cgroup typically outlives both the mm and the task. It is assumed that callers of account_locked_vm() only do accounting against the current task. Callers that need to do accounting against remote tasks should use account_pinned_vm() and associated struct vm_account to hold references to the cgroup. Signed-off-by: Alistair Popple Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org --- mm/util.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/mm/util.c b/mm/util.c index 1ca0dfe..755bada 100644 --- a/mm/util.c +++ b/mm/util.c @@ -589,15 +589,21 @@ int __account_locked_vm(struct mm_struct *mm, unsigned long pages, struct task_struct *task, bool bypass_rlim) { unsigned long locked_vm, limit; + struct pins_cgroup *pins_cg = get_pins_cg(task); int ret = 0; mmap_assert_write_locked(mm); + if (pins_cg && !pins_try_charge(pins_cg, pages)) + return -ENOMEM; + locked_vm = mm->locked_vm; if (!bypass_rlim) { limit = task_rlimit(task, RLIMIT_MEMLOCK) >> PAGE_SHIFT; - if (locked_vm + pages > limit) + if (locked_vm + pages > limit) { + pins_uncharge(pins_cg, pages); ret = -ENOMEM; + } } if (!ret) @@ -607,6 +613,12 @@ int __account_locked_vm(struct mm_struct *mm, unsigned long pages, (void *)_RET_IP_, pages << PAGE_SHIFT, locked_vm << PAGE_SHIFT, task_rlimit(task, RLIMIT_MEMLOCK), ret ? " - exceeded" : ""); + pr_debug("%s: [%d] caller %ps %lu %lu/%lu%s\n", __func__, task->pid, + (void *)_RET_IP_, pages << PAGE_SHIFT, locked_vm << PAGE_SHIFT, + task_rlimit(task, RLIMIT_MEMLOCK), ret ? " - exceeded" : ""); + + if (pins_cg) + put_pins_cg(pins_cg); return ret; } EXPORT_SYMBOL_GPL(__account_locked_vm); @@ -622,8 +634,18 @@ void __unaccount_locked_vm(struct mm_struct *mm, unsigned long pages) { unsigned long locked_vm = mm->locked_vm; + /* + * TODO: Convert book3s vio to use pinned vm to ensure + * unaccounting happens to the correct cgroup. + */ + struct pins_cgroup *pins_cg = get_pins_cg(current); + mmap_assert_write_locked(mm); WARN_ON_ONCE(pages > locked_vm); + if (pins_cg) { + pins_uncharge(pins_cg, pages); + put_pins_cg(pins_cg); + } mm->locked_vm = locked_vm - pages; } EXPORT_SYMBOL_GPL(__unaccount_locked_vm); From patchwork Mon Feb 6 07:47:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alistair Popple X-Patchwork-Id: 53070 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:eb09:0:0:0:0:0 with SMTP id s9csp2106207wrn; Sun, 5 Feb 2023 23:54:23 -0800 (PST) X-Google-Smtp-Source: AK7set8GIw7a2wRX0UhD+41j+bqTrvNt3+x2+nXg4MPsj+ILPTrkEB6bviEqxmBT2YMy+Oy0HQM2 X-Received: by 2002:a17:906:184a:b0:889:ed81:dff7 with SMTP id w10-20020a170906184a00b00889ed81dff7mr18693867eje.9.1675670062957; Sun, 05 Feb 2023 23:54:22 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1675670062; cv=pass; d=google.com; s=arc-20160816; b=t6pOkoa5yKzvGwFn+RV298dzukoaCL0RtQLGAm6eWR01vleQbVcGwhCfpcde4qnbOD axIB57uoVv5rA1anD3DGDMQcKTvu5Wz8lfzgqXxQviQiaajmuhPdysZ48osFMBTMaEVb bKnw0FIy+fh/Fgy8N8feBv6EwlhWU2vVw7+58sP7zDddXm2q7z+oh1HQB6Rcv5rMZx1p J/gDYw2DVp8pQwaawKxetAcQZ+r6+xaw8g7PIjH59+3jGV2k3sqGG6BpbQS0nkNz0E7y 3R6/L7QNAkZY5K9jtQAWkj9KeP8J27F+mH9L5sn6lbhLjQMSLDsJssWaUhc8oQpl5A6J TCkg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:mime-version:content-transfer-encoding :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CGKTTJbiu7jo+yKcPAlQKTQIKt8ia+61r7AgkXzDJs4=; b=B9irwYjBD4TfHTHSuY1zKCzcUzxHcWnQExqcJMJTqb35GjPHzPhWmZC9fSsfozvfYu FV+CmH/5oTs0525CTgbV2zg5p/9enG9R6Gz5fLQ7sHfb31tO5l5clkhR66zUrgES6vUt fOmT2G4WM9CZBORHg7C1jnsckjaak2je5YEB8OIOPBgQdp3MgpV/KD8fCP7Q5kadixsr opYGDzww1ZxYVx5x2bhLGqiPGsPdTJDwxKmr5ujbRuZcF8zpYYCj3/i+5ndLWlngM8kK Z/ZC50pdV9vyEtaTSeXoVxpLVTq1zyZzu9Kxi3B+7IzoduV5wgDixUoy58ZL+dAjWJxd RALA== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=kVvPNwBi; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id 9-20020a170906018900b00888161349e5si15041061ejb.660.2023.02.05.23.53.59; Sun, 05 Feb 2023 23:54:22 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@Nvidia.com header.s=selector2 header.b=kVvPNwBi; arc=pass (i=1 spf=pass spfdomain=nvidia.com dkim=pass dkdomain=nvidia.com dmarc=pass fromdomain=nvidia.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=nvidia.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230080AbjBFHwe (ORCPT + 99 others); Mon, 6 Feb 2023 02:52:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57910 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229954AbjBFHwX (ORCPT ); Mon, 6 Feb 2023 02:52:23 -0500 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (mail-dm6nam11on20603.outbound.protection.outlook.com [IPv6:2a01:111:f400:7eaa::603]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C333D18AA0; Sun, 5 Feb 2023 23:51:50 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Im7MEEVN4FWEx3zCb+jxDxt6oqesrmj8wEvyNhB6xZWYLmNr8YJXfqXWgxp0JvAIIXXdUnrO96MGX7+XrY25hIfkVOGBVsn2Figp5RwnKVNY70RpHmDwh8BhJCEQSoJDTvh9pWSRLJa9mJrdPdI3/timqioSJ2uD8tURUPv571Hfev7ZeKatRw7XEZcFN+Dfl08r4F0Uq+fqLnHaEKJnUOuE4ZGF8WR1sYqFACWL5xks82FH9i91x063H+8KsIGxTQVt0s+3RP9+X2Y+p8t8tPaFR4t/2KDfOMNGR3J+apzlJOvC2Uw8dS2ggut5XjsdygdkpqL6aIKTsa2gpiAu1A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CGKTTJbiu7jo+yKcPAlQKTQIKt8ia+61r7AgkXzDJs4=; b=S3LRHDSF95Tx4J6VA9J6urDze9zWFzHyWsmZiwRPG8JJEaOQOgkPWyZ5TuEI+DHP5JB5ZZfiLSynKg9QXeRz8TyTGWmKjNNGQUdIIiMYxeVZ1rzPnTc6JYwghRZ2SJnqTaP/389fVUpnOhQO1l2UPFGisARyo2Lciofjl+XgzaAJ5k8NSgb/lkKZPNk1RoGyA+h1u8RQklqn3wTZqKzzB5iJ1NFS9ulU19AwzNJZ2p7UQI3DVv48+ewMlfv5bmRLtjhxtXIoQztM8uw2VDRC9ISCSrT8i5KMThjR49aeeUXL6LItBPZ6225fYLdJqNy5OzmqcSmToAQHw65KtyHO5Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CGKTTJbiu7jo+yKcPAlQKTQIKt8ia+61r7AgkXzDJs4=; b=kVvPNwBixSEKdOtpAfVIoKMGsQdXxYk24fgyI253NvLsXcQWCV9659gz3zqc7aYzWX6DrVJfCdezr5YAvVyZt8PlxpG2J5KbszClhFSkw8xAeADMD1gLKJGkafXE9BeTlOAp+Zxs1R6FuJ2t35g4NfxRBbK9CSX9qjPG/A1wkbHG4LJTYm26ACVEBwi/Zx2Co+3SluiTDJTpDHFJv9nk6iX+beV3QuD32Ls4HdQUj33P7Ug/q0oOHp0GypmRt3kaJM57nIfalsiCFYqHdotasEmSHOdlQoYneLf21VI0x26SYY3JMnc58Xfv9QOUq3r3EdpgI4/5wnxUBYQyd69aZw== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DS0PR12MB8573.namprd12.prod.outlook.com (2603:10b6:8:162::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6064.32; Mon, 6 Feb 2023 07:50:20 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::4bd4:de67:b676:67df%6]) with mapi id 15.20.6064.032; Mon, 6 Feb 2023 07:50:20 +0000 From: Alistair Popple To: linux-mm@kvack.org, cgroups@vger.kernel.org Cc: linux-kernel@vger.kernel.org, jgg@nvidia.com, jhubbard@nvidia.com, tjmercier@google.com, hannes@cmpxchg.org, surenb@google.com, mkoutny@suse.com, daniel@ffwll.ch, "Daniel P . Berrange" , Alex Williamson , Alistair Popple , Shuah Khan , linux-kselftest@vger.kernel.org Subject: [PATCH 19/19] selftests/vm: Add pins-cgroup selftest for mlock/mmap Date: Mon, 6 Feb 2023 18:47:56 +1100 Message-Id: <2bd6038b4a1571631e7797ce0f47a133f52acd9c.1675669136.git-series.apopple@nvidia.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: References: X-ClientProxiedBy: SYCPR01CA0045.ausprd01.prod.outlook.com (2603:10c6:10:e::33) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DS0PR12MB8573:EE_ X-MS-Office365-Filtering-Correlation-Id: d11c3ca0-e577-4cea-9a76-08db0816cb58 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 9dWZD2t/+Z7V9QOIUUWaxrSGi0KuhVSIbQ77D2ROgVKVnwxGP52AevErTClN4BqAy21B5fx21j6tBiYlt8u35F60dW8g90c/XKmkVbPhuynbVr6O+4Xghv14ztWgD/29bEkInSoVrxBkn/08YV1MUq9x4RChn3IkP6M+jIQLrUT1scnll48P/ztTwbl0IDBav8icxKqw3/RCPT+PO3kX4NT2mc7s04u7e7KLHTOhOIRxPsusGcYsL6+w1r3dUct98mhm++I1+CKwtb6dFisPWefIHJTICoX53HpZ2KoOnR7lz1J6Kw1jx/cAlLFdzoiygI++gsSkhWTSxui/Jq2ERMn/UjhQYNc3jWXKcgrbJmVfemrw0U84V3963AxNslhkPWNvP3VePQFbPXj+jpugf+QzEUMAZfDn5Uk989t65iJJpKzGdK5zckXmd99UeXwsRYPQXni1J8zzETrAWXnrF4mu2rGw7OHG+AUg4/v0QVWVildJ07/HpedRKRKNeZlmwp2KU+lQBys3n/Bkfu8X/mM/uCdM5GzA5C99yWLxorPDW4JhXFApebCvHSrCvJiQ19m6bFejTrOciu3q1zeRzrRIBNA7BHN+u3vu0JBMwY+JCVbXPHThQ70b+I8B6TaT9LxCml/0t/fhHwHV+NssoA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230025)(4636009)(376002)(396003)(136003)(39860400002)(366004)(346002)(451199018)(54906003)(316002)(86362001)(6666004)(6486002)(38100700002)(186003)(2616005)(6506007)(26005)(478600001)(6512007)(7416002)(5660300002)(83380400001)(2906002)(36756003)(66946007)(8936002)(4326008)(41300700001)(66476007)(8676002)(66556008);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 3j3ry8KiLXLZp8Riv1ALQgPDVgFrUuvNKw26OWlhJphF7+Hc8o5BbQbqnuN/Nw3ZHBrpWrui2J+NXfFlizPM9vjj8Gkk8/nsxRcdCKXUPkmgg0o4NXntKMUjVdiKtZDSUM7DVOfDJuzX++7b42gLeXteGJCy0kG5kM7z3oXOO++mXFnJ6ii5BDy3tLAhDpWmXTz2ZQDLsDK6+xsmS9o4ZklXY9TxRAIn+bIc8tv8SXNFT/ed9QCyK9PMb1t6a9AfxxeybPrD3Uii//c2x5XJBqhrgs64FkXT6WRwJauYz0PaPIe5k/PHr5DRpyyXagjbFWiQQXpIQP7jshIefUMXio4CnV1gNyFVfkuhYqAWD/kcukf/vhibTuhuATZCdiJqPGD/ANt+tvqLMRy68GIE3JmimGZOsZPF/iVWwLQAmhGJbzDAp4hDYsPqIyZs6lxGLj6kB6W3AsdFkJoDOcPmAmQtTYsoKZb5DHKF4AGgM+PYmKrl93CqnaQGuuzFwHVNjiZeC1jmTwhdyTnZf4dY3XtV9l+639GAL/aeIYNjcTjtWDmiPjx/bxvLgQ9WpBc+xwx7dE1C8idBrHUWgr8fBqDY5uFmcPBbvRZRs5HkBCMV7rSDfP6VoEENuNzqOZFoHcTrDBrCySdemrERJHMp14VFyjPf3CoCfDRS9/B0ysJNz9yzSYy/OD0YeXupBzrQ+kuEY/c3alFStwFF6UXLDiugpc2lJ4GotWmF2ANGl1WcCmD6oFbOXZ9Q4O8q6Y95ESbw7YCm5o0w4ymlzSTtu5EitAYdGMIdGgLN725Mkl/D9Is60IXpS0+4GEwnN2tqwD9AzYDUiXaetdkm+Nkd7z/YhkUs+wFMGXJDa27r7b/DPj32HyB8/r8mojOpUQzmZ7onBYXMAC3cqh9KL4DjanR2idx1m3rZMlgP9+Lw/Trucb/WuwIh7SX6gUEZRqd0qnUdwWJjpmt/AF5OkWoFp5F4cW8y4SsQYh5Igw+fN06vC36QFQyRvfkTfE0DABMGB/C+80//St3vH5q9oezb7vFYz2IQ1DIW8wt39149HssNvsH8iiIPh9eU8KQc8eu9ixy+h1mrcjgNIk3mFI143yXkOwNCP98hh9o5R89QmczzB22Z9rRrj3JQ1JIYTD3xVspAB92wOVC1J2mMc7eK+18NzVOq4I8cNdro+7fmvMnSD5ZARXPjWA4QkgIBxepfzCncVnjjXMka/EdSrtidSZSbzBDnjNMejFy3+ATZpC4vr0j8/qxN9oc331BV6BUfbWengLNIjpcIq00lNuNsHqPqBg0xJfrjhOzbAILuXT/pDp+V0+rY4mnIJ1er7y/HBolYTQevQz8MowdOUs6g0GSr9bzaOMnNI8qlmADJFoRv50tYLDcfza44pzces2sluUAsK3NnRId7WaB8emV/AFIgHj1ITn4XornfziLewz5vPTuVaJBEYELm+dHxWkLvCdOt5Tc3gzv6VJg4zcuFKm8xJkagwI2oYfvVuYw2xAuTF3s8AjlPTz3l/PVaWmw1kMtSC1O1S1z3eiQAuzB1e+Xq4C3GcdolcN9UyabSP79S7wsrNNGC09EU0HhObYU9 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: d11c3ca0-e577-4cea-9a76-08db0816cb58 X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Feb 2023 07:50:19.9753 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KClDenTLQ/TczAn7Sbd3dGzzwGDXykfOgI4zkvGApKdmx3a7WPKSSGaevP2sFCVTgUe7bCPuUu4K9rn6fFxZFw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB8573 X-Spam-Status: No, score=-1.5 required=5.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FORGED_SPF_HELO, SPF_HELO_PASS,SPF_NONE autolearn=no autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1757067411889276072?= X-GMAIL-MSGID: =?utf-8?q?1757067411889276072?= Add some basic tests of mlock/mmap cgroup accounting for pinned memory. Signed-off-by: Alistair Popple Cc: Shuah Khan Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org Cc: linux-kselftest@vger.kernel.org Cc: cgroups@vger.kernel.org --- MAINTAINERS | 1 +- tools/testing/selftests/vm/Makefile | 1 +- tools/testing/selftests/vm/pins-cgroup.c | 271 ++++++++++++++++++++++++- 3 files changed, 273 insertions(+) create mode 100644 tools/testing/selftests/vm/pins-cgroup.c diff --git a/MAINTAINERS b/MAINTAINERS index f8526e2..4c4eed9 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5387,6 +5387,7 @@ L: cgroups@vger.kernel.org L: linux-mm@kvack.org S: Maintained F: mm/pins_cgroup.c +F: tools/testing/selftests/vm/pins-cgroup.c CORETEMP HARDWARE MONITORING DRIVER M: Fenghua Yu diff --git a/tools/testing/selftests/vm/Makefile b/tools/testing/selftests/vm/Makefile index 89c14e4..0653720 100644 --- a/tools/testing/selftests/vm/Makefile +++ b/tools/testing/selftests/vm/Makefile @@ -56,6 +56,7 @@ TEST_GEN_PROGS += soft-dirty TEST_GEN_PROGS += split_huge_page_test TEST_GEN_FILES += ksm_tests TEST_GEN_PROGS += ksm_functional_tests +TEST_GEN_FILES += pins-cgroup ifeq ($(MACHINE),x86_64) CAN_BUILD_I386 := $(shell ./../x86/check_cc.sh "$(CC)" ../x86/trivial_32bit_program.c -m32) diff --git a/tools/testing/selftests/vm/pins-cgroup.c b/tools/testing/selftests/vm/pins-cgroup.c new file mode 100644 index 0000000..c2eabc2 --- /dev/null +++ b/tools/testing/selftests/vm/pins-cgroup.c @@ -0,0 +1,271 @@ +// SPDX-License-Identifier: GPL-2.0 +#include "../kselftest_harness.h" + +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define CGROUP_TEMP "/sys/fs/cgroup/pins_XXXXXX" +#define PINS_MAX (-1UL) + +FIXTURE(pins_cg) +{ + char *cg_path; + long page_size; +}; + +static char *cgroup_new(void) +{ + char *cg; + + cg = malloc(sizeof(CGROUP_TEMP)); + strcpy(cg, CGROUP_TEMP); + if (!mkdtemp(cg)) { + perror("Failed to create cgroup"); + return NULL; + } + + return cg; +} + +static int cgroup_add_proc(char *cg, pid_t pid) +{ + char *cg_proc; + FILE *f; + int ret = 0; + + if (asprintf(&cg_proc, "%s/cgroup.procs", cg) < 0) + return -1; + + f = fopen(cg_proc, "w"); + free(cg_proc); + if (!f) + return -1; + + if (fprintf(f, "%ld\n", (long) pid) < 0) + ret = -1; + + fclose(f); + return ret; +} + +static int cgroup_set_limit(char *cg, unsigned long limit) +{ + char *cg_pins_max; + FILE *f; + int ret = 0; + + if (asprintf(&cg_pins_max, "%s/pins.max", cg) < 0) + return -1; + + f = fopen(cg_pins_max, "w"); + free(cg_pins_max); + if (!f) + return -1; + + if (limit != PINS_MAX) { + if (fprintf(f, "%ld\n", limit) < 0) + ret = -1; + } else { + if (fprintf(f, "max\n") < 0) + ret = -1; + } + + fclose(f); + return ret; +} + +FIXTURE_SETUP(pins_cg) +{ + char *cg_subtree_control; + FILE *f; + + if (asprintf(&cg_subtree_control, + "/sys/fs/cgroup/cgroup.subtree_control") < 0) + return; + + f = fopen(cg_subtree_control, "w"); + free(cg_subtree_control); + if (!f) + return; + + fprintf(f, "+pins\n"); + fclose(f); + + self->cg_path = cgroup_new(); + self->page_size = sysconf(_SC_PAGE_SIZE); +} + +FIXTURE_TEARDOWN(pins_cg) +{ + cgroup_add_proc("/sys/fs/cgroup", getpid()); + + rmdir(self->cg_path); + free(self->cg_path); +} + +static long cgroup_pins(char *cg) +{ + long pin_count; + char *cg_pins_current; + FILE *f; + int ret; + + if (asprintf(&cg_pins_current, "%s/pins.current", cg) < 0) + return -1; + + f = fopen(cg_pins_current, "r"); + if (!f) { + printf("Can't open %s\n", cg_pins_current); + getchar(); + free(cg_pins_current); + return -2; + } + + free(cg_pins_current); + + if (fscanf(f, "%ld", &pin_count) == EOF) + ret = -3; + else + ret = pin_count; + + fclose(f); + return ret; +} + +static int set_rlim_memlock(unsigned long size) +{ + struct rlimit rlim_memlock = { + .rlim_cur = size, + .rlim_max = size, + }; + cap_t cap; + cap_value_t capability[1] = { CAP_IPC_LOCK }; + + /* + * Many of the rlimit checks are skipped if a process has + * CAP_IP_LOCK. As this test should be run as root we need to + * explicitly drop it. + */ + cap = cap_get_proc(); + if (!cap) + return -1; + if (cap_set_flag(cap, CAP_EFFECTIVE, 1, capability, CAP_CLEAR)) + return -1; + if (cap_set_proc(cap)) + return -1; + return setrlimit(RLIMIT_MEMLOCK, &rlim_memlock); +} + +TEST_F(pins_cg, basic) +{ + pid_t child_pid; + long page_size = self->page_size; + char *p; + + ASSERT_EQ(cgroup_add_proc(self->cg_path, getpid()), 0); + p = mmap(NULL, 32*page_size, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + ASSERT_NE(p, MAP_FAILED); + + ASSERT_EQ(cgroup_pins(self->cg_path), 0); + memset(p, 0, 16*page_size); + ASSERT_EQ(mlock(p, page_size), 0); + ASSERT_EQ(cgroup_pins(self->cg_path), 1); + ASSERT_EQ(mlock(p + page_size, page_size), 0); + ASSERT_EQ(cgroup_pins(self->cg_path), 2); + ASSERT_EQ(mlock(p, page_size), 0); + ASSERT_EQ(cgroup_pins(self->cg_path), 2); + ASSERT_EQ(mlock(p, 4*page_size), 0); + ASSERT_EQ(cgroup_pins(self->cg_path), 4); + ASSERT_EQ(munlock(p + 2*page_size, 2*page_size), 0); + ASSERT_EQ(cgroup_pins(self->cg_path), 2); + ASSERT_EQ(cgroup_set_limit(self->cg_path, 8), 0); + ASSERT_EQ(mlock(p, 16*page_size), -1); + ASSERT_EQ(errno, ENOMEM); + ASSERT_EQ(cgroup_pins(self->cg_path), 2); + ASSERT_EQ(cgroup_set_limit(self->cg_path, PINS_MAX), 0); + + /* check mremap() a locked region correctly accounts locked pages */ + ASSERT_EQ(mlock(p, 32*page_size), 0); + ASSERT_EQ(cgroup_pins(self->cg_path), 32); + p = mremap(p, 32*page_size, 64*page_size, MREMAP_MAYMOVE); + ASSERT_NE(p, MAP_FAILED); + ASSERT_EQ(cgroup_pins(self->cg_path), 64); + ASSERT_EQ(munmap(p + 32*page_size, 32*page_size), 0) + ASSERT_EQ(cgroup_pins(self->cg_path), 32); + p = mremap(p, 32*page_size, 32*page_size, MREMAP_MAYMOVE | MREMAP_DONTUNMAP); + ASSERT_NE(p, MAP_FAILED); + ASSERT_EQ(cgroup_pins(self->cg_path), 32); + ASSERT_EQ(munlock(p, 32*page_size), 0); + + /* mremap() a locked region should fail if limit exceeded */ + ASSERT_EQ(set_rlim_memlock(32*page_size), 0); + ASSERT_EQ(mlock(p, 32*page_size), 0); + ASSERT_EQ(mremap(p, 32*page_size, 64*page_size, 0), MAP_FAILED); + ASSERT_EQ(munlock(p, 32*page_size), 0); + + /* Exceeds rlimit, expected to fail */ + ASSERT_EQ(set_rlim_memlock(16*page_size), 0); + ASSERT_EQ(mlock(p, 32*page_size), -1); + ASSERT_EQ(errno, ENOMEM); + + /* memory in the child isn't locked so shouldn't increase pin_cg count */ + ASSERT_EQ(mlock(p, 16*page_size), 0); + child_pid = fork(); + if (!child_pid) { + ASSERT_EQ(cgroup_pins(self->cg_path), 16); + ASSERT_EQ(mlock(p, 16*page_size), 0); + ASSERT_EQ(cgroup_pins(self->cg_path), 32); + return; + + } + waitpid(child_pid, NULL, 0); + + /* check that child exit uncharged the pins */ + ASSERT_EQ(cgroup_pins(self->cg_path), 16); +} + +TEST_F(pins_cg, mmap) +{ + char *p; + + ASSERT_EQ(cgroup_add_proc(self->cg_path, getpid()), 0); + p = mmap(NULL, 4*self->page_size, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE | MAP_LOCKED, -1, 0); + ASSERT_NE(p, MAP_FAILED); + ASSERT_EQ(cgroup_pins(self->cg_path), 4); +} + +/* + * Test moving to a different cgroup. + */ +TEST_F(pins_cg, move_cg) +{ + char *p, *new_cg; + + ASSERT_EQ(cgroup_add_proc(self->cg_path, getpid()), 0); + p = mmap(NULL, 16*self->page_size, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + ASSERT_NE(p, MAP_FAILED); + memset(p, 0, 16*self->page_size); + ASSERT_EQ(mlock(p, 16*self->page_size), 0); + ASSERT_EQ(cgroup_pins(self->cg_path), 16); + ASSERT_NE(new_cg = cgroup_new(), NULL); + ASSERT_EQ(cgroup_add_proc(new_cg, getpid()), 0); + ASSERT_EQ(cgroup_pins(new_cg), 16); + ASSERT_EQ(cgroup_add_proc(self->cg_path, getpid()), 0); + rmdir(new_cg); +} +TEST_HARNESS_MAIN