From patchwork Mon May 22 11:08:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 97299 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1366946vqo; Mon, 22 May 2023 04:16:08 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5N51AuAw9XmvsRqfeZaRnednmyefEoekiNuwbXWd17gleK9nCkHxnyytFYewojNXclUZmd X-Received: by 2002:a05:6a20:6a13:b0:101:2ad0:1347 with SMTP id p19-20020a056a206a1300b001012ad01347mr9665499pzk.23.1684754168500; Mon, 22 May 2023 04:16:08 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684754168; cv=none; d=google.com; s=arc-20160816; b=hvblAPlB8SR81OzqqUoTis7i7GroiQ3mZpNbsS/aNq6/R7HE4bFghNpVVh4j49yZBc CiPjIH0fl8gaTiOfAaVp6RqXkMeKtOoynWWmHAq6nEKhbjBdvl6OWyZ06kGbb+jy6HVQ nDt/az74X82+rVoSorT/vJyw28tDDiZI1TRfDVuyQ3CRlJuyiWboIkT89CjfUPP9IRyB USpXtXabFcFBLEpnXmbsU0bzY7gRpR+a4JYhFYpxOQcfzJHQOULuMw82tW9iww91+Cx3 AzYfK5O3cWvdHsudUVnBb0KjAEMn/10IZlCv/OzvUeVnru3hW5C87hN/nqk3amGYDFHU VutA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=C3POfPMkD0T4i+gkKz2qkPU4t7/WZTiyT2qCdNLU280=; b=bqI7oPHjBJzCV9XxwMEUHyudt/bVJF7aQqraLuyDGoB6WGPbFfkBYOOP01dHZfKxiD KCxPkXzyudj+VDQD0X/aFJWTRw5FrdiSgoOhxCuTHPA6YZ/qSehlcx3u4iwNnN901/SC 34mx6rj5S9Ned5se8MefdqmSb1CddbOYvZ6xTWe0yb3bWnYYM81fWFeLOD+0fNQ6Oa+l H7g0mskCIhHh1N3wAKisBoSgs0OpGZE79LJZIvq6hPKlGXStNHZlWv7vYrMrjpohLMnZ OvZ+hIT4eRaGlU8OYHuGnUv1vuiocSTtDH8x02+C2W0ldhCwygOpB1yxPBEKd/Rbsigk plnA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=VBUTLfKb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id n10-20020a17090ade8a00b002532c9b252asi2656528pjv.73.2023.05.22.04.15.53; Mon, 22 May 2023 04:16:08 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=VBUTLfKb; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230470AbjEVLOg (ORCPT + 99 others); Mon, 22 May 2023 07:14:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232524AbjEVLN4 (ORCPT ); Mon, 22 May 2023 07:13:56 -0400 Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA8FA19BF for ; Mon, 22 May 2023 04:08:54 -0700 (PDT) Received: by mail-ej1-x630.google.com with SMTP id a640c23a62f3a-96fffe11714so96048966b.0 for ; Mon, 22 May 2023 04:08:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C3POfPMkD0T4i+gkKz2qkPU4t7/WZTiyT2qCdNLU280=; b=VBUTLfKbApHu6QZhrVrOCB3BjALZb8HbF0H20xKFhc6s2ddiN7suwTGMU0bL+g+sYn RmbQ1cww5mUZenVFKfKM0obeU2G6DDRfd8SBMnH9fmFpOyakPo9ae6ksAd2XakoCBv0E 7Ov6b3cng1MnPemi03HJHLAXu9L7D+fD1mqdF1IW9NYQd6lcGKgDypewZcwaRLFcrhmr yjR9yAKjvtwtrmWZi3nCGabZYIW/EQNSzPSvJbp0e3+m5rd5yon1lw8lX3KGm/ik7TPO pykNacp5zTZge80vp2rYbryAhbquVFUZS46oKag6NjrQQcMLsIE3M8wIwP5IFy1Gt0Ul IcQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C3POfPMkD0T4i+gkKz2qkPU4t7/WZTiyT2qCdNLU280=; b=DWtQWnPxDRa1HKXMy1KSj82cHj8PG0ivfj7kHpBRqlKmAq1c9cwWNx7JuqjB/uyLhd 724tTd3H7bcTKZKiaqW/lsoucF1kp4pp6jRKBGJ49vQ9IWvKvThWQRjPtam1hSyMh5VW KHYlnSUu5y0BnF2MtjX7LRPnLOfrNayGgIgw3g9pBsBv/2GPt5kpS/6Z6HTxKvzbc+Ll kA9gihmshuOG0bMac15MuCSINPSwO0Ulw0XNX0aXjrvgHML0lz3PKclmA/PTkEXX9Hw0 uRQLvshb0HDNwJWdDjCBESsR6legxTOnMkWOYVRdw+ZimhAqrmBfhf/7i3+5G/ZlrDnK 28rQ== X-Gm-Message-State: AC+VfDwcV5lPdrlOBNl+U6j9o7Tbtx+Gp8YlQE3vEnVamPEefOLFT3sL AASTmX4gz81/Qzk/w2IkyPE= X-Received: by 2002:a17:907:1c8b:b0:94e:48ac:9a51 with SMTP id nb11-20020a1709071c8b00b0094e48ac9a51mr11815922ejc.4.1684753732807; Mon, 22 May 2023 04:08:52 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:52 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 1/9] mm: vmalloc: Add va_alloc() helper Date: Mon, 22 May 2023 13:08:41 +0200 Message-Id: <20230522110849.2921-2-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766592786913018301?= X-GMAIL-MSGID: =?utf-8?q?1766592786913018301?= Currently __alloc_vmap_area() function contains an open codded logic that finds and adjusts a VA based on allocation request. Introduce a va_alloc() helper that adjusts found VA only. It will be used later at least in two places. There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 41 ++++++++++++++++++++++++++++------------- 1 file changed, 28 insertions(+), 13 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 31ff782d368b..409285b68a67 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1482,6 +1482,32 @@ adjust_va_to_fit_type(struct rb_root *root, struct list_head *head, return 0; } +static unsigned long +va_alloc(struct vmap_area *va, + struct rb_root *root, struct list_head *head, + unsigned long size, unsigned long align, + unsigned long vstart, unsigned long vend) +{ + unsigned long nva_start_addr; + int ret; + + if (va->va_start > vstart) + nva_start_addr = ALIGN(va->va_start, align); + else + nva_start_addr = ALIGN(vstart, align); + + /* Check the "vend" restriction. */ + if (nva_start_addr + size > vend) + return vend; + + /* Update the free vmap_area. */ + ret = adjust_va_to_fit_type(root, head, va, nva_start_addr, size); + if (WARN_ON_ONCE(ret)) + return vend; + + return nva_start_addr; +} + /* * Returns a start address of the newly allocated area, if success. * Otherwise a vend is returned that indicates failure. @@ -1494,7 +1520,6 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head, bool adjust_search_size = true; unsigned long nva_start_addr; struct vmap_area *va; - int ret; /* * Do not adjust when: @@ -1512,18 +1537,8 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head, if (unlikely(!va)) return vend; - if (va->va_start > vstart) - nva_start_addr = ALIGN(va->va_start, align); - else - nva_start_addr = ALIGN(vstart, align); - - /* Check the "vend" restriction. */ - if (nva_start_addr + size > vend) - return vend; - - /* Update the free vmap_area. */ - ret = adjust_va_to_fit_type(root, head, va, nva_start_addr, size); - if (WARN_ON_ONCE(ret)) + nva_start_addr = va_alloc(va, root, head, size, align, vstart, vend); + if (nva_start_addr == vend) return vend; #if DEBUG_AUGMENT_LOWEST_MATCH_CHECK From patchwork Mon May 22 11:08:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 97298 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1366923vqo; Mon, 22 May 2023 04:16:06 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4zdB8xYaRbEHvmhdtGKJ1TIgPp8BB8jEr9MwWji5jZ+vKTCk7Rvr8ElCYDJj8k7OlRrLQ7 X-Received: by 2002:a05:6a20:8e15:b0:103:e05f:c6d2 with SMTP id y21-20020a056a208e1500b00103e05fc6d2mr11569399pzj.1.1684754166263; Mon, 22 May 2023 04:16:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684754166; cv=none; d=google.com; s=arc-20160816; b=F5iS2edsbChanTzhGkPikPRHFHTCJ5fP9qCt7d8KkY5i+sX/RbS3o2ulz6jQXS/74c 7Ou77eeKrcGPr5rTHShwTW2tFg4tPqWLv9/Z26ZSpEMGDrU4c5xuLFKp0JFd0LWP1xLU bLOEMZqxAbjoFRCbLcaBMnRXtWMdMb/lpGX7aYiG4wnztgc1sWYcUxoYw09hzqH1mWKq peJMvGGu4M2XBewqv0UYSTsH7+D7nGGfRbMhopXJmE/ANBTP6+6pTxBfgfXKXEqTRxTG hqRSDPa+ZZEC1faToo6KQYpu2fNDkKL3RD6enJ7j9MP9rpDpLfHvZIUyK9EVfY/F5jXN +efA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AqE7n0km0cT3SoKgXn9+9ffSGk68u7ZexKpVeyjeuP0=; b=qLv8+H/YOM4fYAesmEKXIf9zEkyyrGfRBAlSrgBpK0BhicjZ2vOKFbVwiUkFvS5m3I N8GRgmqfdX7waxUQ9uE9dId5QHwmShleggr29mfWCygPhu7OFxyRT/qGE9ywD4SQzbZY yvAxgDgleNrnKaRO59bDl4if5lSqY9SIcrfY3TMZ1j9yBilUtEkU2ZFx8w3wWtHIfx+T wwHxAQX2F4mwuJQLSVRC9TTxRvf/wVymenkC5TWNYE/FJQLAgtN4dIAG11glTjDySQg5 vWinyDOpW5nO4C3+UISWxt+S7zDZhHoZ95dl5aA6Q9A2IcKFMtKAQtAZ5pkQ7OgZwPyh uikQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=UI0Yt0lW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id g10-20020a636b0a000000b005348954e079si4650414pgc.95.2023.05.22.04.15.53; Mon, 22 May 2023 04:16:06 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=UI0Yt0lW; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233221AbjEVLOj (ORCPT + 99 others); Mon, 22 May 2023 07:14:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58266 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232442AbjEVLN4 (ORCPT ); Mon, 22 May 2023 07:13:56 -0400 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49A3340E8 for ; Mon, 22 May 2023 04:08:55 -0700 (PDT) Received: by mail-ej1-x632.google.com with SMTP id a640c23a62f3a-96f7377c86aso590975466b.1 for ; Mon, 22 May 2023 04:08:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AqE7n0km0cT3SoKgXn9+9ffSGk68u7ZexKpVeyjeuP0=; b=UI0Yt0lWDXh9ljMMkPAajAGV57h0505ib1ttO6nC+S6/ox2EoI9VGZd7+i0KyUT3Iz UrOeQv6HeCqlYMLmrO9xLceNit+eAAhp+ayorSX9GWMqWetCfy65fNLgsBYIBMUhhJ2L U75zeNwLejw8wEtPfmFV0nYxaTwgfXdcMVWerdFeHWF2BfssxPxEikaABdcefX8aVmFB OzcafrJAiZhR3U1i8vQurAWGDU7VgmaPUGDQ2nyXctKGB+RLSHtg6lFIAG/yBViPwShe HnTgtWcZBRbZXXW3R63z1IHpNcsR0E7wzcXRYrATiECZ/KAKp4PAjxmjbmz2ChABim6V aqqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753733; x=1687345733; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AqE7n0km0cT3SoKgXn9+9ffSGk68u7ZexKpVeyjeuP0=; b=FmKUQSv/MjcYtCNoiHTuepGu3R8lTdL7NCEDNARzsccux8cUpsUKUO0gGbw2xAM2VG EnJskQzntNvpghe1aFBIUnPymWi9pM44dTNPg3qRJqMxPLrOdIuUm+oPaa++Rfrj7DK2 6Bsqw9C6dX4tOdt7oFJfL+pZZKgzsYdKtL7m+fBh4JjHu94i/lDyiLOGrW/Uc3bmXgaD sLp1bUga3EHITkbwgtfiiPLvtmBbPbIL00dYMj3yhAgQM/0sXLzItYxt70GHzADwKZSk 95JLoi8AO3NYYMt66t3oxRO/lnruNWNvv36e9uxfs0Z6J4sDnEhw37qFs7RMvbAGQC/o cHtQ== X-Gm-Message-State: AC+VfDz/REzDP87GihVnDMULruGiFuQLcEbc+obyOMGVJhnMQHMlsXD/ C+dYwoiSwCahUgCCTEebcbc= X-Received: by 2002:a17:907:2d23:b0:96a:928c:d382 with SMTP id gs35-20020a1709072d2300b0096a928cd382mr10346599ejc.48.1684753733572; Mon, 22 May 2023 04:08:53 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:53 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 2/9] mm: vmalloc: Rename adjust_va_to_fit_type() function Date: Mon, 22 May 2023 13:08:42 +0200 Message-Id: <20230522110849.2921-3-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766592784146216333?= X-GMAIL-MSGID: =?utf-8?q?1766592784146216333?= This patch renames the adjust_va_to_fit_type() function to a shorter variant and more expressive. A new name is va_clip(). There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 409285b68a67..5f900efec6a9 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1383,9 +1383,9 @@ classify_va_fit_type(struct vmap_area *va, } static __always_inline int -adjust_va_to_fit_type(struct rb_root *root, struct list_head *head, - struct vmap_area *va, unsigned long nva_start_addr, - unsigned long size) +va_clip(struct rb_root *root, struct list_head *head, + struct vmap_area *va, unsigned long nva_start_addr, + unsigned long size) { struct vmap_area *lva = NULL; enum fit_type type = classify_va_fit_type(va, nva_start_addr, size); @@ -1501,7 +1501,7 @@ va_alloc(struct vmap_area *va, return vend; /* Update the free vmap_area. */ - ret = adjust_va_to_fit_type(root, head, va, nva_start_addr, size); + ret = va_clip(root, head, va, nva_start_addr, size); if (WARN_ON_ONCE(ret)) return vend; @@ -3979,9 +3979,8 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, /* It is a BUG(), but trigger recovery instead. */ goto recovery; - ret = adjust_va_to_fit_type(&free_vmap_area_root, - &free_vmap_area_list, - va, start, size); + ret = va_clip(&free_vmap_area_root, + &free_vmap_area_list, va, start, size); if (WARN_ON_ONCE(unlikely(ret))) /* It is a BUG(), but trigger recovery instead. */ goto recovery; From patchwork Mon May 22 11:08:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 97300 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1366998vqo; Mon, 22 May 2023 04:16:13 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7BACOrZiq020Y5JtkwrIULaOioPxWpce08/qO8d/GuOY4D188zsToI3NuAuCsy8vNRQsc6 X-Received: by 2002:a05:6a21:7888:b0:10b:f331:d1c0 with SMTP id bf8-20020a056a21788800b0010bf331d1c0mr1383570pzc.30.1684754173386; Mon, 22 May 2023 04:16:13 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684754173; cv=none; d=google.com; s=arc-20160816; b=XIT6XCYkPLDXw8FHrdzDI6aK4Yevk35ti50sAoDrv5jxwrbqfi0aIY3JKVmPoiVGVU htQdDaemXKB1/kBqUnpogv1/Kl+l9a2lfkTQLUye4m+eCUZz6u44kem5AlDvwsF6buU6 o4vE9GtZnhgroLhoIHpFtbrTbrqHFQPS0DpNDW4vtZquIGfRYSJ6Qzf+MIAl48SzeyGW 4mIjLY/GzNzGKGpnDpAnHxC80C6VaVBvmNzFEkXMUk19B9l6uCwmXxBIHMR2MoZofxeH zIFIHJQZHl7WWkGd5SiSw9b4OAty24VHAWm4ZI7z3LB90L7F3oEgyu2N91SLBTtixABH 7oxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=sNtfrgi/quI4TC521AyXgwFWeKCoK/ixdzMUo1Hjyag=; b=cdVF5+wiGwSBV6OFXfYH9ayEF4h42yXU0rS4Sr9JJoSuHtXxkpmAPVhwqlMOItG3JG 81ZkQleuP90lEz/yDvmfAJdpSR3RSmTgPGOJ4j5XW2GmCXvmMS08da/drhnGAD4KuWg4 zkmEDq1ua20hnPIjJzOKdbDWR45lHiAlsrp0g3GfeHKYmP5JVuLXi+XjO6PE+oljT40w OPYWriphvd3WK/FdDTuyNMYy0qh+rNN27n30LvVcn4ykRAVnn5A/4GqvoAvRHOjBmBK3 qefQ2pitYej+EhDS/vcSSsrXVQFHQ8pyvCA9f3z+nD3AhBzofoje3GD01rSBa3Zilscb 1jeg== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b="GkR4/dm/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id e18-20020a17090a9a9200b0025072690087si4601373pjp.7.2023.05.22.04.15.59; Mon, 22 May 2023 04:16:13 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b="GkR4/dm/"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233226AbjEVLOm (ORCPT + 99 others); Mon, 22 May 2023 07:14:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232979AbjEVLN5 (ORCPT ); Mon, 22 May 2023 07:13:57 -0400 Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2186F40ED for ; Mon, 22 May 2023 04:08:56 -0700 (PDT) Received: by mail-ej1-x636.google.com with SMTP id a640c23a62f3a-96f683e8855so488847866b.2 for ; Mon, 22 May 2023 04:08:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753734; x=1687345734; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sNtfrgi/quI4TC521AyXgwFWeKCoK/ixdzMUo1Hjyag=; b=GkR4/dm/njRpBQa+KVraZ+7JOlNk7shw4BEphYd/T4MuYuRj9UeHx7STRIkE9qZ3ZV 2oXVDGHFmqIOsQbp8tUn9Y00B46Baxi7Y8dneLOn6DV42d85pL1sEsY68+QnjQ1f2vmJ aDPoq7XcD9rLqsdzZWq3ANly5HjVSjJ29W0p5d8YhCo15pKjujxSAlIMECHBsZDAyuMN tLgaKTuxmHTfb/eGGN0rAdS+/lKXxYAVJyynl/B47Xw/ZEaCyR2lt1lpmnS2FpLh3BXG MgpEFyKwNZW3Q2xr0xt2mb+dk5JSHw7+52EgYFgJevxmwbWAu2YUPoOjuE6sjvnpUmYm ZEog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753734; x=1687345734; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sNtfrgi/quI4TC521AyXgwFWeKCoK/ixdzMUo1Hjyag=; b=FTZ3yfk6jS6vKVXbW+/KmL42W+/A/0LU6hHyovEVYmhf1EB2QBqDn19pmtG8/mQ4Ei hWnhTRwkh55HHMi+wxdu1FsFpjA2IMpVzKW4uEMJpuTBOfdz0qoMzFsnbA5tS90o7Mdm xS2KE+rY+MPoRCrl8f+74hO79VwqCNVdMSfze9cbbqkR2dW8h6Egn78MKFoidwQxY6fB ld/s2KdjEBesyC13rGxOYctu4CZrQVwStEu3Rrcubc3z/MAgkWNUf95rOdTWRelFOQyK mJWrQtwbRt+gzwre07P2ubdUVjVSQ1vlPEot5vhEub/qi/0oJB2am4yKleZXnXwoNjmi xztQ== X-Gm-Message-State: AC+VfDwI+MLuZ2nmwbQF63GjwJOKw9BOOGtMsx7ULvMErwYslkD6CjuK FjXKwbA2Ue0DCk3DX1KRldA= X-Received: by 2002:a17:907:d21:b0:965:4b43:11f1 with SMTP id gn33-20020a1709070d2100b009654b4311f1mr9711913ejc.3.1684753734338; Mon, 22 May 2023 04:08:54 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:54 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 3/9] mm: vmalloc: Move vmap_init_free_space() down in vmalloc.c Date: Mon, 22 May 2023 13:08:43 +0200 Message-Id: <20230522110849.2921-4-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766592792163795471?= X-GMAIL-MSGID: =?utf-8?q?1766592792163795471?= A vmap_init_free_space() is a function that setups a vmap space and is considered as part of initialization phase. Since a main entry which is vmalloc_init(), has been moved down in vmalloc.c it makes sense to follow the pattern. There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) Reviewed-by: Christoph Hellwig Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 82 ++++++++++++++++++++++++++-------------------------- 1 file changed, 41 insertions(+), 41 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 5f900efec6a9..19dcffb0d563 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2416,47 +2416,6 @@ void __init vm_area_register_early(struct vm_struct *vm, size_t align) kasan_populate_early_vm_area_shadow(vm->addr, vm->size); } -static void vmap_init_free_space(void) -{ - unsigned long vmap_start = 1; - const unsigned long vmap_end = ULONG_MAX; - struct vmap_area *busy, *free; - - /* - * B F B B B F - * -|-----|.....|-----|-----|-----|.....|- - * | The KVA space | - * |<--------------------------------->| - */ - list_for_each_entry(busy, &vmap_area_list, list) { - if (busy->va_start - vmap_start > 0) { - free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); - if (!WARN_ON_ONCE(!free)) { - free->va_start = vmap_start; - free->va_end = busy->va_start; - - insert_vmap_area_augment(free, NULL, - &free_vmap_area_root, - &free_vmap_area_list); - } - } - - vmap_start = busy->va_end; - } - - if (vmap_end - vmap_start > 0) { - free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); - if (!WARN_ON_ONCE(!free)) { - free->va_start = vmap_start; - free->va_end = vmap_end; - - insert_vmap_area_augment(free, NULL, - &free_vmap_area_root, - &free_vmap_area_list); - } - } -} - static inline void setup_vmalloc_vm_locked(struct vm_struct *vm, struct vmap_area *va, unsigned long flags, const void *caller) { @@ -4271,6 +4230,47 @@ module_init(proc_vmalloc_init); #endif +static void vmap_init_free_space(void) +{ + unsigned long vmap_start = 1; + const unsigned long vmap_end = ULONG_MAX; + struct vmap_area *busy, *free; + + /* + * B F B B B F + * -|-----|.....|-----|-----|-----|.....|- + * | The KVA space | + * |<--------------------------------->| + */ + list_for_each_entry(busy, &vmap_area_list, list) { + if (busy->va_start - vmap_start > 0) { + free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); + if (!WARN_ON_ONCE(!free)) { + free->va_start = vmap_start; + free->va_end = busy->va_start; + + insert_vmap_area_augment(free, NULL, + &free_vmap_area_root, + &free_vmap_area_list); + } + } + + vmap_start = busy->va_end; + } + + if (vmap_end - vmap_start > 0) { + free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); + if (!WARN_ON_ONCE(!free)) { + free->va_start = vmap_start; + free->va_end = vmap_end; + + insert_vmap_area_augment(free, NULL, + &free_vmap_area_root, + &free_vmap_area_list); + } + } +} + void __init vmalloc_init(void) { struct vmap_area *va; From patchwork Mon May 22 11:08:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 97312 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1374292vqo; Mon, 22 May 2023 04:30:09 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5TuX42iYngNrkLVLzKrOrmbe0rQIXupmvg2xrNzxYnROhNW0yvVYyme6cO3tfsMP1wbXx+ X-Received: by 2002:a05:6a21:6811:b0:e9:5b0a:deff with SMTP id wr17-20020a056a21681100b000e95b0adeffmr9536550pzb.22.1684755009549; Mon, 22 May 2023 04:30:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684755009; cv=none; d=google.com; s=arc-20160816; b=G52X3U2c/HmU8P2Ti7G9PbwSXAo7s/e9zp6hKqVN0nZXGuIGsuVwumPbw8TInJ4s/B u14RZ4yvq6xPyivjphceniZsTwfX86PtaeIqEIvbujWVEYI+OHKfk/JvoIRtbsD7Nl0H ZyK9xtKAXT+VNKmmpWR97SoDNkdlrIj4aoxMpHyLTFLSTDWiDGFzSyuWhZaHVeN6niNi dx1Jnyu5NDStrmTUgVlhoCw8rST7kryFg3XXzNVo12eB5fNInp0z/hQkhGW7Ez7losy+ 0mT5zmmriNVreZL9YyrO7hMToIvQrbCdZAz5kBYqEKIvbMqI+6FDNI28yo/OvF6cl9m8 0Yuw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=mTQo+zUHAx5YpHIVDwfvmELLOewBkYXc4+E9Yxhyo7AvWwTrfA8Ie9ULPQrhnNPpZ8 ywDoTZUMl7vBDZWHwJY1FBeiXHZ/fUTve16ZhodZGI2IlD2Kwbj2JS8piWduAEKKrnW8 lXwiFBKypOSH7k/seuFQSP176Y0PP4FZtsRP1L8BAVJmYpOwcdsXZBe3JlBH8NcGJXx8 tf8aNPTVJ7Fxp6I8Wu2L2QnobBlA5BFlQ44ORD1TLue/zLeNz/hLTJ+UtqhhzzIeZENf rsCkDo31B1jtn0nZlGAsBZkmSm+HXvtSdigqSKzd1biYIvS50bPYZ+TYS2MzBKgAWtnl 8caw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=iDb9tK4T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id i62-20020a638741000000b0053b8f80c65esi2059575pge.730.2023.05.22.04.29.56; Mon, 22 May 2023 04:30:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=iDb9tK4T; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232800AbjEVLOp (ORCPT + 99 others); Mon, 22 May 2023 07:14:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58068 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233096AbjEVLN7 (ORCPT ); Mon, 22 May 2023 07:13:59 -0400 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C8DE1BCF for ; Mon, 22 May 2023 04:08:57 -0700 (PDT) Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-970028cfb6cso94840866b.1 for ; Mon, 22 May 2023 04:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753735; x=1687345735; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=iDb9tK4TOgaJNMq61xrRvzxoVUaPP0VvNEnz6NrY+zacf0FfJnwnuypFPmwH5SLNBS GyEOPDWFsssngIu4YWW0FrNovP9ZjqfzkUf/51MOpeMHMX80L0GnQUdqQZEtrUUDgAN5 CqpASsjwkoQ7U63fy74gThmTbQvI9Ie3XaHwC17zsSXEk9lHt40AkqJDIcGVhr55gFza Ab5qRCRkiZfB7QM1qkbc7/2z6/RGesk6zCBFvjSfwFLXPP4gxqH+BsOk1j0UanBK1aWA e+0Ct1vWALCAGVP46sp1nfM2tzxvrxxrGXz0I5hY5eIN0kmUd5nEpcG0c3MTS3rmdd8s WEcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753735; x=1687345735; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WvsDwavnfIYng0FXX5ciX4/rx3a5CWQd4rOb3eMOJOg=; b=EkECDYmO1cmq7RrmSSdvaPq3iu2YuN+IjQuaqMIMMVcYC+lSqik1QlC5sPR38AcuO6 DQrzdJd4a54m3R0mgdtdVUPfn+jNCGU9NycSiqrvRhOBYLNx7qlkQx+9VXvGpCIx4B98 ty0jwlBP/GlTafI5uWnwUnccsiRH/UEifmAGfFoRjQ2ber3v67U4UZH3o5cU/O1LKF5l paS7n7986s07d/iY+z+BZvnz3DhvtsxJFK4nZMtKOh8QuFxKX2EXeh7i+/tWkQfBOtyO 9k6YWlDDaw8Raycf4+6tpptpcCsTaQa3q4Daxrax/BlNrr6i4BDvd7n5QzNgNb/IxubQ opzA== X-Gm-Message-State: AC+VfDzSUYRIkfJHKrTOwqVlv9l8HE/NZIF+e41P6gNgUnwYhknaeaew Bp1wP6ZvVIm6Xf2jQ1uJMX8= X-Received: by 2002:a17:907:2d0e:b0:96f:cde5:5f5e with SMTP id gs14-20020a1709072d0e00b0096fcde55f5emr3930140ejc.29.1684753735155; Mon, 22 May 2023 04:08:55 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:54 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 4/9] mm: vmalloc: Add a per-CPU-zone infrastructure Date: Mon, 22 May 2023 13:08:44 +0200 Message-Id: <20230522110849.2921-5-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766593668537826175?= X-GMAIL-MSGID: =?utf-8?q?1766593668537826175?= Introduce a basic skeleton that defines a cache entity for a CPU. It consist of bookkeeping data which belong to a CPU zone. A CPU-zone defines a virtual address range with its fixed size. This approach gives a possibility to map any address to a particular CPU-zone it belongs to. Below is a high-level description how it is used in alloc and free paths: alloc-path: in-front-pre-fetch occurs if a cache is empty. A chunk is placed to a free-tree of this CPU. Finally this CPU cache is clipped in order to accomplish a request. A fresh VA resides in a busy-tree of specific CPU-zone it is bound to. free-path: va->va_start is converted to a zone, so a lookup occurs in a bookkeeping zone it belongs to. Once it gets removed a VA is moved to a lazy-tree. It is freed after a TLB flushing. A "cvz_size", that reflects a zone size, is set to ULONG_MAX by this patch. The reason is we would like to follow an old behavior until all dependent changes are in place. There is no a functional change as a result of this patch. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 19dcffb0d563..f6da2590b4de 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -773,6 +773,61 @@ static struct rb_root free_vmap_area_root = RB_ROOT; */ static DEFINE_PER_CPU(struct vmap_area *, ne_fit_preload_node); +/* + * A per-cpu effective vmap cache logic. It allows to pre-fetch + * a memory vmap chunk with a specified size. A bookkeeping data + * is stored and accessed per-CPU. + */ +enum { FREE = 0, BUSY, LAZY, NFBL }; + +#define fbl(z, i, m) z->fbl[i].m +#define fbl_root(z, i) fbl(z, i, root) +#define fbl_head(z, i) fbl(z, i, head) + +#define fbl_lock(z, i) spin_lock(&fbl(z, i, lock)) +#define fbl_unlock(z, i) spin_unlock(&fbl(z, i, lock)) + +struct cpu_vmap_zone { + /* + * FREE, BUSY, LAZY bookkeeping data of this CPU zone. + */ + struct { + struct rb_root root; + struct list_head head; + spinlock_t lock; + } fbl[NFBL]; + + /* + * A list of outstanding lazily-freed areas, ready to + * be drained, which belong(address space) to this CPU + * zone. + */ + struct list_head purge_list; + + /* + * It controls that only one CPU can pre-fetch this + * CPU zone. + */ + atomic_t fill_in_progress; +}; + +static DEFINE_PER_CPU(struct cpu_vmap_zone, cpu_vmap_zone); + +/* Disable a per-cpu caching. */ +static __read_mostly unsigned long cvz_size = ULONG_MAX; + +static inline unsigned int +addr_to_cpu(unsigned long addr) +{ + return (addr / cvz_size) % num_possible_cpus(); +} + +static inline struct cpu_vmap_zone * +__maybe_unused addr_to_cvz(unsigned long addr) +{ + return &per_cpu(cpu_vmap_zone, addr_to_cpu(addr)); +} + static __always_inline unsigned long va_size(struct vmap_area *va) { From patchwork Mon May 22 11:08:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 97303 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1371508vqo; Mon, 22 May 2023 04:24:39 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ44JIi+60LQOqZdUNPTw6/BNV4uBNQMBxleLIQ99w1NrvjEl3pqdlTU50fxpXfrw7xh2gmD X-Received: by 2002:a05:6a20:8421:b0:10b:5305:6a11 with SMTP id c33-20020a056a20842100b0010b53056a11mr4942158pzd.52.1684754679030; Mon, 22 May 2023 04:24:39 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684754679; cv=none; d=google.com; s=arc-20160816; b=leoc/Ej8Pt92J1+S5Hr5S7TuenahD1rlgYK7p27SymatL0V8JNV1UDBzM0/mu5m6XL Y+6GbB1ec0fCCKqrJBHscV4OYXSEx7fVym3RYGUKOjHNjhd32+GcL2O5ODS9K7z2USqZ 5P5LVvkHz/qXT/bKx1noYrSv0bSP6v+smfpfOIrhY2q4AhvD746kt3g+2ha7zPY9e8xs hf0hrKitNmw5Zr17YE0/ZsGvlrk+4s/M64K1aITzRjyBibVUjYimx7jNk25vYerViPLM wSA2FNu7wkYDQHqPh5RlCi1TKuXGcxRkqOaVhiNiiJyTSlCcbK8zyeADtAgOBdl8ILm6 R6FA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=DrvLnO25dwGFFlUS7GxG9ba70dEBhOOiZuR4m0pcTUM=; b=GJOyTsbDdn6GmdEw9KGn4g0fvKH49IPIl1qoMHWy1y0ND7hVV20r+uI4tbzWm4zJz9 sSXbtB50evI+XftPXPvAs2V+94i7lKPt6oVQy5H2ENBAdE6Zb2yH8tLzHGqjETcMR0Bg qFWIJA1IiXe/SAJkhtC/w3N/mOOpzAIBWdODO8soipvvvGvslPi46UgSuinh3XtJx+gT KZtiQJ0XJ/OhKZsj9ugYT4yJWd2F6CULgYIaBwp4i55vKZmCXdIskre9xWF5/dUCmVTw YqsN/Sn40TGXWxdq3hYEplWCLxhX7fA/c7lTCqA81TdzlJzIOvKQaMsJZvBtDaAbHBor 78vA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=flXewQ2y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s4-20020a639244000000b0052c73367c13si4437955pgn.871.2023.05.22.04.24.25; Mon, 22 May 2023 04:24:39 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=flXewQ2y; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233096AbjEVLOt (ORCPT + 99 others); Mon, 22 May 2023 07:14:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58070 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233117AbjEVLOA (ORCPT ); Mon, 22 May 2023 07:14:00 -0400 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8DEC1BD4 for ; Mon, 22 May 2023 04:08:57 -0700 (PDT) Received: by mail-ej1-x62d.google.com with SMTP id a640c23a62f3a-96fab30d1e1so374667766b.0 for ; Mon, 22 May 2023 04:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753736; x=1687345736; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DrvLnO25dwGFFlUS7GxG9ba70dEBhOOiZuR4m0pcTUM=; b=flXewQ2yArAVYlv2skHOfRQSd1S7Cqlwt9MNZZJrqFsSrWWwyouqmQjJ95oq8wCC6C ad1/P8xGdXwlSwJMGJC8rDQai6cooJJwK0WkPCEKTPeV1R4JhMhzlsAxID+3OAx7hyz8 qGIYAx4Fe+Vqa5htZtPnUE8LV6kn4rgy+MrpLn2fFSbuz9hRvORwAIU7W24PqNv/v/A+ Vvb0nbx4byHbsHsQfe2U6VreF5LdyXSZ3hV+iYjamkAgVT/lQoTYVkBdU0CzbUpSU/0c 71/85NUEE602yGx29yMZLdgzUhf6phivNIaZUU/ts0HMXqTu7rh81+XTKlpL8mliOzmO yxdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753736; x=1687345736; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DrvLnO25dwGFFlUS7GxG9ba70dEBhOOiZuR4m0pcTUM=; b=h2Kwnm8vDDqbKunIIATkwt0b+UcxSqEo01h6/45STBs0711TjZwFQEnrawbLF9ANQQ k954fTwA1a0MEBa/y8YLZv0hjgOwbB5ZlQYKKhWh5uk/F7bSuPGMOa99TcSo+9XBjS9I q/amZY1cYcfmpEWQqQqgCAmPkh6WuGXNVyC3MK15e/07qo7Q+lfCKFC3NcQBUlLdzVSL FTlGpe+51SFVH1YOgP0P+uip2uO8XnyoOAgjduYDkj5kqnGvxXkhOwCvJZZa7N8a9tbo uPPxZT3df8jMbXCZ5zJvK2CBgERYqRVNcNnuCY0PDSLcrFmqyX0dw6ZVc5C5ogX6tLHM YXMg== X-Gm-Message-State: AC+VfDzHgwMa8o5V+1pRfW3uKWEUpKPGwIPtvnBnK/7t80kX9aV7tJUZ Vrx1HNQxRVpEw/Ngbl0SVpwsYZ2RnWm2Ew== X-Received: by 2002:a17:907:961e:b0:96f:c545:6fd with SMTP id gb30-20020a170907961e00b0096fc54506fdmr4438428ejc.10.1684753736024; Mon, 22 May 2023 04:08:56 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:55 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 5/9] mm: vmalloc: Insert busy-VA per-cpu zone Date: Mon, 22 May 2023 13:08:45 +0200 Message-Id: <20230522110849.2921-6-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766593322480663073?= X-GMAIL-MSGID: =?utf-8?q?1766593322480663073?= Store busy-VA objects per a CPU zone. A va->va_start address is converted into a correct zone where it is placed and resides. An addr_to_cvz() function is used to do a proper address conversion. Such approach balances VAs across CPUs. That is why an access becomes scalable to number of CPUs in a system. Please note: Since a zone size is set to ULONG_MAX, i.e. everything is bound thus accessed to the CPU_0 so far, this patch does not give any difference comparing with a current behavior. The global vmap_area_lock, vmap_area_root are removed as there is no need in it anymore. The vmap_area_list is still kept and is _empty_. It is exported for a kexec only. The vmallocinfo and vread() have to be reworked to be able to handle multiple zones. As a result of this patch it can handle only one zone, i.e. when cache is disabled. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 ++++++++++++++++++++++++++++++++++----------------- 1 file changed, 84 insertions(+), 43 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index f6da2590b4de..a9170fe19909 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -729,11 +729,9 @@ EXPORT_SYMBOL(vmalloc_to_pfn); #define DEBUG_AUGMENT_LOWEST_MATCH_CHECK 0 -static DEFINE_SPINLOCK(vmap_area_lock); static DEFINE_SPINLOCK(free_vmap_area_lock); /* Export for kexec only */ LIST_HEAD(vmap_area_list); -static struct rb_root vmap_area_root = RB_ROOT; static bool vmap_initialized __read_mostly; static struct rb_root purge_vmap_area_root = RB_ROOT; @@ -823,7 +821,7 @@ addr_to_cpu(unsigned long addr) } static inline struct cpu_vmap_zone * -__maybe_unused addr_to_cvz(unsigned long addr) +addr_to_cvz(unsigned long addr) { return &per_cpu(cpu_vmap_zone, addr_to_cpu(addr)); } @@ -859,10 +857,10 @@ unsigned long vmalloc_nr_pages(void) } /* Look up the first VA which satisfies addr < va_end, NULL if none. */ -static struct vmap_area *find_vmap_area_exceed_addr(unsigned long addr) +static struct vmap_area *find_vmap_area_exceed_addr(unsigned long addr, struct rb_root *root) { struct vmap_area *va = NULL; - struct rb_node *n = vmap_area_root.rb_node; + struct rb_node *n = root->rb_node; addr = (unsigned long)kasan_reset_tag((void *)addr); @@ -1608,12 +1606,14 @@ __alloc_vmap_area(struct rb_root *root, struct list_head *head, */ static void free_vmap_area(struct vmap_area *va) { + struct cpu_vmap_zone *z = addr_to_cvz(va->va_start); + /* * Remove from the busy tree/list. */ - spin_lock(&vmap_area_lock); - unlink_va(va, &vmap_area_root); - spin_unlock(&vmap_area_lock); + fbl_lock(z, BUSY); + unlink_va(va, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); /* * Insert/Merge it back to the free tree/list. @@ -1656,6 +1656,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, int node, gfp_t gfp_mask, unsigned long va_flags) { + struct cpu_vmap_zone *z; struct vmap_area *va; unsigned long freed; unsigned long addr; @@ -1701,9 +1702,11 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, va->vm = NULL; va->flags = va_flags; - spin_lock(&vmap_area_lock); - insert_vmap_area(va, &vmap_area_root, &vmap_area_list); - spin_unlock(&vmap_area_lock); + z = addr_to_cvz(va->va_start); + + fbl_lock(z, BUSY); + insert_vmap_area(va, &fbl_root(z, BUSY), &fbl_head(z, BUSY)); + fbl_unlock(z, BUSY); BUG_ON(!IS_ALIGNED(va->va_start, align)); BUG_ON(va->va_start < vstart); @@ -1926,24 +1929,26 @@ static void free_unmap_vmap_area(struct vmap_area *va) struct vmap_area *find_vmap_area(unsigned long addr) { + struct cpu_vmap_zone *z = addr_to_cvz(addr); struct vmap_area *va; - spin_lock(&vmap_area_lock); - va = __find_vmap_area(addr, &vmap_area_root); - spin_unlock(&vmap_area_lock); + fbl_lock(z, BUSY); + va = __find_vmap_area(addr, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); return va; } static struct vmap_area *find_unlink_vmap_area(unsigned long addr) { + struct cpu_vmap_zone *z = addr_to_cvz(addr); struct vmap_area *va; - spin_lock(&vmap_area_lock); - va = __find_vmap_area(addr, &vmap_area_root); + fbl_lock(z, BUSY); + va = __find_vmap_area(addr, &fbl_root(z, BUSY)); if (va) - unlink_va(va, &vmap_area_root); - spin_unlock(&vmap_area_lock); + unlink_va(va, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); return va; } @@ -2095,14 +2100,17 @@ static void *new_vmap_block(unsigned int order, gfp_t gfp_mask) static void free_vmap_block(struct vmap_block *vb) { + struct cpu_vmap_zone *z; struct vmap_block *tmp; tmp = xa_erase(&vmap_blocks, addr_to_vb_idx(vb->va->va_start)); BUG_ON(tmp != vb); - spin_lock(&vmap_area_lock); - unlink_va(vb->va, &vmap_area_root); - spin_unlock(&vmap_area_lock); + z = addr_to_cvz(vb->va->va_start); + + fbl_lock(z, BUSY); + unlink_va(vb->va, &fbl_root(z, BUSY)); + fbl_unlock(z, BUSY); free_vmap_area_noflush(vb->va); kfree_rcu(vb, rcu_head); @@ -2484,9 +2492,11 @@ static inline void setup_vmalloc_vm_locked(struct vm_struct *vm, static void setup_vmalloc_vm(struct vm_struct *vm, struct vmap_area *va, unsigned long flags, const void *caller) { - spin_lock(&vmap_area_lock); + struct cpu_vmap_zone *z = addr_to_cvz(va->va_start); + + fbl_lock(z, BUSY); setup_vmalloc_vm_locked(vm, va, flags, caller); - spin_unlock(&vmap_area_lock); + fbl_unlock(z, BUSY); } static void clear_vm_uninitialized_flag(struct vm_struct *vm) @@ -3605,6 +3615,7 @@ static void vmap_ram_vread(char *buf, char *addr, int count, unsigned long flags */ long vread(char *buf, char *addr, unsigned long count) { + struct cpu_vmap_zone *z; struct vmap_area *va; struct vm_struct *vm; char *vaddr, *buf_start = buf; @@ -3617,8 +3628,11 @@ long vread(char *buf, char *addr, unsigned long count) if ((unsigned long) addr + count < count) count = -(unsigned long) addr; - spin_lock(&vmap_area_lock); - va = find_vmap_area_exceed_addr((unsigned long)addr); + /* Hooked to CPU0 because a cache is not activated. */ + z = &per_cpu(cpu_vmap_zone, 0); + fbl_lock(z, BUSY); + + va = find_vmap_area_exceed_addr((unsigned long)addr, &fbl_root(z, BUSY)); if (!va) goto finished; @@ -3626,7 +3640,7 @@ long vread(char *buf, char *addr, unsigned long count) if ((unsigned long)addr + count <= va->va_start) goto finished; - list_for_each_entry_from(va, &vmap_area_list, list) { + list_for_each_entry_from(va, &fbl_head(z, BUSY), list) { if (!count) break; @@ -3674,7 +3688,7 @@ long vread(char *buf, char *addr, unsigned long count) count -= n; } finished: - spin_unlock(&vmap_area_lock); + fbl_unlock(z, BUSY); if (buf == buf_start) return 0; @@ -4014,14 +4028,15 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, } /* insert all vm's */ - spin_lock(&vmap_area_lock); for (area = 0; area < nr_vms; area++) { - insert_vmap_area(vas[area], &vmap_area_root, &vmap_area_list); + struct cpu_vmap_zone *z = addr_to_cvz(vas[area]->va_start); + fbl_lock(z, BUSY); + insert_vmap_area(vas[area], &fbl_root(z, BUSY), &fbl_head(z, BUSY)); setup_vmalloc_vm_locked(vms[area], vas[area], VM_ALLOC, pcpu_get_vm_areas); + fbl_unlock(z, BUSY); } - spin_unlock(&vmap_area_lock); /* * Mark allocated areas as accessible. Do it now as a best-effort @@ -4145,24 +4160,24 @@ bool vmalloc_dump_obj(void *object) #ifdef CONFIG_PROC_FS static void *s_start(struct seq_file *m, loff_t *pos) __acquires(&vmap_purge_lock) - __acquires(&vmap_area_lock) + __acquires(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) { mutex_lock(&vmap_purge_lock); - spin_lock(&vmap_area_lock); + fbl_lock((&per_cpu(cpu_vmap_zone, 0)), BUSY); - return seq_list_start(&vmap_area_list, *pos); + return seq_list_start(&fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), *pos); } static void *s_next(struct seq_file *m, void *p, loff_t *pos) { - return seq_list_next(p, &vmap_area_list, pos); + return seq_list_next(p, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), pos); } static void s_stop(struct seq_file *m, void *p) - __releases(&vmap_area_lock) + __releases(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) __releases(&vmap_purge_lock) { - spin_unlock(&vmap_area_lock); + fbl_unlock((&per_cpu(cpu_vmap_zone, 0)), BUSY); mutex_unlock(&vmap_purge_lock); } @@ -4258,7 +4273,7 @@ static int s_show(struct seq_file *m, void *p) * As a final step, dump "unpurged" areas. */ final: - if (list_is_last(&va->list, &vmap_area_list)) + if (list_is_last(&va->list, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY))) show_purge_info(m); return 0; @@ -4289,7 +4304,8 @@ static void vmap_init_free_space(void) { unsigned long vmap_start = 1; const unsigned long vmap_end = ULONG_MAX; - struct vmap_area *busy, *free; + struct vmap_area *free; + struct vm_struct *busy; /* * B F B B B F @@ -4297,12 +4313,12 @@ static void vmap_init_free_space(void) * | The KVA space | * |<--------------------------------->| */ - list_for_each_entry(busy, &vmap_area_list, list) { - if (busy->va_start - vmap_start > 0) { + for (busy = vmlist; busy; busy = busy->next) { + if (busy->addr - vmap_start > 0) { free = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); if (!WARN_ON_ONCE(!free)) { free->va_start = vmap_start; - free->va_end = busy->va_start; + free->va_end = (unsigned long) busy->addr; insert_vmap_area_augment(free, NULL, &free_vmap_area_root, @@ -4310,7 +4326,7 @@ static void vmap_init_free_space(void) } } - vmap_start = busy->va_end; + vmap_start = (unsigned long) busy->addr + busy->size; } if (vmap_end - vmap_start > 0) { @@ -4326,6 +4342,22 @@ static void vmap_init_free_space(void) } } +static void vmap_init_pcpu_zones(void) +{ + struct cpu_vmap_zone *z; + int i, j; + + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + + for (j = 0; j < ARRAY_SIZE(z->fbl); j++) { + INIT_LIST_HEAD(&z->fbl[j].head); + z->fbl[j].root = RB_ROOT; + spin_lock_init(&z->fbl[j].lock); + } + } +} + void __init vmalloc_init(void) { struct vmap_area *va; @@ -4349,8 +4381,15 @@ void __init vmalloc_init(void) INIT_WORK(&p->wq, delayed_vfree_work); } + /* + * Setup per-cpu data before importing vmlist. + */ + vmap_init_pcpu_zones(); + /* Import existing vmlist entries. */ for (tmp = vmlist; tmp; tmp = tmp->next) { + struct cpu_vmap_zone *z; + va = kmem_cache_zalloc(vmap_area_cachep, GFP_NOWAIT); if (WARN_ON_ONCE(!va)) continue; @@ -4358,7 +4397,9 @@ void __init vmalloc_init(void) va->va_start = (unsigned long)tmp->addr; va->va_end = va->va_start + tmp->size; va->vm = tmp; - insert_vmap_area(va, &vmap_area_root, &vmap_area_list); + + z = addr_to_cvz(va->va_start); + insert_vmap_area(va, &fbl_root(z, BUSY), &fbl_head(z, BUSY)); } /* From patchwork Mon May 22 11:08:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 97313 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1374294vqo; Mon, 22 May 2023 04:30:09 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7YEc6DjSCwwhJhclaIBI+DqfZV20/pbP6wjdJlZncrbNt4+ziTcrYZtupSqEx03gKYwtze X-Received: by 2002:a05:6a20:7d87:b0:105:dafa:febd with SMTP id v7-20020a056a207d8700b00105dafafebdmr13076401pzj.21.1684755009554; Mon, 22 May 2023 04:30:09 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684755009; cv=none; d=google.com; s=arc-20160816; b=sRSt8mrtMQdgkGCsgTEt2/IxMAIYhaI5qL6oOGF1PVE8SfMUiQsbj9e6+DHIA1WbrX hTm+RjMSq6AZS6plvHBpTDUmFgfIvQUaZRcMyiPP3pKvpifSqNg/pzjsGMI6zGMCrHTu Sb/cV/oNwgK9YxEfB2ELLsvCRcb8ybP61W8btZyFPGFLVBrZys3nNnGPvHzRY7gv+hQ0 9CPBaz4ktrRQFuRhfIEajHrIUtnVtzY2TEEt/JSCdvQq0D83VXFjPp4EIXNDIowdlnNc p1d7QEwf2z93ZPLWPjhjSazG/b9MJ23ym1HyWalg1TjVU4SQ4AQFf+sdOc+XxlpDhlGg +l+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=o/CF6SD8t4KOzLfI5LisrLifDfr954/t5B1JlATADoU=; b=DuP300vezpGQdhBE80EBFrCbmFp4N/Q3CRQ1FL5SQvoE9qQZMpmhIY6KPQdVrpeUPx e+CTIRC7jTYIvhzc2kijZeM+LPDg+K7DLu4EoOuZQeytYTfwk6coxXUeXTghEq0CMD0o ikCLZg8KU19Tqu2p8CyWybTuoUYR2BXD7sCX6Q9MchhZa3WfvS70Xi1LVX+zEbCRVTrT 5YHWGAEyIhIiivU+TXcevVu2QJB1E4XaUH8WhPBBr/qsk7ziYn4Owvu0ff/NLwKMAohw 1ousTksmrwK3IQz/+36KNI4IaJnr5bM7pwms0NVYeWt9d4UGGc/h+zfocqB7B2qF/9Oy qL1g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=Fi720qzQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id k13-20020a63ab4d000000b0051e011fcd73si38283pgp.237.2023.05.22.04.29.56; Mon, 22 May 2023 04:30:09 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=Fi720qzQ; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231354AbjEVLPM (ORCPT + 99 others); Mon, 22 May 2023 07:15:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232609AbjEVLOC (ORCPT ); Mon, 22 May 2023 07:14:02 -0400 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDAB31BD1 for ; Mon, 22 May 2023 04:08:58 -0700 (PDT) Received: by mail-ed1-x529.google.com with SMTP id 4fb4d7f45d1cf-510b869fe0dso10509970a12.3 for ; Mon, 22 May 2023 04:08:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753737; x=1687345737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o/CF6SD8t4KOzLfI5LisrLifDfr954/t5B1JlATADoU=; b=Fi720qzQUzMzlB1l50k0vAlTsjRzBe/cg9qRDCZSutZdUCFmDwXU1qWUWjnhF6C+YJ IYmwGB/dBSBFcf+4n8lnS3JuELxCnqx/5KzHyZvkVrc0zdw6+T1veg4ZmjJxjg9CxrEV OKjrPtpIsPfNKe3Tk/UTfUWcUqfmc21TZhwMRhoz1hrlNz5snsaWIbFcpPMBqrvXzw+e BjnHOUkDc9QXislazQsjEiQDUHPub3bFS8PaRZShQXGb/Yb2GfHmsHP7ed4HMpnWLyiW rLbMYl8TEH+ya11TbsE8iAZdu8RNYvaI/k8Wdj555MFh7QE1iHfdGfjD0joUYCDSs+QZ 4VAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753737; x=1687345737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o/CF6SD8t4KOzLfI5LisrLifDfr954/t5B1JlATADoU=; b=P06ktIl7nPoWMHbVMHWYfM4gVoS5HzGsi/giE8KswU23lAHx/Pvy2VHdQrFT6xck0J PzjowFZKY+EbtZJfl/o1RvdjeMqdSHGLs/AGBgSlN8truGcliJHOQnQS60PtQNiF9FEX g6g6RI66gcqTnynAsSRw6escN+m6Wvg7bQ/l+MEuEZ+7OSLQEGWgCmkfoGF6ONWf7xxC GioUdSGlzO05JDN1cO4DIl8DRTseIKkluw7ktTgfcZEZkB/UHE3No9nL7PIVY95csyiz 7XBIJB/xUP2jDY1T18Y2pBI8C7sm52YhXKqv4VWsd786sL3wwuPgZU2JT5u3PLB5SdlY eMMg== X-Gm-Message-State: AC+VfDz4ccvktdKc6aTgl75rt56yWjCsD3y2LJDHm4aZv4zvdqKrNiMz 3jXEgihcde3bll9Bt6z6Jro= X-Received: by 2002:a17:907:8a12:b0:93e:fa12:aa1a with SMTP id sc18-20020a1709078a1200b0093efa12aa1amr10143550ejc.1.1684753736842; Mon, 22 May 2023 04:08:56 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:56 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 6/9] mm: vmalloc: Support multiple zones in vmallocinfo Date: Mon, 22 May 2023 13:08:46 +0200 Message-Id: <20230522110849.2921-7-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766593668766679019?= X-GMAIL-MSGID: =?utf-8?q?1766593668766679019?= A global vmap area busy tree has been removed and replaced by multiple per-cpu trees/lists, therefore we need to traversal all per-cpu busy-lists in order to dump all allocated objects. Please note, after this patch, dumped addresses of allocated areas are not sequential. See an example below: 0 1 2 0 1 2 |---|---|---|---|---|---|-> vmap space There 3 CPUs dumping is done per-CPU zone, as you can see address of zone_0 can be ahead of addresses residing in the zone_1 or zone_2. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 90 ++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 45 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index a9170fe19909..dd83deb5ef4f 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -4159,26 +4159,18 @@ bool vmalloc_dump_obj(void *object) #ifdef CONFIG_PROC_FS static void *s_start(struct seq_file *m, loff_t *pos) - __acquires(&vmap_purge_lock) - __acquires(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) { - mutex_lock(&vmap_purge_lock); - fbl_lock((&per_cpu(cpu_vmap_zone, 0)), BUSY); - - return seq_list_start(&fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), *pos); + return *pos < 1 ? (void *)1 : NULL; } static void *s_next(struct seq_file *m, void *p, loff_t *pos) { - return seq_list_next(p, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY), pos); + ++*pos; + return NULL; } static void s_stop(struct seq_file *m, void *p) - __releases(&fbl(&per_cpu(cpu_vmap_zone, 0), BUSY, lock)) - __releases(&vmap_purge_lock) { - fbl_unlock((&per_cpu(cpu_vmap_zone, 0)), BUSY); - mutex_unlock(&vmap_purge_lock); } static void show_numa_info(struct seq_file *m, struct vm_struct *v) @@ -4209,6 +4201,7 @@ static void show_purge_info(struct seq_file *m) { struct vmap_area *va; + mutex_lock(&vmap_purge_lock); spin_lock(&purge_vmap_area_lock); list_for_each_entry(va, &purge_vmap_area_list, list) { seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", @@ -4216,65 +4209,72 @@ static void show_purge_info(struct seq_file *m) va->va_end - va->va_start); } spin_unlock(&purge_vmap_area_lock); + mutex_unlock(&vmap_purge_lock); } static int s_show(struct seq_file *m, void *p) { + struct cpu_vmap_zone *z; struct vmap_area *va; struct vm_struct *v; + int i; - va = list_entry(p, struct vmap_area, list); + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); - if (!va->vm) { - if (va->flags & VMAP_RAM) - seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); + fbl_lock(z, BUSY); + list_for_each_entry(va, &fbl_head(z, BUSY), list) { + if (!va->vm) { + if (va->flags & VMAP_RAM) + seq_printf(m, "0x%pK-0x%pK %7ld vm_map_ram\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); - goto final; - } + continue; + } - v = va->vm; + v = va->vm; - seq_printf(m, "0x%pK-0x%pK %7ld", - v->addr, v->addr + v->size, v->size); + seq_printf(m, "0x%pK-0x%pK %7ld", + v->addr, v->addr + v->size, v->size); - if (v->caller) - seq_printf(m, " %pS", v->caller); + if (v->caller) + seq_printf(m, " %pS", v->caller); - if (v->nr_pages) - seq_printf(m, " pages=%d", v->nr_pages); + if (v->nr_pages) + seq_printf(m, " pages=%d", v->nr_pages); - if (v->phys_addr) - seq_printf(m, " phys=%pa", &v->phys_addr); + if (v->phys_addr) + seq_printf(m, " phys=%pa", &v->phys_addr); - if (v->flags & VM_IOREMAP) - seq_puts(m, " ioremap"); + if (v->flags & VM_IOREMAP) + seq_puts(m, " ioremap"); - if (v->flags & VM_ALLOC) - seq_puts(m, " vmalloc"); + if (v->flags & VM_ALLOC) + seq_puts(m, " vmalloc"); - if (v->flags & VM_MAP) - seq_puts(m, " vmap"); + if (v->flags & VM_MAP) + seq_puts(m, " vmap"); - if (v->flags & VM_USERMAP) - seq_puts(m, " user"); + if (v->flags & VM_USERMAP) + seq_puts(m, " user"); - if (v->flags & VM_DMA_COHERENT) - seq_puts(m, " dma-coherent"); + if (v->flags & VM_DMA_COHERENT) + seq_puts(m, " dma-coherent"); - if (is_vmalloc_addr(v->pages)) - seq_puts(m, " vpages"); + if (is_vmalloc_addr(v->pages)) + seq_puts(m, " vpages"); - show_numa_info(m, v); - seq_putc(m, '\n'); + show_numa_info(m, v); + seq_putc(m, '\n'); + } + fbl_unlock(z, BUSY); + } /* * As a final step, dump "unpurged" areas. */ -final: - if (list_is_last(&va->list, &fbl_head((&per_cpu(cpu_vmap_zone, 0)), BUSY))) - show_purge_info(m); + show_purge_info(m); return 0; } From patchwork Mon May 22 11:08:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 97307 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1372906vqo; Mon, 22 May 2023 04:27:23 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7ICADNwocAgu2LaNtNx8xf6lqznDl+09UC1Db7TKFIqsYC1F3uiZXm2ktNCL4pNLprOPUs X-Received: by 2002:a17:902:c3c4:b0:1ad:eb16:35e2 with SMTP id j4-20020a170902c3c400b001adeb1635e2mr10467854plj.66.1684754843133; Mon, 22 May 2023 04:27:23 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684754843; cv=none; d=google.com; s=arc-20160816; b=JQN7S2aBAjCZ7hOvfSvJUa2MOmhksfuRs10GEtZnIgwN1GHcVQd46o84tUQYeGkJ9q jlLAht0wAEMp7UR54y9mUkARTa0Z2Bztgf9q0S5jkebhPAq7rcCmr5ke1Pe+Tswxrmci ygtIX/aF76JQN9f7bpWCqrOn/0yTY3A95qsNFqw+QHJ9S501tHF19Nu83uPA6IYxXIvO P+bc1IyrMhO069l2RcmT5CZ7A4Fm4Ca8JXme0KkGdpeNGvpcXHK2yjtZF0q3RyMv42d1 Tm0zBJl3aiK3QUvGAyJgRYmdljWZP/sh3BL5z1H+qXqy0+EXEi9trbrZQjgdDx0Numv/ LTnA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=T5CaFruXYOiRENLIpqWmUEWDjIM3GgxQkXpQsKzQbjL/SVwXeNFxrF163DZKrWcKRS oPzkJHVk0MTJrsA2eWnhMD+Lmjd9FaUTUBXtveARPVgWJEOTdBVMwq1gS/jDgqYCUfxr LdVw4SBesq3wJRZiDI76czR+BIBZ51hImyofKrCxltNFtCQOEMSFMD0lY3MW1JSTFge+ ia+XV2pAziq2FcvVVy93PxHZCsqyilbcOaXg/PP6WfXd9jLcYpsbn6i/Z1zBdMti7Xpl Rj4jwEfE9VcynqGdtwgv+l+Hdi9X2Nv7Zp/cYpYojidMii8C4wL5StCRjrhtAUh/pCgp Rjhw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=W+JtWfgK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q5-20020a170902788500b001aaf6228a75si4365703pll.88.2023.05.22.04.27.11; Mon, 22 May 2023 04:27:23 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=W+JtWfgK; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231996AbjEVLPQ (ORCPT + 99 others); Mon, 22 May 2023 07:15:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233017AbjEVLOC (ORCPT ); Mon, 22 May 2023 07:14:02 -0400 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 69C6640F4 for ; Mon, 22 May 2023 04:08:59 -0700 (PDT) Received: by mail-ej1-x62e.google.com with SMTP id a640c23a62f3a-96f99222e80so434939666b.1 for ; Mon, 22 May 2023 04:08:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753738; x=1687345738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=W+JtWfgKVMFA959Uc5C8wiOgCsYc9+SYai8nupDO9KlapBo9VlFnbVWgJAQ004f/QK mZQgkKrt4awtWG7HH1AMVsq0BI7HDBAsUxUanMdXJuT1NjGXA8j91P6MNQVZ7/2q/uOl u8b46ysLxNwbLImafPVAYrKfHx5ge512UBdEXib49XuWenSq7+MfDhUI4cQN2Y29QWPG 8ukh8J/sV1fqVvb+o7S1LN2QlrXlkMocODGZyaNkalUBH3toAXu0mQeN6PrvNuypiv2Q zaaZIwTdJUy/uadYITY2t/C6AJEJ4AgkDslYAlw0hmiXnARf/cNIGN7mPPlYm11bAx6x 3eRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753738; x=1687345738; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fdmHm9JMTKbIdAZzyhemwDZBEVKcMOyv00QDGf853lw=; b=Lx6m+HdMS/Hwxc8nsiCZCqf+5OrtGd0GNPZp7yP3vX4SWQLx+Of9EE6kX9eAqNBPXS ddjXsV3jw40kkkRzWUCSP5nti6tn7MQRSZGOyLPHkk0UySnpPc95qSRzS4QL5tBr97bg llzIQ0YKwGKvpacQOLG8yiCvSRDkDavvl1aJ/n6Tg2VI7dYdHQNCBqrJFCAgB3Y2E0Ev t1ubtbXFgWMAfcUK69lBRAJXMIF3n02lBhbRrqRSswOar6wVGJdI/ojm85W9rPWDb74z Hi9uPOMlVBSWhLrg/yagusFcy9kgo2sr31MrghxeQyE7pvVQqs7P1zeAL4j8/PoW3U7n 7iLg== X-Gm-Message-State: AC+VfDxLxaHdCpzTR14qWJwIxrBc8oQ10n/H8TvjbeHWnM6mytlbJWT3 u+BKDdhk9vYWkl5jA+/OIBU= X-Received: by 2002:a17:907:3eaa:b0:96f:8d00:43be with SMTP id hs42-20020a1709073eaa00b0096f8d0043bemr6754703ejc.0.1684753737692; Mon, 22 May 2023 04:08:57 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:57 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 7/9] mm: vmalloc: Insert lazy-VA per-cpu zone Date: Mon, 22 May 2023 13:08:47 +0200 Message-Id: <20230522110849.2921-8-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766593494187770295?= X-GMAIL-MSGID: =?utf-8?q?1766593494187770295?= Similar to busy VAs, lazy ones are stored per a CPU zone also. Freed address is converted into a correct zone it belongs to and resides there for further handling. Such approach does not require to have any global locking primitive, instead an access becomes scalable to number of CPUs. This patch removes a global purge-lock, global purge-tree and list. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 ++++++++++++++++++++++++++++----------------------- 1 file changed, 71 insertions(+), 56 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index dd83deb5ef4f..fe993c0561dd 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -734,10 +734,6 @@ static DEFINE_SPINLOCK(free_vmap_area_lock); LIST_HEAD(vmap_area_list); static bool vmap_initialized __read_mostly; -static struct rb_root purge_vmap_area_root = RB_ROOT; -static LIST_HEAD(purge_vmap_area_list); -static DEFINE_SPINLOCK(purge_vmap_area_lock); - /* * This kmem_cache is used for vmap_area objects. Instead of * allocating from slab we reuse an object from this cache to @@ -1792,39 +1788,17 @@ static DEFINE_MUTEX(vmap_purge_lock); /* for per-CPU blocks */ static void purge_fragmented_blocks_allcpus(void); -/* - * Purges all lazily-freed vmap areas. - */ -static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) +static unsigned long +purge_cpu_vmap_zone(struct cpu_vmap_zone *z) { - unsigned long resched_threshold; - unsigned int num_purged_areas = 0; - struct list_head local_purge_list; + unsigned long num_purged_areas = 0; struct vmap_area *va, *n_va; - lockdep_assert_held(&vmap_purge_lock); - - spin_lock(&purge_vmap_area_lock); - purge_vmap_area_root = RB_ROOT; - list_replace_init(&purge_vmap_area_list, &local_purge_list); - spin_unlock(&purge_vmap_area_lock); - - if (unlikely(list_empty(&local_purge_list))) + if (list_empty(&z->purge_list)) goto out; - start = min(start, - list_first_entry(&local_purge_list, - struct vmap_area, list)->va_start); - - end = max(end, - list_last_entry(&local_purge_list, - struct vmap_area, list)->va_end); - - flush_tlb_kernel_range(start, end); - resched_threshold = lazy_max_pages() << 1; - spin_lock(&free_vmap_area_lock); - list_for_each_entry_safe(va, n_va, &local_purge_list, list) { + list_for_each_entry_safe(va, n_va, &z->purge_list, list) { unsigned long nr = (va->va_end - va->va_start) >> PAGE_SHIFT; unsigned long orig_start = va->va_start; unsigned long orig_end = va->va_end; @@ -1846,13 +1820,57 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) atomic_long_sub(nr, &vmap_lazy_nr); num_purged_areas++; - - if (atomic_long_read(&vmap_lazy_nr) < resched_threshold) - cond_resched_lock(&free_vmap_area_lock); } spin_unlock(&free_vmap_area_lock); out: + return num_purged_areas; +} + +/* + * Purges all lazily-freed vmap areas. + */ +static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) +{ + unsigned long num_purged_areas = 0; + struct cpu_vmap_zone *z; + int need_purge = 0; + int i; + + lockdep_assert_held(&vmap_purge_lock); + + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + INIT_LIST_HEAD(&z->purge_list); + + if (RB_EMPTY_ROOT(&fbl_root(z, LAZY))) + continue; + + fbl_lock(z, LAZY); + WRITE_ONCE(fbl(z, LAZY, root.rb_node), NULL); + list_replace_init(&fbl_head(z, LAZY), &z->purge_list); + fbl_unlock(z, LAZY); + + start = min(start, + list_first_entry(&z->purge_list, + struct vmap_area, list)->va_start); + + end = max(end, + list_last_entry(&z->purge_list, + struct vmap_area, list)->va_end); + + need_purge++; + } + + if (need_purge) { + flush_tlb_kernel_range(start, end); + + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + num_purged_areas += purge_cpu_vmap_zone(z); + } + } + trace_purge_vmap_area_lazy(start, end, num_purged_areas); return num_purged_areas > 0; } @@ -1870,16 +1888,9 @@ static void purge_vmap_area_lazy(void) static void drain_vmap_area_work(struct work_struct *work) { - unsigned long nr_lazy; - - do { - mutex_lock(&vmap_purge_lock); - __purge_vmap_area_lazy(ULONG_MAX, 0); - mutex_unlock(&vmap_purge_lock); - - /* Recheck if further work is required. */ - nr_lazy = atomic_long_read(&vmap_lazy_nr); - } while (nr_lazy > lazy_max_pages()); + mutex_lock(&vmap_purge_lock); + __purge_vmap_area_lazy(ULONG_MAX, 0); + mutex_unlock(&vmap_purge_lock); } /* @@ -1889,6 +1900,7 @@ static void drain_vmap_area_work(struct work_struct *work) */ static void free_vmap_area_noflush(struct vmap_area *va) { + struct cpu_vmap_zone *z = addr_to_cvz(va->va_start); unsigned long nr_lazy_max = lazy_max_pages(); unsigned long va_start = va->va_start; unsigned long nr_lazy; @@ -1902,10 +1914,9 @@ static void free_vmap_area_noflush(struct vmap_area *va) /* * Merge or place it to the purge tree/list. */ - spin_lock(&purge_vmap_area_lock); - merge_or_add_vmap_area(va, - &purge_vmap_area_root, &purge_vmap_area_list); - spin_unlock(&purge_vmap_area_lock); + fbl_lock(z, LAZY); + merge_or_add_vmap_area(va, &fbl_root(z, LAZY), &fbl_head(z, LAZY)); + fbl_unlock(z, LAZY); trace_free_vmap_area_noflush(va_start, nr_lazy, nr_lazy_max); @@ -4199,17 +4210,21 @@ static void show_numa_info(struct seq_file *m, struct vm_struct *v) static void show_purge_info(struct seq_file *m) { + struct cpu_vmap_zone *z; struct vmap_area *va; + int i; - mutex_lock(&vmap_purge_lock); - spin_lock(&purge_vmap_area_lock); - list_for_each_entry(va, &purge_vmap_area_list, list) { - seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", - (void *)va->va_start, (void *)va->va_end, - va->va_end - va->va_start); + for_each_possible_cpu(i) { + z = per_cpu_ptr(&cpu_vmap_zone, i); + + fbl_lock(z, LAZY); + list_for_each_entry(va, &fbl_head(z, LAZY), list) { + seq_printf(m, "0x%pK-0x%pK %7ld unpurged vm_area\n", + (void *)va->va_start, (void *)va->va_end, + va->va_end - va->va_start); + } + fbl_unlock(z, LAZY); } - spin_unlock(&purge_vmap_area_lock); - mutex_unlock(&vmap_purge_lock); } static int s_show(struct seq_file *m, void *p) From patchwork Mon May 22 11:08:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 97311 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1373917vqo; Mon, 22 May 2023 04:29:20 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ57aq80AJz9oOLFibVG6jsVzJTG7ztzigeScypmDz7StKt1c1gNe7RCVpTaxKaR78BW9GcE X-Received: by 2002:a17:902:ed11:b0:1ab:d89:5ef6 with SMTP id b17-20020a170902ed1100b001ab0d895ef6mr9889660pld.68.1684754960203; Mon, 22 May 2023 04:29:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684754960; cv=none; d=google.com; s=arc-20160816; b=qRoNPQSg7SjsOieN+gFK9F1IrjJy6iL6+BVqQ9PDJu6pykQMi5Tg3OEsLdy11vD5UC 0SVR0pqmoAd48BKxKM7E1KBKXL2QrlclcFJ2tPgaGPvFX7kkisfxMPbSrW+qvZN+wiy4 S1e/q9turYHtiAY7LoWLSry68tJVPXjky1Lf96JTMXOuzrkxwmqz7rgPZ7phJt1cMK0/ fwmSuHs5JilnihdywVUahDwluaFJoGQLJ3FSaFhGDslNYI6/8f291kmT5LD+GvzgbAlf HkVODaYwwZaMh3V6YUDtYAqxsHKnuUeFwTCSKG6NUp6cE9HEq5olPS8EJQCbVR8xWmnY hH5g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=Bj43rlFR3doEiWwuzEBGkzBlL311yB7p6Mp4YxCWuhDmt2VASmXoDFYpz87THJgT1P jwLZzx2CWQ4yUtru4NaW4Pu7vvdi7ua2xheKoV5Wbt90pWTuONIdFlGs2E/8fg/7lsIx ACJDSDQiHuW/jSro/d6mLu2qHJ8eFNTXdllb37uHTzIht8rkewaNt01pykVf9zDROxPG d/orT57NaTsB3NoTtgxoVPuVvzdA7B3K96SvMnsOPpVwwWJqzSZKn4ccnrPRQZGxAcQJ 7Q4VlR56ug+IFbgIcmpt5JoNDgkz7+ikIs6gAF6A5JwEwtbWXDMsf5KGqAtl6J6OJ8M5 msww== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b="Ad+N9/Bx"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a17-20020a170902ecd100b001a1cefdaa33si4614869plh.342.2023.05.22.04.29.07; Mon, 22 May 2023 04:29:20 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b="Ad+N9/Bx"; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233269AbjEVLOz (ORCPT + 99 others); Mon, 22 May 2023 07:14:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233020AbjEVLOC (ORCPT ); Mon, 22 May 2023 07:14:02 -0400 Received: from mail-ej1-x629.google.com (mail-ej1-x629.google.com [IPv6:2a00:1450:4864:20::629]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5C14D1BD3 for ; Mon, 22 May 2023 04:09:00 -0700 (PDT) Received: by mail-ej1-x629.google.com with SMTP id a640c23a62f3a-96fbe7fbdd4so291570466b.3 for ; Mon, 22 May 2023 04:09:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=Ad+N9/BxXUiItff9Ka8gp9m6Ut+hvQw/iiM/znN/E/FjnWbeKMX497Feia+wUTQL7P nn0HMTXM+eAOwCIXtFoVtxCNH0LHncvSzMz7xzwQ1Srikc1cgV966ZS1qa7YdSUE18l3 cG1SVgkhO1ucaJlN4YMO1/+KLZpOHs16VnpaYal4+QLxI/ktvW/6nIUZRewRfRy6T7uj t6Bkz1DHlo2XaUESkC1cNdMlQINss0j4ahV0Ld6LZrYiZAUobHEQcZF2xuwkm88kQ9RE EafR2f2o+2S2vnNz55Q7/rEKNFxFSEEhyB+7TfomnLLk3I+4DN8cvc7+ZtkoKPBd38H7 HuUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CbQClwJJa5Q5OTrsV+8XOhbG6pyJZXEhMwzuSrOaxe4=; b=W9lBUUaziEYILHl3tGuuMchwrn5i6Eo4pq3Z5ljshPKtiOgI4YLIGqfc7WUa4ZnlTp 8IEZuBuX7st7UB6jFh1ItqCI3L0+d9rO4+VOoKV4lv3D8g+S1uFn1WUW32q48byxYMCR RgLf7ovGbQGpACI3uHjrpqUnq/9zC7O1wIuPlvmQDMAPRHwr2IwR2CCdJ+PSJ+m6HHzY Rsxs/37NZN7g5nXwsMqElSVmy2qyuNcy8a5ZhX5aKcu2FHCQGnI2g9wtcqqjnsLGQqp2 YtBLHROc4J6dVbvfVlFOpeqkgoJimiUOyxEKSjWAnPkU8vsp38rp41+7H9UXfebFrdtU 2Zzg== X-Gm-Message-State: AC+VfDzpmXVbb0c2YVj3m0ZgFmJUtMG8kobxUUErdn8uO7n0+NAJ9SnN DFd7QrTTlHZrzMfjafgM7YY= X-Received: by 2002:a17:907:1c07:b0:96f:d8a9:d045 with SMTP id nc7-20020a1709071c0700b0096fd8a9d045mr3465283ejc.59.1684753738602; Mon, 22 May 2023 04:08:58 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:58 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 8/9] mm: vmalloc: Offload free_vmap_area_lock global lock Date: Mon, 22 May 2023 13:08:48 +0200 Message-Id: <20230522110849.2921-9-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766593616934386892?= X-GMAIL-MSGID: =?utf-8?q?1766593616934386892?= Introduce a fast path of allocation sequence, that consists of per-cpu path and fallback mechanism which is used when a request can not be accomplished by fast track. A fast track pre-loads a chunk from a global vmap heap directly into its per-cpu zone, following by clipping the chunk based on allocation request. This technique allows to offload a global free_vmap_area_lock making an allocation path to be serialized to number of CPUs in a system. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 127 +++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 123 insertions(+), 4 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index fe993c0561dd..8054b8bf6c18 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1642,6 +1642,93 @@ preload_this_cpu_lock(spinlock_t *lock, gfp_t gfp_mask, int node) kmem_cache_free(vmap_area_cachep, va); } +static unsigned long +this_cpu_zone_alloc_fill(struct cpu_vmap_zone *z, + unsigned long size, unsigned long align, + gfp_t gfp_mask, int node) +{ + unsigned long addr = VMALLOC_END; + struct vmap_area *va; + + /* + * It still can race. One task sets a progress to + * 1 a second one gets preempted on entry, the first + * zeroed the progress flag and second proceed with + * an extra prefetch. + */ + if (atomic_xchg(&z->fill_in_progress, 1)) + return addr; + + va = kmem_cache_alloc_node(vmap_area_cachep, gfp_mask, node); + if (unlikely(!va)) + goto out; + + spin_lock(&free_vmap_area_lock); + addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, + cvz_size, 1, VMALLOC_START, VMALLOC_END); + spin_unlock(&free_vmap_area_lock); + + if (addr == VMALLOC_END) { + kmem_cache_free(vmap_area_cachep, va); + goto out; + } + + va->va_start = addr; + va->va_end = addr + cvz_size; + + fbl_lock(z, FREE); + va = merge_or_add_vmap_area_augment(va, + &fbl_root(z, FREE), &fbl_head(z, FREE)); + addr = va_alloc(va, &fbl_root(z, FREE), &fbl_head(z, FREE), + size, align, VMALLOC_START, VMALLOC_END); + fbl_unlock(z, FREE); + +out: + atomic_set(&z->fill_in_progress, 0); + return addr; +} + +static unsigned long +this_cpu_zone_alloc(unsigned long size, unsigned long align, gfp_t gfp_mask, int node) +{ + struct cpu_vmap_zone *z = raw_cpu_ptr(&cpu_vmap_zone); + unsigned long extra = align > PAGE_SIZE ? align : 0; + unsigned long addr = VMALLOC_END, left = 0; + + /* + * It is disabled, fallback to a global heap. + */ + if (cvz_size == ULONG_MAX) + return addr; + + /* + * Any allocation bigger/equal than one half of + * a zone-size will fallback to a global heap. + */ + if (cvz_size / (size + extra) < 3) + return addr; + + if (RB_EMPTY_ROOT(&fbl_root(z, FREE))) + goto fill; + + fbl_lock(z, FREE); + addr = __alloc_vmap_area(&fbl_root(z, FREE), &fbl_head(z, FREE), + size, align, VMALLOC_START, VMALLOC_END); + + if (addr == VMALLOC_END) + left = get_subtree_max_size(fbl_root(z, FREE).rb_node); + fbl_unlock(z, FREE); + +fill: + /* + * A low watermark is 3 pages. + */ + if (addr == VMALLOC_END && left < 4 * PAGE_SIZE) + addr = this_cpu_zone_alloc_fill(z, size, align, gfp_mask, node); + + return addr; +} + /* * Allocate a region of KVA of the specified size and alignment, within the * vstart and vend. @@ -1678,11 +1765,21 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, */ kmemleak_scan_area(&va->rb_node, SIZE_MAX, gfp_mask); + /* + * Fast path allocation, start with it. + */ + if (vstart == VMALLOC_START && vend == VMALLOC_END) + addr = this_cpu_zone_alloc(size, align, gfp_mask, node); + else + addr = vend; + retry: - preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node); - addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, - size, align, vstart, vend); - spin_unlock(&free_vmap_area_lock); + if (addr == vend) { + preload_this_cpu_lock(&free_vmap_area_lock, gfp_mask, node); + addr = __alloc_vmap_area(&free_vmap_area_root, &free_vmap_area_list, + size, align, vstart, vend); + spin_unlock(&free_vmap_area_lock); + } trace_alloc_vmap_area(addr, size, align, vstart, vend, addr == vend); @@ -1827,6 +1924,27 @@ purge_cpu_vmap_zone(struct cpu_vmap_zone *z) return num_purged_areas; } +static void +drop_cpu_vmap_cache(struct cpu_vmap_zone *z) +{ + struct vmap_area *va, *n_va; + LIST_HEAD(free_head); + + if (RB_EMPTY_ROOT(&fbl_root(z, FREE))) + return; + + fbl_lock(z, FREE); + WRITE_ONCE(fbl(z, FREE, root.rb_node), NULL); + list_replace_init(&fbl_head(z, FREE), &free_head); + fbl_unlock(z, FREE); + + spin_lock(&free_vmap_area_lock); + list_for_each_entry_safe(va, n_va, &free_head, list) + merge_or_add_vmap_area_augment(va, + &free_vmap_area_root, &free_vmap_area_list); + spin_unlock(&free_vmap_area_lock); +} + /* * Purges all lazily-freed vmap areas. */ @@ -1868,6 +1986,7 @@ static bool __purge_vmap_area_lazy(unsigned long start, unsigned long end) for_each_possible_cpu(i) { z = per_cpu_ptr(&cpu_vmap_zone, i); num_purged_areas += purge_cpu_vmap_zone(z); + drop_cpu_vmap_cache(z); } } From patchwork Mon May 22 11:08:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Uladzislau Rezki X-Patchwork-Id: 97301 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp1367332vqo; Mon, 22 May 2023 04:16:52 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7TC45rSHHJ3bThlxeDzvR3zJ/QQ5Nm2SoIv/Jpln1deoi+vaLzlYIDQnwLSnOS/5Y5RaqT X-Received: by 2002:a17:90a:d80a:b0:250:939f:70a0 with SMTP id a10-20020a17090ad80a00b00250939f70a0mr9429630pjv.14.1684754211907; Mon, 22 May 2023 04:16:51 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1684754211; cv=none; d=google.com; s=arc-20160816; b=NcdudqU/OH3XTKL/o3TCGo7wyUDfQGI7JXYJ0fC2DkA+sEYxM7uzQTmzrc981KT6fM 5FFcyIRhz38nOyAPIPWl5Ogt+RhchBLcUeKyXXQnHTumutNLWetDXuiVCUwDpIoWbcE9 c/amXJqK0rxALbRsZaT9L7IXiYAi1GlvCGH2ThSubKlL2UJoJlpo+s/wp9Wse6J+DUqi LhweRBGgIPjGWZaMgjOr4hUiYaHPimhJvsA9nZyeDVwABj+ZnfHjQjmB+u+bD/gdnpGr goiZfgBIre3+LEjKyos+8KWbcjfZuDzpN7b1CVi33wsDMHP8/i3RIg5/dj6qx3fqjEKH hQQA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=f2bGuuQYL8b+xbvuLn4PcYekQUWt9yPN5iQ314l4lMI=; b=bi042izZylZ2QaJP5JgvJZ6iOAmvF1CtG1nCk68NMg2VcnJJZ/Zl/gshbYYmm3Gdti njggzW4y7/MJouQ6tnremqPZ84KmpvcJfpzLpJe67YAsGXL6HyRKe5+r4DHiJmxsVtwU P8VFjy3ngI1ouB7g9h+igGCGHOFIqNXOtXPRxBbocy6M6tsk0vZWrgt042ucHsj5g6RG oHTwJhmEzNVPgFayRsNes78Ni58SkigDCOvHuVAHj28+PFk5wHmjq/w/89i6/MWFltP4 U/ejn1Xbq54f9qcBP0Pk5eT3nfdwpghTyJN3ZDdb8S+zhUIX2mP6lR348HGGWRS/BFqp OQyA== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=PvB7xNb9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s129-20020a632c87000000b0053488ce246esi4473061pgs.609.2023.05.22.04.16.37; Mon, 22 May 2023 04:16:51 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=PvB7xNb9; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233250AbjEVLOx (ORCPT + 99 others); Mon, 22 May 2023 07:14:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58350 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233118AbjEVLOC (ORCPT ); Mon, 22 May 2023 07:14:02 -0400 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FE4D1BD7 for ; Mon, 22 May 2023 04:09:01 -0700 (PDT) Received: by mail-ej1-x62f.google.com with SMTP id a640c23a62f3a-96f683e8855so488862466b.2 for ; Mon, 22 May 2023 04:09:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=f2bGuuQYL8b+xbvuLn4PcYekQUWt9yPN5iQ314l4lMI=; b=PvB7xNb9ygkZOuxzTtzRZdOhB6MfhJSa/XRAM+BYE1xoLfv+ewUOyX4VD8Bu0FXaOK AXlUTKkyIK2WVRsD39QMuEntv1HpM65yERdXpxKJXgRGMC2pW3R98Ogtx7X6mgbxBHMR ldToVDFezt0XeMtgZh1MaqlUYU2be1G2xPupT/rIWTKOSu/1KFkLMDIX1H3BLSGe29sO Fg3kOpF9fGfKN44fAOhFpBwP9Z7IQ03rJSGvaYH1M1Nh+mhzwxadyfvgGc5HplBYwh3q fzDEE5gFHzKH0XnCa9ZqMPOjoi2odYahJQNJL0GrEIc+/7U+m3d4QYr1A8eWf9czKONh 1f4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684753739; x=1687345739; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=f2bGuuQYL8b+xbvuLn4PcYekQUWt9yPN5iQ314l4lMI=; b=LM+YizE6+kL5AztjvEBeZDF2GFzwha7k4U7RwKQMWpQlC5JMNqhm1KAaaA4hnrtFXG I+ptpKfoouk7VR7SZ4jMnWH5LYpFJbEPIn25e2fgpOTir9iXRdc9x0XOiPxB1LdxkKVw 4JxJiZPhuaMmSJT9IH+CEFap64ewL8A4YegG5M925mtJjfbLzXnbZXmk7BxyGqCDO80q crpYZFF/lkCakuVppPU8RhlbCkwCaUIMlOPz7mkOAUa3CfE7Z+igvy0lxslRqXvH/38T sAPSZIo8sqZarYSnVIjPaVByT64/qm+gvC9vezWjPoAd/w9IQIILTwniNusmkN+z4ogW eltw== X-Gm-Message-State: AC+VfDxni/6dRANrTBhGmZVtLDUdQTocIkGfh3Lvth9xfnRtwkM7H1ap i9al701hyGLJ2IqiC8NtpXA= X-Received: by 2002:a17:907:a42a:b0:96f:a7d9:6e86 with SMTP id sg42-20020a170907a42a00b0096fa7d96e86mr6231547ejc.1.1684753739445; Mon, 22 May 2023 04:08:59 -0700 (PDT) Received: from pc638.lan ([155.137.26.201]) by smtp.gmail.com with ESMTPSA id lx19-20020a170906af1300b0094f07545d40sm2949453ejb.220.2023.05.22.04.08.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 22 May 2023 04:08:59 -0700 (PDT) From: "Uladzislau Rezki (Sony)" To: linux-mm@kvack.org, Andrew Morton Cc: LKML , Baoquan He , Lorenzo Stoakes , Christoph Hellwig , Matthew Wilcox , "Liam R . Howlett" , Dave Chinner , "Paul E . McKenney" , Joel Fernandes , Uladzislau Rezki , Oleksiy Avramchenko Subject: [PATCH 9/9] mm: vmalloc: Scale and activate cvz_size Date: Mon, 22 May 2023 13:08:49 +0200 Message-Id: <20230522110849.2921-10-urezki@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230522110849.2921-1-urezki@gmail.com> References: <20230522110849.2921-1-urezki@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1766592832508853862?= X-GMAIL-MSGID: =?utf-8?q?1766592832508853862?= Scale a zone size to number of CPUs. Each power of two value adds extra 100 * PAGE_SIZE bytes to a zone size. A maximum zone size limit is set to 4M, it is reached if number of CPUs=1024 and PAGE_SIZE=4096 bytes. For 32-bit or single core systems cvz_size is set to ULONG_MAX. It means that a pre-loading technique is deactivated. Signed-off-by: Uladzislau Rezki (Sony) --- mm/vmalloc.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 8054b8bf6c18..a5e6956a8da4 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -806,8 +806,6 @@ struct cpu_vmap_zone { }; static DEFINE_PER_CPU(struct cpu_vmap_zone, cpu_vmap_zone); - -/* Disable a per-cpu caching. */ static __read_mostly unsigned long cvz_size = ULONG_MAX; static inline unsigned int @@ -4490,6 +4488,19 @@ static void vmap_init_pcpu_zones(void) spin_lock_init(&z->fbl[j].lock); } } + + /* + * Scale a zone size to number of CPUs. Each power of two + * value adds extra 100 pages to a zone size. A high-threshold + * limit is set to 4M. It can be reached if number of CPUs=1024 + * whereas a PAGE_SIZE is 4096 bytes. + */ +#if BITS_PER_LONG == 64 + if (i > 1) { + cvz_size = fls(i) * (100 * PAGE_SIZE); + cvz_size = min(cvz_size, (unsigned long) SZ_4M); + } +#endif } void __init vmalloc_init(void)