From patchwork Wed Aug 23 14:14:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrew Stubbs X-Patchwork-Id: 136673 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:a7d1:0:b0:3f2:4152:657d with SMTP id p17csp492231vqm; Wed, 23 Aug 2023 07:18:35 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEcPMrgU04WBu/IWUXjkJyJuUOfbmY2mu7HIW/U/MOI05kpipiEGdlcgBqb4eHDJcHW3lc8 X-Received: by 2002:a17:906:10dc:b0:9a1:bf00:ae4e with SMTP id v28-20020a17090610dc00b009a1bf00ae4emr3970463ejv.54.1692800315028; Wed, 23 Aug 2023 07:18:35 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1692800315; cv=none; d=google.com; s=arc-20160816; b=T9xqRduJsYHj92UutCgtQBMOdjv6UGH071V2d6vR256bl58v5KI4Yvp+1/HnH+dGgi YmMhVzwlzmJHab+/4eaO1s2yY/H1RKU4niP75GEC0bVHnCnjznARKM1mc75fOXl8lq2k J4Zpi5rPY+vTSqidPjz9TuNqNGdzRqJEzpcna8gkRkz7gNFI0DO34PYAO6Qh48QrD6SR PwT5kzk5jQsRB/rU2CbjlxHhlroCdlHMZZRdZ2X7UaNL0EEwsT5eixFOTksdHx4CwpH6 s+C6KgMBnaIoSmq61rThundLDIIKppKiRfx5Tneb6gTz2F9bMCLzvd68mtBE7vUR/S3l +9VA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:content-transfer-encoding :mime-version:references:in-reply-to:message-id:date:subject:to:from :ironport-sdr:dmarc-filter:delivered-to; bh=D8/E1hruSMSkPAr4537dfpjLi/9E+uEmgJoXHRWhQKo=; fh=9xD62naDDQ44CEuxrRV0ts5PrjZies6B3X/DCYdxTRw=; b=uZgtanMQI8o8kiWGjSOcLniQlXllQP+OYv+MTzejNCy5sSybF8YzJQ0VI3Nc1rpQVA vubCCaDWIH8fHMP7vkyZ/k99BLgnooP9F38T1cEuI/LyNvmJPnsyoN5NI+FPmD4fHT6R oI7jUz+OkgC1Wy+G2JXILphaXlNUKuWUzbHKIqs84TRvqnAqhONxSt+8mo2OQFusZxvs Tuih4ra8y1ty3F8uP4LI3eTvNkVrPRQzuh6IyL53LMikqNLmL8UaXY+qErcr3FDtDQeq kAuDOwmTPqJ+3ffJyFfz0dkRqNDYfouA5Mvo0cxelvLJ6G42oRzZea7B/4JvMJ5fSwRV 2I/A== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (server2.sourceware.org. [2620:52:3:1:0:246e:9693:128c]) by mx.google.com with ESMTPS id cd13-20020a170906b34d00b0099347270140si8452259ejb.565.2023.08.23.07.18.34 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Aug 2023 07:18:35 -0700 (PDT) Received-SPF: pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) client-ip=2620:52:3:1:0:246e:9693:128c; Authentication-Results: mx.google.com; spf=pass (google.com: domain of gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org designates 2620:52:3:1:0:246e:9693:128c as permitted sender) smtp.mailfrom="gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org" Received: from server2.sourceware.org (localhost [IPv6:::1]) by sourceware.org (Postfix) with ESMTP id 4DF6C3830B55 for ; Wed, 23 Aug 2023 14:15:52 +0000 (GMT) X-Original-To: gcc-patches@gcc.gnu.org Delivered-To: gcc-patches@gcc.gnu.org Received: from esa1.mentor.iphmx.com (esa1.mentor.iphmx.com [68.232.129.153]) by sourceware.org (Postfix) with ESMTPS id 840F1385AC09 for ; Wed, 23 Aug 2023 14:14:48 +0000 (GMT) DMARC-Filter: OpenDMARC Filter v1.4.2 sourceware.org 840F1385AC09 Authentication-Results: sourceware.org; dmarc=none (p=none dis=none) header.from=codesourcery.com Authentication-Results: sourceware.org; spf=pass smtp.mailfrom=mentor.com X-IronPort-AV: E=Sophos;i="6.01,195,1684828800"; d="scan'208";a="17132456" Received: from orw-gwy-02-in.mentorg.com ([192.94.38.167]) by esa1.mentor.iphmx.com with ESMTP; 23 Aug 2023 06:14:47 -0800 IronPort-SDR: 1oyj0R/5oebkJMXeeUF/CybaqVPNJVV37cy/hVFZIDRokigX0QFTrdSDW+OIVi12iG5k9Q2lLG tQwEpCBQ2U+EGYiRvGFKXN7KZT8p9zRRPFfgvH92702QzVd4TZEvBCgd8H81ywKaw1zMf2xw8H KjhgO4Z3nYCJ6X5xyfxqWEyxm8m2POTFR29ctDxlbM8G+p7aZDt/AetCeqqX8guMVbZbdRddH0 sUGYKistU2URPijMDJ6tFFo3cXUVdvOayYaVsI0YEzBqaFsxsO5xBElL2L6xE3rTPmAxpYqWEQ i4o= From: Andrew Stubbs To: Subject: [PATCH v2 4/6] openmp: -foffload-memory=pinned Date: Wed, 23 Aug 2023 15:14:24 +0100 Message-ID: <20230823141426.320160-5-ams@codesourcery.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230823141426.320160-1-ams@codesourcery.com> References: <20230823141426.320160-1-ams@codesourcery.com> MIME-Version: 1.0 X-Originating-IP: [137.202.0.90] X-ClientProxiedBy: svr-ies-mbx-15.mgc.mentorg.com (139.181.222.15) To svr-ies-mbx-11.mgc.mentorg.com (139.181.222.11) X-Spam-Status: No, score=-12.0 required=5.0 tests=BAYES_00, GIT_PATCH_0, HEADER_FROM_DIFFERENT_DOMAINS, KAM_DMARC_STATUS, SPF_HELO_PASS, SPF_PASS, TXREP autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on server2.sourceware.org X-BeenThere: gcc-patches@gcc.gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Gcc-patches mailing list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: gcc-patches-bounces+ouuuleilei=gmail.com@gcc.gnu.org Sender: "Gcc-patches" X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1775029783290325526 X-GMAIL-MSGID: 1775029783290325526 Implement the -foffload-memory=pinned option such that libgomp is instructed to enable fully-pinned memory at start-up. The option is intended to provide a performance boost to certain offload programs without modifying the code. This feature only works on Linux, at present, and simply calls mlockall to enable always-on memory pinning. It requires that the ulimit feature is set high enough to accommodate all the program's memory usage. In this mode the ompx_pinned_memory_alloc feature is disabled as it is not needed and may conflict. gcc/ChangeLog: * omp-builtins.def (BUILT_IN_GOMP_ENABLE_PINNED_MODE): New. * omp-low.cc (omp_enable_pinned_mode): New function. (execute_lower_omp): Call omp_enable_pinned_mode. libgomp/ChangeLog: * config/linux/allocator.c (always_pinned_mode): New variable. (GOMP_enable_pinned_mode): New function. (linux_memspace_alloc): Disable pinning when always_pinned_mode set. (linux_memspace_calloc): Likewise. (linux_memspace_free): Likewise. (linux_memspace_realloc): Likewise. * libgomp.map: Add GOMP_enable_pinned_mode. * testsuite/libgomp.c/alloc-pinned-7.c: New test. * testsuite/libgomp.c-c++-common/alloc-pinned-1.c: New test. --- gcc/omp-builtins.def | 3 + gcc/omp-low.cc | 66 +++++++++++++++++++ libgomp/config/linux/allocator.c | 26 ++++++++ libgomp/libgomp.map | 1 + .../libgomp.c-c++-common/alloc-pinned-1.c | 28 ++++++++ libgomp/testsuite/libgomp.c/alloc-pinned-7.c | 63 ++++++++++++++++++ 6 files changed, 187 insertions(+) create mode 100644 libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c create mode 100644 libgomp/testsuite/libgomp.c/alloc-pinned-7.c diff --git a/gcc/omp-builtins.def b/gcc/omp-builtins.def index e0f03263db0..1f7280f6b36 100644 --- a/gcc/omp-builtins.def +++ b/gcc/omp-builtins.def @@ -470,3 +470,6 @@ DEF_GOMP_BUILTIN (BUILT_IN_GOMP_WARNING, "GOMP_warning", BT_FN_VOID_CONST_PTR_SIZE, ATTR_NOTHROW_LEAF_LIST) DEF_GOMP_BUILTIN (BUILT_IN_GOMP_ERROR, "GOMP_error", BT_FN_VOID_CONST_PTR_SIZE, ATTR_COLD_NORETURN_NOTHROW_LEAF_LIST) +DEF_GOMP_BUILTIN (BUILT_IN_GOMP_ENABLE_PINNED_MODE, + "GOMP_enable_pinned_mode", + BT_FN_VOID, ATTR_NOTHROW_LIST) diff --git a/gcc/omp-low.cc b/gcc/omp-low.cc index b882df048ef..64bc0662332 100644 --- a/gcc/omp-low.cc +++ b/gcc/omp-low.cc @@ -14683,6 +14683,68 @@ lower_omp (gimple_seq *body, omp_context *ctx) input_location = saved_location; } +/* Emit a constructor function to enable -foffload-memory=pinned + at runtime. Libgomp handles the OS mode setting, but we need to trigger + it by calling GOMP_enable_pinned mode before the program proper runs. */ + +static void +omp_enable_pinned_mode () +{ + static bool visited = false; + if (visited) + return; + visited = true; + + /* Create a new function like this: + + static void __attribute__((constructor)) + __set_pinned_mode () + { + GOMP_enable_pinned_mode (); + } + */ + + tree name = get_identifier ("__set_pinned_mode"); + tree voidfntype = build_function_type_list (void_type_node, NULL_TREE); + tree decl = build_decl (UNKNOWN_LOCATION, FUNCTION_DECL, name, voidfntype); + + TREE_STATIC (decl) = 1; + TREE_USED (decl) = 1; + DECL_ARTIFICIAL (decl) = 1; + DECL_IGNORED_P (decl) = 0; + TREE_PUBLIC (decl) = 0; + DECL_UNINLINABLE (decl) = 1; + DECL_EXTERNAL (decl) = 0; + DECL_CONTEXT (decl) = NULL_TREE; + DECL_INITIAL (decl) = make_node (BLOCK); + BLOCK_SUPERCONTEXT (DECL_INITIAL (decl)) = decl; + DECL_STATIC_CONSTRUCTOR (decl) = 1; + DECL_ATTRIBUTES (decl) = tree_cons (get_identifier ("constructor"), + NULL_TREE, NULL_TREE); + + tree t = build_decl (UNKNOWN_LOCATION, RESULT_DECL, NULL_TREE, + void_type_node); + DECL_ARTIFICIAL (t) = 1; + DECL_IGNORED_P (t) = 1; + DECL_CONTEXT (t) = decl; + DECL_RESULT (decl) = t; + + push_struct_function (decl); + init_tree_ssa (cfun); + + tree calldecl = builtin_decl_explicit (BUILT_IN_GOMP_ENABLE_PINNED_MODE); + gcall *call = gimple_build_call (calldecl, 0); + + gimple_seq seq = NULL; + gimple_seq_add_stmt (&seq, call); + gimple_set_body (decl, gimple_build_bind (NULL_TREE, seq, NULL)); + + cfun->function_end_locus = UNKNOWN_LOCATION; + cfun->curr_properties |= PROP_gimple_any; + pop_cfun (); + cgraph_node::add_new_function (decl, true); +} + /* Main entry point. */ static unsigned int @@ -14739,6 +14801,10 @@ execute_lower_omp (void) for (auto task_stmt : task_cpyfns) finalize_task_copyfn (task_stmt); task_cpyfns.release (); + + if (flag_offload_memory == OFFLOAD_MEMORY_PINNED) + omp_enable_pinned_mode (); + return 0; } diff --git a/libgomp/config/linux/allocator.c b/libgomp/config/linux/allocator.c index edcde9f1e81..8205d67c7a2 100644 --- a/libgomp/config/linux/allocator.c +++ b/libgomp/config/linux/allocator.c @@ -51,11 +51,28 @@ #include #include "libgomp.h" +static bool always_pinned_mode = false; + +/* This function is called by the compiler when -foffload-memory=pinned + is used. */ + +void +GOMP_enable_pinned_mode () +{ + if (mlockall (MCL_CURRENT | MCL_FUTURE) != 0) + gomp_error ("failed to pin all memory (ulimit too low?)"); + else + always_pinned_mode = true; +} + static void * linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin) { (void)memspace; + /* Explicit pinning may not be required. */ + pin = pin && !always_pinned_mode; + if (pin) { void *addr = mmap (NULL, size, PROT_READ | PROT_WRITE, @@ -79,6 +96,9 @@ linux_memspace_alloc (omp_memspace_handle_t memspace, size_t size, int pin) static void * linux_memspace_calloc (omp_memspace_handle_t memspace, size_t size, int pin) { + /* Explicit pinning may not be required. */ + pin = pin && !always_pinned_mode; + if (pin) return linux_memspace_alloc (memspace, size, pin); else @@ -91,6 +111,9 @@ linux_memspace_free (omp_memspace_handle_t memspace, void *addr, size_t size, { (void)memspace; + /* Explicit pinning may not be required. */ + pin = pin && !always_pinned_mode; + if (pin) munmap (addr, size); else @@ -101,6 +124,9 @@ static void * linux_memspace_realloc (omp_memspace_handle_t memspace, void *addr, size_t oldsize, size_t size, int oldpin, int pin) { + /* Explicit pinning may not be required. */ + pin = pin && !always_pinned_mode; + if (oldpin && pin) { void *newaddr = mremap (addr, oldsize, size, MREMAP_MAYMOVE); diff --git a/libgomp/libgomp.map b/libgomp/libgomp.map index ce6b719a57f..d27d3a30f47 100644 --- a/libgomp/libgomp.map +++ b/libgomp/libgomp.map @@ -406,6 +406,7 @@ GOMP_5.0.1 { global: GOMP_alloc; GOMP_free; + GOMP_enable_pinned_mode; } GOMP_5.0; GOMP_5.1 { diff --git a/libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c b/libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c new file mode 100644 index 00000000000..e0e08019bff --- /dev/null +++ b/libgomp/testsuite/libgomp.c-c++-common/alloc-pinned-1.c @@ -0,0 +1,28 @@ +/* { dg-do run } */ +/* { dg-additional-options "-foffload-memory=pinned" } */ +/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */ + +#if __cplusplus +#define EXTERNC extern "C" +#else +#define EXTERNC +#endif + +/* Intercept the libgomp initialization call to check it happens. */ + +int good = 0; + +EXTERNC void +GOMP_enable_pinned_mode () +{ + good = 1; +} + +int +main () +{ + if (!good) + __builtin_exit (1); + + return 0; +} diff --git a/libgomp/testsuite/libgomp.c/alloc-pinned-7.c b/libgomp/testsuite/libgomp.c/alloc-pinned-7.c new file mode 100644 index 00000000000..350bcd36c5a --- /dev/null +++ b/libgomp/testsuite/libgomp.c/alloc-pinned-7.c @@ -0,0 +1,63 @@ +/* { dg-do run } */ +/* { dg-additional-options "-foffload-memory=pinned" } */ + +/* { dg-xfail-run-if "Pinning not implemented on this host" { ! *-*-linux-gnu } } */ + +/* Test that -foffload-memory=pinned works. */ + +#include +#include + +#ifdef __linux__ +#include +#include + +#include + +int +get_pinned_mem () +{ + int pid = getpid (); + char buf[100]; + sprintf (buf, "/proc/%d/status", pid); + + FILE *proc = fopen (buf, "r"); + if (!proc) + abort (); + while (fgets (buf, 100, proc)) + { + int val; + if (sscanf (buf, "VmLck: %d", &val)) + { + fclose (proc); + return val; + } + } + abort (); +} +#else +int +get_pinned_mem () +{ + return 0; +} + +#define mlockall(...) 0 +#endif + +#include + +int +main () +{ + // Sanity check + if (get_pinned_mem () == 0) + { + /* -foffload-memory=pinned has failed, but maybe that's because + isufficient pinned memory was available. */ + if (mlockall (MCL_CURRENT | MCL_FUTURE) == 0) + abort (); + } + + return 0; +}