From patchwork Fri Feb 2 23:54:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charlie Jenkins X-Patchwork-Id: 196096 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp763535dyc; Fri, 2 Feb 2024 15:56:54 -0800 (PST) X-Google-Smtp-Source: AGHT+IHz8u5UAfu5A5KiZCxZXZ9of/0QYYHVjvJPBoGNrUEuNBi4/gcDar4R20ouQ5TpbBUdQBE+ X-Received: by 2002:ac8:60d6:0:b0:42c:e47:8359 with SMTP id i22-20020ac860d6000000b0042c0e478359mr417331qtm.21.1706918213931; Fri, 02 Feb 2024 15:56:53 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706918213; cv=pass; d=google.com; s=arc-20160816; b=Jz4IJM/8OVfNyWILdTsJ/vmLhDJFlm2Cf1CIjEJatLlG/RWRXp1ObhWQuv9azXy3hg vJ8pYnwh8gnCnayt10emvaE7sc9AYV3LoSXlnSZtEQvtMyerz9XRwZeUkc6Qy02pu23m EVC781U0op3e4KS+vc0ED+1Touu+b2GZFgPRwTxL/6VWwMWpjNzjDZUZ9Idt2ePnJ6nn bnVHHnJkoZCrmNInUW0hE3prphP95BQuKfyr2CxeBI7kHxlDyxFuLU9vXdKp3ttMrS/u PKSO4axPIpDLRZSpdGgkM8nwJnG37Liov0WAsa/3g2saPCSK6C1QktZq6JkfpfOJAadV nGqQ== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :subject:date:from:dkim-signature; bh=/1MLhuldF93A/E3O+pm9/YZf3cQu2JpI7YOHHh8M+ks=; fh=BmRIS3b4+yd50bfPWGA5+rBzxXfZRbk78rWVkQjseYU=; b=S90Re4qj1IPy3BdXP5KwQjFmbe9uIaEYkVszxf+M57+FQ0T/MgoFEkWq2/SrkciDId QJbpShc7QM6q8TPEYvJqQnMmmFrcGKsC0NYaB9HvBXQD6eXVwmRzJKoDxx5qFgWf6zKT oeHS7MQWP6Q89WVdbjQP68FwMaWnqN4bx1KMPALVA8M+vH3/Jv1ZfJUJi+6O/9N5I8UB JCiv4Lj1M0CYPi31kqdrq1fj2vl8vD5JzESmBwoSJz4EXzMsqkUXCHxk0AOb/Dw+rM9E TCXXvVW2ngWbFR510jFjbHI0NbVxGBHmMoFxHThffSNpw5FA3DaHkmGqpoicBQopvCBO bikQ==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=3XXlubJS; arc=pass (i=1 spf=pass spfdomain=rivosinc.com dkim=pass dkdomain=rivosinc-com.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-50733-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-50733-ouuuleilei=gmail.com@vger.kernel.org" X-Forwarded-Encrypted: i=1; AJvYcCVpsdeimfA9aFhGZo1idRaXAr6kMbaBhXQFFb7M2Scb14WMHlJAS0YsU4dogEyssylIqJILHtl02d/iPRpktKMgy2EKpw== Received: from ny.mirrors.kernel.org (ny.mirrors.kernel.org. [147.75.199.223]) by mx.google.com with ESMTPS id e2-20020ac85982000000b0042c0fb47a11si171101qte.281.2024.02.02.15.56.53 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 15:56:53 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-50733-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) client-ip=147.75.199.223; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=3XXlubJS; arc=pass (i=1 spf=pass spfdomain=rivosinc.com dkim=pass dkdomain=rivosinc-com.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-50733-ouuuleilei=gmail.com@vger.kernel.org designates 147.75.199.223 as permitted sender) smtp.mailfrom="linux-kernel+bounces-50733-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ny.mirrors.kernel.org (Postfix) with ESMTPS id 3D2C01C23332 for ; Fri, 2 Feb 2024 23:55:26 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 5B2C812F374; Fri, 2 Feb 2024 23:55:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="3XXlubJS" Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C00B212C7E6 for ; Fri, 2 Feb 2024 23:54:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706918098; cv=none; b=aOKWHFzDIjuhORNLpO0EXOchHnZ9BcEmlPcXHVwrbaRBpYcIfdRu7v4zRWuKuZJa8EKGWmDBVXAL1B0/K7nYofFmzbX62dhr7GGFXTOjE6QPVPswp0pE9tJejyRlxeYCwjDML+muOjggVulbB2S4CKWFfaflVRmtemhGufVhv6E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706918098; c=relaxed/simple; bh=EJLAQjoAaveZcHBafYijrvolB9ZWj2bzuPRUhvoIegQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=NidT828gOzyRO2qi7//Sg4SjAYfMmKLPcxca+66yxaPXw+EiTgVed/ET4PID9QchxNid6+lAkDiLd8T3msOjEEK0wZjId/Lt33Jru/2EIggaQsUYA9yJWGGKlfSfCO6MkvmVpSpGmUO3+VAq7vps5uSu250c4zkN8MMEl07AB3E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=3XXlubJS; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1d751bc0c15so22925995ad.2 for ; Fri, 02 Feb 2024 15:54:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1706918096; x=1707522896; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=/1MLhuldF93A/E3O+pm9/YZf3cQu2JpI7YOHHh8M+ks=; b=3XXlubJSvxYOqf0Uymbuxh1wvk8Hedjsvw/3DukfOsr8TfPOjWnWmeiMLhWxgKD1v/ drHtrX50HfYZ6lDr2EE92oYHgAvi+2s86uJQheyieTu2ZZ9AK0ZYKdtWpdzdPBljOEZr 544JVNT3X7df232VO283qCBOLWnMrom34T0egMmB2TxomLmYsKxq6Qgsg+iebUtjSt67 HqwqanKSw5UA6Wx/rx+bQrv7qDekLlb+lqwaqOuaZPezrnqAue9vzDETObWgTnmozTtH NlBdeIT7EKGmxuCfcw0FXOSjpkMQgJAGAKZn3Qpy7SUMZRZJmOlX9QeSSbSeKttSY0Sh q0mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706918096; x=1707522896; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/1MLhuldF93A/E3O+pm9/YZf3cQu2JpI7YOHHh8M+ks=; b=ZB3++9EUQ3K9K/HdgrZZCAWGFjKtn4xTayOg2Z40L3yen5/rmPvxjgDmJ5n1RPCrLc JWULxGwOcA+6Hg2ZJ7XRXgExEHva3anbpycxvs1u32vXHUPBkXCkQxuMd8YcasfxOLGw R9MwMxh0zHjbYtvacEkPQdOzl5rC3vubLt4VIBc8XeKQg1e4/naZ0G9KE1dVp/48fy4a lUkawGKvHcTqw5TqYieoxtUEYqZ4RR4cVEpQ+LYFWC8uLsnUHLrcZiK7ITVAWeu74pXw ui1d1rlkb0EIeqYjHO8amRY8P3oNYIoHMvpI68yP/Pg3EDBXEKdek5ErfYYlkJx4pJPg d2+w== X-Gm-Message-State: AOJu0YzEeAH211rqYTL9mBPeXd9zQ96kujjZVwklSTgZa/kUhdn89xMM m282MF7rZi8Iiup2XWTVxYaeywVkSc5QWy/2WNkMuZtt3dWMz/cR8vBgKjEMmI8= X-Received: by 2002:a17:902:bb16:b0:1d9:7a79:c35a with SMTP id im22-20020a170902bb1600b001d97a79c35amr2200249plb.35.1706918096176; Fri, 02 Feb 2024 15:54:56 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCWXRw1NDqFZj5R1OPn+mE5bcPhpsNgwjgE6bHW7KDP+lpfxY9uwZ4X1PpKnXjgcutHPAW7/V1cbJjc8baznAl/8B5GMjAFx78zCbu1+x+kc7EOesSZDlXdUdw0zg/dpcLGW3uoo9aBggOPILURQ9FvHsV1SlFPua43+txomnVGl//dA7WIBVJYygdJs8b7fiC8cja1KCesna0Tr1Qniy59mGsBEHpQKX+xSKF34KGwnh2xvS2i3ccBDMJAFy4fpBp7CN+Ra4lDi6lnpZmrZTwB7JtVeMn2VNWadSkTAsvhVRwih2bjoMLGubk/cm1egLYvutCT30w+qSQ== Received: from charlie.ba.rivosinc.com ([64.71.180.162]) by smtp.gmail.com with ESMTPSA id u1-20020a17090282c100b001d8f81ece98sm2133607plz.104.2024.02.02.15.54.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 15:54:55 -0800 (PST) From: Charlie Jenkins Date: Fri, 02 Feb 2024 15:54:52 -0800 Subject: [PATCH v3 1/2] riscv: lib: Introduce has_fast_misaligned_access function Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240202-disable_misaligned_probe_config-v3-1-c44f91f03bb6@rivosinc.com> References: <20240202-disable_misaligned_probe_config-v3-0-c44f91f03bb6@rivosinc.com> In-Reply-To: <20240202-disable_misaligned_probe_config-v3-0-c44f91f03bb6@rivosinc.com> To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Jisheng Zhang , Evan Green , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Eric Biggers , Elliot Berman Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Charlie Jenkins X-Mailer: b4 0.12.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1706918093; l=2638; i=charlie@rivosinc.com; s=20231120; h=from:subject:message-id; bh=EJLAQjoAaveZcHBafYijrvolB9ZWj2bzuPRUhvoIegQ=; b=PLwP0vQ99Z+7IGjRrd6DwcVhXXGqxnXdGfaaYbpjQXIm11Du3c0rwMO/IAoNkGCtIJCgwhfGo 14cUsMbyb7LBWToiVkJFcuLp7tEVlDeIh2FEd6BBfXainkYwSh+K4RA X-Developer-Key: i=charlie@rivosinc.com; a=ed25519; pk=t4RSWpMV1q5lf/NWIeR9z58bcje60/dbtxxmoSfBEcs= X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789833473217005751 X-GMAIL-MSGID: 1789833473217005751 Create has_fast_misaligned_access to avoid needing to explicitly check the fast_misaligned_access_speed_key static key. Signed-off-by: Charlie Jenkins Reviewed-by: Evan Green --- arch/riscv/include/asm/cpufeature.h | 15 ++++++++++----- arch/riscv/lib/csum.c | 7 ++----- 2 files changed, 12 insertions(+), 10 deletions(-) diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index 5a626ed2c47a..eb3ac304fc42 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0-only */ /* - * Copyright 2022-2023 Rivos, Inc + * Copyright 2022-2024 Rivos, Inc */ #ifndef _ASM_CPUFEATURE_H @@ -28,8 +28,6 @@ struct riscv_isainfo { DECLARE_PER_CPU(struct riscv_cpuinfo, riscv_cpuinfo); -DECLARE_PER_CPU(long, misaligned_access_speed); - /* Per-cpu ISA extensions. */ extern struct riscv_isainfo hart_isa[NR_CPUS]; @@ -53,6 +51,15 @@ static inline bool check_unaligned_access_emulated(int cpu) static inline void unaligned_emulation_finish(void) {} #endif +DECLARE_PER_CPU(long, misaligned_access_speed); + +DECLARE_STATIC_KEY_FALSE(fast_misaligned_access_speed_key); + +static __always_inline bool has_fast_misaligned_accesses(void) +{ + return static_branch_likely(&fast_misaligned_access_speed_key); +} + unsigned long riscv_get_elf_hwcap(void); struct riscv_isa_ext_data { @@ -135,6 +142,4 @@ static __always_inline bool riscv_cpu_has_extension_unlikely(int cpu, const unsi return __riscv_isa_extension_available(hart_isa[cpu].isa, ext); } -DECLARE_STATIC_KEY_FALSE(fast_misaligned_access_speed_key); - #endif diff --git a/arch/riscv/lib/csum.c b/arch/riscv/lib/csum.c index af3df5274ccb..47392a83239d 100644 --- a/arch/riscv/lib/csum.c +++ b/arch/riscv/lib/csum.c @@ -3,7 +3,7 @@ * Checksum library * * Influenced by arch/arm64/lib/csum.c - * Copyright (C) 2023 Rivos Inc. + * Copyright (C) 2023-2024 Rivos Inc. */ #include #include @@ -318,10 +318,7 @@ unsigned int do_csum(const unsigned char *buff, int len) * branches. The largest chunk of overlap was delegated into the * do_csum_common function. */ - if (static_branch_likely(&fast_misaligned_access_speed_key)) - return do_csum_no_alignment(buff, len); - - if (((unsigned long)buff & OFFSET_MASK) == 0) + if (has_fast_misaligned_accesses() || (((unsigned long)buff & OFFSET_MASK) == 0)) return do_csum_no_alignment(buff, len); return do_csum_with_alignment(buff, len); From patchwork Fri Feb 2 23:54:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Charlie Jenkins X-Patchwork-Id: 196094 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:9bc1:b0:106:209c:c626 with SMTP id op1csp763191dyc; Fri, 2 Feb 2024 15:55:40 -0800 (PST) X-Google-Smtp-Source: AGHT+IH+UUsceRPFSppOOz46Zp8HhppRpKZPuk/sw6LelHYP2IU71wDCxNFTJQf5eyHLaDj/wdVL X-Received: by 2002:a2e:be08:0:b0:2d0:9b1c:649e with SMTP id z8-20020a2ebe08000000b002d09b1c649emr145559ljq.31.1706918139861; Fri, 02 Feb 2024 15:55:39 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1706918139; cv=pass; d=google.com; s=arc-20160816; b=t+uCWlOCLotvqlb0q+00KTQz9p7pRN6KuefEvqx3V/MC8QXARx39YU/GCwOu85zMTs rHo+JnQQeRQoPW2kCpZJMb7S+jgOccfpAySsV3jIpR5AEuf4EhhkilSh0zFxcTNaPVGE FQVXsl/Eomg3IYU9fQgiMHxqr3i+gSuxkutpuiD0xTWrbvifeh/UuV+hsC50xgErbww9 cf0LAih+omz2yKsJ+IP8DA1/U0vhdAgjVvvyyRxqgbK2aoeLFOkUIr4zogwa2BvvHxdM 04JPXax8FwA5XahsaSGZiv9ODO9rCyewR9cnr8rEJUCypuf7SwAWcO851yHvinDau9Gj z49A== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:list-unsubscribe:list-subscribe:list-id:precedence :subject:date:from:dkim-signature; bh=xunksNOigL1rXi22PNXZoQDfg/C2wRrV7gzfa+vwFao=; fh=9QNd2nymF3BqjLlRxcmmjofHubeHuoYbcXPZU0sTHi8=; b=b+EuGNlQhyCVQEvCEAzSHY9uhYj6HXoW8oe6caUDH+Eb4tNa4p0ainwW6V+iOIQGE7 skgER6thjHLtv7OM9hTC2Hz/NvGlMPMp5ICMag2q9f/lG2NufndqCi6g4YgEn3t6CLDG 7PXk3rw0qXYOhkztDq5g+13Du8yQpkqbZ5t9XC+AsSPTbQc22EDnc4MT6E+7SlIjUmd0 R4Y3KVWB5isjpYMlGzmosB0fBZZp2Y+kCfajZX2fM/ldmxqL64XYzzjgHgR4PJAPSSvb FbqAAZOytUvk0Ugw0yWjqmmkX9jChl0DUoJNkaMtDpbl3qzs0af7wATitKfofyd4c6z2 Z+ww==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=U0R5hav+; arc=pass (i=1 spf=pass spfdomain=rivosinc.com dkim=pass dkdomain=rivosinc-com.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-50734-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-50734-ouuuleilei=gmail.com@vger.kernel.org" X-Forwarded-Encrypted: i=1; AJvYcCW2eTMti9DWinQqESot6F+/cGaaY1KTumm0awZqnvg7DTq7TXHqwO7oqnSORWJTig1iRY8A6qgjWarVdzA+1ltoWet5ZQ== Received: from am.mirrors.kernel.org (am.mirrors.kernel.org. [2604:1380:4601:e00::3]) by mx.google.com with ESMTPS id y26-20020a50e61a000000b0056019f36d87si102316edm.351.2024.02.02.15.55.39 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 15:55:39 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-50734-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) client-ip=2604:1380:4601:e00::3; Authentication-Results: mx.google.com; dkim=pass header.i=@rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=U0R5hav+; arc=pass (i=1 spf=pass spfdomain=rivosinc.com dkim=pass dkdomain=rivosinc-com.20230601.gappssmtp.com); spf=pass (google.com: domain of linux-kernel+bounces-50734-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:4601:e00::3 as permitted sender) smtp.mailfrom="linux-kernel+bounces-50734-ouuuleilei=gmail.com@vger.kernel.org" Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by am.mirrors.kernel.org (Postfix) with ESMTPS id 3D7991F29629 for ; Fri, 2 Feb 2024 23:55:39 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 7B21D12F38C; Fri, 2 Feb 2024 23:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b="U0R5hav+" Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42BE485946 for ; Fri, 2 Feb 2024 23:54:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706918100; cv=none; b=MyC4BFNUxgS2smYoDXqgYhH4rvOr0Cml2gUQQmDarXvSn+6CIZ1pztH+f68yCN5ffhsjJC1zy2ED1shUam2KKc9OLVpFbL/e/6jhgX28yMUKI9MpLPGcbmpIEBE1ejHIbdNNqk9+sAfijdb7SIgMKoUJOLCMqW2xoqh+45ZpTNA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706918100; c=relaxed/simple; bh=vwhBmHfF/EKObUwb3qKoX49eRqG1MgvOSHuJsJTDdRo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Rg9cae+qzU5RFn6LnLunjj98GWXm42//tRiWe7YLUmehkUj8NGGjtJzksTdPPpjU3ehoz+RwX5Hvq6FOzCmQRw6foOA6jqj8Uj9S1Mu5P3ARlyKXegkDj2oJGW62fYxCffHWlijPBC77oxreebkPT7KqSLmefr9Kk2eDhfKDgtM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com; spf=pass smtp.mailfrom=rivosinc.com; dkim=pass (2048-bit key) header.d=rivosinc-com.20230601.gappssmtp.com header.i=@rivosinc-com.20230601.gappssmtp.com header.b=U0R5hav+; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=rivosinc.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=rivosinc.com Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-1d746856d85so20821265ad.0 for ; Fri, 02 Feb 2024 15:54:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1706918097; x=1707522897; darn=vger.kernel.org; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:from:to:cc:subject:date:message-id :reply-to; bh=xunksNOigL1rXi22PNXZoQDfg/C2wRrV7gzfa+vwFao=; b=U0R5hav+AWzFHUU849BcG44sBQcsL1/gyOVUnLZ1SOr8a5UpKU4B32nDF+6krnxvcL rV2dOUDnuv9VqeG3Wx2olOBM0DHaWh65z4jNejRKDTVWkBJpty5QuLVC194a5jzLHLwY JfnoBEE5g3YXdMYHO1nT+A0Kybfk84wNT+TjYslAO2xP1fbcTSU6p6dDcbgFqtYCyfM+ tMuFu1Wyy2/9PvvOGWbgZkp9vjXA4Bywv0g/ySG733d6nM0ILi9z+wu2WxYgeVrjnD/j Qgkcl2WEm2Bu5FwAyMtrWc8WICGJXvYRTSecZXwv//Cy2zO/FRsYGicgJPTqv0qgMIbn GQFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706918097; x=1707522897; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :mime-version:subject:date:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xunksNOigL1rXi22PNXZoQDfg/C2wRrV7gzfa+vwFao=; b=otdrwFeAeRpEwLZUiRDBbFCyw6dzCc9nf8D8rTu2Cc2noywBSCVO8TGF/awr0NzbQv P1bbe6qbkU1/JuQrFu11ybSlM/xjEEp/RLJkraiHA9BOfqnIQj4uwwplDdH15Wm3v5nR drPjyVg2XOtiFCWkMkhRltdzIEl2EEIaSzNWx+4r/yueLAShnqzqtA/iietlVEj0qSlX gCR+FyQ9UpEDTPcIk2s0Sqwo55QB8jwt/y8mynt3Dyug9WM4feedU/dhVZJB+po7st8X DQhVBcZfVbhvcdRPX+lALXyVrf/PjkQH2M4LaaBwbcYxnOS9U+wD7tupoFtftPKM7B39 k2XQ== X-Gm-Message-State: AOJu0Yz2p7G+xlW9FWSAWAoJ5qsWtRwmmsCOlJUPxfXzJvtlTTyxDVpP RPpEfhgJAaV6CID1CtqZcmKYsUBPDs4bdSkMRxFO2nDytQVBRhocGdXpzDh7VNM= X-Received: by 2002:a17:903:1210:b0:1d8:debb:4125 with SMTP id l16-20020a170903121000b001d8debb4125mr3316154plh.38.1706918097358; Fri, 02 Feb 2024 15:54:57 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCUngsmC9mmyAjdvI5TTu9ggcNSXbmqX/s809OxaHO7G3/6T1jktG55f6M5eOHDC9iFwztIn/oqWXyXAPkpjpdMucEQN3HPjSkiZyZTbKwlqG6Rq3NirkiCKXKpa9sD0jeozME1MBNcshWKkmYZ9tZIUySIXk246uhabAM5cus1y9r2BjT9Kdljzu24iH8BmHlFMssm0Ge1tR+MDjULLEsHt7jgsQ3l7M2KrGPVvwd7Om44NSqvGi729GqqWFuaXvDa6Qouki+NyZO0Urt3xyFUTKA5uJzha2DDyQODaQTi/XD3+nGXqkTTxZayTF3GMNOt9CQr7Xqq3SA== Received: from charlie.ba.rivosinc.com ([64.71.180.162]) by smtp.gmail.com with ESMTPSA id u1-20020a17090282c100b001d8f81ece98sm2133607plz.104.2024.02.02.15.54.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 15:54:56 -0800 (PST) From: Charlie Jenkins Date: Fri, 02 Feb 2024 15:54:53 -0800 Subject: [PATCH v3 2/2] riscv: Disable misaligned access probe when CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20240202-disable_misaligned_probe_config-v3-2-c44f91f03bb6@rivosinc.com> References: <20240202-disable_misaligned_probe_config-v3-0-c44f91f03bb6@rivosinc.com> In-Reply-To: <20240202-disable_misaligned_probe_config-v3-0-c44f91f03bb6@rivosinc.com> To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Jisheng Zhang , Evan Green , =?utf-8?b?Q2zDqW1lbnQgTMOpZ2Vy?= , Eric Biggers , Elliot Berman Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Charlie Jenkins X-Mailer: b4 0.12.3 X-Developer-Signature: v=1; a=ed25519-sha256; t=1706918093; l=19916; i=charlie@rivosinc.com; s=20231120; h=from:subject:message-id; bh=vwhBmHfF/EKObUwb3qKoX49eRqG1MgvOSHuJsJTDdRo=; b=QBayGrWdc/19uAkWgBJuvk/KwxtyVs+VZzXjoNqgXRlxbhfDMcIcM//li0qGe6y6xyRNUJszt IDoFebO2PCBCiQ9tPXk2cBHc8ktUcbEg4p1nf96VFtUzqma0JiKZlSi X-Developer-Key: i=charlie@rivosinc.com; a=ed25519; pk=t4RSWpMV1q5lf/NWIeR9z58bcje60/dbtxxmoSfBEcs= X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1789833395121968746 X-GMAIL-MSGID: 1789833395121968746 When CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS is selected, the cpus can be set to have fast misaligned access without needing to probe. To avoid some ifdefs, move unalignment probing code into its own file and make CONFIG_RISCV_MISALIGNED depend on CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS. Signed-off-by: Charlie Jenkins Reviewed-by: Evan Green --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/cpufeature.h | 7 + arch/riscv/kernel/Makefile | 3 + arch/riscv/kernel/cpufeature.c | 255 -------------------------- arch/riscv/kernel/misaligned_access_speed.c | 265 ++++++++++++++++++++++++++++ arch/riscv/kernel/sys_hwprobe.c | 4 + 6 files changed, 280 insertions(+), 255 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index bffbd869a068..3223d2d08f74 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -690,6 +690,7 @@ config THREAD_SIZE_ORDER config RISCV_MISALIGNED bool "Support misaligned load/store traps for kernel and userspace" select SYSCTL_ARCH_UNALIGN_ALLOW + depends on !HAVE_EFFICIENT_UNALIGNED_ACCESS default y help Say Y here if you want the kernel to embed support for misaligned diff --git a/arch/riscv/include/asm/cpufeature.h b/arch/riscv/include/asm/cpufeature.h index eb3ac304fc42..44734e5169b1 100644 --- a/arch/riscv/include/asm/cpufeature.h +++ b/arch/riscv/include/asm/cpufeature.h @@ -51,6 +51,12 @@ static inline bool check_unaligned_access_emulated(int cpu) static inline void unaligned_emulation_finish(void) {} #endif +#ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS +static __always_inline bool has_fast_misaligned_accesses(void) +{ + return true; +} +#else DECLARE_PER_CPU(long, misaligned_access_speed); DECLARE_STATIC_KEY_FALSE(fast_misaligned_access_speed_key); @@ -59,6 +65,7 @@ static __always_inline bool has_fast_misaligned_accesses(void) { return static_branch_likely(&fast_misaligned_access_speed_key); } +#endif unsigned long riscv_get_elf_hwcap(void); diff --git a/arch/riscv/kernel/Makefile b/arch/riscv/kernel/Makefile index f71910718053..ffba5ecf12c2 100644 --- a/arch/riscv/kernel/Makefile +++ b/arch/riscv/kernel/Makefile @@ -62,6 +62,9 @@ obj-y += tests/ obj-$(CONFIG_MMU) += vdso.o vdso/ obj-$(CONFIG_RISCV_MISALIGNED) += traps_misaligned.o +ifneq ($(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS), y) +obj-y += misaligned_access_speed.o +endif obj-$(CONFIG_FPU) += fpu.o obj-$(CONFIG_RISCV_ISA_V) += vector.o obj-$(CONFIG_RISCV_ISA_V) += kernel_mode_vector.o diff --git a/arch/riscv/kernel/cpufeature.c b/arch/riscv/kernel/cpufeature.c index 89920f84d0a3..319670af5704 100644 --- a/arch/riscv/kernel/cpufeature.c +++ b/arch/riscv/kernel/cpufeature.c @@ -11,7 +11,6 @@ #include #include #include -#include #include #include #include @@ -21,20 +20,12 @@ #include #include #include -#include #include #include #include -#include "copy-unaligned.h" - #define NUM_ALPHA_EXTS ('z' - 'a' + 1) -#define MISALIGNED_ACCESS_JIFFIES_LG2 1 -#define MISALIGNED_BUFFER_SIZE 0x4000 -#define MISALIGNED_BUFFER_ORDER get_order(MISALIGNED_BUFFER_SIZE) -#define MISALIGNED_COPY_SIZE ((MISALIGNED_BUFFER_SIZE / 2) - 0x80) - unsigned long elf_hwcap __read_mostly; /* Host ISA bitmap */ @@ -43,11 +34,6 @@ static DECLARE_BITMAP(riscv_isa, RISCV_ISA_EXT_MAX) __read_mostly; /* Per-cpu ISA extensions. */ struct riscv_isainfo hart_isa[NR_CPUS]; -/* Performance information */ -DEFINE_PER_CPU(long, misaligned_access_speed); - -static cpumask_t fast_misaligned_access; - /** * riscv_isa_extension_base() - Get base extension word * @@ -706,247 +692,6 @@ unsigned long riscv_get_elf_hwcap(void) return hwcap; } -static int check_unaligned_access(void *param) -{ - int cpu = smp_processor_id(); - u64 start_cycles, end_cycles; - u64 word_cycles; - u64 byte_cycles; - int ratio; - unsigned long start_jiffies, now; - struct page *page = param; - void *dst; - void *src; - long speed = RISCV_HWPROBE_MISALIGNED_SLOW; - - if (check_unaligned_access_emulated(cpu)) - return 0; - - /* Make an unaligned destination buffer. */ - dst = (void *)((unsigned long)page_address(page) | 0x1); - /* Unalign src as well, but differently (off by 1 + 2 = 3). */ - src = dst + (MISALIGNED_BUFFER_SIZE / 2); - src += 2; - word_cycles = -1ULL; - /* Do a warmup. */ - __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); - preempt_disable(); - start_jiffies = jiffies; - while ((now = jiffies) == start_jiffies) - cpu_relax(); - - /* - * For a fixed amount of time, repeatedly try the function, and take - * the best time in cycles as the measurement. - */ - while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { - start_cycles = get_cycles64(); - /* Ensure the CSR read can't reorder WRT to the copy. */ - mb(); - __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); - /* Ensure the copy ends before the end time is snapped. */ - mb(); - end_cycles = get_cycles64(); - if ((end_cycles - start_cycles) < word_cycles) - word_cycles = end_cycles - start_cycles; - } - - byte_cycles = -1ULL; - __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); - start_jiffies = jiffies; - while ((now = jiffies) == start_jiffies) - cpu_relax(); - - while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { - start_cycles = get_cycles64(); - mb(); - __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); - mb(); - end_cycles = get_cycles64(); - if ((end_cycles - start_cycles) < byte_cycles) - byte_cycles = end_cycles - start_cycles; - } - - preempt_enable(); - - /* Don't divide by zero. */ - if (!word_cycles || !byte_cycles) { - pr_warn("cpu%d: rdtime lacks granularity needed to measure unaligned access speed\n", - cpu); - - return 0; - } - - if (word_cycles < byte_cycles) - speed = RISCV_HWPROBE_MISALIGNED_FAST; - - ratio = div_u64((byte_cycles * 100), word_cycles); - pr_info("cpu%d: Ratio of byte access time to unaligned word access is %d.%02d, unaligned accesses are %s\n", - cpu, - ratio / 100, - ratio % 100, - (speed == RISCV_HWPROBE_MISALIGNED_FAST) ? "fast" : "slow"); - - per_cpu(misaligned_access_speed, cpu) = speed; - - /* - * Set the value of fast_misaligned_access of a CPU. These operations - * are atomic to avoid race conditions. - */ - if (speed == RISCV_HWPROBE_MISALIGNED_FAST) - cpumask_set_cpu(cpu, &fast_misaligned_access); - else - cpumask_clear_cpu(cpu, &fast_misaligned_access); - - return 0; -} - -static void check_unaligned_access_nonboot_cpu(void *param) -{ - unsigned int cpu = smp_processor_id(); - struct page **pages = param; - - if (smp_processor_id() != 0) - check_unaligned_access(pages[cpu]); -} - -DEFINE_STATIC_KEY_FALSE(fast_misaligned_access_speed_key); - -static void modify_unaligned_access_branches(cpumask_t *mask, int weight) -{ - if (cpumask_weight(mask) == weight) - static_branch_enable_cpuslocked(&fast_misaligned_access_speed_key); - else - static_branch_disable_cpuslocked(&fast_misaligned_access_speed_key); -} - -static void set_unaligned_access_static_branches_except_cpu(int cpu) -{ - /* - * Same as set_unaligned_access_static_branches, except excludes the - * given CPU from the result. When a CPU is hotplugged into an offline - * state, this function is called before the CPU is set to offline in - * the cpumask, and thus the CPU needs to be explicitly excluded. - */ - - cpumask_t fast_except_me; - - cpumask_and(&fast_except_me, &fast_misaligned_access, cpu_online_mask); - cpumask_clear_cpu(cpu, &fast_except_me); - - modify_unaligned_access_branches(&fast_except_me, num_online_cpus() - 1); -} - -static void set_unaligned_access_static_branches(void) -{ - /* - * This will be called after check_unaligned_access_all_cpus so the - * result of unaligned access speed for all CPUs will be available. - * - * To avoid the number of online cpus changing between reading - * cpu_online_mask and calling num_online_cpus, cpus_read_lock must be - * held before calling this function. - */ - - cpumask_t fast_and_online; - - cpumask_and(&fast_and_online, &fast_misaligned_access, cpu_online_mask); - - modify_unaligned_access_branches(&fast_and_online, num_online_cpus()); -} - -static int lock_and_set_unaligned_access_static_branch(void) -{ - cpus_read_lock(); - set_unaligned_access_static_branches(); - cpus_read_unlock(); - - return 0; -} - -arch_initcall_sync(lock_and_set_unaligned_access_static_branch); - -static int riscv_online_cpu(unsigned int cpu) -{ - static struct page *buf; - - /* We are already set since the last check */ - if (per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_UNKNOWN) - goto exit; - - buf = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); - if (!buf) { - pr_warn("Allocation failure, not measuring misaligned performance\n"); - return -ENOMEM; - } - - check_unaligned_access(buf); - __free_pages(buf, MISALIGNED_BUFFER_ORDER); - -exit: - set_unaligned_access_static_branches(); - - return 0; -} - -static int riscv_offline_cpu(unsigned int cpu) -{ - set_unaligned_access_static_branches_except_cpu(cpu); - - return 0; -} - -/* Measure unaligned access on all CPUs present at boot in parallel. */ -static int check_unaligned_access_all_cpus(void) -{ - unsigned int cpu; - unsigned int cpu_count = num_possible_cpus(); - struct page **bufs = kzalloc(cpu_count * sizeof(struct page *), - GFP_KERNEL); - - if (!bufs) { - pr_warn("Allocation failure, not measuring misaligned performance\n"); - return 0; - } - - /* - * Allocate separate buffers for each CPU so there's no fighting over - * cache lines. - */ - for_each_cpu(cpu, cpu_online_mask) { - bufs[cpu] = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); - if (!bufs[cpu]) { - pr_warn("Allocation failure, not measuring misaligned performance\n"); - goto out; - } - } - - /* Check everybody except 0, who stays behind to tend jiffies. */ - on_each_cpu(check_unaligned_access_nonboot_cpu, bufs, 1); - - /* Check core 0. */ - smp_call_on_cpu(0, check_unaligned_access, bufs[0], true); - - /* - * Setup hotplug callbacks for any new CPUs that come online or go - * offline. - */ - cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "riscv:online", - riscv_online_cpu, riscv_offline_cpu); - -out: - unaligned_emulation_finish(); - for_each_cpu(cpu, cpu_online_mask) { - if (bufs[cpu]) - __free_pages(bufs[cpu], MISALIGNED_BUFFER_ORDER); - } - - kfree(bufs); - return 0; -} - -arch_initcall(check_unaligned_access_all_cpus); - void riscv_user_isa_enable(void) { if (riscv_cpu_has_extension_unlikely(smp_processor_id(), RISCV_ISA_EXT_ZICBOZ)) diff --git a/arch/riscv/kernel/misaligned_access_speed.c b/arch/riscv/kernel/misaligned_access_speed.c new file mode 100644 index 000000000000..b725c07dd1af --- /dev/null +++ b/arch/riscv/kernel/misaligned_access_speed.c @@ -0,0 +1,265 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright 2024 Rivos Inc. + */ + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "copy-unaligned.h" + +#define MISALIGNED_ACCESS_JIFFIES_LG2 1 +#define MISALIGNED_BUFFER_SIZE 0x4000 +#define MISALIGNED_BUFFER_ORDER get_order(MISALIGNED_BUFFER_SIZE) +#define MISALIGNED_COPY_SIZE ((MISALIGNED_BUFFER_SIZE / 2) - 0x80) + +DEFINE_PER_CPU(long, misaligned_access_speed); + +static cpumask_t fast_misaligned_access; + +static int check_unaligned_access(void *param) +{ + int cpu = smp_processor_id(); + u64 start_cycles, end_cycles; + u64 word_cycles; + u64 byte_cycles; + int ratio; + unsigned long start_jiffies, now; + struct page *page = param; + void *dst; + void *src; + long speed = RISCV_HWPROBE_MISALIGNED_SLOW; + + if (check_unaligned_access_emulated(cpu)) + return 0; + + /* Make an unaligned destination buffer. */ + dst = (void *)((unsigned long)page_address(page) | 0x1); + /* Unalign src as well, but differently (off by 1 + 2 = 3). */ + src = dst + (MISALIGNED_BUFFER_SIZE / 2); + src += 2; + word_cycles = -1ULL; + /* Do a warmup. */ + __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); + preempt_disable(); + start_jiffies = jiffies; + while ((now = jiffies) == start_jiffies) + cpu_relax(); + + /* + * For a fixed amount of time, repeatedly try the function, and take + * the best time in cycles as the measurement. + */ + while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { + start_cycles = get_cycles64(); + /* Ensure the CSR read can't reorder WRT to the copy. */ + mb(); + __riscv_copy_words_unaligned(dst, src, MISALIGNED_COPY_SIZE); + /* Ensure the copy ends before the end time is snapped. */ + mb(); + end_cycles = get_cycles64(); + if ((end_cycles - start_cycles) < word_cycles) + word_cycles = end_cycles - start_cycles; + } + + byte_cycles = -1ULL; + __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); + start_jiffies = jiffies; + while ((now = jiffies) == start_jiffies) + cpu_relax(); + + while (time_before(jiffies, now + (1 << MISALIGNED_ACCESS_JIFFIES_LG2))) { + start_cycles = get_cycles64(); + mb(); + __riscv_copy_bytes_unaligned(dst, src, MISALIGNED_COPY_SIZE); + mb(); + end_cycles = get_cycles64(); + if ((end_cycles - start_cycles) < byte_cycles) + byte_cycles = end_cycles - start_cycles; + } + + preempt_enable(); + + /* Don't divide by zero. */ + if (!word_cycles || !byte_cycles) { + pr_warn("cpu%d: rdtime lacks granularity needed to measure unaligned access speed\n", + cpu); + + return 0; + } + + if (word_cycles < byte_cycles) + speed = RISCV_HWPROBE_MISALIGNED_FAST; + + ratio = div_u64((byte_cycles * 100), word_cycles); + pr_info("cpu%d: Ratio of byte access time to unaligned word access is %d.%02d, unaligned accesses are %s\n", + cpu, + ratio / 100, + ratio % 100, + (speed == RISCV_HWPROBE_MISALIGNED_FAST) ? "fast" : "slow"); + + per_cpu(misaligned_access_speed, cpu) = speed; + + /* + * Set the value of fast_misaligned_access of a CPU. These operations + * are atomic to avoid race conditions. + */ + if (speed == RISCV_HWPROBE_MISALIGNED_FAST) + cpumask_set_cpu(cpu, &fast_misaligned_access); + else + cpumask_clear_cpu(cpu, &fast_misaligned_access); + + return 0; +} + +static void check_unaligned_access_nonboot_cpu(void *param) +{ + unsigned int cpu = smp_processor_id(); + struct page **pages = param; + + if (smp_processor_id() != 0) + check_unaligned_access(pages[cpu]); +} + +DEFINE_STATIC_KEY_FALSE(fast_misaligned_access_speed_key); + +static void modify_unaligned_access_branches(cpumask_t *mask, int weight) +{ + if (cpumask_weight(mask) == weight) + static_branch_enable_cpuslocked(&fast_misaligned_access_speed_key); + else + static_branch_disable_cpuslocked(&fast_misaligned_access_speed_key); +} + +static void set_unaligned_access_static_branches_except_cpu(int cpu) +{ + /* + * Same as set_unaligned_access_static_branches, except excludes the + * given CPU from the result. When a CPU is hotplugged into an offline + * state, this function is called before the CPU is set to offline in + * the cpumask, and thus the CPU needs to be explicitly excluded. + */ + + cpumask_t fast_except_me; + + cpumask_and(&fast_except_me, &fast_misaligned_access, cpu_online_mask); + cpumask_clear_cpu(cpu, &fast_except_me); + + modify_unaligned_access_branches(&fast_except_me, num_online_cpus() - 1); +} + +static void set_unaligned_access_static_branches(void) +{ + /* + * This will be called after check_unaligned_access_all_cpus so the + * result of unaligned access speed for all CPUs will be available. + * + * To avoid the number of online cpus changing between reading + * cpu_online_mask and calling num_online_cpus, cpus_read_lock must be + * held before calling this function. + */ + + cpumask_t fast_and_online; + + cpumask_and(&fast_and_online, &fast_misaligned_access, cpu_online_mask); + + modify_unaligned_access_branches(&fast_and_online, num_online_cpus()); +} + +static int lock_and_set_unaligned_access_static_branch(void) +{ + cpus_read_lock(); + set_unaligned_access_static_branches(); + cpus_read_unlock(); + + return 0; +} + +arch_initcall_sync(lock_and_set_unaligned_access_static_branch); + +static int riscv_online_cpu(unsigned int cpu) +{ + static struct page *buf; + + /* We are already set since the last check */ + if (per_cpu(misaligned_access_speed, cpu) != RISCV_HWPROBE_MISALIGNED_UNKNOWN) + goto exit; + + buf = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); + if (!buf) { + pr_warn("Allocation failure, not measuring misaligned performance\n"); + return -ENOMEM; + } + + check_unaligned_access(buf); + __free_pages(buf, MISALIGNED_BUFFER_ORDER); + +exit: + set_unaligned_access_static_branches(); + + return 0; +} + +static int riscv_offline_cpu(unsigned int cpu) +{ + set_unaligned_access_static_branches_except_cpu(cpu); + + return 0; +} + +/* Measure unaligned access on all CPUs present at boot in parallel. */ +static int check_unaligned_access_all_cpus(void) +{ + unsigned int cpu; + unsigned int cpu_count = num_possible_cpus(); + struct page **bufs = kzalloc(cpu_count * sizeof(struct page *), + GFP_KERNEL); + + if (!bufs) { + pr_warn("Allocation failure, not measuring misaligned performance\n"); + return 0; + } + + /* + * Allocate separate buffers for each CPU so there's no fighting over + * cache lines. + */ + for_each_cpu(cpu, cpu_online_mask) { + bufs[cpu] = alloc_pages(GFP_KERNEL, MISALIGNED_BUFFER_ORDER); + if (!bufs[cpu]) { + pr_warn("Allocation failure, not measuring misaligned performance\n"); + goto out; + } + } + + /* Check everybody except 0, who stays behind to tend jiffies. */ + on_each_cpu(check_unaligned_access_nonboot_cpu, bufs, 1); + + /* Check core 0. */ + smp_call_on_cpu(0, check_unaligned_access, bufs[0], true); + + /* + * Setup hotplug callbacks for any new CPUs that come online or go + * offline. + */ + cpuhp_setup_state_nocalls(CPUHP_AP_ONLINE_DYN, "riscv:online", + riscv_online_cpu, riscv_offline_cpu); + +out: + unaligned_emulation_finish(); + for_each_cpu(cpu, cpu_online_mask) { + if (bufs[cpu]) + __free_pages(bufs[cpu], MISALIGNED_BUFFER_ORDER); + } + + kfree(bufs); + return 0; +} + +arch_initcall(check_unaligned_access_all_cpus); diff --git a/arch/riscv/kernel/sys_hwprobe.c b/arch/riscv/kernel/sys_hwprobe.c index a7c56b41efd2..d9bd24776a3e 100644 --- a/arch/riscv/kernel/sys_hwprobe.c +++ b/arch/riscv/kernel/sys_hwprobe.c @@ -149,6 +149,9 @@ static bool hwprobe_ext0_has(const struct cpumask *cpus, unsigned long ext) static u64 hwprobe_misaligned(const struct cpumask *cpus) { +#ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS + return RISCV_HWPROBE_MISALIGNED_FAST; +#else int cpu; u64 perf = -1ULL; @@ -168,6 +171,7 @@ static u64 hwprobe_misaligned(const struct cpumask *cpus) return RISCV_HWPROBE_MISALIGNED_UNKNOWN; return perf; +#endif } static void hwprobe_one_pair(struct riscv_hwprobe *pair,