From patchwork Mon Dec 19 06:40:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yuan, Perry" X-Patchwork-Id: 34413 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:adf:e747:0:0:0:0:0 with SMTP id c7csp2229882wrn; Sun, 18 Dec 2022 22:44:00 -0800 (PST) X-Google-Smtp-Source: AA0mqf5xRtf5/8XQWgvKtNfBMptFqRxCjG3YzKI/6/ka+43RUBLkBYvz6Un1FrDyP282nHP8Ti6B X-Received: by 2002:a05:6a20:320e:b0:aa:7548:883d with SMTP id y14-20020a056a20320e00b000aa7548883dmr72875979pzc.11.1671432240028; Sun, 18 Dec 2022 22:44:00 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1671432240; cv=pass; d=google.com; s=arc-20160816; b=qtFVzJp92V3LTTdjmcVlOft+7XOieTSZKPBkMHpWmD97ehMa8GfVjsz3zr4RuG//F5 PsF7nWfGYf+XxmD8DA/67P3RWCASGzIh2P89WmyO5+5foTsDqn1TXDwj9H0fh/gNvIa/ 4k90QxSS5W48ZQfP2SSgRmMuANLBpJ2TIZiMTWKZMnGO3lg5I47g2UsOY0mvi1jwiS19 XDRC7w7m1PffvGaDb0tzQFIhAGyj0YM50stx0NKLhemUURP8EOPShWaSsguHehBcUfuH raOO5CcQl8d72HJEon6GjRerC0/bqsBg7oiIiWOBmH8u4g5ywPOy397uxy3LRpywUdxs OXKg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=mmPJhrVa+osWNqnUzgjy+W/3VTg+emW+vxV3duJrg6M=; b=H32APF/eg/LD7Lwsbz178ke0fypA3gSs3uM50u8X+kwCYCSYvFu+1HpiES+hw8VZPI NACQWdkHD/Uc7YUkwYrIZvPieVI9j1X+0wymJIpkap68AkVGDtYMlD27Bj7Q8KU6WHuK WmTdfHdKhtQyboyt70Ni6jd92utSvGUuJ8fyJPZyCuOAhnpKsBLGD4BeVmIAxfox203B 7daQMt6LRGz9RUHUo7L4CWqrtG/hL41/rRTvTy6hgD0+cVaBHpjhUilOALxYgykGuIUr DJGdXhi2se2UKOxljeEFyVhLCiezBfBtLaJhR78C5sncfuy8QUW9YuAy7JK9O8iUXp4I UFrw== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=IMUIAK3X; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id q22-20020a056a00089600b0053e9c7939basi9585064pfj.188.2022.12.18.22.43.47; Sun, 18 Dec 2022 22:44:00 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=IMUIAK3X; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231543AbiLSGmq (ORCPT + 99 others); Mon, 19 Dec 2022 01:42:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231421AbiLSGmQ (ORCPT ); Mon, 19 Dec 2022 01:42:16 -0500 Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2078.outbound.protection.outlook.com [40.107.244.78]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2DE24D2C4; Sun, 18 Dec 2022 22:41:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=d4yIhARclZfNJ/ONWH+Y5o8eLP9GJD7WFkyAzYxAcmAclva0NL9r7uUSjQNIu6oECCVRmwqG2axsR74X1xcbLLfnasVGebZ31uB76CAnSZxXiu3Bfcujl3i4qjdk3Mm2rr9v5QGP2qReVQOzC412VtdB7m1V9Nc0UxZn/BAgByU1kriK1LFIw9v4vD7G16tdS4c6RmBJg5E4RLkY13/keHoZGEExOSQpvJcTmfEiMVd+NzBEKUFKPO9EOWtuumDRtFi44w3xpsvJ60ALq+nUNKXrWjruv4foc0T+StrejenT+/M/jpzp3J4ih5cPX0uCnwwaSMOW9WypXTNmxFPWlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mmPJhrVa+osWNqnUzgjy+W/3VTg+emW+vxV3duJrg6M=; b=ax8BS4jyauo9O9O1uIzdcUPxrbFgo4WUwyVr7j3PmhwCw76yxrlRlu9R3tXHF+BEdKBUQ/BLwMybjYkwOz0P+oUdY5uqEXT/mVdZkhq1Nq/BEGKZq2/dOEDHWMSQlklOYSWy0jwEJhfn5kP8JsKUG0QUTZvuvqfwEUGdZ2n6bhLJ6xidH6ljDmEnFzitys5xHZG/K91Dxb4VT28NxPVkiewv/RW9hDrG015iHzThNFxotHic82NWXkPPB4ZMhdMUrNdes6zmbInxmwDhZ79G0J9cz7YWl3VEnEFp6LSTX1LROnpD5IepaYo5eAPsqT7tLLkkF0dXai5zuRjFYqwAAg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=intel.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mmPJhrVa+osWNqnUzgjy+W/3VTg+emW+vxV3duJrg6M=; b=IMUIAK3XTtQZwmEnBW1HxaPdoSaGOWo50ASdU4oyb86t7qg98qhtQqkso6o1xDHFv+l8tzk9dX2gcl6CO2js8PdpN853TOerlTrw3+OSivTLVYiav2J7sXQWlCWDUPTHhSz97RGKl+4UxmtXUhQFmI4rxtzbi1qhr0DNYeT7YW4= Received: from DM6PR21CA0013.namprd21.prod.outlook.com (2603:10b6:5:174::23) by CH0PR12MB5252.namprd12.prod.outlook.com (2603:10b6:610:d3::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5924.16; Mon, 19 Dec 2022 06:41:44 +0000 Received: from DM6NAM11FT098.eop-nam11.prod.protection.outlook.com (2603:10b6:5:174:cafe::6c) by DM6PR21CA0013.outlook.office365.com (2603:10b6:5:174::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5944.9 via Frontend Transport; Mon, 19 Dec 2022 06:41:44 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by DM6NAM11FT098.mail.protection.outlook.com (10.13.173.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.5924.16 via Frontend Transport; Mon, 19 Dec 2022 06:41:44 +0000 Received: from pyuan-Cloudripper.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.34; Mon, 19 Dec 2022 00:41:21 -0600 From: Perry Yuan To: , , , CC: , , , , , , , , Subject: [PATCH v8 06/13] cpufreq: amd-pstate: implement Pstate EPP support for the AMD processors Date: Mon, 19 Dec 2022 14:40:35 +0800 Message-ID: <20221219064042.661122-7-perry.yuan@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20221219064042.661122-1-perry.yuan@amd.com> References: <20221219064042.661122-1-perry.yuan@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM6NAM11FT098:EE_|CH0PR12MB5252:EE_ X-MS-Office365-Filtering-Correlation-Id: 3199ee32-38ed-48fc-be9e-08dae18c182e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: /It6wlW0oamWDOkIxnctFgaW0lRDKHL15ZItFAFCxJGot7MxZAdthOPHNdMQPOA+R9hwKLN1ekghqa4Ks54hh0xGMGtrjKgpPyewOH0aJSoqFeUoEQA0hLB/26FkDFxk3Ha7LoJW/XLasItG5IQSq6+0ji784xKYg+y9SYJFzdTyVxqMeE7y3t/N8bDrrqQabyOc+KJWVt7A8gOt62w8MblktdPw2VPciWjaHaoR0dz8y/lo6PYzijU45e2rb8t6RySe8pdX3nyhtlolzhfXYqy/ph/H80O7PlByGyXdfuCNGgqhWV9Q+P3658lzh9OQtGOZfoZe5lXPsuEGDEOQ13/hG8jmD6rdT++P4pQoG6QqHI0BiZX2PFwwbmVdF6TSS+Ldipoh4nUPwTcQkaai9MZwgGOMMMghASH6Uv7Q+4uhefLe+dTrpaktvrTL7KJsMy3/t+aGuAF/fSGmGErnqTd1L/oV5GBKJlR47s8mpa+9jR2MIcTZ5xr4O8H8fs0FQtoBkVYl7cCcPro8gwyqzb4ax/O+5uQ23nq/BwqbR99pBIvT/mCF9uaC6OyleU9AQjBa3wFzf1MZHniEKl7F5c8CbxMoZVWth7px5Qnoz9yJeM805D/kMdy7Vzc5X/nQtNZMLzrpp0NWfqL7l4VeRjTgU9jqVaolh0jIJKQudiOSVD/xxTjQrJxRbMazb9DaADI6xxV6bB9RwpXv7MMsE0EA9pBdvdb2q5kdYl3vSbw= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230022)(4636009)(396003)(136003)(376002)(346002)(39860400002)(451199015)(40470700004)(46966006)(36840700001)(40480700001)(2906002)(82310400005)(16526019)(316002)(40460700003)(4326008)(1076003)(70206006)(8676002)(70586007)(54906003)(110136005)(336012)(7696005)(2616005)(86362001)(36860700001)(36756003)(478600001)(5660300002)(81166007)(83380400001)(6666004)(26005)(8936002)(426003)(47076005)(82740400003)(41300700001)(186003)(356005)(44832011)(30864003)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Dec 2022 06:41:44.2952 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 3199ee32-38ed-48fc-be9e-08dae18c182e X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT098.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH0PR12MB5252 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,SPF_HELO_PASS,SPF_PASS autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1752623732308173420?= X-GMAIL-MSGID: =?utf-8?q?1752623732308173420?= From: Perry Yuan Add EPP driver support for AMD SoCs which support a dedicated MSR for CPPC. EPP is used by the DPM controller to configure the frequency that a core operates at during short periods of activity. The SoC EPP targets are configured on a scale from 0 to 255 where 0 represents maximum performance and 255 represents maximum efficiency. The amd-pstate driver exports profile string names to userspace that are tied to specific EPP values. The balance_performance string (0x80) provides the best balance for efficiency versus power on most systems, but users can choose other strings to meet their needs as well. $ cat /sys/devices/system/cpu/cpufreq/policy0/energy_performance_available_preferences default performance balance_performance balance_power power $ cat /sys/devices/system/cpu/cpufreq/policy0/energy_performance_preference balance_performance To enable the driver,it needs to add `amd_pstate=active` to kernel command line and kernel will load the active mode epp driver Signed-off-by: Perry Yuan --- drivers/cpufreq/amd-pstate.c | 447 ++++++++++++++++++++++++++++++++++- include/linux/amd-pstate.h | 10 + 2 files changed, 451 insertions(+), 6 deletions(-) diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c index 861a905f9324..66b39457a312 100644 --- a/drivers/cpufreq/amd-pstate.c +++ b/drivers/cpufreq/amd-pstate.c @@ -59,7 +59,10 @@ * we disable it by default to go acpi-cpufreq on these processors and add a * module parameter to be able to enable it manually for debugging. */ +static struct cpufreq_driver *default_pstate_driver; static struct cpufreq_driver amd_pstate_driver; +static struct cpufreq_driver amd_pstate_epp_driver; +static struct amd_cpudata **all_cpu_data; static int cppc_state = AMD_PSTATE_DISABLE; static inline int get_mode_idx_from_str(const char *str, size_t size) @@ -70,9 +73,128 @@ static inline int get_mode_idx_from_str(const char *str, size_t size) if (!strncmp(str, amd_pstate_mode_string[i], size)) return i; } + return -EINVAL; } +/** + * struct amd_pstate_params - global parameters for the performance control + * @ cppc_boost_disabled wheher the core performance boost disabled + */ +struct amd_pstate_params { + bool cppc_boost_disabled; +}; + +static struct amd_pstate_params global_params; + +static DEFINE_MUTEX(amd_pstate_limits_lock); +static DEFINE_MUTEX(amd_pstate_driver_lock); + +static s16 amd_pstate_get_epp(struct amd_cpudata *cpudata, u64 cppc_req_cached) +{ + u64 epp; + int ret; + + if (boot_cpu_has(X86_FEATURE_CPPC)) { + if (!cppc_req_cached) { + epp = rdmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, + &cppc_req_cached); + if (epp) + return epp; + } + epp = (cppc_req_cached >> 24) & 0xFF; + } else { + ret = cppc_get_epp_perf(cpudata->cpu, &epp); + if (ret < 0) { + pr_debug("Could not retrieve energy perf value (%d)\n", ret); + return -EIO; + } + } + + return (s16)(epp & 0xff); +} + +static int amd_pstate_get_energy_pref_index(struct amd_cpudata *cpudata) +{ + s16 epp; + int index = -EINVAL; + + epp = amd_pstate_get_epp(cpudata, 0); + if (epp < 0) + return epp; + + switch (epp) { + case HWP_EPP_PERFORMANCE: + index = EPP_INDEX_PERFORMANCE; + break; + case HWP_EPP_BALANCE_PERFORMANCE: + index = EPP_INDEX_BALANCE_PERFORMANCE; + break; + case HWP_EPP_BALANCE_POWERSAVE: + index = EPP_INDEX_BALANCE_POWERSAVE; + break; + case HWP_EPP_POWERSAVE: + index = EPP_INDEX_POWERSAVE; + break; + default: + break; + } + + return index; +} + +static int amd_pstate_set_epp(struct amd_cpudata *cpudata, u32 epp) +{ + int ret; + struct cppc_perf_ctrls perf_ctrls; + + if (boot_cpu_has(X86_FEATURE_CPPC)) { + u64 value = READ_ONCE(cpudata->cppc_req_cached); + + value &= ~GENMASK_ULL(31, 24); + value |= (u64)epp << 24; + WRITE_ONCE(cpudata->cppc_req_cached, value); + + ret = wrmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, value); + if (!ret) + cpudata->epp_cached = epp; + } else { + perf_ctrls.energy_perf = epp; + ret = cppc_set_epp_perf(cpudata->cpu, &perf_ctrls, 1); + if (ret) { + pr_debug("failed to set energy perf value (%d)\n", ret); + return ret; + } + cpudata->epp_cached = epp; + } + + return ret; +} + +static int amd_pstate_set_energy_pref_index(struct amd_cpudata *cpudata, + int pref_index) +{ + int epp = -EINVAL; + int ret; + + if (!pref_index) { + pr_debug("EPP pref_index is invalid\n"); + return -EINVAL; + } + + if (epp == -EINVAL) + epp = epp_values[pref_index]; + + if (epp > 0 && cpudata->policy == CPUFREQ_POLICY_PERFORMANCE) { + pr_debug("EPP cannot be set under performance policy\n"); + return -EBUSY; + } + + ret = amd_pstate_set_epp(cpudata, epp); + + return ret; +} + static inline int pstate_enable(bool enable) { return wrmsrl_safe(MSR_AMD_CPPC_ENABLE, enable); @@ -81,11 +203,21 @@ static inline int pstate_enable(bool enable) static int cppc_enable(bool enable) { int cpu, ret = 0; + struct cppc_perf_ctrls perf_ctrls; for_each_present_cpu(cpu) { ret = cppc_set_enable(cpu, enable); if (ret) return ret; + + /* Enable autonomous mode for EPP */ + if (cppc_state == AMD_PSTATE_ACTIVE) { + /* Set desired perf as zero to allow EPP firmware control */ + perf_ctrls.desired_perf = 0; + ret = cppc_set_perf(cpu, &perf_ctrls); + if (ret) + return ret; + } } return ret; @@ -429,7 +561,7 @@ static void amd_pstate_boost_init(struct amd_cpudata *cpudata) return; cpudata->boost_supported = true; - amd_pstate_driver.boost_enabled = true; + default_pstate_driver->boost_enabled = true; } static void amd_perf_ctl_reset(unsigned int cpu) @@ -603,10 +735,61 @@ static ssize_t show_amd_pstate_highest_perf(struct cpufreq_policy *policy, return sprintf(&buf[0], "%u\n", perf); } +static ssize_t show_energy_performance_available_preferences( + struct cpufreq_policy *policy, char *buf) +{ + int i = 0; + int offset = 0; + + while (energy_perf_strings[i] != NULL) + offset += sysfs_emit_at(buf, offset, "%s ", energy_perf_strings[i++]); + + sysfs_emit_at(buf, offset, "\n"); + + return offset; +} + +static ssize_t store_energy_performance_preference( + struct cpufreq_policy *policy, const char *buf, size_t count) +{ + struct amd_cpudata *cpudata = policy->driver_data; + char str_preference[21]; + ssize_t ret; + + ret = sscanf(buf, "%20s", str_preference); + if (ret != 1) + return -EINVAL; + + ret = match_string(energy_perf_strings, -1, str_preference); + if (ret < 0) + return -EINVAL; + + mutex_lock(&amd_pstate_limits_lock); + ret = amd_pstate_set_energy_pref_index(cpudata, ret); + mutex_unlock(&amd_pstate_limits_lock); + + return ret ?: count; +} + +static ssize_t show_energy_performance_preference( + struct cpufreq_policy *policy, char *buf) +{ + struct amd_cpudata *cpudata = policy->driver_data; + int preference; + + preference = amd_pstate_get_energy_pref_index(cpudata); + if (preference < 0) + return preference; + + return sysfs_emit(buf, "%s\n", energy_perf_strings[preference]); +} + cpufreq_freq_attr_ro(amd_pstate_max_freq); cpufreq_freq_attr_ro(amd_pstate_lowest_nonlinear_freq); cpufreq_freq_attr_ro(amd_pstate_highest_perf); +cpufreq_freq_attr_rw(energy_performance_preference); +cpufreq_freq_attr_ro(energy_performance_available_preferences); static struct freq_attr *amd_pstate_attr[] = { &amd_pstate_max_freq, @@ -615,6 +798,235 @@ static struct freq_attr *amd_pstate_attr[] = { NULL, }; +static struct freq_attr *amd_pstate_epp_attr[] = { + &amd_pstate_max_freq, + &amd_pstate_lowest_nonlinear_freq, + &amd_pstate_highest_perf, + &energy_performance_preference, + &energy_performance_available_preferences, + NULL, +}; + +static inline void update_boost_state(void) +{ + u64 misc_en; + struct amd_cpudata *cpudata; + + cpudata = all_cpu_data[0]; + rdmsrl(MSR_K7_HWCR, misc_en); + global_params.cppc_boost_disabled = misc_en & BIT_ULL(25); +} + +static int amd_pstate_init_cpu(unsigned int cpunum) +{ + struct amd_cpudata *cpudata; + + cpudata = all_cpu_data[cpunum]; + if (!cpudata) { + cpudata = kzalloc(sizeof(*cpudata), GFP_KERNEL); + if (!cpudata) + return -ENOMEM; + WRITE_ONCE(all_cpu_data[cpunum], cpudata); + + cpudata->cpu = cpunum; + } + + cpudata->epp_policy = 0; + pr_debug("controlling: cpu %d\n", cpunum); + return 0; +} + +static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy) +{ + int min_freq, max_freq, nominal_freq, lowest_nonlinear_freq, ret; + struct amd_cpudata *cpudata; + struct device *dev; + int rc; + u64 value; + + rc = amd_pstate_init_cpu(policy->cpu); + if (rc) + return rc; + + cpudata = all_cpu_data[policy->cpu]; + + dev = get_cpu_device(policy->cpu); + if (!dev) + goto free_cpudata1; + + rc = amd_pstate_init_perf(cpudata); + if (rc) + goto free_cpudata1; + + min_freq = amd_get_min_freq(cpudata); + max_freq = amd_get_max_freq(cpudata); + nominal_freq = amd_get_nominal_freq(cpudata); + lowest_nonlinear_freq = amd_get_lowest_nonlinear_freq(cpudata); + if (min_freq < 0 || max_freq < 0 || min_freq > max_freq) { + dev_err(dev, "min_freq(%d) or max_freq(%d) value is incorrect\n", + min_freq, max_freq); + ret = -EINVAL; + goto free_cpudata1; + } + + policy->min = min_freq; + policy->max = max_freq; + + policy->cpuinfo.min_freq = min_freq; + policy->cpuinfo.max_freq = max_freq; + /* It will be updated by governor */ + policy->cur = policy->cpuinfo.min_freq; + + /* Initial processor data capability frequencies */ + cpudata->max_freq = max_freq; + cpudata->min_freq = min_freq; + cpudata->nominal_freq = nominal_freq; + cpudata->lowest_nonlinear_freq = lowest_nonlinear_freq; + + policy->driver_data = cpudata; + + cpudata->epp_cached = amd_pstate_get_epp(cpudata, value); + + policy->min = policy->cpuinfo.min_freq; + policy->max = policy->cpuinfo.max_freq; + + /* + * Set the policy to powersave to provide a valid fallback value in case + * the default cpufreq governor is neither powersave nor performance. + */ + policy->policy = CPUFREQ_POLICY_POWERSAVE; + + if (boot_cpu_has(X86_FEATURE_CPPC)) { + policy->fast_switch_possible = true; + ret = rdmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_REQ, &value); + if (ret) + return ret; + WRITE_ONCE(cpudata->cppc_req_cached, value); + + ret = rdmsrl_on_cpu(cpudata->cpu, MSR_AMD_CPPC_CAP1, &value); + if (ret) + return ret; + WRITE_ONCE(cpudata->cppc_cap1_cached, value); + } + amd_pstate_boost_init(cpudata); + + return 0; + +free_cpudata1: + kfree(cpudata); + return ret; +} + +static int amd_pstate_epp_cpu_exit(struct cpufreq_policy *policy) +{ + pr_debug("CPU %d exiting\n", policy->cpu); + policy->fast_switch_possible = false; + return 0; +} + +static void amd_pstate_update_max_freq(unsigned int cpu) +{ + struct cpufreq_policy *policy = cpufreq_cpu_get(cpu); + + if (!policy) + return; + + refresh_frequency_limits(policy); + cpufreq_cpu_put(policy); +} + +static void amd_pstate_epp_update_limits(unsigned int cpu) +{ + mutex_lock(&amd_pstate_driver_lock); + update_boost_state(); + if (global_params.cppc_boost_disabled) { + for_each_possible_cpu(cpu) + amd_pstate_update_max_freq(cpu); + } else { + cpufreq_update_policy(cpu); + } + mutex_unlock(&amd_pstate_driver_lock); +} + +static void amd_pstate_epp_init(unsigned int cpu) +{ + struct amd_cpudata *cpudata = all_cpu_data[cpu]; + u32 max_perf, min_perf; + u64 value; + s16 epp; + + max_perf = READ_ONCE(cpudata->highest_perf); + min_perf = READ_ONCE(cpudata->lowest_perf); + + value = READ_ONCE(cpudata->cppc_req_cached); + + if (cpudata->policy == CPUFREQ_POLICY_PERFORMANCE) + min_perf = max_perf; + + /* Initial min/max values for CPPC Performance Controls Register */ + value &= ~AMD_CPPC_MIN_PERF(~0L); + value |= AMD_CPPC_MIN_PERF(min_perf); + + value &= ~AMD_CPPC_MAX_PERF(~0L); + value |= AMD_CPPC_MAX_PERF(max_perf); + + /* CPPC EPP feature require to set zero to the desire perf bit */ + value &= ~AMD_CPPC_DES_PERF(~0L); + value |= AMD_CPPC_DES_PERF(0); + + if (cpudata->epp_policy == cpudata->policy) + goto skip_epp; + + cpudata->epp_policy = cpudata->policy; + + if (cpudata->policy == CPUFREQ_POLICY_PERFORMANCE) { + epp = amd_pstate_get_epp(cpudata, value); + if (epp < 0) + goto skip_epp; + /* force the epp value to be zero for performance policy */ + epp = 0; + } else { + /* Get BIOS pre-defined epp value */ + epp = amd_pstate_get_epp(cpudata, value); + if (epp) + goto skip_epp; + } + /* Set initial EPP value */ + if (boot_cpu_has(X86_FEATURE_CPPC)) { + value &= ~GENMASK_ULL(31, 24); + value |= (u64)epp << 24; + } + +skip_epp: + WRITE_ONCE(cpudata->cppc_req_cached, value); + amd_pstate_set_epp(cpudata, epp); +} + +static int amd_pstate_epp_set_policy(struct cpufreq_policy *policy) +{ + struct amd_cpudata *cpudata; + + if (!policy->cpuinfo.max_freq) + return -ENODEV; + + pr_debug("set_policy: cpuinfo.max %u policy->max %u\n", + policy->cpuinfo.max_freq, policy->max); + + cpudata = all_cpu_data[policy->cpu]; + cpudata->policy = policy->policy; + + amd_pstate_epp_init(policy->cpu); + + return 0; +} + +static int amd_pstate_epp_verify_policy(struct cpufreq_policy_data *policy) +{ + cpufreq_verify_within_cpu_limits(policy); + pr_debug("policy_max =%d, policy_min=%d\n", policy->max, policy->min); + return 0; +} + static struct cpufreq_driver amd_pstate_driver = { .flags = CPUFREQ_CONST_LOOPS | CPUFREQ_NEED_UPDATE_LIMITS, .verify = amd_pstate_verify, @@ -628,8 +1040,20 @@ static struct cpufreq_driver amd_pstate_driver = { .attr = amd_pstate_attr, }; +static struct cpufreq_driver amd_pstate_epp_driver = { + .flags = CPUFREQ_CONST_LOOPS, + .verify = amd_pstate_epp_verify_policy, + .setpolicy = amd_pstate_epp_set_policy, + .init = amd_pstate_epp_cpu_init, + .exit = amd_pstate_epp_cpu_exit, + .update_limits = amd_pstate_epp_update_limits, + .name = "amd_pstate_epp", + .attr = amd_pstate_epp_attr, +}; + static int __init amd_pstate_init(void) { + static struct amd_cpudata **cpudata; int ret; if (boot_cpu_data.x86_vendor != X86_VENDOR_AMD) @@ -656,7 +1080,8 @@ static int __init amd_pstate_init(void) /* capability check */ if (boot_cpu_has(X86_FEATURE_CPPC)) { pr_debug("AMD CPPC MSR based functionality is supported\n"); - amd_pstate_driver.adjust_perf = amd_pstate_adjust_perf; + if (cppc_state == AMD_PSTATE_PASSIVE) + default_pstate_driver->adjust_perf = amd_pstate_adjust_perf; } else { pr_debug("AMD CPPC shared memory based functionality is supported\n"); static_call_update(amd_pstate_enable, cppc_enable); @@ -664,17 +1089,21 @@ static int __init amd_pstate_init(void) static_call_update(amd_pstate_update_perf, cppc_update_perf); } + cpudata = vzalloc(array_size(sizeof(void *), num_possible_cpus())); + if (!cpudata) + return -ENOMEM; + WRITE_ONCE(all_cpu_data, cpudata); + /* enable amd pstate feature */ ret = amd_pstate_enable(true); if (ret) { - pr_err("failed to enable amd-pstate with return %d\n", ret); + pr_err("failed to enable with return %d\n", ret); return ret; } - ret = cpufreq_register_driver(&amd_pstate_driver); + ret = cpufreq_register_driver(default_pstate_driver); if (ret) - pr_err("failed to register amd_pstate_driver with return %d\n", - ret); + pr_err("failed to register with return %d\n", ret); return ret; } @@ -696,6 +1125,12 @@ static int __init amd_pstate_param(char *str) if (cppc_state == AMD_PSTATE_DISABLE) pr_info("driver is explicitly disabled\n"); + if (cppc_state == AMD_PSTATE_ACTIVE) + default_pstate_driver = &amd_pstate_epp_driver; + + if (cppc_state == AMD_PSTATE_PASSIVE) + default_pstate_driver = &amd_pstate_driver; + return 0; } diff --git a/include/linux/amd-pstate.h b/include/linux/amd-pstate.h index 922d05a13902..fe1aef743c09 100644 --- a/include/linux/amd-pstate.h +++ b/include/linux/amd-pstate.h @@ -47,6 +47,10 @@ struct amd_aperf_mperf { * @prev: Last Aperf/Mperf/tsc count value read from register * @freq: current cpu frequency value * @boost_supported: check whether the Processor or SBIOS supports boost mode + * @epp_policy: Last saved policy used to set energy-performance preference + * @epp_cached: Cached CPPC energy-performance preference value + * @policy: Cpufreq policy value + * @cppc_cap1_cached Cached MSR_AMD_CPPC_CAP1 register value * * The amd_cpudata is key private data for each CPU thread in AMD P-State, and * represents all the attributes and goals that AMD P-State requests at runtime. @@ -72,6 +76,12 @@ struct amd_cpudata { u64 freq; bool boost_supported; + + /* EPP feature related attributes*/ + s16 epp_policy; + s16 epp_cached; + u32 policy; + u64 cppc_cap1_cached; }; /**