From patchwork Thu Oct 5 17:20:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Arvind Yadav X-Patchwork-Id: 148951 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:612c:2016:b0:403:3b70:6f57 with SMTP id fe22csp453178vqb; Thu, 5 Oct 2023 10:26:01 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFjGyonxFl4jeXwRimdNAE+RgjbQ6ekLpf0Vfgq5RlN7zkN4b1BaSu2ZPKvlrb6T/RuVb3Q X-Received: by 2002:a17:902:c952:b0:1c0:d17a:bfef with SMTP id i18-20020a170902c95200b001c0d17abfefmr6331302pla.30.1696526761107; Thu, 05 Oct 2023 10:26:01 -0700 (PDT) ARC-Seal: i=2; a=rsa-sha256; t=1696526761; cv=pass; d=google.com; s=arc-20160816; b=ZJK1iu1XwEfpp2rizw4uIxa5+7+vWdB6XWBnLjQz//z6mCeBZ7tYjWRxE+ozdHLxc0 Dv+FTM4tH151G4a149BvDEwPlYMQRlkxj3DK8llMd1yx/CkUuYRpAEOlDzE3W4VRmfdG QTzMV9MS/02UDHXoe1Ee4J/g8SkGPjnbl8OnSMpkaU7wKbQVuCVsct5U/0Ab4GKOkVe4 MC49ecKBi0Gr437C6uATZm12bLbgTUDSzz74cVOoK85uAEnbFgrDCuZq8L6BPw3H9gf1 Gj5AnjNsRV5q8R/KOw37NzJ3bxEo8yS4ywG46gMhJKHH3NfGdc6g9WMneEq+BB18pHSR WpXw== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=WztqDgo7TjJyCM3T7lwugZwY+t5sFQZRgrk1r0fcRzk=; fh=ay2M3ioaqOwwrS5OY1CvUKcgO9QZSdOXrxpSTc1vX2w=; b=s37Cz+8f8dhuQ1QNOH3SOZ/YXDBbKb+EQcDXJnqB4YsYcpq+KrNCvLF5l+OMD4Iyw/ fP1lovKX/+6owaWiyl1hmradi4qSWTPQZXz1UaEAgRkMtcCNDOsinu8+4S+hjBUOHP9p 31Y4lv9sBjBpE2MhOjXKynFrhutp06ZSVbw1b1seV3W+kS90eymgRZRM5m3tU2fNpcPj dS9jicHJzYZeV0Y5D6vFi1tsQj8H+weigLofu/Gt49+ztx3MWN4HAELJV+TTPV2yLCWQ L9j1xB01W6dqrfi7VLcGEZ6VXSHRtfRouzuL+R1y96YLepjKg0A+0mvVxY0Y2u0b5OjL 9m4Q== ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=JU6UP30W; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: from lipwig.vger.email (lipwig.vger.email. [2620:137:e000::3:3]) by mx.google.com with ESMTPS id kz7-20020a170902f9c700b001b89fd6bec4si1787765plb.144.2023.10.05.10.26.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Oct 2023 10:26:01 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) client-ip=2620:137:e000::3:3; Authentication-Results: mx.google.com; dkim=pass header.i=@amd.com header.s=selector1 header.b=JU6UP30W; arc=pass (i=1 spf=pass spfdomain=amd.com dmarc=pass fromdomain=amd.com); spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::3:3 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=QUARANTINE sp=QUARANTINE dis=NONE) header.from=amd.com Received: from out1.vger.email (depot.vger.email [IPv6:2620:137:e000::3:0]) by lipwig.vger.email (Postfix) with ESMTP id 7B494807BEF8; Thu, 5 Oct 2023 10:25:58 -0700 (PDT) X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.103.10 at lipwig.vger.email Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231643AbjJERZn (ORCPT + 18 others); Thu, 5 Oct 2023 13:25:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49998 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230365AbjJERZC (ORCPT ); Thu, 5 Oct 2023 13:25:02 -0400 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (mail-co1nam11on2064.outbound.protection.outlook.com [40.107.220.64]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB99E115 for ; Thu, 5 Oct 2023 10:21:03 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hq8R6Jvw60mxBVMTqoDGN6tn6wBe8JDD0lePzbtK7LBWlHScN96g6CIbpPHpXwzzNAOvb6CksrPQjJDpmLTcVqo8c3jGXu8gZFcMszO7bIgAdaWXhgKeGXKyuA3tXSxz6zvEQqpjh70/W5V/CoQA7GTTQp83sqDUU2IMYvASE7g4GOPQUuF0tETCLzGQWOu3dr//m0+zj55FQm8R97QlobGovamgLKhpQLXqIyyhw6t38UaMWKIQuTTZmtquyrf4cvPfjxGjqJZ5tzS80CarzQUfl6nDi3wm2s2QEusxxBPcGysCVAa6QCIfRzKXUVVtbP+f7j+AsVTCcZ+KsvOEHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WztqDgo7TjJyCM3T7lwugZwY+t5sFQZRgrk1r0fcRzk=; b=SyBPfBsSUdt0kNPQA5fECFbLLCWrQ+gEaPMtxCVSKc2MPws+euhXRHKFTBtKwQiE2zKY1PPOzPM3LCSxsLMfgPykjABlVnzfQiCo1haN5UwMHAB1kRUk9dGDMoUKG/cbucpo81qjqSV09H0Wi/BLgmnCTJvtrbifEXRWWPIYnrBqFcFoiP44A/3S1Qq6M1SFLERiO8x6GevSYXtaIxEbeYuq91xk3yg3Lgcu0uNtkE0nkcEtqsuSGIRlA6CWapWQ1FWgM1sTDWLQa/SMtxw+JQVrOK6JN5jBwUyQqH6T8yDjgl+K6vwXvKmu1jTthgkh3Ihb5p8S2hxQk0GEl7jjMA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=gmail.com smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WztqDgo7TjJyCM3T7lwugZwY+t5sFQZRgrk1r0fcRzk=; b=JU6UP30WdvvFYQfcj/3okYmD1/i3vPR1h/Uw8tADioj/EDLN8fM9WVwZN9FmYdcoWenteuGfPFL7LAOWYNAQEQv6L44PP6C6LBXRREsfvgNbST1jDL0FM7yhnU8mAdPzC7ZTGt2OQghw4/RSWsPmhLkfjJsdDRIKWF2yPIbte54= Received: from CH2PR04CA0018.namprd04.prod.outlook.com (2603:10b6:610:52::28) by IA0PR12MB8646.namprd12.prod.outlook.com (2603:10b6:208:489::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.28; Thu, 5 Oct 2023 17:20:57 +0000 Received: from CY4PEPF0000EDD1.namprd03.prod.outlook.com (2603:10b6:610:52:cafe::3e) by CH2PR04CA0018.outlook.office365.com (2603:10b6:610:52::28) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6838.37 via Frontend Transport; Thu, 5 Oct 2023 17:20:57 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; pr=C Received: from SATLEXMB04.amd.com (165.204.84.17) by CY4PEPF0000EDD1.mail.protection.outlook.com (10.167.241.205) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.6838.22 via Frontend Transport; Thu, 5 Oct 2023 17:20:57 +0000 Received: from rtg-Artic.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.27; Thu, 5 Oct 2023 12:20:53 -0500 From: Arvind Yadav To: , , , , , , , CC: , , , Arvind Yadav , Christian Koenig Subject: [PATCH v4 1/1] drm/amdkfd: get doorbell's absolute offset based on the db_size Date: Thu, 5 Oct 2023 22:50:11 +0530 Message-ID: <20231005172011.9271-2-Arvind.Yadav@amd.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231005172011.9271-1-Arvind.Yadav@amd.com> References: <20231005172011.9271-1-Arvind.Yadav@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CY4PEPF0000EDD1:EE_|IA0PR12MB8646:EE_ X-MS-Office365-Filtering-Correlation-Id: 40f9d98c-de55-46a4-6a04-08dbc5c7704a X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: aYXaw66nZPEoQ8DQSCVvsjG+I1czSbJZeeD7Z0ablXsX+GcFdbsNVZIyDIejpcT3sOdL7cSDBUCIZy1BQhtdwD8tsZayKdIMbG7kJty6C5tp0V5+90h/9si/uvrbk/t395cmoIBrCPBIJvQeDmM/m1SulnJHYDwFX//N2QwZd7hd+nCR0Or6IsLTT5J7vMfPJa021Dqk6nH5vI3ThC5f4natEy5SoqvebBYIh8FlDN1Q2DEw/wsfvKhdzqasw2VKMUoHOfSQqJgJMoWDQvE6trIXc3o1zXyY+bYkcINW3EgYF3a6XQHX+nagB+QHTIDLXv49bENEXlk8C9JSHdvt5G+1YYYinQdHoI40rb4YD8/FclXuIrNtly1n+qdCYb7sOZd4xJsymNojSYMc4mWUQwRAVMP+Gg6DVDgkHPwq34vEZIJOojlrLpJ51fcpajt+sjG7yoBAFeWQinv2XrWVYP0qq8M1s0N8VffL4itowG0+sUg4UtSjC6eAT60c3m6C78lM6UvZhWojnzjCQuFEey2R/QYZy8Wl46ynynZJSHNln9naOLXR14YBUgWPeI9Zp5B9OPh7y8bmhHpWDwN9S0a0fiIuNgNqpQpyqcg/PBbwHfPfKjYnwLYxid2DTSQEQ7t9u9BObAKGhBOooOy/CfO2TNzAbdYqD6OWZe+oXkKvhGdhMLbUdGL+V3QmhVtKCGHVYAyu8apXKBgSe65Gg4or7idy778PxIIBU8rFsHkkR7363/6jiit7eqoK93ZSfOX0QvDTXss+NYVUkmzAOw== X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(13230031)(4636009)(376002)(346002)(136003)(396003)(39860400002)(230922051799003)(64100799003)(451199024)(82310400011)(186009)(1800799009)(36840700001)(40470700004)(46966006)(40460700003)(40480700001)(6666004)(2616005)(478600001)(7696005)(36860700001)(47076005)(82740400003)(81166007)(86362001)(356005)(41300700001)(83380400001)(2906002)(426003)(336012)(1076003)(26005)(16526019)(36756003)(70586007)(54906003)(5660300002)(316002)(70206006)(8936002)(110136005)(8676002)(4326008)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Oct 2023 17:20:57.5052 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 40f9d98c-de55-46a4-6a04-08dbc5c7704a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CY4PEPF0000EDD1.namprd03.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB8646 X-Spam-Status: No, score=-0.9 required=5.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lipwig.vger.email Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.6.4 (lipwig.vger.email [0.0.0.0]); Thu, 05 Oct 2023 10:25:58 -0700 (PDT) X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1778937245383641806 X-GMAIL-MSGID: 1778937245383641806 Here, Adding db_size in byte to find the doorbell's absolute offset for both 32-bit and 64-bit doorbell sizes. So that doorbell offset will be aligned based on the doorbell size. v2: - Addressed the review comment from Felix. v3: - Adding doorbell_size as parameter to get db absolute offset. v4: Squash the two patches into one. Cc: Christian Koenig Cc: Alex Deucher Signed-off-by: Shashank Sharma Signed-off-by: Arvind Yadav Reviewed-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h | 5 +++-- drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c | 13 +++++++++---- .../gpu/drm/amd/amdkfd/kfd_device_queue_manager.c | 3 ++- drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c | 10 ++++++++-- .../gpu/drm/amd/amdkfd/kfd_process_queue_manager.c | 3 ++- 5 files changed, 24 insertions(+), 10 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h index 09f6727e7c73..4a8b33f55f6b 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell.h @@ -357,8 +357,9 @@ int amdgpu_doorbell_init(struct amdgpu_device *adev); void amdgpu_doorbell_fini(struct amdgpu_device *adev); int amdgpu_doorbell_create_kernel_doorbells(struct amdgpu_device *adev); uint32_t amdgpu_doorbell_index_on_bar(struct amdgpu_device *adev, - struct amdgpu_bo *db_bo, - uint32_t doorbell_index); + struct amdgpu_bo *db_bo, + uint32_t doorbell_index, + uint32_t db_size); #define RDOORBELL32(index) amdgpu_mm_rdoorbell(adev, (index)) #define WDOORBELL32(index, v) amdgpu_mm_wdoorbell(adev, (index), (v)) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c index da4be0bbb446..6690f5a72f4d 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_doorbell_mgr.c @@ -114,19 +114,24 @@ void amdgpu_mm_wdoorbell64(struct amdgpu_device *adev, u32 index, u64 v) * @adev: amdgpu_device pointer * @db_bo: doorbell object's bo * @db_index: doorbell relative index in this doorbell object + * @db_size: doorbell size is in byte * * returns doorbell's absolute index in BAR */ uint32_t amdgpu_doorbell_index_on_bar(struct amdgpu_device *adev, - struct amdgpu_bo *db_bo, - uint32_t doorbell_index) + struct amdgpu_bo *db_bo, + uint32_t doorbell_index, + uint32_t db_size) { int db_bo_offset; db_bo_offset = amdgpu_bo_gpu_offset_no_check(db_bo); - /* doorbell index is 32 bit but doorbell's size is 64-bit, so *2 */ - return db_bo_offset / sizeof(u32) + doorbell_index * 2; + /* doorbell index is 32 bit but doorbell's size can be 32 bit + * or 64 bit, so *db_size(in byte)/4 for alignment. + */ + return db_bo_offset / sizeof(u32) + doorbell_index * + DIV_ROUND_UP(db_size, 4); } /** diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c index 0d3d538b64eb..e07652e72496 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c @@ -407,7 +407,8 @@ static int allocate_doorbell(struct qcm_process_device *qpd, q->properties.doorbell_off = amdgpu_doorbell_index_on_bar(dev->adev, qpd->proc_doorbells, - q->doorbell_id); + q->doorbell_id, + dev->kfd->device_info.doorbell_size); return 0; } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c index 7b38537c7c99..05c74887fd6f 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_doorbell.c @@ -161,7 +161,10 @@ void __iomem *kfd_get_kernel_doorbell(struct kfd_dev *kfd, if (inx >= KFD_MAX_NUM_OF_QUEUES_PER_PROCESS) return NULL; - *doorbell_off = amdgpu_doorbell_index_on_bar(kfd->adev, kfd->doorbells, inx); + *doorbell_off = amdgpu_doorbell_index_on_bar(kfd->adev, + kfd->doorbells, + inx, + kfd->device_info.doorbell_size); inx *= 2; pr_debug("Get kernel queue doorbell\n" @@ -240,7 +243,10 @@ phys_addr_t kfd_get_process_doorbells(struct kfd_process_device *pdd) return 0; } - first_db_index = amdgpu_doorbell_index_on_bar(adev, pdd->qpd.proc_doorbells, 0); + first_db_index = amdgpu_doorbell_index_on_bar(adev, + pdd->qpd.proc_doorbells, + 0, + pdd->dev->kfd->device_info.doorbell_size); return adev->doorbell.base + first_db_index * sizeof(uint32_t); } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c index adb5e4bdc0b2..77649392e233 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c @@ -377,7 +377,8 @@ int pqm_create_queue(struct process_queue_manager *pqm, */ uint32_t first_db_index = amdgpu_doorbell_index_on_bar(pdd->dev->adev, pdd->qpd.proc_doorbells, - 0); + 0, + pdd->dev->kfd->device_info.doorbell_size); *p_doorbell_offset_in_process = (q->properties.doorbell_off - first_db_index) * sizeof(uint32_t);