From patchwork Thu Feb 29 20:07:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dimitri Sivanich X-Patchwork-Id: 208526 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a05:7301:2097:b0:108:e6aa:91d0 with SMTP id gs23csp685674dyb; Thu, 29 Feb 2024 13:15:38 -0800 (PST) X-Forwarded-Encrypted: i=3; AJvYcCVwbPSV+YfEs7Vs3Cmzg2ew9YZjddtNoNeGax8Jl+bLTHuNca8jzpCy4i2LUUtgUhCtfgHPAd7QCp0mkKmpI7JSTIr8Uw== X-Google-Smtp-Source: AGHT+IFGh8xVxTzADEbYLQ+C4WqPu8J0KL1ypEcU2MPpjFALxXrJBGDzlHXc26dcuJWtldhSYg89 X-Received: by 2002:a05:6a20:a083:b0:1a1:cdd:1ab with SMTP id r3-20020a056a20a08300b001a10cdd01abmr2634736pzj.40.1709241337953; Thu, 29 Feb 2024 13:15:37 -0800 (PST) ARC-Seal: i=2; a=rsa-sha256; t=1709241337; cv=pass; d=google.com; s=arc-20160816; b=C3DjsdmWZnWYIqkebQgfen19egecwGVPIn0jtm3nwkz4/DPbuk4Ehsh62V0e02tKVS cgsCuuYlOBDkZX0xoyb9PC6CTuOmtoWfE+aKFbBRKM5N6jhUIOrdJic9GlY2JYGeX4XC nIyHMCV2AlnYMYYqXrgyg52R10k/dWHazLhRnIEYPfgJ1S58EeDV38/6EAYeVtxXodYQ Ba9eMPf6/zLxuteGwljmpJ4q3stzlm6XC/y5nryoX2UhPZkH7io+9GKFO6J9JT597lVg RlGg+ME2D8JhDtFOEm1yssnhB3bLja5cqBlUpsG+7GsyaTojzjafNttttEQtOZdqSJBy egRg== ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-disposition:mime-version:list-unsubscribe:list-subscribe :list-id:precedence:message-id:subject:cc:to:from:date :dkim-signature; bh=bTL4DOSmyNxQGe0QwLYqgTCiTE3h293KECpT2Cd+njk=; fh=XcFjdjFTaNtz88QPe6LSj9P32kiohGo6scefG7jmvdI=; b=pUrU+naSOMMpCFwUPa9ezWeLlSSnJbiww341o39Y41KWs6pMrOCvMFM142P5eNVaiG jOwkoB8Z1GhKH7HI7Du4rSrdg0hDsl5DHfbflGCpUyaYCVU5ZCQ/FiXh7PWKnCLrYNp5 wJLvkDyf5SV5XrETUvcUrtuK/Jeu7DVHFBGOfNNpneCYTZ9z97zHgwPO+tZsPVLXP6e6 P5R36BNQwIPEaWMXHGGpnhN3rll/dftDDKEoBaYemtoVuYKQ5wQCUwv7n6TMRvnPQnJ5 x+aSxOw7oej4tmnMegLA50qPHu4dRMnMa6CGXBHv9h/rKsq2F6rsG5y2yJOsyTIjxDgj CkjA==; dara=google.com ARC-Authentication-Results: i=2; mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=kzPXwjK1; arc=pass (i=1 spf=pass spfdomain=hpe.com dkim=pass dkdomain=hpe.com dmarc=pass fromdomain=hpe.com); spf=pass (google.com: domain of linux-kernel+bounces-87555-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-87555-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: from sv.mirrors.kernel.org (sv.mirrors.kernel.org. [2604:1380:45e3:2400::1]) by mx.google.com with ESMTPS id n9-20020a6543c9000000b005dc1bb540cesi2110199pgp.76.2024.02.29.13.15.37 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Feb 2024 13:15:37 -0800 (PST) Received-SPF: pass (google.com: domain of linux-kernel+bounces-87555-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) client-ip=2604:1380:45e3:2400::1; Authentication-Results: mx.google.com; dkim=pass header.i=@hpe.com header.s=pps0720 header.b=kzPXwjK1; arc=pass (i=1 spf=pass spfdomain=hpe.com dkim=pass dkdomain=hpe.com dmarc=pass fromdomain=hpe.com); spf=pass (google.com: domain of linux-kernel+bounces-87555-ouuuleilei=gmail.com@vger.kernel.org designates 2604:1380:45e3:2400::1 as permitted sender) smtp.mailfrom="linux-kernel+bounces-87555-ouuuleilei=gmail.com@vger.kernel.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=hpe.com Received: from smtp.subspace.kernel.org (wormhole.subspace.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by sv.mirrors.kernel.org (Postfix) with ESMTPS id 33557288870 for ; Thu, 29 Feb 2024 21:11:55 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1FBF914404D; Thu, 29 Feb 2024 21:03:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=hpe.com header.i=@hpe.com header.b="kzPXwjK1" Received: from mx0a-002e3701.pphosted.com (mx0a-002e3701.pphosted.com [148.163.147.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F184014373A; Thu, 29 Feb 2024 21:03:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.147.86 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709240594; cv=none; b=Pq2FBAZyY+s0XsGgrnFsNwjIqHRVTe50OOEmyZrWrvhiEsC2G+TET0eZV1XU8whCSx+SH5VOAqDBmQqq27Dn2XEsbwULY7cbe8BvFzcCpLlFJmaBomEnzE3MSZwv05U9JxRlFEW1rwd9m2pDVjxBriE1hJFCP33Uc90OxBs2Ffg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1709240594; c=relaxed/simple; bh=VTjWIjnaot8bvp8Uz7YSb+Wu+VwzB7BvnAiwu0uiHUE=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition; b=aA5Kx72eCTQTlcfN85Ih+XhIu4XF4ziOO3F3Zx1Tun8tw96TIREO2CLLU2ZHPTvqDXf+FHPFlKHsHC+dYAz1pm48C3JNfsBcJ8j0yZBgUibj9Cc0+EWKyRPK2ruKLwpnFxIlWWYx9HbrqDopnnj8ljt5aKIXMwhqUVwjy3Jkrhw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hpe.com; spf=pass smtp.mailfrom=hpe.com; dkim=pass (2048-bit key) header.d=hpe.com header.i=@hpe.com header.b=kzPXwjK1; arc=none smtp.client-ip=148.163.147.86 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=hpe.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=hpe.com Received: from pps.filterd (m0150241.ppops.net [127.0.0.1]) by mx0a-002e3701.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 41TFBuaH010561; Thu, 29 Feb 2024 20:07:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hpe.com; h=date : from : to : cc : subject : message-id : mime-version : content-type; s=pps0720; bh=bTL4DOSmyNxQGe0QwLYqgTCiTE3h293KECpT2Cd+njk=; b=kzPXwjK1SZsCC4A2UXBWK91e9uFF/hCuTPi/3IZkSdWrHbLVAxL10mh32LdEw+Qf5/ik at81CsioejMhAKCSvyosSXQTrD63rc1D1WG3QZE84heShZI+kdUW4LvQ0dzZE4esF0OQ /GLydvzxR5es2XSb3ehAa2Sd4tkM+ZqWsYOjmbT1nfxPMjNPWoDTqxkvuPohvhABOSFU e1sc33lrbdKqeMAkVfZzwza1x2KLjLatL3e6oHtFcInLnKCYtXEyqtvWrOqwpzmj/fb/ qthu2c9EMSRUd+FI9oRgFnlP/q2d3V9q4YlabwIFqRkVDvEs4B8R3/II53iogipiTrde 7w== Received: from p1lg14879.it.hpe.com ([16.230.97.200]) by mx0a-002e3701.pphosted.com (PPS) with ESMTPS id 3wjvbxb4w5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 29 Feb 2024 20:07:30 +0000 Received: from p1lg14885.dc01.its.hpecorp.net (unknown [10.119.18.236]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by p1lg14879.it.hpe.com (Postfix) with ESMTPS id 9BB85130F8; Thu, 29 Feb 2024 20:07:29 +0000 (UTC) Received: from hpe.com (unknown [16.231.227.39]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by p1lg14885.dc01.its.hpecorp.net (Postfix) with ESMTPS id A83CB8005F1; Thu, 29 Feb 2024 20:07:26 +0000 (UTC) Date: Thu, 29 Feb 2024 14:07:24 -0600 From: Dimitri Sivanich To: Dimitri Sivanich , David Woodhouse , Lu Baolu , Joerg Roedel , Will Deacon , Robin Murphy , Thomas Gleixner , iommu@lists.linux.dev Cc: linux-kernel@vger.kernel.org, Steve Wahl , Russ Anderson Subject: [PATCH] Allocate DMAR fault interrupts locally Message-ID: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline X-Proofpoint-GUID: kaKI-Rt3ddMThGjjOtXFFOFMzOETu6v1 X-Proofpoint-ORIG-GUID: kaKI-Rt3ddMThGjjOtXFFOFMzOETu6v1 X-HPE-SCL: -1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.1011,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2024-02-29_06,2024-02-29_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 adultscore=0 mlxscore=0 mlxlogscore=995 priorityscore=1501 impostorscore=0 phishscore=0 suspectscore=0 lowpriorityscore=0 bulkscore=0 malwarescore=0 clxscore=1011 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2402290155 X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1792269445555206762 X-GMAIL-MSGID: 1792269445555206762 The Intel IOMMU code currently tries to allocate all DMAR fault interrupt vectors on the boot cpu. On large systems with high DMAR counts this results in vector exhaustion, and most of the vectors are not initially allocated socket local. Instead, have a cpu on each node do the vector allocation for the DMARs on that node. The boot cpu still does the allocation for its node during its boot sequence. Signed-off-by: Dimitri Sivanich --- drivers/iommu/intel/dmar.c | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/intel/dmar.c b/drivers/iommu/intel/dmar.c index 23cb80d62a9a..41ef72ba7509 100644 --- a/drivers/iommu/intel/dmar.c +++ b/drivers/iommu/intel/dmar.c @@ -2108,8 +2108,12 @@ int __init enable_drhd_fault_handling(void) */ for_each_iommu(iommu, drhd) { u32 fault_status; - int ret = dmar_set_interrupt(iommu); + int ret; + if (iommu->node != cpu_to_node(smp_processor_id())) + continue; + + ret = dmar_set_interrupt(iommu); if (ret) { pr_err("DRHD %Lx: failed to enable fault, interrupt, ret %d\n", (unsigned long long)drhd->reg_base_addr, ret); @@ -2192,6 +2196,34 @@ static int __init dmar_free_unused_resources(void) late_initcall(dmar_free_unused_resources); +#ifdef CONFIG_X86_LOCAL_APIC +static void __init irq_remap_enable_fault_handling_thr(struct work_struct *work) +{ + irq_remap_enable_fault_handling(); +} + +static int __init assign_dmar_vectors(void) +{ + struct work_struct irq_remap_work; + int nid; + + INIT_WORK(&irq_remap_work, irq_remap_enable_fault_handling_thr); + cpus_read_lock(); + for_each_online_node(nid) { + /* Boot cpu dmar vectors are assigned before the rest */ + if (nid == cpu_to_node(get_boot_cpu_id())) + continue; + schedule_work_on(cpumask_first(cpumask_of_node(nid)), + &irq_remap_work); + flush_work(&irq_remap_work); + } + cpus_read_unlock(); + return 0; +} + +arch_initcall(assign_dmar_vectors); +#endif /* CONFIG_X86_LOCAL_APIC */ + /* * DMAR Hotplug Support * For more details, please refer to Intel(R) Virtualization Technology