| Message ID | 3817d810-e0f1-8ef8-0bbd-663b919ca49b@cybernetics.com |
|---|---|
| State | New |
| Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org>
Delivered-To: ouuuleilei@gmail.com
Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp3641221vqo;
Tue, 25 Apr 2023 13:04:17 -0700 (PDT)
X-Google-Smtp-Source:
AKy350bvAX1szjpoOAFXYV9BDbaHtxHsJLtMFwHwrQxkBeCu/Sh1aWEQfsBpDHXT4mQBRK+wFNqd
X-Received: by 2002:a17:90a:b102:b0:246:f8a8:af02 with SMTP id
z2-20020a17090ab10200b00246f8a8af02mr17924115pjq.5.1682453056693;
Tue, 25 Apr 2023 13:04:16 -0700 (PDT)
ARC-Seal: i=1; a=rsa-sha256; t=1682453056; cv=none;
d=google.com; s=arc-20160816;
b=HslFJtxC95v3QdS2zwfrs/p51TADS0jRkEVKIVHRjos9j8vsXXLNJbU/9jE6v4MeHv
eUR3MVj3iUqaFm6/ewBHLXFOvToiIYZWbMT6q85tZd1R4c2Bsd8PcW6lqvnZ28X3ch3g
CY70XIfmBY1s59ebzXtD2Qoj0hpd5HfPEwhtKiFCMpOwE20ikx0R8kQyKv4M9Icybpbc
sxoictxWfnLSSl05tJutMkt4jBZU2EUCviQsGaGgMCDxnzzdOlSC3WFguNR/MsjH6Ugu
Q/62KwAf3v15wd4q3hhU7msVSMY+2+xCOmWo+MBJ5Ol94VW1MimvgXDjaSFNfmD1E2wQ
Oamg==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com;
s=arc-20160816;
h=list-id:precedence:content-transfer-encoding:subject:from:cc:to
:content-language:user-agent:mime-version:date:message-id
:dkim-signature;
bh=NKoTw0X5ggn4GP4rCHMZBRbxnmfIguEqeaihKcA5REk=;
b=RBABw9ZYPgkKHKvPJaNDvAYHrf0M74CX3WTsew3NsCf0+pdeuafxGx8WNbeqrzPoOG
YOi0g2ILl4QGGZ3Dt0azo7y4yh3vjnbn/zJdf+jBhQcq57W+bXUXzS+L6FR8Onfl5Tw+
POfe5f1i2LRIoddzr137QFQFN9g8Kkts4nt8XNSCQeyDz4xKW/ADtby+cuH3/qciAeSC
VxR0UAYhCFJ1luf0hpCDKb0DYEhGBlBqdYvWHbf/oNO0h0Ma/NIsWiqo/0A1z0D4FAin
Ot6yd0wnPeiXT6RXKzW+tJ5+hU956ahFKS+HjGd83nliRptuqWCPAYp1gowyYIAMpf9V
+v7g==
ARC-Authentication-Results: i=1; mx.google.com;
dkim=pass header.i=@cybernetics.com header.s=mail header.b=XjJvQNfR;
spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org
designates 2620:137:e000::1:20 as permitted sender)
smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cybernetics.com
Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20])
by mx.google.com with ESMTP id
l7-20020a6542c7000000b004dacaa17ecbsi13620698pgp.559.2023.04.25.13.03.37;
Tue, 25 Apr 2023 13:04:16 -0700 (PDT)
Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org
designates 2620:137:e000::1:20 as permitted sender)
client-ip=2620:137:e000::1:20;
Authentication-Results: mx.google.com;
dkim=pass header.i=@cybernetics.com header.s=mail header.b=XjJvQNfR;
spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org
designates 2620:137:e000::1:20 as permitted sender)
smtp.mailfrom=linux-kernel-owner@vger.kernel.org;
dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=cybernetics.com
Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand
id S236150AbjDYT5Q (ORCPT <rfc822;zxc52fgh@gmail.com> + 99 others);
Tue, 25 Apr 2023 15:57:16 -0400
Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58914 "EHLO
lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org
with ESMTP id S234754AbjDYT5O (ORCPT
<rfc822;linux-kernel@vger.kernel.org>);
Tue, 25 Apr 2023 15:57:14 -0400
X-Greylist: delayed 1817 seconds by postgrey-1.37 at
lindbergh.monkeyblade.net; Tue, 25 Apr 2023 12:57:12 PDT
Received: from mail.cybernetics.com (mail.cybernetics.com [72.215.153.18])
by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC25C19A2
for <linux-kernel@vger.kernel.org>;
Tue, 25 Apr 2023 12:57:12 -0700 (PDT)
X-ASG-Debug-ID: 1682450813-1cf4391215313690001-xx1T2L
Received: from cybernetics.com ([10.10.4.126]) by mail.cybernetics.com with
ESMTP id UJmp3BejnK1cjIIr; Tue, 25 Apr 2023 15:26:53 -0400 (EDT)
X-Barracuda-Envelope-From: tonyb@cybernetics.com
X-ASG-Whitelist: Client
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed;
d=cybernetics.com; s=mail;
bh=NKoTw0X5ggn4GP4rCHMZBRbxnmfIguEqeaihKcA5REk=;
h=Content-Transfer-Encoding:Content-Type:Subject:From:Cc:To:Content-Language:
MIME-Version:Date:Message-ID;
b=XjJvQNfRCimTBY0NYTJiqaV9m5jO6rqZsItwTv2jCDwOE
avvzTgWVFYvLTXnDb4GXZREDk1+OzndudyR/3Q9FlskrpRtPaFaczQNSQ12HBhOF1DvPeBiMCO0zE
njol8uefnZ/keDtObWbQu/fhnYfYYmZHFqIc9bRsxOj06gv4o=
Received: from [10.157.2.224] (HELO [192.168.200.1])
by cybernetics.com (CommuniGate Pro SMTP 7.1.1)
with ESMTPS id 12590162; Tue, 25 Apr 2023 15:26:53 -0400
Message-ID: <3817d810-e0f1-8ef8-0bbd-663b919ca49b@cybernetics.com>
Date: Tue, 25 Apr 2023 15:26:53 -0400
MIME-Version: 1.0
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101
Thunderbird/102.10.0
Content-Language: en-US
To: Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
Dave Hansen <dave.hansen@linux.intel.com>, x86@kernel.org
Cc: "H. Peter Anvin" <hpa@zytor.com>,
Mario Limonciello <mario.limonciello@amd.com>,
Tom Lendacky <thomas.lendacky@amd.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
From: Tony Battersby <tonyb@cybernetics.com>
Subject: [PATCH RFC] x86/cpu: fix intermittent lockup on poweroff
Content-Type: text/plain; charset=UTF-8
X-ASG-Orig-Subj: [PATCH RFC] x86/cpu: fix intermittent lockup on poweroff
Content-Transfer-Encoding: 7bit
X-Barracuda-Connect: UNKNOWN[10.10.4.126]
X-Barracuda-Start-Time: 1682450813
X-Barracuda-URL: https://10.10.4.122:443/cgi-mod/mark.cgi
X-Barracuda-BRTS-Status: 1
X-Virus-Scanned: by bsmtpd at cybernetics.com
X-Barracuda-Scan-Msg-Size: 1697
X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED,
DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,SPF_HELO_NONE,SPF_PASS,
T_SCC_BODY_TEXT_LINE,URIBL_BLOCKED autolearn=ham autolearn_force=no
version=3.4.6
X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on
lindbergh.monkeyblade.net
Precedence: bulk
List-ID: <linux-kernel.vger.kernel.org>
X-Mailing-List: linux-kernel@vger.kernel.org
X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?=
X-GMAIL-THRID: =?utf-8?q?1764178577694427736?=
X-GMAIL-MSGID: =?utf-8?q?1764179896442095303?=
|
| Series |
[RFC] x86/cpu: fix intermittent lockup on poweroff
|
|
Commit Message
Tony Battersby
April 25, 2023, 7:26 p.m. UTC
In stop_this_cpu(), make sure the CPUID leaf exists before accessing the
leaf. This fixes a lockup on poweroff 50% of the time due to the wrong
branch being taken randomly on some CPUs (seen on Supermicro X8DTH-6F
with Intel Xeon X5650).
Fixes: 08f253ec3767 ("x86/cpu: Clear SME feature flag when not in use")
Cc: <stable@vger.kernel.org> # 5.18+
Signed-off-by: Tony Battersby <tonyb@cybernetics.com>
---
NOTE: I don't have any AMD CPUs to test, so I was unable to fully test
this patch. Could someone with an AMD CPU that supports SME please test
this and make sure it calls native_wbinvd()?
arch/x86/kernel/process.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
Comments
On 4/25/23 15:39, Borislav Petkov wrote: > On Tue, Apr 25, 2023 at 03:26:53PM -0400, Tony Battersby wrote: >> In stop_this_cpu(), make sure the CPUID leaf exists before accessing the >> leaf. This fixes a lockup on poweroff 50% of the time due to the wrong >> branch being taken randomly on some CPUs (seen on Supermicro X8DTH-6F >> with Intel Xeon X5650). > Please send the output of > > cpuid -r > > from that machine. > > Thx. > I have attached the output for the Xeon X5650 and Xeon X5690 (one is running a slightly older kernel). I have two different systems with Supermicro X8DTH-6F motherboards and different speed CPUs, and I have seen the problem on both of them. Tony Battersby Cybernetics CPU 0: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x00200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000000 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000 CPU 1: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x02200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000002 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000002 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000002 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000002 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000002 CPU 2: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x04200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000004 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000004 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000004 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000004 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000004 CPU 3: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x10200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000010 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000010 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000010 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000010 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000010 CPU 4: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x12200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000012 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000012 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000012 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000012 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000012 CPU 5: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x14200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000014 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000014 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000014 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000014 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000014 CPU 6: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x01200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000001 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000001 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000001 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000001 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000001 CPU 7: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x03200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000003 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000003 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000003 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000003 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000003 CPU 8: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x05200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000005 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000005 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000005 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000005 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000005 CPU 9: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x11200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000011 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000011 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000011 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000011 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000011 CPU 10: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x13200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000013 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000013 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000013 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000013 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000013 CPU 11: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x15200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000015 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000015 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000015 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30353635 ebx=0x20402020 ecx=0x37362e32 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000015 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000015 CPU 0: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x00200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000000 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000 CPU 1: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x02200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000002 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000002 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000002 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000002 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000002 CPU 2: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x04200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000004 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000004 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000004 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000004 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000004 CPU 3: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x10200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000010 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000010 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000010 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000010 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000010 CPU 4: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x12200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000012 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000012 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000012 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000012 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000012 CPU 5: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x14200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000014 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000014 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000014 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000014 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000014 CPU 6: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x01200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000001 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000001 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000001 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000001 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000001 CPU 7: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x03200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000003 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000003 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000003 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000003 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000003 CPU 8: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x05200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000005 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000005 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000005 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000005 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000005 CPU 9: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x11200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000011 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000011 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000011 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000011 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000011 CPU 10: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x13200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000013 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000013 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000013 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000013 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000013 CPU 11: 0x00000000 0x00: eax=0x0000000b ebx=0x756e6547 ecx=0x6c65746e edx=0x49656e69 0x00000001 0x00: eax=0x000206c2 ebx=0x15200800 ecx=0x029ee3ff edx=0xbfebfbff 0x00000002 0x00: eax=0x55035a01 ebx=0x00f0b2ff ecx=0x00000000 edx=0x00ca0000 0x00000003 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000004 0x00: eax=0x3c004121 ebx=0x01c0003f ecx=0x0000003f edx=0x00000000 0x00000004 0x01: eax=0x3c004122 ebx=0x00c0003f ecx=0x0000007f edx=0x00000000 0x00000004 0x02: eax=0x3c004143 ebx=0x01c0003f ecx=0x000001ff edx=0x00000000 0x00000004 0x03: eax=0x3c07c163 ebx=0x03c0003f ecx=0x00002fff edx=0x00000002 0x00000005 0x00: eax=0x00000040 ebx=0x00000040 ecx=0x00000003 edx=0x00001120 0x00000006 0x00: eax=0x00000007 ebx=0x00000002 ecx=0x00000009 edx=0x00000000 0x00000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x9c000000 0x00000008 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x00000009 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x0000000a 0x00: eax=0x07300403 ebx=0x00000004 ecx=0x00000000 edx=0x00000603 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000015 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000015 0x20000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000015 0x80000000 0x00: eax=0x80000008 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000001 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000001 edx=0x2c100800 0x80000002 0x00: eax=0x65746e49 ebx=0x2952286c ecx=0x6f655820 edx=0x2952286e 0x80000003 0x00: eax=0x55504320 ebx=0x20202020 ecx=0x20202020 edx=0x58202020 0x80000004 0x00: eax=0x30393635 ebx=0x20402020 ecx=0x37342e33 edx=0x007a4847 0x80000005 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80000006 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x01006040 edx=0x00000000 0x80000007 0x00: eax=0x00000000 ebx=0x00000000 ecx=0x00000000 edx=0x00000100 0x80000008 0x00: eax=0x00003028 ebx=0x00000000 ecx=0x00000000 edx=0x00000000 0x80860000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000015 0xc0000000 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000015
On 4/25/23 12:26, Tony Battersby wrote: > - if (cpuid_eax(0x8000001f) & BIT(0)) > + if (c->extended_cpuid_level >= 0x8000001f && > + (cpuid_eax(0x8000001f) & BIT(0))) > native_wbinvd(); Oh, so the existing code is running into the > If a value entered for CPUID.EAX is higher than the maximum input > value for basic or extended function for that processor then the data > for the highest basic information leaf is returned behavior. It's basically looking at BIT(0) of some random extended leaf. Probably 0x80000008 based on your 'cpuid -r' output.
On 4/25/23 13:03, Dave Hansen wrote: > On 4/25/23 12:26, Tony Battersby wrote: >> - if (cpuid_eax(0x8000001f) & BIT(0)) >> + if (c->extended_cpuid_level >= 0x8000001f && >> + (cpuid_eax(0x8000001f) & BIT(0))) >> native_wbinvd(); > Oh, so the existing code is running into the > >> If a value entered for CPUID.EAX is higher than the maximum input >> value for basic or extended function for that processor then the data >> for the highest basic information leaf is returned > behavior. It's basically looking at BIT(0) of some random extended > leaf. Probably 0x80000008 based on your 'cpuid -r' output. Whoops, 0x80000008 isn't a "basic information leaf". If 'cpuid -r' dumps all the basic leaves, that would mean the "highest basic information leaf" is 0x0000000b: > 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000 which does have BIT(0) set. So, that at least explains how WBINVD gets called in the first place. But (as tglx noted on IRC) it doesn't really explain the lockup. WBINVD should work everywhere and it won't #UD or something because the CPUID check was botched.
On Tue, Apr 25 2023 at 13:03, Dave Hansen wrote: > On 4/25/23 12:26, Tony Battersby wrote: >> - if (cpuid_eax(0x8000001f) & BIT(0)) >> + if (c->extended_cpuid_level >= 0x8000001f && >> + (cpuid_eax(0x8000001f) & BIT(0))) >> native_wbinvd(); > > Oh, so the existing code is running into the > >> If a value entered for CPUID.EAX is higher than the maximum input >> value for basic or extended function for that processor then the data >> for the highest basic information leaf is returned > behavior. It's basically looking at BIT(0) of some random extended > leaf. Probably 0x80000008 based on your 'cpuid -r' output. Right, accessing that leaf without checking whether it exists is wrong, but that does not explain the hang itself. The only consequence of looking at bit 0 of some random other leaf is that all CPUs which run stop_this_cpu() issue WBINVD in parallel, which is slow but should not be a fatal issue. Tony observed this is a 50% chance to hang, which means this is a timing issue. Now there are two things to investigate: 1) Does the system go south if enough CPUs issue WBINVD concurrently? That should be trivial to analyze by enforcing concurreny on a WBINVD in an IPI via a global synchronization bit or such 2) The first thing stop_this_cpu() does is to clear its own bit in the CPU online mask. The CPU which controls shutdown/reboot waits for num_online_cpus() to drop down to one, which means it can proceed _before_ the other CPUs have actually reached HALT. That's not a new thing. It has been that way forever. Just the WBINVD might cause enough delay to create problems. That should be trivial to analyze too by just waiting on the control side for e.g 100ms after num_online_cpus() dropped down to one. The patch itself is correct as is, but as it does not explain the underlying problem. There is a real serious issue underneath. Thanks, tglx
On Tue, Apr 25, 2023 at 01:34:48PM -0700, Dave Hansen wrote: > > 0x0000000b 0x00: eax=0x00000001 ebx=0x00000002 ecx=0x00000100 edx=0x00000000 > > which does have BIT(0) set. Yeah, or the last basic leaf 0x0000000b 0x01: eax=0x00000005 ebx=0x0000000c ecx=0x00000201 edx=0x00000000 but that one has bit 0 set too, so yeah, patch is correct. I'll queue it after -rc1. But Tony, please change your commit message to say something like this: "Check the maximum supported extended CPUID level because on Intel, querying an invalid extended CPUID leaf returns the values of the maximum basic CPUID leaf and if bit 0 in EAX is set, the check falsely evaluates to true and WBINVD is wrongly executed where it shouldn't." Thx.
On 4/25/23 14:05, Thomas Gleixner wrote: > The only consequence of looking at bit 0 of some random other leaf is > that all CPUs which run stop_this_cpu() issue WBINVD in parallel, which > is slow but should not be a fatal issue. > > Tony observed this is a 50% chance to hang, which means this is a timing > issue. I _think_ the system in question is a dual-socket Westmere. I don't see any obvious errata that we could pin this on: > https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-5600-specification-update.pdf Andi Kleen had an interesting theory. WBINVD is a pretty expensive operation. It's possible that it has some degenerative behavior when it's called on a *bunch* of CPUs all at once (which this path can do). If the instruction takes too long, it could trigger one of the CPU's internal lockup detectors and trigger a machine check. At that point, all hell breaks loose. I don't know the cache coherency protocol well enough to say for sure, but I wonder if there's a storm of cache coherency traffic as all those lines get written back. One of the CPUs gets starved from making enough forward progress and trips a CPU-internal watchdog. Andi also says that it _should_ log something in the machine check banks when this happens so there should be at least some kind of breadcrumb. Either way, I'm hoping this hand waving satiates tglx's morbid curiosity about hardware that came out from before I even worked at Intel. ;)
On Tue, Apr 25 2023 at 15:29, Dave Hansen wrote: > On 4/25/23 14:05, Thomas Gleixner wrote: >> The only consequence of looking at bit 0 of some random other leaf is >> that all CPUs which run stop_this_cpu() issue WBINVD in parallel, which >> is slow but should not be a fatal issue. >> >> Tony observed this is a 50% chance to hang, which means this is a timing >> issue. > > I _think_ the system in question is a dual-socket Westmere. I don't see > any obvious errata that we could pin this on: > >> https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-5600-specification-update.pdf > > Andi Kleen had an interesting theory. WBINVD is a pretty expensive > operation. It's possible that it has some degenerative behavior when > it's called on a *bunch* of CPUs all at once (which this path can do). > If the instruction takes too long, it could trigger one of the CPU's > internal lockup detectors and trigger a machine check. At that point, > all hell breaks loose. > > I don't know the cache coherency protocol well enough to say for sure, > but I wonder if there's a storm of cache coherency traffic as all those > lines get written back. One of the CPUs gets starved from making enough > forward progress and trips a CPU-internal watchdog. > > Andi also says that it _should_ log something in the machine check banks > when this happens so there should be at least some kind of breadcrumb. > > Either way, I'm hoping this hand waving satiates tglx's morbid curiosity > about hardware that came out from before I even worked at Intel. ;) No, it does not. :) There is no reason to believe that this is just a problem of CPUs which were released long time ago. If there is an issue with concurrent WBINVD then this needs to be addressed independently of Tony's observations. Aside of that the allowance for the control CPU to make progress based on the early clearing of the CPU online bit is still a possibility to explain the wreckage just based on timing. The reason why I insist on a proper analysis is definitely not morbid curiosity. The real reason is that I fundamentally hate problems being handwaved away. It's a matter of fact that all problems which are not root caused keep coming back and not necessarily in debuggable ways. Tony's 50% case is golden compared to the once in a blue moon issues. I outlined the debug options already. So just throw them at the problem instead of indulging in handwaing theories. Thanks, tglx
On April 25, 2023 3:29:49 PM PDT, Dave Hansen <dave.hansen@intel.com> wrote: >On 4/25/23 14:05, Thomas Gleixner wrote: >> The only consequence of looking at bit 0 of some random other leaf is >> that all CPUs which run stop_this_cpu() issue WBINVD in parallel, which >> is slow but should not be a fatal issue. >> >> Tony observed this is a 50% chance to hang, which means this is a timing >> issue. > >I _think_ the system in question is a dual-socket Westmere. I don't see >any obvious errata that we could pin this on: > >> https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-5600-specification-update.pdf > >Andi Kleen had an interesting theory. WBINVD is a pretty expensive >operation. It's possible that it has some degenerative behavior when >it's called on a *bunch* of CPUs all at once (which this path can do). >If the instruction takes too long, it could trigger one of the CPU's >internal lockup detectors and trigger a machine check. At that point, >all hell breaks loose. > >I don't know the cache coherency protocol well enough to say for sure, >but I wonder if there's a storm of cache coherency traffic as all those >lines get written back. One of the CPUs gets starved from making enough >forward progress and trips a CPU-internal watchdog. > >Andi also says that it _should_ log something in the machine check banks >when this happens so there should be at least some kind of breadcrumb. > >Either way, I'm hoping this hand waving satiates tglx's morbid curiosity >about hardware that came out from before I even worked at Intel. ;) "Pretty expensive" doesn't really cover it. It is by far the longest time an x86 CPU can block out all outside events.
On 4/25/23 17:05, Thomas Gleixner wrote: > On Tue, Apr 25 2023 at 13:03, Dave Hansen wrote: > >> On 4/25/23 12:26, Tony Battersby wrote: >>> - if (cpuid_eax(0x8000001f) & BIT(0)) >>> + if (c->extended_cpuid_level >= 0x8000001f && >>> + (cpuid_eax(0x8000001f) & BIT(0))) >>> native_wbinvd(); >> Oh, so the existing code is running into the >> >>> If a value entered for CPUID.EAX is higher than the maximum input >>> value for basic or extended function for that processor then the data >>> for the highest basic information leaf is returned >> behavior. It's basically looking at BIT(0) of some random extended >> leaf. Probably 0x80000008 based on your 'cpuid -r' output. > Right, accessing that leaf without checking whether it exists is wrong, > but that does not explain the hang itself. > > The only consequence of looking at bit 0 of some random other leaf is > that all CPUs which run stop_this_cpu() issue WBINVD in parallel, which > is slow but should not be a fatal issue. > > Tony observed this is a 50% chance to hang, which means this is a timing > issue. > > Now there are two things to investigate: > > 1) Does the system go south if enough CPUs issue WBINVD concurrently? > > That should be trivial to analyze by enforcing concurreny on a > WBINVD in an IPI via a global synchronization bit or such > > 2) The first thing stop_this_cpu() does is to clear its own bit in the > CPU online mask. > > The CPU which controls shutdown/reboot waits for num_online_cpus() > to drop down to one, which means it can proceed _before_ the other > CPUs have actually reached HALT. > > That's not a new thing. It has been that way forever. Just the > WBINVD might cause enough delay to create problems. > > That should be trivial to analyze too by just waiting on the > control side for e.g 100ms after num_online_cpus() dropped down to > one. > For test #1, I have never used IPI before, so I would have to look into how to do that. Or you could send me a patch to test if you still want the test done. But test #2 produced results, so maybe it is not necessary. For test #2, I re-enabled native_wbinvd() by reverting the patch that I sent, and then I applied the following patch: diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c index 375b33ecafa2..1a9b225c85b6 100644 --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -212,6 +212,7 @@ static void native_stop_other_cpus(int wait) udelay(1); } + mdelay(100); local_irq_save(flags); disable_local_APIC(); mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); With that I got a successful power-off 10 times in a row. Let me know if there is anything else I can test. I will resend my patch later with a different description. Tony Battersby Cybernetics
Tony! On Wed, Apr 26 2023 at 10:45, Tony Battersby wrote: > On 4/25/23 17:05, Thomas Gleixner wrote: > For test #1, I have never used IPI before, so I would have to look into > how to do that. Or you could send me a patch to test if you still want > the test done. But test #2 produced results, so maybe it is not > necessary. I think we can spare that exercise. > For test #2, I re-enabled native_wbinvd() by reverting the patch that I > sent, and then I applied the following patch: > > diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c > index 375b33ecafa2..1a9b225c85b6 100644 > --- a/arch/x86/kernel/smp.c > +++ b/arch/x86/kernel/smp.c > @@ -212,6 +212,7 @@ static void native_stop_other_cpus(int wait) > udelay(1); > } > > + mdelay(100); > local_irq_save(flags); > disable_local_APIC(); > mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); > > With that I got a successful power-off 10 times in a row. Thanks for trying this! The problem really seems to be that the control CPU goes off before the other CPUs have finished and depending on timing that causes the wreckage. Otherwise the mdelay(100) would not have helped at all. But looking at it, that num_online_cpus() == 1 check in stop_other_cpus() is fragile as hell independent of that wbinvd() issue. Something like the completely untested below should cure that. Thanks, tglx --- arch/x86/include/asm/cpu.h | 2 ++ arch/x86/kernel/process.c | 10 ++++++++++ arch/x86/kernel/smp.c | 15 ++++++++++++--- 3 files changed, 24 insertions(+), 3 deletions(-) --- a/arch/x86/include/asm/cpu.h +++ b/arch/x86/include/asm/cpu.h @@ -98,4 +98,6 @@ extern u64 x86_read_arch_cap_msr(void); int intel_find_matching_signature(void *mc, unsigned int csig, int cpf); int intel_microcode_sanity_check(void *mc, bool print_err, int hdr_type); +extern atomic_t stop_cpus_count; + #endif /* _ASM_X86_CPU_H */ --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -752,6 +752,8 @@ bool xen_set_default_idle(void) } #endif +atomic_t stop_cpus_count; + void __noreturn stop_this_cpu(void *dummy) { local_irq_disable(); @@ -776,6 +778,14 @@ void __noreturn stop_this_cpu(void *dumm */ if (cpuid_eax(0x8000001f) & BIT(0)) native_wbinvd(); + + /* + * native_stop_other_cpus() will write to @stop_cpus_count after + * observing that it went down to zero, which will invalidate the + * cacheline on this CPU. + */ + atomic_dec(&stop_cpus_count); + for (;;) { /* * Use native_halt() so that memory contents don't change --- a/arch/x86/kernel/smp.c +++ b/arch/x86/kernel/smp.c @@ -27,6 +27,7 @@ #include <asm/mmu_context.h> #include <asm/proto.h> #include <asm/apic.h> +#include <asm/cpu.h> #include <asm/idtentry.h> #include <asm/nmi.h> #include <asm/mce.h> @@ -171,6 +172,8 @@ static void native_stop_other_cpus(int w if (atomic_cmpxchg(&stopping_cpu, -1, safe_smp_processor_id()) != -1) return; + atomic_set(&stop_cpus_count, num_online_cpus() - 1); + /* sync above data before sending IRQ */ wmb(); @@ -183,12 +186,12 @@ static void native_stop_other_cpus(int w * CPUs reach shutdown state. */ timeout = USEC_PER_SEC; - while (num_online_cpus() > 1 && timeout--) + while (atomic_read(&stop_cpus_count) > 0 && timeout--) udelay(1); } /* if the REBOOT_VECTOR didn't work, try with the NMI */ - if (num_online_cpus() > 1) { + if (atomic_read(&stop_cpus_count) > 0) { /* * If NMI IPI is enabled, try to register the stop handler * and send the IPI. In any case try to wait for the other @@ -208,7 +211,7 @@ static void native_stop_other_cpus(int w * one or more CPUs do not reach shutdown state. */ timeout = USEC_PER_MSEC * 10; - while (num_online_cpus() > 1 && (wait || timeout--)) + while (atomic_read(&stop_cpus_count) > 0 && (wait || timeout--)) udelay(1); } @@ -216,6 +219,12 @@ static void native_stop_other_cpus(int w disable_local_APIC(); mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); local_irq_restore(flags); + + /* + * Ensure that the cache line is invalidated on the other CPUs. See + * comment vs. SME in stop_this_cpu(). + */ + atomic_set(&stop_cpus_count, INT_MAX); } /*
On 4/26/23 12:37, Thomas Gleixner wrote: > The problem really seems to be that the control CPU goes off before the > other CPUs have finished and depending on timing that causes the > wreckage. Otherwise the mdelay(100) would not have helped at all. > > But looking at it, that num_online_cpus() == 1 check in > stop_other_cpus() is fragile as hell independent of that wbinvd() issue. > > Something like the completely untested below should cure that. > > Thanks, > > tglx > --- > arch/x86/include/asm/cpu.h | 2 ++ > arch/x86/kernel/process.c | 10 ++++++++++ > arch/x86/kernel/smp.c | 15 ++++++++++++--- > 3 files changed, 24 insertions(+), 3 deletions(-) > > --- a/arch/x86/include/asm/cpu.h > +++ b/arch/x86/include/asm/cpu.h > @@ -98,4 +98,6 @@ extern u64 x86_read_arch_cap_msr(void); > int intel_find_matching_signature(void *mc, unsigned int csig, int cpf); > int intel_microcode_sanity_check(void *mc, bool print_err, int hdr_type); > > +extern atomic_t stop_cpus_count; > + > #endif /* _ASM_X86_CPU_H */ > --- a/arch/x86/kernel/process.c > +++ b/arch/x86/kernel/process.c > @@ -752,6 +752,8 @@ bool xen_set_default_idle(void) > } > #endif > > +atomic_t stop_cpus_count; > + > void __noreturn stop_this_cpu(void *dummy) > { > local_irq_disable(); > @@ -776,6 +778,14 @@ void __noreturn stop_this_cpu(void *dumm > */ > if (cpuid_eax(0x8000001f) & BIT(0)) > native_wbinvd(); > + > + /* > + * native_stop_other_cpus() will write to @stop_cpus_count after > + * observing that it went down to zero, which will invalidate the > + * cacheline on this CPU. > + */ > + atomic_dec(&stop_cpus_count); > + > for (;;) { > /* > * Use native_halt() so that memory contents don't change > --- a/arch/x86/kernel/smp.c > +++ b/arch/x86/kernel/smp.c > @@ -27,6 +27,7 @@ > #include <asm/mmu_context.h> > #include <asm/proto.h> > #include <asm/apic.h> > +#include <asm/cpu.h> > #include <asm/idtentry.h> > #include <asm/nmi.h> > #include <asm/mce.h> > @@ -171,6 +172,8 @@ static void native_stop_other_cpus(int w > if (atomic_cmpxchg(&stopping_cpu, -1, safe_smp_processor_id()) != -1) > return; > > + atomic_set(&stop_cpus_count, num_online_cpus() - 1); > + > /* sync above data before sending IRQ */ > wmb(); > > @@ -183,12 +186,12 @@ static void native_stop_other_cpus(int w > * CPUs reach shutdown state. > */ > timeout = USEC_PER_SEC; > - while (num_online_cpus() > 1 && timeout--) > + while (atomic_read(&stop_cpus_count) > 0 && timeout--) > udelay(1); > } > > /* if the REBOOT_VECTOR didn't work, try with the NMI */ > - if (num_online_cpus() > 1) { > + if (atomic_read(&stop_cpus_count) > 0) { > /* > * If NMI IPI is enabled, try to register the stop handler > * and send the IPI. In any case try to wait for the other > @@ -208,7 +211,7 @@ static void native_stop_other_cpus(int w > * one or more CPUs do not reach shutdown state. > */ > timeout = USEC_PER_MSEC * 10; > - while (num_online_cpus() > 1 && (wait || timeout--)) > + while (atomic_read(&stop_cpus_count) > 0 && (wait || timeout--)) > udelay(1); > } > > @@ -216,6 +219,12 @@ static void native_stop_other_cpus(int w > disable_local_APIC(); > mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); > local_irq_restore(flags); > + > + /* > + * Ensure that the cache line is invalidated on the other CPUs. See > + * comment vs. SME in stop_this_cpu(). > + */ > + atomic_set(&stop_cpus_count, INT_MAX); > } > > /* > Tested-by: Tony Battersby <tonyb@cybernetics.com> 10 successful poweroffs in a row with wbinvd() enabled. As I mentioned before though, I don't have an AMD CPU to test the SME cache invalidation logic. I will reply with my patch with an updated title and description. Tony
On 4/26/23 12:37, Tony Battersby wrote: > On 4/26/23 12:37, Thomas Gleixner wrote: >> The problem really seems to be that the control CPU goes off before the >> other CPUs have finished and depending on timing that causes the >> wreckage. Otherwise the mdelay(100) would not have helped at all. >> >> But looking at it, that num_online_cpus() == 1 check in >> stop_other_cpus() is fragile as hell independent of that wbinvd() issue. >> >> Something like the completely untested below should cure that. >> >> Thanks, >> >> tglx >> --- >> arch/x86/include/asm/cpu.h | 2 ++ >> arch/x86/kernel/process.c | 10 ++++++++++ >> arch/x86/kernel/smp.c | 15 ++++++++++++--- >> 3 files changed, 24 insertions(+), 3 deletions(-) >> >> --- a/arch/x86/include/asm/cpu.h >> +++ b/arch/x86/include/asm/cpu.h >> @@ -98,4 +98,6 @@ extern u64 x86_read_arch_cap_msr(void); >> int intel_find_matching_signature(void *mc, unsigned int csig, int cpf); >> int intel_microcode_sanity_check(void *mc, bool print_err, int hdr_type); >> >> +extern atomic_t stop_cpus_count; >> + >> #endif /* _ASM_X86_CPU_H */ >> --- a/arch/x86/kernel/process.c >> +++ b/arch/x86/kernel/process.c >> @@ -752,6 +752,8 @@ bool xen_set_default_idle(void) >> } >> #endif >> >> +atomic_t stop_cpus_count; >> + >> void __noreturn stop_this_cpu(void *dummy) >> { >> local_irq_disable(); >> @@ -776,6 +778,14 @@ void __noreturn stop_this_cpu(void *dumm >> */ >> if (cpuid_eax(0x8000001f) & BIT(0)) >> native_wbinvd(); >> + >> + /* >> + * native_stop_other_cpus() will write to @stop_cpus_count after >> + * observing that it went down to zero, which will invalidate the >> + * cacheline on this CPU. >> + */ >> + atomic_dec(&stop_cpus_count); This is probably going to pull in a cache line and cause the problem the native_wbinvd() is trying to avoid. Thanks, Tom >> + >> for (;;) { >> /* >> * Use native_halt() so that memory contents don't change >> --- a/arch/x86/kernel/smp.c >> +++ b/arch/x86/kernel/smp.c >> @@ -27,6 +27,7 @@ >> #include <asm/mmu_context.h> >> #include <asm/proto.h> >> #include <asm/apic.h> >> +#include <asm/cpu.h> >> #include <asm/idtentry.h> >> #include <asm/nmi.h> >> #include <asm/mce.h> >> @@ -171,6 +172,8 @@ static void native_stop_other_cpus(int w >> if (atomic_cmpxchg(&stopping_cpu, -1, safe_smp_processor_id()) != -1) >> return; >> >> + atomic_set(&stop_cpus_count, num_online_cpus() - 1); >> + >> /* sync above data before sending IRQ */ >> wmb(); >> >> @@ -183,12 +186,12 @@ static void native_stop_other_cpus(int w >> * CPUs reach shutdown state. >> */ >> timeout = USEC_PER_SEC; >> - while (num_online_cpus() > 1 && timeout--) >> + while (atomic_read(&stop_cpus_count) > 0 && timeout--) >> udelay(1); >> } >> >> /* if the REBOOT_VECTOR didn't work, try with the NMI */ >> - if (num_online_cpus() > 1) { >> + if (atomic_read(&stop_cpus_count) > 0) { >> /* >> * If NMI IPI is enabled, try to register the stop handler >> * and send the IPI. In any case try to wait for the other >> @@ -208,7 +211,7 @@ static void native_stop_other_cpus(int w >> * one or more CPUs do not reach shutdown state. >> */ >> timeout = USEC_PER_MSEC * 10; >> - while (num_online_cpus() > 1 && (wait || timeout--)) >> + while (atomic_read(&stop_cpus_count) > 0 && (wait || timeout--)) >> udelay(1); >> } >> >> @@ -216,6 +219,12 @@ static void native_stop_other_cpus(int w >> disable_local_APIC(); >> mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); >> local_irq_restore(flags); >> + >> + /* >> + * Ensure that the cache line is invalidated on the other CPUs. See >> + * comment vs. SME in stop_this_cpu(). >> + */ >> + atomic_set(&stop_cpus_count, INT_MAX); >> } >> >> /* >> > Tested-by: Tony Battersby <tonyb@cybernetics.com> > > 10 successful poweroffs in a row with wbinvd() enabled. As I mentioned > before though, I don't have an AMD CPU to test the SME cache > invalidation logic. > > I will reply with my patch with an updated title and description. > > Tony > >
On 4/26/23 10:51, Tom Lendacky wrote: >>> + /* >>> + * native_stop_other_cpus() will write to @stop_cpus_count after >>> + * observing that it went down to zero, which will invalidate the >>> + * cacheline on this CPU. >>> + */ >>> + atomic_dec(&stop_cpus_count); > > This is probably going to pull in a cache line and cause the problem the > native_wbinvd() is trying to avoid. Is one _more_ cacheline really the problem? Or is having _any_ cacheline pulled in a problem? What about the text page containing the WBINVD? How about all the page table pages that are needed to resolve %RIP to a physical address? What about the mds_idle_clear_cpu_buffers() code that snuck into native_halt()? > ffffffff810ede4c: 0f 09 wbinvd > ffffffff810ede4e: 8b 05 e4 3b a7 02 mov 0x2a73be4(%rip),%eax # ffffffff83b61a38 <mds_idle_clear> > ffffffff810ede54: 85 c0 test %eax,%eax > ffffffff810ede56: 7e 07 jle ffffffff810ede5f <stop_this_cpu+0x9f> > ffffffff810ede58: 0f 00 2d b1 75 13 01 verw 0x11375b1(%rip) # ffffffff82225410 <ds.6688> > ffffffff810ede5f: f4 hlt > ffffffff810ede60: eb ec jmp ffffffff810ede4e <stop_this_cpu+0x8e> > ffffffff810ede62: e8 59 40 1a 00 callq ffffffff81291ec0 <trace_hardirqs_off> > ffffffff810ede67: eb 85 jmp ffffffff810eddee <stop_this_cpu+0x2e> > ffffffff810ede69: 0f 1f 80 00 00 00 00 nopl 0x0(%rax)
On 4/26/23 13:15, Dave Hansen wrote: > On 4/26/23 10:51, Tom Lendacky wrote: >>>> + /* >>>> + * native_stop_other_cpus() will write to @stop_cpus_count after >>>> + * observing that it went down to zero, which will invalidate the >>>> + * cacheline on this CPU. >>>> + */ >>>> + atomic_dec(&stop_cpus_count); >> >> This is probably going to pull in a cache line and cause the problem the >> native_wbinvd() is trying to avoid. > > Is one _more_ cacheline really the problem? The answer is it depends. If the cacheline ends up modified/dirty, then it can be a problem. > > Or is having _any_ cacheline pulled in a problem? What about the text > page containing the WBINVD? How about all the page table pages that are > needed to resolve %RIP to a physical address? It's been a while since I looked into all this, but text and page table pages didn't present any problems because they weren't modified, but stack memory was. Doing a plain wbinvd() resulted in calls to the paravirt support and stack data from the call to wbinvd() ended up in some page structs in the kexec kernel (applicable to zen1 and zen2). Using native_wbinvd() eliminated the stack data changes after the WBINVD and didn't end up with any corruption following a kexec. > > What about the mds_idle_clear_cpu_buffers() code that snuck into > native_halt()? Luckily that is all inline and using a static branch which isn't enabled for AMD and should just jmp to the hlt, so no modified cache lines. Thanks, Tom > >> ffffffff810ede4c: 0f 09 wbinvd >> ffffffff810ede4e: 8b 05 e4 3b a7 02 mov 0x2a73be4(%rip),%eax # ffffffff83b61a38 <mds_idle_clear> >> ffffffff810ede54: 85 c0 test %eax,%eax >> ffffffff810ede56: 7e 07 jle ffffffff810ede5f <stop_this_cpu+0x9f> >> ffffffff810ede58: 0f 00 2d b1 75 13 01 verw 0x11375b1(%rip) # ffffffff82225410 <ds.6688> >> ffffffff810ede5f: f4 hlt >> ffffffff810ede60: eb ec jmp ffffffff810ede4e <stop_this_cpu+0x8e> >> ffffffff810ede62: e8 59 40 1a 00 callq ffffffff81291ec0 <trace_hardirqs_off> >> ffffffff810ede67: eb 85 jmp ffffffff810eddee <stop_this_cpu+0x2e> >> ffffffff810ede69: 0f 1f 80 00 00 00 00 nopl 0x0(%rax) >
On Wed, Apr 26 2023 at 12:51, Tom Lendacky wrote: > On 4/26/23 12:37, Tony Battersby wrote: >>> + /* >>> + * native_stop_other_cpus() will write to @stop_cpus_count after >>> + * observing that it went down to zero, which will invalidate the >>> + * cacheline on this CPU. >>> + */ >>> + atomic_dec(&stop_cpus_count); > > This is probably going to pull in a cache line and cause the problem the > native_wbinvd() is trying to avoid. The comment above this atomic_dec() explains why this is _not_ a problem. Here is the counterpart in native_stop_other_cpus(): >>> @@ -216,6 +219,12 @@ static void native_stop_other_cpus(int w >>> disable_local_APIC(); >>> mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); >>> local_irq_restore(flags); >>> + >>> + /* >>> + * Ensure that the cache line is invalidated on the other CPUs. See >>> + * comment vs. SME in stop_this_cpu(). >>> + */ >>> + atomic_set(&stop_cpus_count, INT_MAX); That happens _after_ all the other CPUs did the atomic_dec() as the control CPU waits for it to become 0. As this makes the cacheline exclusive on the control CPU the dirty cacheline on the CPU which did the last atomic_dec() is invalidated. As the atomic_dec() is obviously serialized via the lock prefix there can be only one dirty copy on some other CPU at the time when the control CPU writes to it. After that the only dirty copy is on the control CPU, no? Thanks, tglx
> > > This is probably going to pull in a cache line and cause the problem the > > > native_wbinvd() is trying to avoid. > > > > Is one _more_ cacheline really the problem? > > The answer is it depends. If the cacheline ends up modified/dirty, then it > can be a problem. I haven't followed this all in detail, but if any dirty cache line a problem you probably would need to be sure that any possible NMI user (like perf or watchdogs) is disabled at this point, otherwise you could still get NMIs here. I don't think perf currently has a mechanism to do that other than to offline the CPU. Also there are of course machine checks and SMIs that could still happen, but I guess there's nothing you could do about them. -Andi
Andi! On Wed, Apr 26 2023 at 15:02, Andi Kleen wrote: >> > > This is probably going to pull in a cache line and cause the problem the >> > > native_wbinvd() is trying to avoid. >> > >> > Is one _more_ cacheline really the problem? >> >> The answer is it depends. If the cacheline ends up modified/dirty, then it >> can be a problem. > > I haven't followed this all in detail, but if any dirty cache line a > problem you probably would need to be sure that any possible NMI user > (like perf or watchdogs) is disabled at this point, otherwise you could > still get NMIs here. > > I don't think perf currently has a mechanism to do that other > than to offline the CPU. stop_this_cpu() disable_local_APIC() apic_soft_disable() clear_local_APIC() v = apic_read(APIC_LVTPC); apic_write(APIC_LVTPC, v | APIC_LVT_MASKED); So after that point the PMU can't raise NMIs anymore which includes the default perf based NMI watchdog, no? External NMIs are a different problem, but they kinda fall into the same category as: > Also there are of course machine checks and SMIs that could still happen, > but I guess there's nothing you could do about them. Though external NMIs could be disabled via outb(0x70,...) if paranoid enough. Albeit if there is an external NMI based watchdog raising the NMI during this kexec() scenario then the outcome is probably as undefined as in the MCE case independent of the SEV dirty cacheline concern. Thanks, tglx
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index b650cde3f64d..26aa32e8f636 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -754,13 +754,15 @@ bool xen_set_default_idle(void) void __noreturn stop_this_cpu(void *dummy) { + struct cpuinfo_x86 *c = this_cpu_ptr(&cpu_info); + local_irq_disable(); /* * Remove this CPU: */ set_cpu_online(smp_processor_id(), false); disable_local_APIC(); - mcheck_cpu_clear(this_cpu_ptr(&cpu_info)); + mcheck_cpu_clear(c); /* * Use wbinvd on processors that support SME. This provides support @@ -774,7 +776,8 @@ void __noreturn stop_this_cpu(void *dummy) * Test the CPUID bit directly because the machine might've cleared * X86_FEATURE_SME due to cmdline options. */ - if (cpuid_eax(0x8000001f) & BIT(0)) + if (c->extended_cpuid_level >= 0x8000001f && + (cpuid_eax(0x8000001f) & BIT(0))) native_wbinvd(); for (;;) { /*