Message ID | 20230729003643.1053367-1-seanjc@google.com |
---|---|
Headers |
Return-Path: <linux-kernel-owner@vger.kernel.org> Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:918b:0:b0:3e4:2afc:c1 with SMTP id s11csp861422vqg; Fri, 28 Jul 2023 22:40:11 -0700 (PDT) X-Google-Smtp-Source: APBJJlFXEjAlIhBEb/4vWxfjISrc/eGrHZRxswp9XFEQCXddulPkeC30SZOqoDF2tQNbXfrws4V7 X-Received: by 2002:a05:6a20:6a0f:b0:130:b19d:ec1f with SMTP id p15-20020a056a206a0f00b00130b19dec1fmr3966997pzk.11.1690609211362; Fri, 28 Jul 2023 22:40:11 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1690609211; cv=none; d=google.com; s=arc-20160816; b=YVi8ffFG1zfCigMUHfM5nY+Ln79xab12ecUSQxcU7TXV6mLr21MgPI5hn+MhRCEKJ1 pOJ0s+L5hJj5ezz0xhEhq7cklzB8xnP0YoTLL/mFIbJaiaC1VOZZKuHM8lBBqf1WzhBx 29BHvnDxGSH/3il56HzjKQxWg6vlQVk3xQkT48S0sb1fuI5vWxbisQbd4NTDjioi2QjN oKSl5f3E10NmFEuP8REH9ir9lkhdzjr8DXtiWqAi+0TD01RG3wdW1ysqiyLJQVCUiuNh Epj5ZaRHqPzUnbxlOju/oYe+4EAfzmhnwBrgQSJH2mj9kOPsr4ZIEiv6UBV8U3tVbnsJ KmKg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:cc:to:from:subject:message-id:mime-version:date :reply-to:dkim-signature; bh=dMhJ/b6DMkuq/YdDrUWNaUZgSpR6tSiB89ISkM72Wx4=; fh=Ctt0ydInMKwei8YFBFbw7rU4p8H0G/T51ZI/KkO6Md4=; b=PzpZ18/dH1zr5Hxt+2PBr0pS8GnCPouKclva3tT+Zcl8chTZaf12fQq1f15Y0FTPUx hTGM1KaHL18TNszVJfvnhMW6Uv6S8dCqPoE3VbdCpJO3+4rMlh8s23a0PVTs01CkpCie LwKIEsoRfhWW86oZLv4XbMEjT95Hc01miti2YwJAYl9FgKxiaRj1pmpOWnuT/M6BsHfL add/Yf3ZLq1bEFv+8GluIzY8NgTl/4RDb6xqr2R9MpBGV6MaL89VALmV3r2yaF1Z+Wl1 EqiFuxHxQaoiimCOpR6fk5ZGga5gC4L3Cp2bfhbFV/kiZnVtaHoX/a2DTkxhd50o3FNI YWqw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=uDNZSukB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id a13-20020a634d0d000000b0055c5b8a8888si4082946pgb.684.2023.07.28.22.39.58; Fri, 28 Jul 2023 22:40:11 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20221208 header.b=uDNZSukB; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229740AbjG2Agv (ORCPT <rfc822;hanasaki@gmail.com> + 99 others); Fri, 28 Jul 2023 20:36:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57842 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230075AbjG2Ags (ORCPT <rfc822;linux-kernel@vger.kernel.org>); Fri, 28 Jul 2023 20:36:48 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8A8630F9 for <linux-kernel@vger.kernel.org>; Fri, 28 Jul 2023 17:36:46 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-5844e92ee6bso26604677b3.3 for <linux-kernel@vger.kernel.org>; Fri, 28 Jul 2023 17:36:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1690591006; x=1691195806; h=cc:to:from:subject:message-id:mime-version:date:reply-to:from:to:cc :subject:date:message-id:reply-to; bh=dMhJ/b6DMkuq/YdDrUWNaUZgSpR6tSiB89ISkM72Wx4=; b=uDNZSukByS7JyZ2jTEcrBQl7mGqhWoKqA2jPxWizvz2EyIAtNwjBDsbWnq8K25aP1c aGUNlsQCf/llPt5k93kDpL4/X2KOWZMToFa5soFdm8SQZ80FiVSihf9oTl/ob06d4uLB A89y28pB2flFbHcOhZSaDKEr1iY46y9zNb3+YdGFthAy7fQZVnSH2+uJlM3jA3U9b+s0 1lob8zXM5sLoRCYLPUaqo5Y8s5wf+iEZDOMb/VHGGR5MKtEIbQJv8IwQ6aUQjSQXD4nL iyyyuFW0dkdEnlSBWkk1bEMBBgfS5s9+l9tIhsWOiBjr/xL04hP/43Ge/C3IBQpzoNAN hyAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690591006; x=1691195806; h=cc:to:from:subject:message-id:mime-version:date:reply-to :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dMhJ/b6DMkuq/YdDrUWNaUZgSpR6tSiB89ISkM72Wx4=; b=HAk992tjJvyuZj8gFpnfNWT7owe+OTJ4DIx+LC+yUAKYS9y6V1a/V9XFen6xZfWIO4 4RYa834iy56AJg1yjrxAZ8ZbC2onLaFwZXhNLuU+OM32N7b7v+wgr9OLbrr3ITdX6msh DktVf3Qb8oPdDGojlcTVPs8XBpRFqBtIsEdjQqnnxB2TlVLyt29NZMfGGOcFOAwVTdZo ev6nbEc08wnKXcsQcDOsPYIY+jhXtyH7v47aD0uIp8EA9fj+VJ4Rs2VU9iBfQ4J3L9mM T6/SyCdWqcrsNktVpCphdmiIxSjBl6FaKQld35zVcBe9+UZAd3SkUBiDZNJ7kHk/K09A 2jyA== X-Gm-Message-State: ABy/qLZ+shXYH0t5oTINIvZErmKRpG9iUDcBJz9rGuqUPtZZVKPxswPI rlDbzFa6issNskHAvcz7BKaih91hVNk= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:19c2:0:b0:d0d:1563:58f2 with SMTP id 185-20020a2519c2000000b00d0d156358f2mr17145ybz.2.1690591006063; Fri, 28 Jul 2023 17:36:46 -0700 (PDT) Reply-To: Sean Christopherson <seanjc@google.com> Date: Fri, 28 Jul 2023 17:36:09 -0700 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.487.g6d72f3e995-goog Message-ID: <20230729003643.1053367-1-seanjc@google.com> Subject: [PATCH v4 00/34] KVM: selftests: Guest printf and asserts overhaul From: Sean Christopherson <seanjc@google.com> To: Paolo Bonzini <pbonzini@redhat.com>, Marc Zyngier <maz@kernel.org>, Oliver Upton <oliver.upton@linux.dev>, Christian Borntraeger <borntraeger@linux.ibm.com>, Janosch Frank <frankja@linux.ibm.com>, Claudio Imbrenda <imbrenda@linux.ibm.com> Cc: kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, Sean Christopherson <seanjc@google.com>, Thomas Huth <thuth@redhat.com>, " =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= " <philmd@linaro.org>, Aaron Lewis <aaronlewis@google.com> Content-Type: text/plain; charset="UTF-8" X-Spam-Status: No, score=-9.6 required=5.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,RCVD_IN_DNSWL_NONE, SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: <linux-kernel.vger.kernel.org> X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: INBOX X-GMAIL-THRID: 1772732244240249949 X-GMAIL-MSGID: 1772732244240249949 |
Series |
KVM: selftests: Guest printf and asserts overhaul
|
|
Message
Sean Christopherson
July 29, 2023, 12:36 a.m. UTC
This is effectively v4 of Aaron's series to add printf capabilities to the guest[*]. I also pulled in Thomas' patch to rename ASSERT_EQ() to TEST_ASSERT_EQ(), mainly so that we can decide on a common output format for both host and guest asserts in a single series, but also so that all these basically treewide patches are contained in a single series. Note, Aaron did all of the heavy lifting, I just mopped up. The core code is pretty much unchanged from Aaron's v3, v4 massages the assert code a bit and converts all the tests. I initially did the conversion in one big patch (the flag is ugly), but after hitting a nasty bug (see "Add a shameful hack to preserve/clobber GPRs across ucall") that occurred purely due to the compiler using registers differently, I decided splitting it up was probably for the best (even though I still think it probably wasn't worth the effort). Other maintainers, I want to get this into 6.6, hell or high water. The ability to use proper asserts in the guest is super nice, and given how much selftests code is written via CTRL-C + CTRL-V, the sooner we switch over the better. My thought is to apply this whole thing to kvm-x86/selftests early next week, and then create a tag to make that part of branch immutable. That would allow other architectures to pull the code into their trees, e.g. if an arch is gaining a big selftest or something. Apologies for not giving advance warning, I was originally thinking we could leisurely convert to the printf-based asserts, but then realized that we would probably never get rid of the old crud if we tried that approach. Any objections, or better ideas? Thanks! Oh, and tested on Intel, AMD, and whatever flavor of ARM we have. Compile tested on s390 and RISC-V. [*] https://lore.kernel.org/all/20230607224520.4164598-1-aaronlewis@google.com Aaron Lewis (5): KVM: selftests: Add strnlen() to the string overrides KVM: selftests: Add guest_snprintf() to KVM selftests KVM: selftests: Add additional pages to the guest to accommodate ucall KVM: selftests: Add string formatting options to ucall KVM: selftests: Add a selftest for guest prints and formatted asserts Sean Christopherson (28): KVM: selftests: Make TEST_ASSERT_EQ() output look like normal TEST_ASSERT() KVM: selftests: Add a shameful hack to preserve/clobber GPRs across ucall KVM: selftests: Add formatted guest assert support in ucall framework KVM: selftests: Convert aarch_timer to printf style GUEST_ASSERT KVM: selftests: Convert debug-exceptions to printf style GUEST_ASSERT KVM: selftests: Convert ARM's hypercalls test to printf style GUEST_ASSERT KVM: selftests: Convert ARM's page fault test to printf style GUEST_ASSERT KVM: selftests: Convert ARM's vGIC IRQ test to printf style GUEST_ASSERT KVM: selftests: Convert the memslot performance test to printf guest asserts KVM: selftests: Convert s390's memop test to printf style GUEST_ASSERT KVM: selftests: Convert s390's tprot test to printf style GUEST_ASSERT KVM: selftests: Convert set_memory_region_test to printf-based GUEST_ASSERT KVM: selftests: Convert steal_time test to printf style GUEST_ASSERT KVM: selftests: Convert x86's CPUID test to printf style GUEST_ASSERT KVM: selftests: Convert the Hyper-V extended hypercalls test to printf asserts KVM: selftests: Convert the Hyper-V feature test to printf style GUEST_ASSERT KVM: selftests: Convert x86's KVM paravirt test to printf style GUEST_ASSERT KVM: selftests: Convert the MONITOR/MWAIT test to use printf guest asserts KVM: selftests: Convert x86's nested exceptions test to printf guest asserts KVM: selftests: Convert x86's set BSP ID test to printf style guest asserts KVM: selftests: Convert the nSVM software interrupt test to printf guest asserts KVM: selftests: Convert x86's TSC MSRs test to use printf guest asserts KVM: selftests: Convert the x86 userspace I/O test to printf guest assert KVM: selftests: Convert VMX's PMU capabilities test to printf guest asserts KVM: selftests: Convert x86's XCR0 test to use printf-based guest asserts KVM: selftests: Rip out old, param-based guest assert macros KVM: selftests: Print out guest RIP on unhandled exception KVM: selftests: Use GUEST_FAIL() in ARM's arch timer helpers Thomas Huth (1): KVM: selftests: Rename the ASSERT_EQ macro tools/testing/selftests/kvm/Makefile | 3 + .../selftests/kvm/aarch64/aarch32_id_regs.c | 8 +- .../selftests/kvm/aarch64/arch_timer.c | 22 +- .../selftests/kvm/aarch64/debug-exceptions.c | 8 +- .../selftests/kvm/aarch64/hypercalls.c | 20 +- .../selftests/kvm/aarch64/page_fault_test.c | 17 +- .../testing/selftests/kvm/aarch64/vgic_irq.c | 3 +- .../testing/selftests/kvm/guest_print_test.c | 221 +++++++++++++ .../kvm/include/aarch64/arch_timer.h | 12 +- .../testing/selftests/kvm/include/test_util.h | 18 +- .../selftests/kvm/include/ucall_common.h | 97 +++--- .../testing/selftests/kvm/lib/guest_sprintf.c | 307 ++++++++++++++++++ tools/testing/selftests/kvm/lib/kvm_util.c | 6 +- .../selftests/kvm/lib/string_override.c | 9 + .../testing/selftests/kvm/lib/ucall_common.c | 44 +++ .../selftests/kvm/lib/x86_64/processor.c | 18 +- .../testing/selftests/kvm/lib/x86_64/ucall.c | 32 +- .../selftests/kvm/max_guest_memory_test.c | 2 +- .../testing/selftests/kvm/memslot_perf_test.c | 4 +- tools/testing/selftests/kvm/s390x/cmma_test.c | 62 ++-- tools/testing/selftests/kvm/s390x/memop.c | 13 +- tools/testing/selftests/kvm/s390x/tprot.c | 11 +- .../selftests/kvm/set_memory_region_test.c | 21 +- tools/testing/selftests/kvm/steal_time.c | 20 +- .../testing/selftests/kvm/x86_64/cpuid_test.c | 12 +- .../x86_64/dirty_log_page_splitting_test.c | 18 +- .../x86_64/exit_on_emulation_failure_test.c | 2 +- .../kvm/x86_64/hyperv_extended_hypercalls.c | 3 +- .../selftests/kvm/x86_64/hyperv_features.c | 29 +- .../selftests/kvm/x86_64/kvm_pv_test.c | 8 +- .../selftests/kvm/x86_64/monitor_mwait_test.c | 35 +- .../kvm/x86_64/nested_exceptions_test.c | 16 +- .../kvm/x86_64/recalc_apic_map_test.c | 6 +- .../selftests/kvm/x86_64/set_boot_cpu_id.c | 6 +- .../kvm/x86_64/svm_nested_soft_inject_test.c | 22 +- .../selftests/kvm/x86_64/tsc_msrs_test.c | 34 +- .../selftests/kvm/x86_64/userspace_io_test.c | 10 +- .../vmx_exception_with_invalid_guest_state.c | 2 +- .../selftests/kvm/x86_64/vmx_pmu_caps_test.c | 31 +- .../selftests/kvm/x86_64/xapic_state_test.c | 8 +- .../selftests/kvm/x86_64/xcr0_cpuid_test.c | 29 +- .../selftests/kvm/x86_64/xen_vmcall_test.c | 20 +- 42 files changed, 938 insertions(+), 331 deletions(-) create mode 100644 tools/testing/selftests/kvm/guest_print_test.c create mode 100644 tools/testing/selftests/kvm/lib/guest_sprintf.c base-commit: fdf0eaf11452d72945af31804e2a1048ee1b574c
Comments
On Fri, 28 Jul 2023 17:36:09 -0700, Sean Christopherson wrote: > This is effectively v4 of Aaron's series to add printf capabilities to > the guest[*]. I also pulled in Thomas' patch to rename ASSERT_EQ() to > TEST_ASSERT_EQ(), mainly so that we can decide on a common output format > for both host and guest asserts in a single series, but also so that all > these basically treewide patches are contained in a single series. > > Note, Aaron did all of the heavy lifting, I just mopped up. The core code > is pretty much unchanged from Aaron's v3, v4 massages the assert code a > bit and converts all the tests. > > [...] Applied to kvm-x86 selftests and officially started this runaway train. I'll give this a few days to stew in linux-next, and assuming no fireworks, will create a tag and make everything up to that point immutable. I will follow-up with all the in-flight selftests series I can find to coordinate, though I'm hoping we'll get lucky and just not have any major conflitcs. Please yell asap if you encounter issues and/or object to any of this. Note, I technically applied patch 1 from Thomas' original posting, but captured it here too as I figured that'd be the least confusing thing. [1/36] KVM: selftests: Rename the ASSERT_EQ macro https://github.com/kvm-x86/linux/commit/6d85f51a1f08 [2/36] KVM: selftests: Make TEST_ASSERT_EQ() output look like normal TEST_ASSERT() https://github.com/kvm-x86/linux/commit/b145c58d95ff [3/36] KVM: selftests: Add a shameful hack to preserve/clobber GPRs across ucall https://github.com/kvm-x86/linux/commit/6783ca4105a7 [4/36] KVM: selftests: Add strnlen() to the string overrides https://github.com/kvm-x86/linux/commit/a1c1b55e116c [5/36] KVM: selftests: Add guest_snprintf() to KVM selftests https://github.com/kvm-x86/linux/commit/e5119382499c [6/36] KVM: selftests: Add additional pages to the guest to accommodate ucall https://github.com/kvm-x86/linux/commit/215a681710a5 [7/36] KVM: selftests: Add string formatting options to ucall https://github.com/kvm-x86/linux/commit/57e5c1fef5ec [8/36] KVM: selftests: Add formatted guest assert support in ucall framework https://github.com/kvm-x86/linux/commit/289c2b4db8f3 [9/36] KVM: selftests: Add arch ucall.h and inline simple arch hooks https://github.com/kvm-x86/linux/commit/b35f4c73d389 [10/36] KVM: selftests: Add #define of expected KVM exit reason for ucall https://github.com/kvm-x86/linux/commit/edb5b700f9f8 [11/36] KVM: selftests: Add a selftest for guest prints and formatted asserts https://github.com/kvm-x86/linux/commit/5d1d46f9d56f [12/36] KVM: selftests: Convert aarch_timer to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/db44e1c871bc [13/36] KVM: selftests: Convert debug-exceptions to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/bac9aeecc387 [14/36] KVM: selftests: Convert ARM's hypercalls test to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/af5b41b97f1c [15/36] KVM: selftests: Convert ARM's page fault test to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/df27f6b45454 [16/36] KVM: selftests: Convert ARM's vGIC IRQ test to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/d0ad3bacc523 [17/36] KVM: selftests: Convert the memslot performance test to printf guest asserts https://github.com/kvm-x86/linux/commit/c55a475d5fc4 [18/36] KVM: selftests: Convert s390's memop test to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/428c76c769fa [19/36] KVM: selftests: Convert s390's tprot test to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/5f82bbab84ad [20/36] KVM: selftests: Convert set_memory_region_test to printf-based GUEST_ASSERT https://github.com/kvm-x86/linux/commit/9291c9cef5b5 [21/36] KVM: selftests: Convert steal_time test to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/3d9bd831175e [22/36] KVM: selftests: Convert x86's CPUID test to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/06b651d250e5 [23/36] KVM: selftests: Convert the Hyper-V extended hypercalls test to printf asserts https://github.com/kvm-x86/linux/commit/82cb0ed66d4e [24/36] KVM: selftests: Convert the Hyper-V feature test to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/8d1d3ce604e5 [25/36] KVM: selftests: Convert x86's KVM paravirt test to printf style GUEST_ASSERT https://github.com/kvm-x86/linux/commit/bf6c760b9df3 [26/36] KVM: selftests: Convert the MONITOR/MWAIT test to use printf guest asserts https://github.com/kvm-x86/linux/commit/0f52e4aaa614 [27/36] KVM: selftests: Convert x86's nested exceptions test to printf guest asserts https://github.com/kvm-x86/linux/commit/b13a307ce3c6 [28/36] KVM: selftests: Convert x86's set BSP ID test to printf style guest asserts https://github.com/kvm-x86/linux/commit/40b319d6b4e1 [29/36] KVM: selftests: Convert the nSVM software interrupt test to printf guest asserts https://github.com/kvm-x86/linux/commit/a925f7994281 [30/36] KVM: selftests: Convert x86's TSC MSRs test to use printf guest asserts https://github.com/kvm-x86/linux/commit/847ae0795514 [31/36] KVM: selftests: Convert the x86 userspace I/O test to printf guest assert https://github.com/kvm-x86/linux/commit/417bfd0c820f [32/36] KVM: selftests: Convert VMX's PMU capabilities test to printf guest asserts https://github.com/kvm-x86/linux/commit/30a6e0b4553d [33/36] KVM: selftests: Convert x86's XCR0 test to use printf-based guest asserts https://github.com/kvm-x86/linux/commit/4e15c38a1aca [34/36] KVM: selftests: Rip out old, param-based guest assert macros https://github.com/kvm-x86/linux/commit/7ce7f8e75418 [35/36] KVM: selftests: Print out guest RIP on unhandled exception https://github.com/kvm-x86/linux/commit/6f321017c84b [36/36] KVM: selftests: Use GUEST_FAIL() in ARM's arch timer helpers https://github.com/kvm-x86/linux/commit/a05c4c2bd8b5 -- https://github.com/kvm-x86/linux/tree/next https://github.com/kvm-x86/linux/tree/fixes