From patchwork Mon Apr 17 09:59:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: zhaotianrui X-Patchwork-Id: 8286 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:b0ea:0:b0:3b6:4342:cba0 with SMTP id b10csp2012540vqo; Mon, 17 Apr 2023 03:22:59 -0700 (PDT) X-Google-Smtp-Source: AKy350ZfijL37+KQshQsyVjBMdUWqRxVDoSvp2dMNB5OJzpT6xauAxqKXr+v7GtyzNyWMO2Q+Hy7 X-Received: by 2002:a17:902:c7cd:b0:19a:a520:b203 with SMTP id r13-20020a170902c7cd00b0019aa520b203mr10067075pla.25.1681726979160; Mon, 17 Apr 2023 03:22:59 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1681726979; cv=none; d=google.com; s=arc-20160816; b=vFeu2zAni8bz0MbEmpYcmc6/8m5CkW/soURR8pYllowzkAOXVw1r9xWIoXxi01CUFq Us2Y0vfD2P3DaqhAM+CSC/eMMaw4hektHkOqty8+7ern6pDMtXHCQYZ32mDYgS9IeL2d ceOlNUT/Dvf66c8EhAJ3HsHytjEOKRjwoSIq2X4zDIfdhmnRcKY+XVFbSutXzpa9yhWt sQdnITe6UX7+s98SrAynkhgeQov7VRKHRIVE+yu1KQNGpanm5nhtquK3ShjU/tRped62 0sDQTPt86NUKkg7BAbv5tsCh12N+32jnKDUPf9GTaBbSxDDtaqd0Ix5Sw2mr1VJiy21S FBlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :message-id:date:subject:cc:to:from; bh=GGY06tJ8gO24kSp/t/Vi770nYb9h0QqEQTT7SMsDuEw=; b=meU875FSmFQZh1uV1sYC+atVs9LiyZjj8fVmpfL/W+Wwqixr2PL0kt/cYY9oZv9nKZ PmvcgJtY6vDAnQ/IsmFdgB2as9Q1aSc5M9v+oLjoVQ0qroAZsATUJ6Y+ZA2UXzgQA7Wi JwB+tnv6a0k1umkDfPAybDcdk+njUREV9xkllloz2VQz+t2l23scKQhzMf3P/aV/JKvw TCYoloMMDXQapY4SsbYIotsVNDfNEiV6EvQuwJLS6VSvaPau9bCK0pHF2qQv/pRdi1Q6 bkK5j/AeWop10IOW+wYDGTQR//Xn5eAsQqsCVfuEDSdqfAudHEJnBBvZQmvboU5eaOoJ 43/g== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id u9-20020a170902b28900b00194bcc88b52si11093049plr.363.2023.04.17.03.22.45; Mon, 17 Apr 2023 03:22:59 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229878AbjDQKB2 (ORCPT + 99 others); Mon, 17 Apr 2023 06:01:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34598 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229657AbjDQKBV (ORCPT ); Mon, 17 Apr 2023 06:01:21 -0400 Received: from loongson.cn (mail.loongson.cn [114.242.206.163]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 45C0A40E1; Mon, 17 Apr 2023 03:00:40 -0700 (PDT) Received: from loongson.cn (unknown [10.2.5.185]) by gateway (Coremail) with SMTP id _____8AxEk6YGD1k2eUdAA--.34821S3; Mon, 17 Apr 2023 17:59:52 +0800 (CST) Received: from localhost.localdomain (unknown [10.2.5.185]) by localhost.localdomain (Coremail) with SMTP id AQAAf8Bx57yWGD1kua0qAA--.48254S2; Mon, 17 Apr 2023 17:59:50 +0800 (CST) From: Tianrui Zhao To: Paolo Bonzini Cc: Huacai Chen , WANG Xuerui , Greg Kroah-Hartman , loongarch@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, Jens Axboe , Mark Brown , Alex Deucher , Oliver Upton , maobibo@loongson.cn, Xi Ruoyao , zhaotianrui@loongson.cn Subject: [PATCH v7 00/30] Add KVM LoongArch support Date: Mon, 17 Apr 2023 17:59:20 +0800 Message-Id: <20230417095950.875228-1-zhaotianrui@loongson.cn> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-CM-TRANSID: AQAAf8Bx57yWGD1kua0qAA--.48254S2 X-CM-SenderInfo: p2kd03xldq233l6o00pqjv00gofq/ X-Coremail-Antispam: 1Uk129KBjvJXoWxKFWkZFWrGr4kGry7Jr1DKFg_yoWDGrWDpF W7urn8Gr4DGrWSq395t3s7Zr15XF1xGrWag3Wav348CrW2qry8ZFWkKr9FvF9rA3ykJry0 qr1rKw1Yg3WUAaDanT9S1TB71UUUU8DqnTZGkaVYY2UrUUUUj1kv1TuYvTs0mT0YCTnIWj qI5I8CrVACY4xI64kE6c02F40Ex7xfYxn0WfASr-VFAUDa7-sFnT9fnUUIcSsGvfJTRUUU b4AFc2x0x2IEx4CE42xK8VAvwI8IcIk0rVWrJVCq3wA2ocxC64kIII0Yj41l84x0c7CEw4 AK67xGY2AK021l84ACjcxK6xIIjxv20xvE14v26w1j6s0DM28EF7xvwVC0I7IYx2IY6xkF 7I0E14v26r4UJVWxJr1l84ACjcxK6I8E87Iv67AKxVWxJr0_GcWl84ACjcxK6I8E87Iv6x kF7I0E14v26F4UJVW0owAaw2AFwI0_GFv_Wryle2I262IYc4CY6c8Ij28IcVAaY2xG8wAq jxCEc2xF0cIa020Ex4CE44I27wAqx4xG64xvF2IEw4CE5I8CrVC2j2WlYx0E74AGY7Cv6c x26rWlOx8S6xCaFVCjc4AY6r1j6r4UM4x0Y48IcxkI7VAKI48JMxkF7I0En4kS14v26r4a 6rW5MxAIw28IcxkI7VAKI48JMxAIw28IcVCjz48v1sIEY20_WwCFx2IqxVCFs4IE7xkEbV WUJVW8JwCFI7km07C267AKxVW8ZVWrXwC20s026c02F40E14v26r1j6r18MI8I3I0E7480 Y4vE14v26r106r1rMI8E67AF67kF1VAFwI0_Jw0_GFylIxkGc2Ij64vIr41lIxAIcVC0I7 IYx2IY67AKxVWDJVCq3wCI42IY6xIIjxv20xvEc7CjxVAFwI0_Gr1j6F4UJwCI42IY6xAI w20EY4v20xvaj40_Jr0_JF4lIxAIcVC2z280aVAFwI0_Gr1j6F4UJwCI42IY6I8E87Iv6x kF7I0E14v26r4UJVWxJrUvcSsGvfC2KfnxnUUI43ZEXa7IU8AcTPUUUUU== X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,SPF_HELO_PASS, SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1763416860022793253?= X-GMAIL-MSGID: =?utf-8?q?1763418549253623782?= This series adds KVM LoongArch support. Loongson 3A5000 supports hardware assisted virtualization. With cpu virtualization, there are separate hw-supported user mode and kernel mode in guest mode. With memory virtualization, there are two-level hw mmu table for guest mode and host mode. Also there is separate hw cpu timer with consant frequency in guest mode, so that vm can migrate between hosts with different freq. Currently, we are able to boot LoongArch Linux Guests. Few key aspects of KVM LoongArch added by this series are: 1. Enable kvm hardware function when kvm module is loaded. 2. Implement VM and vcpu related ioctl interface such as vcpu create, vcpu run etc. GET_ONE_REG/SET_ONE_REG ioctl commands are use to get general registers one by one. 3. Hardware access about MMU, timer and csr are emulated in kernel. 4. Hardwares such as mmio and iocsr device are emulated in user space such as APIC, IPI, pci devices etc. The running environment of LoongArch virt machine: 1. Cross tools to build kernel and uefi: $ wget https://github.com/loongson/build-tools/releases/download/2022.09.06/loongarch64-clfs-6.3-cross-tools-gcc-glibc.tar.xz tar -vxf loongarch64-clfs-6.3-cross-tools-gcc-glibc.tar.xz -C /opt export PATH=/opt/cross-tools/bin:$PATH export LD_LIBRARY_PATH=/opt/cross-tools/lib:$LD_LIBRARY_PATH export LD_LIBRARY_PATH=/opt/cross-tools/loongarch64-unknown-linux-gnu/lib/:$LD_LIBRARY_PATH 2. This series is based on the linux source code: https://github.com/loongson/linux-loongarch-kvm Build command: git checkout kvm-loongarch make ARCH=loongarch CROSS_COMPILE=loongarch64-unknown-linux-gnu- loongson3_defconfig make ARCH=loongarch CROSS_COMPILE=loongarch64-unknown-linux-gnu- 3. QEMU hypervisor with LoongArch supported: https://github.com/loongson/qemu Build command: git checkout kvm-loongarch ./configure --target-list="loongarch64-softmmu" --enable-kvm make 4. Uefi bios of LoongArch virt machine: Link: https://github.com/tianocore/edk2-platforms/tree/master/Platform/Loongson/LoongArchQemuPkg#readme 5. you can also access the binary files we have already build: https://github.com/yangxiaojuan-loongson/qemu-binary The command to boot loongarch virt machine: $ qemu-system-loongarch64 -machine virt -m 4G -cpu la464 \ -smp 1 -bios QEMU_EFI.fd -kernel vmlinuz.efi -initrd ramdisk \ -serial stdio -monitor telnet:localhost:4495,server,nowait \ -append "root=/dev/ram rdinit=/sbin/init console=ttyS0,115200" \ --nographic Changes for v7: 1. Fix the kvm_save/restore_hw_gcsr compiling warnings reported by kernel test robot. The report link is: https://lore.kernel.org/oe-kbuild-all/202304131526.iXfLaVZc-lkp@intel.com/ 2. Fix loongarch kvm trace related compiling problems. Changes for v6: 1. Fix the Documentation/virt/kvm/api.rst compile warning about loongarch parts. Changes for v5: 1. Implement get/set mp_state ioctl interface, and only the KVM_MP_STATE_RUNNABLE state is supported now, and other states will be completed in the future. The state is also used when vcpu run idle instruction, if vcpu state is changed to RUNNABLE, the vcpu will have the possibility to be woken up. 2. Supplement kvm document about loongarch-specific part, such as add api introduction for GET/SET_ONE_REG, GET/SET_FPU, GET/SET_MP_STATE, etc. 3. Improve the kvm_switch_to_guest function in switch.S, remove the previous tmp,tmp1 arguments and replace it with t0,t1 reg. Changes for v4: 1. Add a csr_need_update flag in _vcpu_put, as most csr registers keep unchanged during process context switch, so we need not to update it every time. We can do this only if the soft csr is different form hardware. That is to say all of csrs should update after vcpu enter guest, as for set_csr_ioctl, we have written soft csr to keep consistent with hardware. 2. Improve get/set_csr_ioctl interface, we set SW or HW or INVALID flag for all csrs according to it's features when kvm init. In get/set_csr_ioctl, if csr is HW, we use gcsrrd/ gcsrwr instruction to access it, else if csr is SW, we use software to emulate it, and others return false. 3. Add set_hw_gcsr function in csr_ops.S, and it is used in set_csr_ioctl. We have splited hw gcsr into three parts, so we can calculate the code offset by gcsrid and jump here to run the gcsrwr instruction. We use this function to make the code easier and avoid to use the previous SET_HW_GCSR(XXX) interface. 4. Improve kvm mmu functions, such as flush page table and make clean page table interface. Changes for v3: 1. Remove the vpid array list in kvm_vcpu_arch and use a vpid variable here, because a vpid will never be recycled if a vCPU migrates from physical CPU A to B and back to A. 2. Make some constant variables in kvm_context to global such as vpid_mask, guest_eentry, enter_guest, etc. 3. Add some new tracepoints, such as kvm_trace_idle, kvm_trace_cache, kvm_trace_gspr, etc. 4. There are some duplicate codes in kvm_handle_exit and kvm_vcpu_run, so we move it to a new function kvm_pre_enter_guest. 5. Change the RESUME_HOST, RESUME_GUEST value, return 1 for resume guest and "<= 0" for resume host. 6. Fcsr and fpu registers are saved/restored together. Changes for v2: 1. Seprate the original patch-01 and patch-03 into small patches, and the patches mainly contain kvm module init, module exit, vcpu create, vcpu run, etc. 2. Remove the original KVM_{GET,SET}_CSRS ioctl in the kvm uapi header, and we use the common KVM_{GET,SET}_ONE_REG to access register. 3. Use BIT(x) to replace the "1 << n_bits" statement. Tianrui Zhao (30): LoongArch: KVM: Add kvm related header files LoongArch: KVM: Implement kvm module related interface LoongArch: KVM: Implement kvm hardware enable, disable interface LoongArch: KVM: Implement VM related functions LoongArch: KVM: Add vcpu related header files LoongArch: KVM: Implement vcpu create and destroy interface LoongArch: KVM: Implement vcpu run interface LoongArch: KVM: Implement vcpu handle exit interface LoongArch: KVM: Implement vcpu get, vcpu set registers LoongArch: KVM: Implement vcpu ENABLE_CAP ioctl interface LoongArch: KVM: Implement fpu related operations for vcpu LoongArch: KVM: Implement vcpu interrupt operations LoongArch: KVM: Implement misc vcpu related interfaces LoongArch: KVM: Implement vcpu load and vcpu put operations LoongArch: KVM: Implement vcpu status description LoongArch: KVM: Implement update VM id function LoongArch: KVM: Implement virtual machine tlb operations LoongArch: KVM: Implement vcpu timer operations LoongArch: KVM: Implement kvm mmu operations LoongArch: KVM: Implement handle csr excption LoongArch: KVM: Implement handle iocsr exception LoongArch: KVM: Implement handle idle exception LoongArch: KVM: Implement handle gspr exception LoongArch: KVM: Implement handle mmio exception LoongArch: KVM: Implement handle fpu exception LoongArch: KVM: Implement kvm exception vector LoongArch: KVM: Implement vcpu world switch LoongArch: KVM: Implement probe virtualization when loongarch cpu init LoongArch: KVM: Enable kvm config and add the makefile LoongArch: KVM: Supplement kvm document about loongarch-specific part Documentation/virt/kvm/api.rst | 71 +- arch/loongarch/Kbuild | 1 + arch/loongarch/Kconfig | 2 + arch/loongarch/configs/loongson3_defconfig | 2 + arch/loongarch/include/asm/cpu-features.h | 22 + arch/loongarch/include/asm/cpu-info.h | 13 + arch/loongarch/include/asm/inst.h | 16 + arch/loongarch/include/asm/kvm_csr.h | 58 ++ arch/loongarch/include/asm/kvm_host.h | 268 +++++++ arch/loongarch/include/asm/kvm_types.h | 11 + arch/loongarch/include/asm/kvm_vcpu.h | 114 +++ arch/loongarch/include/asm/loongarch.h | 213 ++++- arch/loongarch/include/uapi/asm/kvm.h | 107 +++ arch/loongarch/kernel/asm-offsets.c | 32 + arch/loongarch/kernel/cpu-probe.c | 53 ++ arch/loongarch/kvm/Kconfig | 38 + arch/loongarch/kvm/Makefile | 22 + arch/loongarch/kvm/csr_ops.S | 76 ++ arch/loongarch/kvm/exit.c | 707 +++++++++++++++++ arch/loongarch/kvm/interrupt.c | 126 +++ arch/loongarch/kvm/main.c | 340 ++++++++ arch/loongarch/kvm/mmu.c | 730 +++++++++++++++++ arch/loongarch/kvm/switch.S | 301 +++++++ arch/loongarch/kvm/timer.c | 266 +++++++ arch/loongarch/kvm/tlb.c | 31 + arch/loongarch/kvm/trace.h | 168 ++++ arch/loongarch/kvm/vcpu.c | 882 +++++++++++++++++++++ arch/loongarch/kvm/vm.c | 79 ++ arch/loongarch/kvm/vmid.c | 65 ++ include/uapi/linux/kvm.h | 9 + 30 files changed, 4808 insertions(+), 15 deletions(-) create mode 100644 arch/loongarch/include/asm/kvm_csr.h create mode 100644 arch/loongarch/include/asm/kvm_host.h create mode 100644 arch/loongarch/include/asm/kvm_types.h create mode 100644 arch/loongarch/include/asm/kvm_vcpu.h create mode 100644 arch/loongarch/include/uapi/asm/kvm.h create mode 100644 arch/loongarch/kvm/Kconfig create mode 100644 arch/loongarch/kvm/Makefile create mode 100644 arch/loongarch/kvm/csr_ops.S create mode 100644 arch/loongarch/kvm/exit.c create mode 100644 arch/loongarch/kvm/interrupt.c create mode 100644 arch/loongarch/kvm/main.c create mode 100644 arch/loongarch/kvm/mmu.c create mode 100644 arch/loongarch/kvm/switch.S create mode 100644 arch/loongarch/kvm/timer.c create mode 100644 arch/loongarch/kvm/tlb.c create mode 100644 arch/loongarch/kvm/trace.h create mode 100644 arch/loongarch/kvm/vcpu.c create mode 100644 arch/loongarch/kvm/vm.c create mode 100644 arch/loongarch/kvm/vmid.c