From patchwork Fri Jun 23 22:55:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Gerst X-Patchwork-Id: 112365 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp6085958vqr; Fri, 23 Jun 2023 15:58:04 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ46C3KGDX/3yiWVs/2ZMkrqXvThHQZnJ1JWyt/3PeSIh1cJd8NehNexKeHhimFTLPpWP7Ro X-Received: by 2002:a05:620a:2b88:b0:762:5179:5d97 with SMTP id dz8-20020a05620a2b8800b0076251795d97mr21011357qkb.12.1687561084607; Fri, 23 Jun 2023 15:58:04 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687561084; cv=none; d=google.com; s=arc-20160816; b=xM9tIynI7U2wurFqgu2YN+WU8MYOXyB7MvBOLJBZmpvzzerZ68PdabsbkO5nh3SHJ8 pqeeSDagtwAefTd8ILRh/qAq9UpRebCFoo2sfOi3hnS0mao6VqgalYSPzydlFWcA7bfv IH3xKh30bYOvYSJoRxCzE5hfZmZVKXmVfIGI1QgG5S8KxmNi5GU2NzHO0A8A4eeonuEG yFu6EBcm5mrdoOorBRjeQ/0Pld7EITuRsytQHgNAjhJ6R3wnVJB17WtSzzV4Cw4PfNJN pkAlxU8uvM7fPGAmYzVfOd0t2GLBbYmg8GJ69pObtTc4IGz4fjP2jSzHyHrrMbFCn2xY 61eA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=AR6g0KQRb2O6Ea75fiXFjt5uQ0E9Sud+2f8NF24oWis=; fh=1MOKIhXcT/1REp5muKdhBqlIgZb5kZhhBfeNS3pKSFc=; b=WctUezO29rniWs/MWn9eaGRQ7We6eVpTv7iU4SkfB2nSiclVd1VgO3A8/NWFfxelt9 pXeFx0A7287DdUsq3nnqPJZ8K3PlNsrgjYm/NsZPylVQdA0vsk8fepMuThXU5Q/cChV1 hzC6JQEjzV4Yu/SrGRfk5FBRIuY7Y1geswqBLwTMyPGHS5aJSH7Wb9lTE2EOJoyvy4ks /m0iJXrj0o79pdcnlQ/OK4rTArIucWO46voqsjha0BqW7DcJUcr5QE65LbN6G2JYPUCY ECYlOAln7ADxSINZG2YM3m0M6WKwByufUwfibH75pRXY/H3Fr3Oym24dPQRKZeAWyhZI oOow== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=Fqs5BbTM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id z1-20020a63c041000000b005538ca839casi339487pgi.546.2023.06.23.15.57.50; Fri, 23 Jun 2023 15:58:04 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=Fqs5BbTM; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232263AbjFWW4V (ORCPT + 99 others); Fri, 23 Jun 2023 18:56:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231626AbjFWW4M (ORCPT ); Fri, 23 Jun 2023 18:56:12 -0400 Received: from mail-ot1-x32d.google.com (mail-ot1-x32d.google.com [IPv6:2607:f8b0:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F8582727 for ; Fri, 23 Jun 2023 15:55:37 -0700 (PDT) Received: by mail-ot1-x32d.google.com with SMTP id 46e09a7af769-6b720fd0e60so1089127a34.1 for ; Fri, 23 Jun 2023 15:55:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687560936; x=1690152936; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AR6g0KQRb2O6Ea75fiXFjt5uQ0E9Sud+2f8NF24oWis=; b=Fqs5BbTMlJj5MYjaBUKQsQlninT9VZmCArJ2HxdceUWS00ICgJID4w2Dq8TyOjVogr 04GoDeSjUMoxo4FrZEZ04tlnpMfE2iR/CdFXS6mbB8Nb08CrNE10naZ6AnQaWw/js6Pp GHnnvCKOTUmEzm40e9vG8IQr1bhcLzVaEpuPiuXq1oV+0fm5dtMfgDrc9hNinWJsxMtq 989rzk6icXGq7lcMvOB+h4oEM9eSIgzauQTc+gWS8Vn05iO7dUh6J+G6koxWR3iHMxAS 7CJj35HaR2DKiZiemXsDmrkO198MifN+FlGyqYxhWXUm4PzBLr7JKPtL6CkitMkkh7Yi 3ZhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687560936; x=1690152936; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AR6g0KQRb2O6Ea75fiXFjt5uQ0E9Sud+2f8NF24oWis=; b=E/joQfkW23W6mjjxJCI4wNtan+yW+mET18Lxu/SdKc3qmgrv4M1lkGkJ7CF4oxWCfj faXgIeNyTb/t4d1o4CtOgI4XgFewCpa3fpdHVUMFJVXAlBfo6Eg70RUGIJgTWBFkFVH/ bN9+ej6fbFFcoUPgQAEujFotPHdtJd21Pc/Cp3guUbKX1ruEsffspF3kXG6JeNKJ9Vyr +C8nNzcFzD/QGtlFgqeidONVk8wW9XLp+dl9AZrMsqy0OMRiOeMJN7yfbeBaUJ7GWHDX l4Jdr6Vvort3VvVo6D2GGYpeDmTeX1cG8Jhb2nczunbCDATqI4F8czOPTUj5ohCPYuMu xhVw== X-Gm-Message-State: AC+VfDxF7I4CZlcGUWrWgvGYKzYxrGNiTnPcGKluL+Exv9FqqfdXZok7 o6oh8ExMm/Rb2+luYvzK9rvHuJkN/A== X-Received: by 2002:a05:6830:60c:b0:6b7:24f7:ec0e with SMTP id w12-20020a056830060c00b006b724f7ec0emr2799713oti.26.1687560935722; Fri, 23 Jun 2023 15:55:35 -0700 (PDT) Received: from citadel.. (047-026-243-217.res.spectrum.com. [47.26.243.217]) by smtp.gmail.com with ESMTPSA id b21-20020a9d5d15000000b006af9d8af435sm267479oti.50.2023.06.23.15.55.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 15:55:35 -0700 (PDT) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Thomas Gleixner , Borislav Petkov , "H . Peter Anvin" , Peter Zijlstra , Sami Tolvanen , alyssa.milburn@linux.intel.com, keescook@chromium.org, jpoimboe@kernel.org, joao@overdrivepizza.com, tim.c.chen@linux.intel.com, Brian Gerst Subject: [PATCH v2 1/2] x86/32: Remove schedule_tail_wrapper() Date: Fri, 23 Jun 2023 18:55:28 -0400 Message-ID: <20230623225529.34590-2-brgerst@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230623225529.34590-1-brgerst@gmail.com> References: <20230623225529.34590-1-brgerst@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769536052043766700?= X-GMAIL-MSGID: =?utf-8?q?1769536052043766700?= The unwinder expects a return address at the very top of the kernel stack just below pt_regs and before any stack frame is created. Instead of calling a wrapper, set up a return address as if ret_from_fork() was called from the syscall entry code. Signed-off-by: Brian Gerst --- arch/x86/entry/entry_32.S | 33 ++++++++++----------------------- 1 file changed, 10 insertions(+), 23 deletions(-) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index 91397f58ac30..e56123f03a79 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -719,26 +719,6 @@ SYM_CODE_START(__switch_to_asm) SYM_CODE_END(__switch_to_asm) .popsection -/* - * The unwinder expects the last frame on the stack to always be at the same - * offset from the end of the page, which allows it to validate the stack. - * Calling schedule_tail() directly would break that convention because its an - * asmlinkage function so its argument has to be pushed on the stack. This - * wrapper creates a proper "end of stack" frame header before the call. - */ -.pushsection .text, "ax" -SYM_FUNC_START(schedule_tail_wrapper) - FRAME_BEGIN - - pushl %eax - call schedule_tail - popl %eax - - FRAME_END - RET -SYM_FUNC_END(schedule_tail_wrapper) -.popsection - /* * A newly forked process directly context switches into this address. * @@ -748,16 +728,23 @@ SYM_FUNC_END(schedule_tail_wrapper) */ .pushsection .text, "ax" SYM_CODE_START(ret_from_fork) - call schedule_tail_wrapper + /* return address for the stack unwinder */ + pushl $.Lsyscall_32_done + + FRAME_BEGIN + pushl %eax + call schedule_tail + addl $4, %esp + FRAME_END testl %ebx, %ebx jnz 1f /* kernel threads are uncommon */ 2: /* When we fork, we trace the syscall return in the child, too. */ - movl %esp, %eax + leal 4(%esp), %eax call syscall_exit_to_user_mode - jmp .Lsyscall_32_done + RET /* kernel thread */ 1: movl %edi, %eax From patchwork Fri Jun 23 22:55:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brian Gerst X-Patchwork-Id: 112375 Return-Path: Delivered-To: ouuuleilei@gmail.com Received: by 2002:a59:994d:0:b0:3d9:f83d:47d9 with SMTP id k13csp6089339vqr; Fri, 23 Jun 2023 16:04:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7vF/66hNz/tgwNmnmwJ+5JFIn4UN4ezxxdtgyIEfStumNYpqxOXyd5DhOe6M8MPdxSJ9EX X-Received: by 2002:a05:6808:1188:b0:39c:93ba:cb92 with SMTP id j8-20020a056808118800b0039c93bacb92mr27379002oil.8.1687561488512; Fri, 23 Jun 2023 16:04:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1687561488; cv=none; d=google.com; s=arc-20160816; b=vSjoaHwlmlU0ffa4o5IVfmUv/R8tWW6jji0bAk9p6ASsYa+mPWmMK5U/1MS/cm7zVa G6a+4YhdHx+BbwCqTSOm56oQRzEsCEx4AhLpFyWJQc56r0o61PYRQoBYq9TS4kmkEs2E LtB7SX+dlEg04ugkMtuDFZVG7TXgd6CnZOT2ydKkkeQM+9SP5qg8PUTuc5t8NIsfgof4 7hBtVcqp9SRpwSiH272d4EDLTuioLcEek6uguQfhp4ixZDEmP4ohJoRTk0h3Q1DCyZ3Y kZA52QkkcL2pU2NUSNGGeV/rYDrdjGw3jZ1eA6SKeJ2mB1bNsyVt8JAxOCjmUO3LPzaE WCJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=list-id:precedence:content-transfer-encoding:mime-version :references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature; bh=G1t8K4z72yZRYWSkZLR5CJTJ/xmjTVHKNdyx/pQkrmM=; fh=1MOKIhXcT/1REp5muKdhBqlIgZb5kZhhBfeNS3pKSFc=; b=jqKb4wYi0v3p9civEef7pxnhVHe6T8nfJU/rD1MWkPhJfhVcFlJSYaLuTuBwVC/Oel ySyLgdYSvVtdWxoozsDu/XljbrmnmZnwa16JCpibVWx3FmAomeIeZy0ua1wsicpinOhG 2pc7CYOO6BT2DYQqYBAoBCLtRE54cLyqWKVh/2U9s9Qss49fgguNguxp57zjL1l28CKB 64wTVsooem8qnE7t3uPViGtNy5hewB84baqrGhzTsaIncrJ+YZ/X6ycg1lsfa84qVjRy QCNUGuATsyb6OxgwczZR95U09aXjIGjbE3/jSoKl5UbjsVsfKnpv1sCfvvIAcUQHt8D/ ax+A== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=MRdk8HnL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from out1.vger.email (out1.vger.email. [2620:137:e000::1:20]) by mx.google.com with ESMTP id s21-20020a056a00179500b006669ba69adasi165188pfg.238.2023.06.23.16.04.35; Fri, 23 Jun 2023 16:04:48 -0700 (PDT) Received-SPF: pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) client-ip=2620:137:e000::1:20; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20221208 header.b=MRdk8HnL; spf=pass (google.com: domain of linux-kernel-owner@vger.kernel.org designates 2620:137:e000::1:20 as permitted sender) smtp.mailfrom=linux-kernel-owner@vger.kernel.org; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232199AbjFWW4Y (ORCPT + 99 others); Fri, 23 Jun 2023 18:56:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231683AbjFWW4M (ORCPT ); Fri, 23 Jun 2023 18:56:12 -0400 Received: from mail-oa1-x33.google.com (mail-oa1-x33.google.com [IPv6:2001:4860:4864:20::33]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E87B0272C for ; Fri, 23 Jun 2023 15:55:38 -0700 (PDT) Received: by mail-oa1-x33.google.com with SMTP id 586e51a60fabf-19fa4346498so972115fac.1 for ; Fri, 23 Jun 2023 15:55:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687560937; x=1690152937; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G1t8K4z72yZRYWSkZLR5CJTJ/xmjTVHKNdyx/pQkrmM=; b=MRdk8HnLfuJdykO+ryiKWiAwgn2CxWTStVczZvMNRx+0qk3mWXzOkMR4klyU7tRS9A 7g5TgacdyoLPCxzqe871bg1/jnsooYJ5VG4ExdRpKrPm/WBBzampZOSfcps0CZs5T5c3 9iTAEIG+SkQQdEZxruqWpDoLBuvzkyPm6oMHe+o35997i3x6c9EB4gbCk6lfyHc5G4oQ J8Yudzwqa6lzMuNdwGP4OWUxdD9p8f/NbQzV2Kzm1ruzP6w+FXY7MVon09/IUtS2abda B5bE3l9kL9/mXT7ds0SGkOrttL6csR/f22gbPWf+TaEH6AUGCssKwng1mPYHCtY3BXj8 vVPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687560937; x=1690152937; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G1t8K4z72yZRYWSkZLR5CJTJ/xmjTVHKNdyx/pQkrmM=; b=FqA5/Bo/qa96xNndJS+leZxQQEGSIphfTAcl7jk3U7w+H5zw6+4d9FIuu1bPYZM36g kkldDgPlyhwSZ18DSe91S4TxmA/o/0Q5BG+hoIXL2OuIigcj59iBsvE5+26XMpNsws3h FNrOWDrWtk9BrlKY51gRr4KVgWuFToMgbV3unf0c9x/q6dKA2DAGeM5s+9lgXM0Ecbfj hgdkgssQXSlpTcVROkW/ycCfdsNWAse9LnUzgoQsFAj4Im5P/NbbRbAmd2EZXclksh/g eIftgXJw4hZkT3lhy0E2kP/xiJ9+Lqk7kTpuZkvk0L+xHf7Dhb9uTUL2fc/5WkZCKnGv Tzjw== X-Gm-Message-State: AC+VfDwQUYuwaTB/5psDMOEGjrN6N6Nkc3Wqf5YtvP4/HdtoGXu1vOxO b6GLiudWRYxt0BBZCUSZZxrKRMp8cA== X-Received: by 2002:a05:6870:44c1:b0:1a9:b19f:7a4 with SMTP id t1-20020a05687044c100b001a9b19f07a4mr18947212oai.10.1687560937270; Fri, 23 Jun 2023 15:55:37 -0700 (PDT) Received: from citadel.. (047-026-243-217.res.spectrum.com. [47.26.243.217]) by smtp.gmail.com with ESMTPSA id b21-20020a9d5d15000000b006af9d8af435sm267479oti.50.2023.06.23.15.55.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 15:55:36 -0700 (PDT) From: Brian Gerst To: linux-kernel@vger.kernel.org, x86@kernel.org Cc: Thomas Gleixner , Borislav Petkov , "H . Peter Anvin" , Peter Zijlstra , Sami Tolvanen , alyssa.milburn@linux.intel.com, keescook@chromium.org, jpoimboe@kernel.org, joao@overdrivepizza.com, tim.c.chen@linux.intel.com, Brian Gerst Subject: [PATCH v2 2/2] x86: Rewrite ret_from_fork() in C Date: Fri, 23 Jun 2023 18:55:29 -0400 Message-ID: <20230623225529.34590-3-brgerst@gmail.com> X-Mailer: git-send-email 2.41.0 In-Reply-To: <20230623225529.34590-1-brgerst@gmail.com> References: <20230623225529.34590-1-brgerst@gmail.com> MIME-Version: 1.0 X-Spam-Status: No, score=-2.1 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,DKIM_VALID_EF,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE,SPF_HELO_NONE,SPF_PASS,T_SCC_BODY_TEXT_LINE autolearn=ham autolearn_force=no version=3.4.6 X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on lindbergh.monkeyblade.net Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= X-GMAIL-THRID: =?utf-8?q?1769536475555860041?= X-GMAIL-MSGID: =?utf-8?q?1769536475555860041?= When kCFI is enabled, special handling is needed for the indirect call to the kernel thread function. Rewrite the ret_from_fork() function in C so that the compiler can properly handle the indirect call. Suggested-by: Peter Zijlstra (Intel) Signed-off-by: Brian Gerst Tested-by: Petr Mladek --- arch/x86/entry/entry_32.S | 30 ++++++++--------------------- arch/x86/entry/entry_64.S | 33 ++++++++------------------------ arch/x86/include/asm/switch_to.h | 4 +++- arch/x86/kernel/process.c | 22 ++++++++++++++++++++- 4 files changed, 40 insertions(+), 49 deletions(-) diff --git a/arch/x86/entry/entry_32.S b/arch/x86/entry/entry_32.S index e56123f03a79..6e6af42e044a 100644 --- a/arch/x86/entry/entry_32.S +++ b/arch/x86/entry/entry_32.S @@ -727,36 +727,22 @@ SYM_CODE_END(__switch_to_asm) * edi: kernel thread arg */ .pushsection .text, "ax" -SYM_CODE_START(ret_from_fork) +SYM_CODE_START(ret_from_fork_asm) + movl %esp, %edx /* regs */ + /* return address for the stack unwinder */ pushl $.Lsyscall_32_done FRAME_BEGIN - pushl %eax - call schedule_tail + /* prev already in EAX */ + movl %ebx, %ecx /* fn */ + pushl %edi /* fn_arg */ + call ret_from_fork addl $4, %esp FRAME_END - testl %ebx, %ebx - jnz 1f /* kernel threads are uncommon */ - -2: - /* When we fork, we trace the syscall return in the child, too. */ - leal 4(%esp), %eax - call syscall_exit_to_user_mode RET - - /* kernel thread */ -1: movl %edi, %eax - CALL_NOSPEC ebx - /* - * A kernel thread is allowed to return here after successfully - * calling kernel_execve(). Exit to userspace to complete the execve() - * syscall. - */ - movl $0, PT_EAX(%esp) - jmp 2b -SYM_CODE_END(ret_from_fork) +SYM_CODE_END(ret_from_fork_asm) .popsection SYM_ENTRY(__begin_SYSENTER_singlestep_region, SYM_L_GLOBAL, SYM_A_NONE) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index f31e286c2977..91f6818884fa 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -284,36 +284,19 @@ SYM_FUNC_END(__switch_to_asm) * r12: kernel thread arg */ .pushsection .text, "ax" - __FUNC_ALIGN -SYM_CODE_START_NOALIGN(ret_from_fork) - UNWIND_HINT_END_OF_STACK +SYM_CODE_START(ret_from_fork_asm) + UNWIND_HINT_REGS ANNOTATE_NOENDBR // copy_thread CALL_DEPTH_ACCOUNT - movq %rax, %rdi - call schedule_tail /* rdi: 'prev' task parameter */ - testq %rbx, %rbx /* from kernel_thread? */ - jnz 1f /* kernel threads are uncommon */ + movq %rax, %rdi /* prev */ + movq %rsp, %rsi /* regs */ + movq %rbx, %rdx /* fn */ + movq %r12, %rcx /* fn_arg */ + call ret_from_fork -2: - UNWIND_HINT_REGS - movq %rsp, %rdi - call syscall_exit_to_user_mode /* returns with IRQs disabled */ jmp swapgs_restore_regs_and_return_to_usermode - -1: - /* kernel thread */ - UNWIND_HINT_END_OF_STACK - movq %r12, %rdi - CALL_NOSPEC rbx - /* - * A kernel thread is allowed to return here after successfully - * calling kernel_execve(). Exit to userspace to complete the execve() - * syscall. - */ - movq $0, RAX(%rsp) - jmp 2b -SYM_CODE_END(ret_from_fork) +SYM_CODE_END(ret_from_fork_asm) .popsection .macro DEBUG_ENTRY_ASSERT_IRQS_OFF diff --git a/arch/x86/include/asm/switch_to.h b/arch/x86/include/asm/switch_to.h index 5c91305d09d2..f42dbf17f52b 100644 --- a/arch/x86/include/asm/switch_to.h +++ b/arch/x86/include/asm/switch_to.h @@ -12,7 +12,9 @@ struct task_struct *__switch_to_asm(struct task_struct *prev, __visible struct task_struct *__switch_to(struct task_struct *prev, struct task_struct *next); -asmlinkage void ret_from_fork(void); +asmlinkage void ret_from_fork_asm(void); +__visible void ret_from_fork(struct task_struct *prev, struct pt_regs *regs, + int (*fn)(void *), void *fn_arg); /* * This is the structure pointed to by thread.sp for an inactive task. The diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index cc7a642f8c9d..001e6dad9a48 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include #include @@ -136,6 +137,25 @@ static int set_new_tls(struct task_struct *p, unsigned long tls) return do_set_thread_area_64(p, ARCH_SET_FS, tls); } +__visible noinstr void ret_from_fork(struct task_struct *prev, struct pt_regs *regs, + int (*fn)(void *), void *fn_arg) +{ + schedule_tail(prev); + + /* Is this a kernel thread? */ + if (unlikely(fn)) { + fn(fn_arg); + /* + * A kernel thread is allowed to return here after successfully + * calling kernel_execve(). Exit to userspace to complete the + * execve() syscall. + */ + regs->ax = 0; + } + + syscall_exit_to_user_mode(regs); +} + int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) { unsigned long clone_flags = args->flags; @@ -152,7 +172,7 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) frame = &fork_frame->frame; frame->bp = encode_frame_pointer(childregs); - frame->ret_addr = (unsigned long) ret_from_fork; + frame->ret_addr = (unsigned long) ret_from_fork_asm; p->thread.sp = (unsigned long) fork_frame; p->thread.io_bitmap = NULL; p->thread.iopl_warn = 0;