unimore/kernel-hacking-2024-linux-steffo
unimore/kernel-hacking-2024-linux-steffo
1
Fork 0
Code Pull requests 5 Wiki Activity
kernel-hacking-2024-linux-s.../arch/riscv/kernel/process.c

236 lines
6.2 KiB
C
Raw Normal View History

treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 120 Based on 1 normalized pattern(s): this program is free software you can redistribute it and or modify it under the terms of the gnu general public license as published by the free software foundation either version 2 of the license or at your option any later version this program is distributed in the hope that it will be useful but without any warranty without even the implied warranty of merchantability or fitness for a particular purpose see the gnu general public license for more details you should have received a copy of the gnu general public license along with this program if not see the file copying or write to the free software foundation inc extracted by the scancode license scanner the SPDX license identifier GPL-2.0-or-later has been chosen to replace the boilerplate/reference in 12 file(s). Signed-off-by: Thomas Gleixner <tglx@linutronix.de> Reviewed-by: Richard Fontana <rfontana@redhat.com> Reviewed-by: Allison Randal <allison@lohutok.net> Cc: linux-spdx@vger.kernel.org Link: https://lkml.kernel.org/r/20190523091651.231300438@linutronix.de Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2019-05-23 09:14:57 +00:00
// SPDX-License-Identifier: GPL-2.0-or-later
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
/*
* Copyright (C) 2009 Sunplus Core Technology Co., Ltd.
* Chen Liqin <liqin.chen@sunplusct.com>
* Lennox Wu <lennox.wu@sunplusct.com>
* Copyright (C) 2012 Regents of the University of California
* Copyright (C) 2017 SiFive
*/
riscv: add missing header file includes sparse identifies several missing prototypes caused by missing preprocessor include directives: arch/riscv/kernel/cpufeature.c:16:6: warning: symbol 'has_fpu' was not declared. Should it be static? arch/riscv/kernel/process.c:26:6: warning: symbol 'arch_cpu_idle' was not declared. Should it be static? arch/riscv/kernel/reset.c:15:6: warning: symbol 'pm_power_off' was not declared. Should it be static? arch/riscv/kernel/syscall_table.c:15:6: warning: symbol 'sys_call_table' was not declared. Should it be static? arch/riscv/kernel/traps.c:149:13: warning: symbol 'trap_init' was not declared. Should it be static? arch/riscv/kernel/vdso.c:54:5: warning: symbol 'arch_setup_additional_pages' was not declared. Should it be static? arch/riscv/kernel/smp.c:64:6: warning: symbol 'arch_match_cpu_phys_id' was not declared. Should it be static? arch/riscv/kernel/module-sections.c:89:5: warning: symbol 'module_frob_arch_sections' was not declared. Should it be static? arch/riscv/mm/context.c:42:6: warning: symbol 'switch_mm' was not declared. Should it be static? Fix by including the appropriate header files in the appropriate source files. This patch should have no functional impact. Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
2019-10-17 22:21:28 +00:00
#include <linux/cpu.h>
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
#include <linux/kernel.h>
#include <linux/sched.h>
riscv: process: Fix no prototype for show_regs Include header file to fix the following W=1 compilation warning: arch/riscv/kernel/process.c:78:6: warning: no previous prototype for ‘show_regs’ [-Wmissing-prototypes] 78 | void show_regs(struct pt_regs *regs) | ^~~~~~~~~ Reported-by: Hulk Robot <hulkci@huawei.com> Signed-off-by: Nanyong Sun <sunnanyong@huawei.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-03-05 11:33:30 +00:00
#include <linux/sched/debug.h>
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
#include <linux/sched/task_stack.h>
#include <linux/tick.h>
#include <linux/ptrace.h>
riscv: use linux/uaccess.h, not asm/uaccess.h... Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
2017-12-04 23:01:01 +00:00
#include <linux/uaccess.h>
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
#include <asm/unistd.h>
#include <asm/processor.h>
#include <asm/csr.h>
riscv: Add dump stack in show_regs Like commit 1149aad10b1e ("arm64: Add dump_backtrace() in show_regs"), dump the stack in riscv show_regs as common code expects. Reviewed-by: Atish Patra <atish.patra@wdc.com> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-01-11 12:40:12 +00:00
#include <asm/stacktrace.h>
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
#include <asm/string.h>
#include <asm/switch_to.h>
riscv: add missing header file includes sparse identifies several missing prototypes caused by missing preprocessor include directives: arch/riscv/kernel/cpufeature.c:16:6: warning: symbol 'has_fpu' was not declared. Should it be static? arch/riscv/kernel/process.c:26:6: warning: symbol 'arch_cpu_idle' was not declared. Should it be static? arch/riscv/kernel/reset.c:15:6: warning: symbol 'pm_power_off' was not declared. Should it be static? arch/riscv/kernel/syscall_table.c:15:6: warning: symbol 'sys_call_table' was not declared. Should it be static? arch/riscv/kernel/traps.c:149:13: warning: symbol 'trap_init' was not declared. Should it be static? arch/riscv/kernel/vdso.c:54:5: warning: symbol 'arch_setup_additional_pages' was not declared. Should it be static? arch/riscv/kernel/smp.c:64:6: warning: symbol 'arch_match_cpu_phys_id' was not declared. Should it be static? arch/riscv/kernel/module-sections.c:89:5: warning: symbol 'module_frob_arch_sections' was not declared. Should it be static? arch/riscv/mm/context.c:42:6: warning: symbol 'switch_mm' was not declared. Should it be static? Fix by including the appropriate header files in the appropriate source files. This patch should have no functional impact. Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
2019-10-17 22:21:28 +00:00
#include <asm/thread_info.h>
RISC-V: Enable CPU_IDLE drivers We force select CPU_PM and provide asm/cpuidle.h so that we can use CPU IDLE drivers for Linux RISC-V kernel. Signed-off-by: Anup Patel <anup.patel@wdc.com> Signed-off-by: Anup Patel <apatel@vetanamicro.com> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Atish Patra <atishp@rivosinc.com> Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-02-10 05:49:40 +00:00
#include <asm/cpuidle.h>
riscv: Add task switch support for vector This patch adds task switch support for vector. It also supports all lengths of vlen. Suggested-by: Andrew Waterman <andrew@sifive.com> Co-developed-by: Nick Knight <nick.knight@sifive.com> Signed-off-by: Nick Knight <nick.knight@sifive.com> Co-developed-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Co-developed-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Signed-off-by: Vineet Gupta <vineetg@rivosinc.com> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Reviewed-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Tested-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20230605110724.21391-11-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-06-05 11:07:07 +00:00
#include <asm/vector.h>
riscv: add support for PR_SET_UNALIGN and PR_GET_UNALIGN Now that trap support is ready to handle misalignment errors in S-mode, allow the user to control the behavior of misaligned accesses using prctl(PR_SET_UNALIGN). Add an align_ctl flag in thread_struct which will be used to determine if we should SIGBUS the process or not on such fault. Signed-off-by: Clément Léger <cleger@rivosinc.com> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Link: https://lore.kernel.org/r/20231004151405.521596-9-cleger@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-10-04 15:14:05 +00:00
#include <asm/cpufeature.h>
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
riscv: Enable per-task stack canaries This enables the use of per-task stack canary values if GCC has support for emitting the stack canary reference relative to the value of tp, which holds the task struct pointer in the riscv kernel. After compare arm64 and x86 implementations, seems arm64's is more flexible and readable. The key point is how gcc get the offset of stack_canary from gs/el0_sp. x86: Use a fix offset from gs, not flexible. struct fixed_percpu_data { /* * GCC hardcodes the stack canary as %gs:40. Since the * irq_stack is the object at %gs:0, we reserve the bottom * 48 bytes of the irq stack for the canary. */ char gs_base[40]; // :( unsigned long stack_canary; }; arm64: Use -mstack-protector-guard-offset & guard-reg gcc options: -mstack-protector-guard=sysreg -mstack-protector-guard-reg=sp_el0 -mstack-protector-guard-offset=xxx riscv: Use -mstack-protector-guard-offset & guard-reg gcc options: -mstack-protector-guard=tls -mstack-protector-guard-reg=tp -mstack-protector-guard-offset=xxx GCC's implementation has been merged: commit c931e8d5a96463427040b0d11f9c4352ac22b2b0 Author: Cooper Qu <cooper.qu@linux.alibaba.com> Date: Mon Jul 13 16:15:08 2020 +0800 RISC-V: Add support for TLS stack protector canary access In the end, these codes are inserted by gcc before return: * 0xffffffe00020b396 <+120>: ld a5,1008(tp) # 0x3f0 * 0xffffffe00020b39a <+124>: xor a5,a5,a4 * 0xffffffe00020b39c <+126>: mv a0,s5 * 0xffffffe00020b39e <+128>: bnez a5,0xffffffe00020b61c <_do_fork+766> 0xffffffe00020b3a2 <+132>: ld ra,136(sp) 0xffffffe00020b3a4 <+134>: ld s0,128(sp) 0xffffffe00020b3a6 <+136>: ld s1,120(sp) 0xffffffe00020b3a8 <+138>: ld s2,112(sp) 0xffffffe00020b3aa <+140>: ld s3,104(sp) 0xffffffe00020b3ac <+142>: ld s4,96(sp) 0xffffffe00020b3ae <+144>: ld s5,88(sp) 0xffffffe00020b3b0 <+146>: ld s6,80(sp) 0xffffffe00020b3b2 <+148>: ld s7,72(sp) 0xffffffe00020b3b4 <+150>: addi sp,sp,144 0xffffffe00020b3b6 <+152>: ret ... * 0xffffffe00020b61c <+766>: auipc ra,0x7f8 * 0xffffffe00020b620 <+770>: jalr -1764(ra) # 0xffffffe000a02f38 <__stack_chk_fail> Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Cooper Qu <cooper.qu@linux.alibaba.com> Reviewed-by: Kees Cook <keescook@chromium.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2020-12-17 16:29:18 +00:00
#if defined(CONFIG_STACKPROTECTOR) && !defined(CONFIG_STACKPROTECTOR_PER_TASK)
riscv: Add STACKPROTECTOR supported The -fstack-protector & -fstack-protector-strong features are from gcc. The patch only add basic kernel support to stack-protector feature and some arch could have its own solution such as ARM64_PTR_AUTH. After enabling STACKPROTECTOR and STACKPROTECTOR_STRONG, the .text size is expanded from 0x7de066 to 0x81fb32 (only 5%) to add canary checking code. Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Reviewed-by: Kees Cook <keescook@chromium.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2020-07-10 16:19:57 +00:00
#include <linux/stackprotector.h>
unsigned long __stack_chk_guard __read_mostly;
EXPORT_SYMBOL(__stack_chk_guard);
#endif
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
extern asmlinkage void ret_from_fork(void);
riscv: Fix warning by declaring arch_cpu_idle() as noinstr The following warning appears when using ftrace: [89855.443413] RCU not on for: arch_cpu_idle+0x0/0x1c [89855.445640] WARNING: CPU: 5 PID: 0 at include/linux/trace_recursion.h:162 arch_ftrace_ops_list_func+0x208/0x228 [89855.445824] Modules linked in: xt_conntrack(E) nft_chain_nat(E) xt_MASQUERADE(E) nf_conntrack_netlink(E) xt_addrtype(E) nft_compat(E) nf_tables(E) nfnetlink(E) br_netfilter(E) cfg80211(E) nls_iso8859_1(E) ofpart(E) redboot(E) cmdlinepart(E) cfi_cmdset_0001(E) virtio_net(E) cfi_probe(E) cfi_util(E) 9pnet_virtio(E) gen_probe(E) net_failover(E) virtio_rng(E) failover(E) 9pnet(E) physmap(E) map_funcs(E) chipreg(E) mtd(E) uio_pdrv_genirq(E) uio(E) dm_multipath(E) scsi_dh_rdac(E) scsi_dh_emc(E) scsi_dh_alua(E) drm(E) efi_pstore(E) backlight(E) ip_tables(E) x_tables(E) raid10(E) raid456(E) async_raid6_recov(E) async_memcpy(E) async_pq(E) async_xor(E) xor(E) async_tx(E) raid6_pq(E) raid1(E) raid0(E) virtio_blk(E) [89855.451563] CPU: 5 PID: 0 Comm: swapper/5 Tainted: G E 6.8.0-rc6ubuntu-defconfig #2 [89855.451726] Hardware name: riscv-virtio,qemu (DT) [89855.451899] epc : arch_ftrace_ops_list_func+0x208/0x228 [89855.452016] ra : arch_ftrace_ops_list_func+0x208/0x228 [89855.452119] epc : ffffffff8016b216 ra : ffffffff8016b216 sp : ffffaf808090fdb0 [89855.452171] gp : ffffffff827c7680 tp : ffffaf808089ad40 t0 : ffffffff800c0dd8 [89855.452216] t1 : 0000000000000001 t2 : 0000000000000000 s0 : ffffaf808090fe30 [89855.452306] s1 : 0000000000000000 a0 : 0000000000000026 a1 : ffffffff82cd6ac8 [89855.452423] a2 : ffffffff800458c8 a3 : ffffaf80b1870640 a4 : 0000000000000000 [89855.452646] a5 : 0000000000000000 a6 : 00000000ffffffff a7 : ffffffffffffffff [89855.452698] s2 : ffffffff82766872 s3 : ffffffff80004caa s4 : ffffffff80ebea90 [89855.452743] s5 : ffffaf808089bd40 s6 : 8000000a00006e00 s7 : 0000000000000008 [89855.452787] s8 : 0000000000002000 s9 : 0000000080043700 s10: 0000000000000000 [89855.452831] s11: 0000000000000000 t3 : 0000000000100000 t4 : 0000000000000064 [89855.452874] t5 : 000000000000000c t6 : ffffaf80b182dbfc [89855.452929] status: 0000000200000100 badaddr: 0000000000000000 cause: 0000000000000003 [89855.453053] [<ffffffff8016b216>] arch_ftrace_ops_list_func+0x208/0x228 [89855.453191] [<ffffffff8000e082>] ftrace_call+0x8/0x22 [89855.453265] [<ffffffff800a149c>] do_idle+0x24c/0x2ca [89855.453357] [<ffffffff8000da54>] return_to_handler+0x0/0x26 [89855.453429] [<ffffffff8000b716>] smp_callin+0x92/0xb6 [89855.453785] ---[ end trace 0000000000000000 ]--- To fix this, mark arch_cpu_idle() as noinstr, like it is done in commit a9cbc1b471d2 ("s390/idle: mark arch_cpu_idle() noinstr"). Reported-by: Evgenii Shatokhin <e.shatokhin@yadro.com> Closes: https://lore.kernel.org/linux-riscv/51f21b87-ebed-4411-afbc-c00d3dea2bab@yadro.com/ Fixes: cfbc4f81c9d0 ("riscv: Select ARCH_WANTS_NO_INSTR") Signed-off-by: Alexandre Ghiti <alexghiti@rivosinc.com> Reviewed-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Andy Chiu <andy.chiu@sifive.com> Acked-by: Puranjay Mohan <puranjay12@gmail.com> Link: https://lore.kernel.org/r/20240326203017.310422-2-alexghiti@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-03-26 20:30:16 +00:00
void noinstr arch_cpu_idle(void)
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
{
RISC-V: Enable CPU_IDLE drivers We force select CPU_PM and provide asm/cpuidle.h so that we can use CPU IDLE drivers for Linux RISC-V kernel. Signed-off-by: Anup Patel <anup.patel@wdc.com> Signed-off-by: Anup Patel <apatel@vetanamicro.com> Reviewed-by: Guo Ren <guoren@kernel.org> Reviewed-by: Atish Patra <atishp@rivosinc.com> Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-02-10 05:49:40 +00:00
cpu_do_idle();
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
}
riscv: add support for PR_SET_UNALIGN and PR_GET_UNALIGN Now that trap support is ready to handle misalignment errors in S-mode, allow the user to control the behavior of misaligned accesses using prctl(PR_SET_UNALIGN). Add an align_ctl flag in thread_struct which will be used to determine if we should SIGBUS the process or not on such fault. Signed-off-by: Clément Léger <cleger@rivosinc.com> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Link: https://lore.kernel.org/r/20231004151405.521596-9-cleger@rivosinc.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-10-04 15:14:05 +00:00
int set_unalign_ctl(struct task_struct *tsk, unsigned int val)
{
if (!unaligned_ctl_available())
return -EINVAL;
tsk->thread.align_ctl = val;
return 0;
}
int get_unalign_ctl(struct task_struct *tsk, unsigned long adr)
{
if (!unaligned_ctl_available())
return -EINVAL;
return put_user(tsk->thread.align_ctl, (unsigned long __user *)adr);
}
riscv: Add dump stack in show_regs Like commit 1149aad10b1e ("arm64: Add dump_backtrace() in show_regs"), dump the stack in riscv show_regs as common code expects. Reviewed-by: Atish Patra <atish.patra@wdc.com> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-01-11 12:40:12 +00:00
void __show_regs(struct pt_regs *regs)
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
{
show_regs_print_info(KERN_DEFAULT);
riscv: Improve __show_regs Show the function symbols of epc and ra to improve the readability of crash reports, and align the printing formats about the raw epc value. Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-01-11 12:40:13 +00:00
if (!user_mode(regs)) {
pr_cont("epc : %pS\n", (void *)regs->epc);
pr_cont(" ra : %pS\n", (void *)regs->ra);
}
pr_cont("epc : " REG_FMT " ra : " REG_FMT " sp : " REG_FMT "\n",
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 12:10:32 +00:00
regs->epc, regs->ra, regs->sp);
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
pr_cont(" gp : " REG_FMT " tp : " REG_FMT " t0 : " REG_FMT "\n",
regs->gp, regs->tp, regs->t0);
pr_cont(" t1 : " REG_FMT " t2 : " REG_FMT " s0 : " REG_FMT "\n",
regs->t1, regs->t2, regs->s0);
pr_cont(" s1 : " REG_FMT " a0 : " REG_FMT " a1 : " REG_FMT "\n",
regs->s1, regs->a0, regs->a1);
pr_cont(" a2 : " REG_FMT " a3 : " REG_FMT " a4 : " REG_FMT "\n",
regs->a2, regs->a3, regs->a4);
pr_cont(" a5 : " REG_FMT " a6 : " REG_FMT " a7 : " REG_FMT "\n",
regs->a5, regs->a6, regs->a7);
pr_cont(" s2 : " REG_FMT " s3 : " REG_FMT " s4 : " REG_FMT "\n",
regs->s2, regs->s3, regs->s4);
pr_cont(" s5 : " REG_FMT " s6 : " REG_FMT " s7 : " REG_FMT "\n",
regs->s5, regs->s6, regs->s7);
pr_cont(" s8 : " REG_FMT " s9 : " REG_FMT " s10: " REG_FMT "\n",
regs->s8, regs->s9, regs->s10);
pr_cont(" s11: " REG_FMT " t3 : " REG_FMT " t4 : " REG_FMT "\n",
regs->s11, regs->t3, regs->t4);
pr_cont(" t5 : " REG_FMT " t6 : " REG_FMT "\n",
regs->t5, regs->t6);
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 12:10:32 +00:00
pr_cont("status: " REG_FMT " badaddr: " REG_FMT " cause: " REG_FMT "\n",
regs->status, regs->badaddr, regs->cause);
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
}
riscv: Add dump stack in show_regs Like commit 1149aad10b1e ("arm64: Add dump_backtrace() in show_regs"), dump the stack in riscv show_regs as common code expects. Reviewed-by: Atish Patra <atish.patra@wdc.com> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-01-11 12:40:12 +00:00
void show_regs(struct pt_regs *regs)
{
__show_regs(regs);
if (!user_mode(regs))
dump_backtrace(regs, NULL, KERN_DEFAULT);
}
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
riscv: compat: Add hw capability check for elf Detect hardware COMPAT (32bit U-mode) capability in rv64. If not support COMPAT mode in hw, compat_elf_check_arch would return false by compat_binfmt_elf.c Add CLASS to enhance (compat_)elf_check_arch to distinguish 32BIT/64BIT elf. Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Guo Ren <guoren@kernel.org> Tested-by: Heiko Stuebner <heiko@sntech.de> Link: https://lore.kernel.org/r/20220405071314.3225832-16-guoren@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-04-05 07:13:09 +00:00
#ifdef CONFIG_COMPAT
static bool compat_mode_supported __read_mostly;
bool compat_elf_check_arch(Elf32_Ehdr *hdr)
{
return compat_mode_supported &&
hdr->e_machine == EM_RISCV &&
hdr->e_ident[EI_CLASS] == ELFCLASS32;
}
static int __init compat_mode_detect(void)
{
unsigned long tmp = csr_read(CSR_STATUS);
csr_write(CSR_STATUS, (tmp & ~SR_UXL) | SR_UXL_32);
compat_mode_supported =
(csr_read(CSR_STATUS) & SR_UXL) == SR_UXL_32;
csr_write(CSR_STATUS, tmp);
pr_info("riscv: ELF compat mode %s",
riscv: compat: s/failed/unsupported if compat mode isn't supported When compat mode isn't supported(I believe this is the most case now), kernel will emit somthing as: [ 0.050407] riscv: ELF compat mode failed This msg may make users think there's something wrong with the kernel itself, replace "failed" with "unsupported" to make it clear. In fact this is the real compat_mode_supported meaning. After the patch, the msg would be: [ 0.050407] riscv: ELF compat mode unsupported Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Acked-by: Guo Ren <guoren@kernel.org> Link: https://lore.kernel.org/r/20220821141819.3804-1-jszhang@kernel.org/ Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-08-21 14:18:19 +00:00
compat_mode_supported ? "supported" : "unsupported");
riscv: compat: Add hw capability check for elf Detect hardware COMPAT (32bit U-mode) capability in rv64. If not support COMPAT mode in hw, compat_elf_check_arch would return false by compat_binfmt_elf.c Add CLASS to enhance (compat_)elf_check_arch to distinguish 32BIT/64BIT elf. Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Guo Ren <guoren@kernel.org> Tested-by: Heiko Stuebner <heiko@sntech.de> Link: https://lore.kernel.org/r/20220405071314.3225832-16-guoren@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-04-05 07:13:09 +00:00
return 0;
}
early_initcall(compat_mode_detect);
#endif
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
void start_thread(struct pt_regs *regs, unsigned long pc,
unsigned long sp)
{
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 12:10:32 +00:00
regs->status = SR_PIE;
riscv: Turn has_fpu into a static key if FPU=y The has_fpu check sits at hot code path: switch_to(). Currently, has_fpu is a bool variable if FPU=y, switch_to() checks it each time, we can optimize out this check by turning the has_fpu into a static key. Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
2021-05-12 14:55:45 +00:00
if (has_fpu()) {
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 12:10:32 +00:00
regs->status |= SR_FS_INITIAL;
riscv: Correct the initialized flow of FP register The following two reasons cause FP registers are sometimes not initialized before starting the user program. 1. Currently, the FP context is initialized in flush_thread() function and we expect these initial values to be restored to FP register when doing FP context switch. However, the FP context switch only occurs in switch_to function. Hence, if this process does not be scheduled out and scheduled in before entering the user space, the FP registers have no chance to initialize. 2. In flush_thread(), the state of reg->sstatus.FS inherits from the parent. Hence, the state of reg->sstatus.FS may be dirty. If this process is scheduled out during flush_thread() and initializing the FP register, the fstate_save() in switch_to will corrupt the FP context which has been initialized until flush_thread(). To solve the 1st case, the initialization of the FP register will be completed in start_thread(). It makes sure all FP registers are initialized before starting the user program. For the 2nd case, the state of reg->sstatus.FS in start_thread will be set to SR_FS_OFF to prevent this process from corrupting FP context in doing context save. The FP state is set to SR_FS_INITIAL in start_trhead(). Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Fixes: 7db91e57a0acd ("RISC-V: Task implementation") Cc: stable@vger.kernel.org [paul.walmsley@sifive.com: fixed brace alignment issue reported by checkpatch] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-08-14 08:23:52 +00:00
/*
* Restore the initial value to the FP register
* before starting the user program.
*/
fstate_restore(current, regs);
}
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 12:10:32 +00:00
regs->epc = pc;
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
regs->sp = sp;
riscv: compat: process: Add UXL_32 support in start_thread If the current task is in COMPAT mode, set SR_UXL_32 in status for returning userspace. We need CONFIG _COMPAT to prevent compiling errors with rv32 defconfig. Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Guo Ren <guoren@kernel.org> Tested-by: Heiko Stuebner <heiko@sntech.de> Link: https://lore.kernel.org/r/20220405071314.3225832-14-guoren@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-04-05 07:13:07 +00:00
#ifdef CONFIG_64BIT
regs->status &= ~SR_UXL;
if (is_compat_task())
regs->status |= SR_UXL_32;
else
regs->status |= SR_UXL_64;
#endif
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
}
void flush_thread(void)
{
Allow to disable FPU support FPU codes have been separated from common part in previous patches. This patch add the CONFIG_FPU option and some stubs, so that a no-FPU configuration is allowed. Signed-off-by: Alan Kao <alankao@andestech.com> Cc: Greentime Hu <greentime@andestech.com> Cc: Vincent Chen <vincentc@andestech.com> Cc: Zong Li <zong@andestech.com> Cc: Nick Hu <nickhu@andestech.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-10-09 02:18:33 +00:00
#ifdef CONFIG_FPU
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
/*
riscv: Correct the initialized flow of FP register The following two reasons cause FP registers are sometimes not initialized before starting the user program. 1. Currently, the FP context is initialized in flush_thread() function and we expect these initial values to be restored to FP register when doing FP context switch. However, the FP context switch only occurs in switch_to function. Hence, if this process does not be scheduled out and scheduled in before entering the user space, the FP registers have no chance to initialize. 2. In flush_thread(), the state of reg->sstatus.FS inherits from the parent. Hence, the state of reg->sstatus.FS may be dirty. If this process is scheduled out during flush_thread() and initializing the FP register, the fstate_save() in switch_to will corrupt the FP context which has been initialized until flush_thread(). To solve the 1st case, the initialization of the FP register will be completed in start_thread(). It makes sure all FP registers are initialized before starting the user program. For the 2nd case, the state of reg->sstatus.FS in start_thread will be set to SR_FS_OFF to prevent this process from corrupting FP context in doing context save. The FP state is set to SR_FS_INITIAL in start_trhead(). Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Fixes: 7db91e57a0acd ("RISC-V: Task implementation") Cc: stable@vger.kernel.org [paul.walmsley@sifive.com: fixed brace alignment issue reported by checkpatch] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-08-14 08:23:52 +00:00
* Reset FPU state and context
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
* frm: round to nearest, ties to even (IEEE default)
* fflags: accrued exceptions cleared
*/
riscv: Correct the initialized flow of FP register The following two reasons cause FP registers are sometimes not initialized before starting the user program. 1. Currently, the FP context is initialized in flush_thread() function and we expect these initial values to be restored to FP register when doing FP context switch. However, the FP context switch only occurs in switch_to function. Hence, if this process does not be scheduled out and scheduled in before entering the user space, the FP registers have no chance to initialize. 2. In flush_thread(), the state of reg->sstatus.FS inherits from the parent. Hence, the state of reg->sstatus.FS may be dirty. If this process is scheduled out during flush_thread() and initializing the FP register, the fstate_save() in switch_to will corrupt the FP context which has been initialized until flush_thread(). To solve the 1st case, the initialization of the FP register will be completed in start_thread(). It makes sure all FP registers are initialized before starting the user program. For the 2nd case, the state of reg->sstatus.FS in start_thread will be set to SR_FS_OFF to prevent this process from corrupting FP context in doing context save. The FP state is set to SR_FS_INITIAL in start_trhead(). Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Reviewed-by: Anup Patel <anup@brainfault.org> Reviewed-by: Christoph Hellwig <hch@lst.de> Fixes: 7db91e57a0acd ("RISC-V: Task implementation") Cc: stable@vger.kernel.org [paul.walmsley@sifive.com: fixed brace alignment issue reported by checkpatch] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-08-14 08:23:52 +00:00
fstate_off(current, task_pt_regs(current));
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
memset(&current->thread.fstate, 0, sizeof(current->thread.fstate));
Allow to disable FPU support FPU codes have been separated from common part in previous patches. This patch add the CONFIG_FPU option and some stubs, so that a no-FPU configuration is allowed. Signed-off-by: Alan Kao <alankao@andestech.com> Cc: Greentime Hu <greentime@andestech.com> Cc: Vincent Chen <vincentc@andestech.com> Cc: Zong Li <zong@andestech.com> Cc: Nick Hu <nickhu@andestech.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Palmer Dabbelt <palmer@sifive.com>
2018-10-09 02:18:33 +00:00
#endif
riscv: Add task switch support for vector This patch adds task switch support for vector. It also supports all lengths of vlen. Suggested-by: Andrew Waterman <andrew@sifive.com> Co-developed-by: Nick Knight <nick.knight@sifive.com> Signed-off-by: Nick Knight <nick.knight@sifive.com> Co-developed-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Co-developed-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Signed-off-by: Vineet Gupta <vineetg@rivosinc.com> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Reviewed-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Tested-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20230605110724.21391-11-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-06-05 11:07:07 +00:00
#ifdef CONFIG_RISCV_ISA_V
/* Reset vector state */
riscv: Add prctl controls for userspace vector management This patch add two riscv-specific prctls, to allow usespace control the use of vector unit: * PR_RISCV_V_SET_CONTROL: control the permission to use Vector at next, or all following execve for a thread. Turning off a thread's Vector live is not possible since libraries may have registered ifunc that may execute Vector instructions. * PR_RISCV_V_GET_CONTROL: get the same permission setting for the current thread, and the setting for following execve(s). Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Greentime Hu <greentime.hu@sifive.com> Reviewed-by: Vincent Chen <vincent.chen@sifive.com> Link: https://lore.kernel.org/r/20230605110724.21391-22-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-06-05 11:07:18 +00:00
riscv_v_vstate_ctrl_init(current);
riscv: Add task switch support for vector This patch adds task switch support for vector. It also supports all lengths of vlen. Suggested-by: Andrew Waterman <andrew@sifive.com> Co-developed-by: Nick Knight <nick.knight@sifive.com> Signed-off-by: Nick Knight <nick.knight@sifive.com> Co-developed-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Co-developed-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Signed-off-by: Vineet Gupta <vineetg@rivosinc.com> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Reviewed-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Tested-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20230605110724.21391-11-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-06-05 11:07:07 +00:00
riscv_v_vstate_off(task_pt_regs(current));
kfree(current->thread.vstate.datap);
memset(&current->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
riscv: sched: defer restoring Vector context for user User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times. The cost of this is that we must disable preemption and mark vector as busy during vstate_{save,restore}. Because then the V context will not get restored back immediately when a trap-causing context switch happens in the middle of vstate_{save,restore}. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Acked-by: Conor Dooley <conor.dooley@microchip.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-5-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-15 05:59:23 +00:00
clear_tsk_thread_flag(current, TIF_RISCV_V_DEFER_RESTORE);
riscv: Add task switch support for vector This patch adds task switch support for vector. It also supports all lengths of vlen. Suggested-by: Andrew Waterman <andrew@sifive.com> Co-developed-by: Nick Knight <nick.knight@sifive.com> Signed-off-by: Nick Knight <nick.knight@sifive.com> Co-developed-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Co-developed-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Signed-off-by: Vineet Gupta <vineetg@rivosinc.com> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Reviewed-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Tested-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20230605110724.21391-11-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-06-05 11:07:07 +00:00
#endif
}
void arch_release_task_struct(struct task_struct *tsk)
{
/* Free the vector context of datap. */
if (has_vector())
riscv: vector: use kmem_cache to manage vector context The allocation size of thread.vstate.datap is always riscv_v_vsize. So it is possbile to use kmem_cache_* to manage the allocation. This gives users more information regarding allocation of vector context via /proc/slabinfo. And it potentially reduces the latency of the first-use trap because of the allocation caches. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-10-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-15 05:59:28 +00:00
riscv_v_thread_free(tsk);
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
}
int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src)
{
fstate_save(src, task_pt_regs(src));
*dst = *src;
riscv: Add task switch support for vector This patch adds task switch support for vector. It also supports all lengths of vlen. Suggested-by: Andrew Waterman <andrew@sifive.com> Co-developed-by: Nick Knight <nick.knight@sifive.com> Signed-off-by: Nick Knight <nick.knight@sifive.com> Co-developed-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Co-developed-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Signed-off-by: Vineet Gupta <vineetg@rivosinc.com> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Reviewed-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Tested-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20230605110724.21391-11-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-06-05 11:07:07 +00:00
/* clear entire V context, including datap for a new task */
memset(&dst->thread.vstate, 0, sizeof(struct __riscv_v_ext_state));
riscv: vector: allow kernel-mode Vector with preemption Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide riscv_v_flags to let context save/restore routine track context status. Context tracking happens whenever the core starts its in-kernel Vector executions. An active (dirty) kernel task's V contexts will be saved to memory whenever a trap-introduced context switch happens. Or, when a softirq, which happens to nest on top of it, uses Vector. Context retoring happens when the execution transfer back to the original Kernel context where it first enable preempt_v. Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking of per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context switch overhead. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-11-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-15 05:59:29 +00:00
memset(&dst->thread.kernel_vstate, 0, sizeof(struct __riscv_v_ext_state));
riscv: sched: defer restoring Vector context for user User will use its Vector registers only after the kernel really returns to the userspace. So we can delay restoring Vector registers as long as we are still running in kernel mode. So, add a thread flag to indicates the need of restoring Vector and do the restore at the last arch-specific exit-to-user hook. This save the context restoring cost when we switch over multiple processes that run V in kernel mode. For example, if the kernel performs a context swicth from A->B->C, and returns to C's userspace, then there is no need to restore B's V-register. Besides, this also prevents us from repeatedly restoring V context when executing kernel-mode Vector multiple times. The cost of this is that we must disable preemption and mark vector as busy during vstate_{save,restore}. Because then the V context will not get restored back immediately when a trap-causing context switch happens in the middle of vstate_{save,restore}. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Acked-by: Conor Dooley <conor.dooley@microchip.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-5-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-15 05:59:23 +00:00
clear_tsk_thread_flag(dst, TIF_RISCV_V_DEFER_RESTORE);
riscv: Add task switch support for vector This patch adds task switch support for vector. It also supports all lengths of vlen. Suggested-by: Andrew Waterman <andrew@sifive.com> Co-developed-by: Nick Knight <nick.knight@sifive.com> Signed-off-by: Nick Knight <nick.knight@sifive.com> Co-developed-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Co-developed-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Signed-off-by: Vineet Gupta <vineetg@rivosinc.com> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Reviewed-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Tested-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20230605110724.21391-11-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-06-05 11:07:07 +00:00
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
return 0;
}
fork: Pass struct kernel_clone_args into copy_thread With io_uring we have started supporting tasks that are for most purposes user space tasks that exclusively run code in kernel mode. The kernel task that exec's init and tasks that exec user mode helpers are also user mode tasks that just run kernel code until they call kernel execve. Pass kernel_clone_args into copy_thread so these oddball tasks can be supported more cleanly and easily. v2: Fix spelling of kenrel_clone_args on h8300 Link: https://lkml.kernel.org/r/20220506141512.516114-2-ebiederm@xmission.com Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2022-04-08 23:07:50 +00:00
int copy_thread(struct task_struct *p, const struct kernel_clone_args *args)
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
{
fork: Pass struct kernel_clone_args into copy_thread With io_uring we have started supporting tasks that are for most purposes user space tasks that exclusively run code in kernel mode. The kernel task that exec's init and tasks that exec user mode helpers are also user mode tasks that just run kernel code until they call kernel execve. Pass kernel_clone_args into copy_thread so these oddball tasks can be supported more cleanly and easily. v2: Fix spelling of kenrel_clone_args on h8300 Link: https://lkml.kernel.org/r/20220506141512.516114-2-ebiederm@xmission.com Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2022-04-08 23:07:50 +00:00
unsigned long clone_flags = args->flags;
unsigned long usp = args->stack;
unsigned long tls = args->tls;
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
struct pt_regs *childregs = task_pt_regs(p);
riscv: process: fix kernel info leakage thread_struct's s[12] may contain random kernel memory content, which may be finally leaked to userspace. This is a security hole. Fix it by clearing the s[12] array in thread_struct when fork. As for kthread case, it's better to clear the s[12] array as well. Fixes: 7db91e57a0ac ("RISC-V: Task implementation") Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Tested-by: Guo Ren <guoren@kernel.org> Link: https://lore.kernel.org/r/20221029113450.4027-1-jszhang@kernel.org Reviewed-by: Guo Ren <guoren@kernel.org> Link: https://lore.kernel.org/r/CAJF2gTSdVyAaM12T%2B7kXAdRPGS4VyuO08X1c7paE-n4Fr8OtRA@mail.gmail.com/ Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2022-10-29 11:34:50 +00:00
memset(&p->thread.s, 0, sizeof(p->thread.s));
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
/* p->thread holds context to be restored by __switch_to() */
fork: Generalize PF_IO_WORKER handling Add fn and fn_arg members into struct kernel_clone_args and test for them in copy_thread (instead of testing for PF_KTHREAD | PF_IO_WORKER). This allows any task that wants to be a user space task that only runs in kernel mode to use this functionality. The code on x86 is an exception and still retains a PF_KTHREAD test because x86 unlikely everything else handles kthreads slightly differently than user space tasks that start with a function. The functions that created tasks that start with a function have been updated to set ".fn" and ".fn_arg" instead of ".stack" and ".stack_size". These functions are fork_idle(), create_io_thread(), kernel_thread(), and user_mode_thread(). Link: https://lkml.kernel.org/r/20220506141512.516114-4-ebiederm@xmission.com Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2022-04-12 15:18:48 +00:00
if (unlikely(args->fn)) {
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
/* Kernel thread */
memset(childregs, 0, sizeof(struct pt_regs));
riscv: abstract out CSR names for supervisor vs machine mode Many of the privileged CSRs exist in a supervisor and machine version that are used very similarly. Provide versions of the CSR names and fields that map to either the S-mode or M-mode variant depending on a new CONFIG_RISCV_M_MODE kconfig symbol. Contains contributions from Damien Le Moal <Damien.LeMoal@wdc.com> and Paul Walmsley <paul.walmsley@sifive.com>. Signed-off-by: Christoph Hellwig <hch@lst.de> Acked-by: Thomas Gleixner <tglx@linutronix.de> # for drivers/clocksource, drivers/irqchip [paul.walmsley@sifive.com: updated to apply] Signed-off-by: Paul Walmsley <paul.walmsley@sifive.com>
2019-10-28 12:10:32 +00:00
/* Supervisor/Machine, irqs on: */
childregs->status = SR_PP | SR_PIE;
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
fork: Generalize PF_IO_WORKER handling Add fn and fn_arg members into struct kernel_clone_args and test for them in copy_thread (instead of testing for PF_KTHREAD | PF_IO_WORKER). This allows any task that wants to be a user space task that only runs in kernel mode to use this functionality. The code on x86 is an exception and still retains a PF_KTHREAD test because x86 unlikely everything else handles kthreads slightly differently than user space tasks that start with a function. The functions that created tasks that start with a function have been updated to set ".fn" and ".fn_arg" instead of ".stack" and ".stack_size". These functions are fork_idle(), create_io_thread(), kernel_thread(), and user_mode_thread(). Link: https://lkml.kernel.org/r/20220506141512.516114-4-ebiederm@xmission.com Signed-off-by: "Eric W. Biederman" <ebiederm@xmission.com>
2022-04-12 15:18:48 +00:00
p->thread.s[0] = (unsigned long)args->fn;
p->thread.s[1] = (unsigned long)args->fn_arg;
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
} else {
*childregs = *(current_pt_regs());
riscv: Add task switch support for vector This patch adds task switch support for vector. It also supports all lengths of vlen. Suggested-by: Andrew Waterman <andrew@sifive.com> Co-developed-by: Nick Knight <nick.knight@sifive.com> Signed-off-by: Nick Knight <nick.knight@sifive.com> Co-developed-by: Guo Ren <guoren@linux.alibaba.com> Signed-off-by: Guo Ren <guoren@linux.alibaba.com> Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Co-developed-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Ruinland Tsai <ruinland.tsai@sifive.com> Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Signed-off-by: Vineet Gupta <vineetg@rivosinc.com> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Conor Dooley <conor.dooley@microchip.com> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Reviewed-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Tested-by: Heiko Stuebner <heiko.stuebner@vrull.eu> Link: https://lore.kernel.org/r/20230605110724.21391-11-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-06-05 11:07:07 +00:00
/* Turn off status.VS */
riscv_v_vstate_off(childregs);
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
if (usp) /* User fork */
childregs->sp = usp;
if (clone_flags & CLONE_SETTLS)
riscv: Implement copy_thread_tls This is required for clone3 which passes the TLS value through a struct rather than a register. Signed-off-by: Amanieu d'Antras <amanieu@gmail.com> Cc: linux-riscv@lists.infradead.org Cc: <stable@vger.kernel.org> # 5.3.x Link: https://lore.kernel.org/r/20200102172413.654385-6-amanieu@gmail.com Signed-off-by: Christian Brauner <christian.brauner@ubuntu.com>
2020-01-02 17:24:11 +00:00
childregs->tp = tls;
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
childregs->a0 = 0; /* Return value of fork() */
riscv: entry: Consolidate ret_from_kernel_thread into ret_from_fork The ret_from_kernel_thread() behaves similarly with ret_from_fork(), the only difference is whether call the fn(arg) or not, this can be achieved by testing fn is NULL or not, I.E s0 is 0 or not. Many architectures have done the same thing, it makes entry.S more clean. Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Reviewed-by: Guo Ren <guoren@kernel.org> Tested-by: Guo Ren <guoren@kernel.org> Signed-off-by: Guo Ren <guoren@kernel.org> Link: https://lore.kernel.org/r/20230222033021.983168-7-guoren@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-02-22 03:30:20 +00:00
p->thread.s[0] = 0;
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
}
riscv: Add support for kernel mode vector Add kernel_vector_begin() and kernel_vector_end() function declarations and corresponding definitions in kernel_mode_vector.c These are needed to wrap uses of vector in kernel mode. Co-developed-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Vincent Chen <vincent.chen@sifive.com> Signed-off-by: Greentime Hu <greentime.hu@sifive.com> Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Reviewed-by: Eric Biggers <ebiggers@google.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-2-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-15 05:59:20 +00:00
p->thread.riscv_v_flags = 0;
riscv: vector: allow kernel-mode Vector with preemption Add kernel_vstate to keep track of kernel-mode Vector registers when trap introduced context switch happens. Also, provide riscv_v_flags to let context save/restore routine track context status. Context tracking happens whenever the core starts its in-kernel Vector executions. An active (dirty) kernel task's V contexts will be saved to memory whenever a trap-introduced context switch happens. Or, when a softirq, which happens to nest on top of it, uses Vector. Context retoring happens when the execution transfer back to the original Kernel context where it first enable preempt_v. Also, provide a config CONFIG_RISCV_ISA_V_PREEMPTIVE to give users an option to disable preemptible kernel-mode Vector at build time. Users with constraint memory may want to disable this config as preemptible kernel-mode Vector needs extra space for tracking of per thread's kernel-mode V context. Or, users might as well want to disable it if all kernel-mode Vector code is time sensitive and cannot tolerate context switch overhead. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-11-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-15 05:59:29 +00:00
if (has_vector())
riscv_v_thread_alloc(p);
riscv: entry: Consolidate ret_from_kernel_thread into ret_from_fork The ret_from_kernel_thread() behaves similarly with ret_from_fork(), the only difference is whether call the fn(arg) or not, this can be achieved by testing fn is NULL or not, I.E s0 is 0 or not. Many architectures have done the same thing, it makes entry.S more clean. Signed-off-by: Jisheng Zhang <jszhang@kernel.org> Reviewed-by: Björn Töpel <bjorn@rivosinc.com> Reviewed-by: Guo Ren <guoren@kernel.org> Tested-by: Guo Ren <guoren@kernel.org> Signed-off-by: Guo Ren <guoren@kernel.org> Link: https://lore.kernel.org/r/20230222033021.983168-7-guoren@kernel.org Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2023-02-22 03:30:20 +00:00
p->thread.ra = (unsigned long)ret_from_fork;
RISC-V: Task implementation This patch contains the implementation of tasks on RISC-V, most of which is involved in task switching. Signed-off-by: Palmer Dabbelt <palmer@dabbelt.com>
2017-07-11 01:04:30 +00:00
p->thread.sp = (unsigned long)childregs; /* kernel sp */
return 0;
}
riscv: vector: use kmem_cache to manage vector context The allocation size of thread.vstate.datap is always riscv_v_vsize. So it is possbile to use kmem_cache_* to manage the allocation. This gives users more information regarding allocation of vector context via /proc/slabinfo. And it potentially reduces the latency of the first-use trap because of the allocation caches. Signed-off-by: Andy Chiu <andy.chiu@sifive.com> Tested-by: Björn Töpel <bjorn@rivosinc.com> Tested-by: Lad Prabhakar <prabhakar.mahadev-lad.rj@bp.renesas.com> Link: https://lore.kernel.org/r/20240115055929.4736-10-andy.chiu@sifive.com Signed-off-by: Palmer Dabbelt <palmer@rivosinc.com>
2024-01-15 05:59:28 +00:00
void __init arch_task_cache_init(void)
{
riscv_v_setup_ctx_cache();
}
Copy permalink