Merge branch kvm-arm64/pkvm-6.10 into kvmarm-master/next
* kvm-arm64/pkvm-6.10: (25 commits) : . : At last, a bunch of pKVM patches, courtesy of Fuad Tabba. : From the cover letter: : : "This series is a bit of a bombay-mix of patches we've been : carrying. There's no one overarching theme, but they do improve : the code by fixing existing bugs in pKVM, refactoring code to : make it more readable and easier to re-use for pKVM, or adding : functionality to the existing pKVM code upstream." : . KVM: arm64: Force injection of a data abort on NISV MMIO exit KVM: arm64: Restrict supported capabilities for protected VMs KVM: arm64: Refactor setting the return value in kvm_vm_ioctl_enable_cap() KVM: arm64: Document the KVM/arm64-specific calls in hypercalls.rst KVM: arm64: Rename firmware pseudo-register documentation file KVM: arm64: Reformat/beautify PTP hypercall documentation KVM: arm64: Clarify rationale for ZCR_EL1 value restored on guest exit KVM: arm64: Introduce and use predicates that check for protected VMs KVM: arm64: Add is_pkvm_initialized() helper KVM: arm64: Simplify vgic-v3 hypercalls KVM: arm64: Move setting the page as dirty out of the critical section KVM: arm64: Change kvm_handle_mmio_return() return polarity KVM: arm64: Fix comment for __pkvm_vcpu_init_traps() KVM: arm64: Prevent kmemleak from accessing .hyp.data KVM: arm64: Do not map the host fpsimd state to hyp in pKVM KVM: arm64: Rename __tlb_switch_to_{guest,host}() in VHE KVM: arm64: Support TLB invalidation in guest context KVM: arm64: Avoid BBM when changing only s/w bits in Stage-2 PTE KVM: arm64: Check for PTE validity when checking for executable/cacheable KVM: arm64: Avoid BUG-ing from the host abort path ... Signed-off-by: Marc Zyngier <maz@kernel.org>
This commit is contained in:
commit
8540bd1b99
33 changed files with 513 additions and 344 deletions
|
@ -6894,6 +6894,13 @@ Note that KVM does not skip the faulting instruction as it does for
|
|||
KVM_EXIT_MMIO, but userspace has to emulate any change to the processing state
|
||||
if it decides to decode and emulate the instruction.
|
||||
|
||||
This feature isn't available to protected VMs, as userspace does not
|
||||
have access to the state that is required to perform the emulation.
|
||||
Instead, a data abort exception is directly injected in the guest.
|
||||
Note that although KVM_CAP_ARM_NISV_TO_USER will be reported if
|
||||
queried outside of a protected VM context, the feature will not be
|
||||
exposed if queried on a protected VM file descriptor.
|
||||
|
||||
::
|
||||
|
||||
/* KVM_EXIT_X86_RDMSR / KVM_EXIT_X86_WRMSR */
|
||||
|
|
138
Documentation/virt/kvm/arm/fw-pseudo-registers.rst
Normal file
138
Documentation/virt/kvm/arm/fw-pseudo-registers.rst
Normal file
|
@ -0,0 +1,138 @@
|
|||
.. SPDX-License-Identifier: GPL-2.0
|
||||
|
||||
=======================================
|
||||
ARM firmware pseudo-registers interface
|
||||
=======================================
|
||||
|
||||
KVM handles the hypercall services as requested by the guests. New hypercall
|
||||
services are regularly made available by the ARM specification or by KVM (as
|
||||
vendor services) if they make sense from a virtualization point of view.
|
||||
|
||||
This means that a guest booted on two different versions of KVM can observe
|
||||
two different "firmware" revisions. This could cause issues if a given guest
|
||||
is tied to a particular version of a hypercall service, or if a migration
|
||||
causes a different version to be exposed out of the blue to an unsuspecting
|
||||
guest.
|
||||
|
||||
In order to remedy this situation, KVM exposes a set of "firmware
|
||||
pseudo-registers" that can be manipulated using the GET/SET_ONE_REG
|
||||
interface. These registers can be saved/restored by userspace, and set
|
||||
to a convenient value as required.
|
||||
|
||||
The following registers are defined:
|
||||
|
||||
* KVM_REG_ARM_PSCI_VERSION:
|
||||
|
||||
KVM implements the PSCI (Power State Coordination Interface)
|
||||
specification in order to provide services such as CPU on/off, reset
|
||||
and power-off to the guest.
|
||||
|
||||
- Only valid if the vcpu has the KVM_ARM_VCPU_PSCI_0_2 feature set
|
||||
(and thus has already been initialized)
|
||||
- Returns the current PSCI version on GET_ONE_REG (defaulting to the
|
||||
highest PSCI version implemented by KVM and compatible with v0.2)
|
||||
- Allows any PSCI version implemented by KVM and compatible with
|
||||
v0.2 to be set with SET_ONE_REG
|
||||
- Affects the whole VM (even if the register view is per-vcpu)
|
||||
|
||||
* KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1:
|
||||
Holds the state of the firmware support to mitigate CVE-2017-5715, as
|
||||
offered by KVM to the guest via a HVC call. The workaround is described
|
||||
under SMCCC_ARCH_WORKAROUND_1 in [1].
|
||||
|
||||
Accepted values are:
|
||||
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_NOT_AVAIL:
|
||||
KVM does not offer
|
||||
firmware support for the workaround. The mitigation status for the
|
||||
guest is unknown.
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_AVAIL:
|
||||
The workaround HVC call is
|
||||
available to the guest and required for the mitigation.
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_NOT_REQUIRED:
|
||||
The workaround HVC call
|
||||
is available to the guest, but it is not needed on this VCPU.
|
||||
|
||||
* KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2:
|
||||
Holds the state of the firmware support to mitigate CVE-2018-3639, as
|
||||
offered by KVM to the guest via a HVC call. The workaround is described
|
||||
under SMCCC_ARCH_WORKAROUND_2 in [1]_.
|
||||
|
||||
Accepted values are:
|
||||
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_AVAIL:
|
||||
A workaround is not
|
||||
available. KVM does not offer firmware support for the workaround.
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_UNKNOWN:
|
||||
The workaround state is
|
||||
unknown. KVM does not offer firmware support for the workaround.
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_AVAIL:
|
||||
The workaround is available,
|
||||
and can be disabled by a vCPU. If
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_ENABLED is set, it is active for
|
||||
this vCPU.
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_REQUIRED:
|
||||
The workaround is always active on this vCPU or it is not needed.
|
||||
|
||||
|
||||
Bitmap Feature Firmware Registers
|
||||
---------------------------------
|
||||
|
||||
Contrary to the above registers, the following registers exposes the
|
||||
hypercall services in the form of a feature-bitmap to the userspace. This
|
||||
bitmap is translated to the services that are available to the guest.
|
||||
There is a register defined per service call owner and can be accessed via
|
||||
GET/SET_ONE_REG interface.
|
||||
|
||||
By default, these registers are set with the upper limit of the features
|
||||
that are supported. This way userspace can discover all the usable
|
||||
hypercall services via GET_ONE_REG. The user-space can write-back the
|
||||
desired bitmap back via SET_ONE_REG. The features for the registers that
|
||||
are untouched, probably because userspace isn't aware of them, will be
|
||||
exposed as is to the guest.
|
||||
|
||||
Note that KVM will not allow the userspace to configure the registers
|
||||
anymore once any of the vCPUs has run at least once. Instead, it will
|
||||
return a -EBUSY.
|
||||
|
||||
The pseudo-firmware bitmap register are as follows:
|
||||
|
||||
* KVM_REG_ARM_STD_BMAP:
|
||||
Controls the bitmap of the ARM Standard Secure Service Calls.
|
||||
|
||||
The following bits are accepted:
|
||||
|
||||
Bit-0: KVM_REG_ARM_STD_BIT_TRNG_V1_0:
|
||||
The bit represents the services offered under v1.0 of ARM True Random
|
||||
Number Generator (TRNG) specification, ARM DEN0098.
|
||||
|
||||
* KVM_REG_ARM_STD_HYP_BMAP:
|
||||
Controls the bitmap of the ARM Standard Hypervisor Service Calls.
|
||||
|
||||
The following bits are accepted:
|
||||
|
||||
Bit-0: KVM_REG_ARM_STD_HYP_BIT_PV_TIME:
|
||||
The bit represents the Paravirtualized Time service as represented by
|
||||
ARM DEN0057A.
|
||||
|
||||
* KVM_REG_ARM_VENDOR_HYP_BMAP:
|
||||
Controls the bitmap of the Vendor specific Hypervisor Service Calls.
|
||||
|
||||
The following bits are accepted:
|
||||
|
||||
Bit-0: KVM_REG_ARM_VENDOR_HYP_BIT_FUNC_FEAT
|
||||
The bit represents the ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID
|
||||
and ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID function-ids.
|
||||
|
||||
Bit-1: KVM_REG_ARM_VENDOR_HYP_BIT_PTP:
|
||||
The bit represents the Precision Time Protocol KVM service.
|
||||
|
||||
Errors:
|
||||
|
||||
======= =============================================================
|
||||
-ENOENT Unknown register accessed.
|
||||
-EBUSY Attempt a 'write' to the register after the VM has started.
|
||||
-EINVAL Invalid bitmap written to the register.
|
||||
======= =============================================================
|
||||
|
||||
.. [1] https://developer.arm.com/-/media/developer/pdf/ARM_DEN_0070A_Firmware_interfaces_for_mitigating_CVE-2017-5715.pdf
|
|
@ -1,138 +1,46 @@
|
|||
.. SPDX-License-Identifier: GPL-2.0
|
||||
|
||||
=======================
|
||||
ARM Hypercall Interface
|
||||
=======================
|
||||
===============================================
|
||||
KVM/arm64-specific hypercalls exposed to guests
|
||||
===============================================
|
||||
|
||||
KVM handles the hypercall services as requested by the guests. New hypercall
|
||||
services are regularly made available by the ARM specification or by KVM (as
|
||||
vendor services) if they make sense from a virtualization point of view.
|
||||
This file documents the KVM/arm64-specific hypercalls which may be
|
||||
exposed by KVM/arm64 to guest operating systems. These hypercalls are
|
||||
issued using the HVC instruction according to version 1.1 of the Arm SMC
|
||||
Calling Convention (DEN0028/C):
|
||||
|
||||
This means that a guest booted on two different versions of KVM can observe
|
||||
two different "firmware" revisions. This could cause issues if a given guest
|
||||
is tied to a particular version of a hypercall service, or if a migration
|
||||
causes a different version to be exposed out of the blue to an unsuspecting
|
||||
guest.
|
||||
https://developer.arm.com/docs/den0028/c
|
||||
|
||||
In order to remedy this situation, KVM exposes a set of "firmware
|
||||
pseudo-registers" that can be manipulated using the GET/SET_ONE_REG
|
||||
interface. These registers can be saved/restored by userspace, and set
|
||||
to a convenient value as required.
|
||||
All KVM/arm64-specific hypercalls are allocated within the "Vendor
|
||||
Specific Hypervisor Service Call" range with a UID of
|
||||
``28b46fb6-2ec5-11e9-a9ca-4b564d003a74``. This UID should be queried by the
|
||||
guest using the standard "Call UID" function for the service range in
|
||||
order to determine that the KVM/arm64-specific hypercalls are available.
|
||||
|
||||
The following registers are defined:
|
||||
``ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID``
|
||||
---------------------------------------------
|
||||
|
||||
* KVM_REG_ARM_PSCI_VERSION:
|
||||
Provides a discovery mechanism for other KVM/arm64 hypercalls.
|
||||
|
||||
KVM implements the PSCI (Power State Coordination Interface)
|
||||
specification in order to provide services such as CPU on/off, reset
|
||||
and power-off to the guest.
|
||||
+---------------------+-------------------------------------------------------------+
|
||||
| Presence: | Mandatory for the KVM/arm64 UID |
|
||||
+---------------------+-------------------------------------------------------------+
|
||||
| Calling convention: | HVC32 |
|
||||
+---------------------+----------+--------------------------------------------------+
|
||||
| Function ID: | (uint32) | 0x86000000 |
|
||||
+---------------------+----------+--------------------------------------------------+
|
||||
| Arguments: | None |
|
||||
+---------------------+----------+----+---------------------------------------------+
|
||||
| Return Values: | (uint32) | R0 | Bitmap of available function numbers 0-31 |
|
||||
| +----------+----+---------------------------------------------+
|
||||
| | (uint32) | R1 | Bitmap of available function numbers 32-63 |
|
||||
| +----------+----+---------------------------------------------+
|
||||
| | (uint32) | R2 | Bitmap of available function numbers 64-95 |
|
||||
| +----------+----+---------------------------------------------+
|
||||
| | (uint32) | R3 | Bitmap of available function numbers 96-127 |
|
||||
+---------------------+----------+----+---------------------------------------------+
|
||||
|
||||
- Only valid if the vcpu has the KVM_ARM_VCPU_PSCI_0_2 feature set
|
||||
(and thus has already been initialized)
|
||||
- Returns the current PSCI version on GET_ONE_REG (defaulting to the
|
||||
highest PSCI version implemented by KVM and compatible with v0.2)
|
||||
- Allows any PSCI version implemented by KVM and compatible with
|
||||
v0.2 to be set with SET_ONE_REG
|
||||
- Affects the whole VM (even if the register view is per-vcpu)
|
||||
``ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID``
|
||||
----------------------------------------
|
||||
|
||||
* KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1:
|
||||
Holds the state of the firmware support to mitigate CVE-2017-5715, as
|
||||
offered by KVM to the guest via a HVC call. The workaround is described
|
||||
under SMCCC_ARCH_WORKAROUND_1 in [1].
|
||||
|
||||
Accepted values are:
|
||||
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_NOT_AVAIL:
|
||||
KVM does not offer
|
||||
firmware support for the workaround. The mitigation status for the
|
||||
guest is unknown.
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_AVAIL:
|
||||
The workaround HVC call is
|
||||
available to the guest and required for the mitigation.
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_1_NOT_REQUIRED:
|
||||
The workaround HVC call
|
||||
is available to the guest, but it is not needed on this VCPU.
|
||||
|
||||
* KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2:
|
||||
Holds the state of the firmware support to mitigate CVE-2018-3639, as
|
||||
offered by KVM to the guest via a HVC call. The workaround is described
|
||||
under SMCCC_ARCH_WORKAROUND_2 in [1]_.
|
||||
|
||||
Accepted values are:
|
||||
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_AVAIL:
|
||||
A workaround is not
|
||||
available. KVM does not offer firmware support for the workaround.
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_UNKNOWN:
|
||||
The workaround state is
|
||||
unknown. KVM does not offer firmware support for the workaround.
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_AVAIL:
|
||||
The workaround is available,
|
||||
and can be disabled by a vCPU. If
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_ENABLED is set, it is active for
|
||||
this vCPU.
|
||||
KVM_REG_ARM_SMCCC_ARCH_WORKAROUND_2_NOT_REQUIRED:
|
||||
The workaround is always active on this vCPU or it is not needed.
|
||||
|
||||
|
||||
Bitmap Feature Firmware Registers
|
||||
---------------------------------
|
||||
|
||||
Contrary to the above registers, the following registers exposes the
|
||||
hypercall services in the form of a feature-bitmap to the userspace. This
|
||||
bitmap is translated to the services that are available to the guest.
|
||||
There is a register defined per service call owner and can be accessed via
|
||||
GET/SET_ONE_REG interface.
|
||||
|
||||
By default, these registers are set with the upper limit of the features
|
||||
that are supported. This way userspace can discover all the usable
|
||||
hypercall services via GET_ONE_REG. The user-space can write-back the
|
||||
desired bitmap back via SET_ONE_REG. The features for the registers that
|
||||
are untouched, probably because userspace isn't aware of them, will be
|
||||
exposed as is to the guest.
|
||||
|
||||
Note that KVM will not allow the userspace to configure the registers
|
||||
anymore once any of the vCPUs has run at least once. Instead, it will
|
||||
return a -EBUSY.
|
||||
|
||||
The pseudo-firmware bitmap register are as follows:
|
||||
|
||||
* KVM_REG_ARM_STD_BMAP:
|
||||
Controls the bitmap of the ARM Standard Secure Service Calls.
|
||||
|
||||
The following bits are accepted:
|
||||
|
||||
Bit-0: KVM_REG_ARM_STD_BIT_TRNG_V1_0:
|
||||
The bit represents the services offered under v1.0 of ARM True Random
|
||||
Number Generator (TRNG) specification, ARM DEN0098.
|
||||
|
||||
* KVM_REG_ARM_STD_HYP_BMAP:
|
||||
Controls the bitmap of the ARM Standard Hypervisor Service Calls.
|
||||
|
||||
The following bits are accepted:
|
||||
|
||||
Bit-0: KVM_REG_ARM_STD_HYP_BIT_PV_TIME:
|
||||
The bit represents the Paravirtualized Time service as represented by
|
||||
ARM DEN0057A.
|
||||
|
||||
* KVM_REG_ARM_VENDOR_HYP_BMAP:
|
||||
Controls the bitmap of the Vendor specific Hypervisor Service Calls.
|
||||
|
||||
The following bits are accepted:
|
||||
|
||||
Bit-0: KVM_REG_ARM_VENDOR_HYP_BIT_FUNC_FEAT
|
||||
The bit represents the ARM_SMCCC_VENDOR_HYP_KVM_FEATURES_FUNC_ID
|
||||
and ARM_SMCCC_VENDOR_HYP_CALL_UID_FUNC_ID function-ids.
|
||||
|
||||
Bit-1: KVM_REG_ARM_VENDOR_HYP_BIT_PTP:
|
||||
The bit represents the Precision Time Protocol KVM service.
|
||||
|
||||
Errors:
|
||||
|
||||
======= =============================================================
|
||||
-ENOENT Unknown register accessed.
|
||||
-EBUSY Attempt a 'write' to the register after the VM has started.
|
||||
-EINVAL Invalid bitmap written to the register.
|
||||
======= =============================================================
|
||||
|
||||
.. [1] https://developer.arm.com/-/media/developer/pdf/ARM_DEN_0070A_Firmware_interfaces_for_mitigating_CVE-2017-5715.pdf
|
||||
See ptp_kvm.rst
|
||||
|
|
|
@ -7,6 +7,7 @@ ARM
|
|||
.. toctree::
|
||||
:maxdepth: 2
|
||||
|
||||
fw-pseudo-registers
|
||||
hyp-abi
|
||||
hypercalls
|
||||
pvtime
|
||||
|
|
|
@ -7,19 +7,29 @@ PTP_KVM is used for high precision time sync between host and guests.
|
|||
It relies on transferring the wall clock and counter value from the
|
||||
host to the guest using a KVM-specific hypercall.
|
||||
|
||||
* ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID: 0x86000001
|
||||
``ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID``
|
||||
----------------------------------------
|
||||
|
||||
This hypercall uses the SMC32/HVC32 calling convention:
|
||||
Retrieve current time information for the specific counter. There are no
|
||||
endianness restrictions.
|
||||
|
||||
ARM_SMCCC_VENDOR_HYP_KVM_PTP_FUNC_ID
|
||||
============== ======== =====================================
|
||||
Function ID: (uint32) 0x86000001
|
||||
Arguments: (uint32) KVM_PTP_VIRT_COUNTER(0)
|
||||
KVM_PTP_PHYS_COUNTER(1)
|
||||
Return Values: (int32) NOT_SUPPORTED(-1) on error, or
|
||||
(uint32) Upper 32 bits of wall clock time (r0)
|
||||
(uint32) Lower 32 bits of wall clock time (r1)
|
||||
(uint32) Upper 32 bits of counter (r2)
|
||||
(uint32) Lower 32 bits of counter (r3)
|
||||
Endianness: No Restrictions.
|
||||
============== ======== =====================================
|
||||
+---------------------+-------------------------------------------------------+
|
||||
| Presence: | Optional |
|
||||
+---------------------+-------------------------------------------------------+
|
||||
| Calling convention: | HVC32 |
|
||||
+---------------------+----------+--------------------------------------------+
|
||||
| Function ID: | (uint32) | 0x86000001 |
|
||||
+---------------------+----------+----+---------------------------------------+
|
||||
| Arguments: | (uint32) | R1 | ``KVM_PTP_VIRT_COUNTER (0)`` |
|
||||
| | | +---------------------------------------+
|
||||
| | | | ``KVM_PTP_PHYS_COUNTER (1)`` |
|
||||
+---------------------+----------+----+---------------------------------------+
|
||||
| Return Values: | (int32) | R0 | ``NOT_SUPPORTED (-1)`` on error, else |
|
||||
| | | | upper 32 bits of wall clock time |
|
||||
| +----------+----+---------------------------------------+
|
||||
| | (uint32) | R1 | Lower 32 bits of wall clock time |
|
||||
| +----------+----+---------------------------------------+
|
||||
| | (uint32) | R2 | Upper 32 bits of counter |
|
||||
| +----------+----+---------------------------------------+
|
||||
| | (uint32) | R3 | Lower 32 bits of counter |
|
||||
+---------------------+----------+----+---------------------------------------+
|
||||
|
|
|
@ -73,10 +73,8 @@ enum __kvm_host_smccc_func {
|
|||
__KVM_HOST_SMCCC_FUNC___kvm_tlb_flush_vmid_range,
|
||||
__KVM_HOST_SMCCC_FUNC___kvm_flush_cpu_context,
|
||||
__KVM_HOST_SMCCC_FUNC___kvm_timer_set_cntvoff,
|
||||
__KVM_HOST_SMCCC_FUNC___vgic_v3_read_vmcr,
|
||||
__KVM_HOST_SMCCC_FUNC___vgic_v3_write_vmcr,
|
||||
__KVM_HOST_SMCCC_FUNC___vgic_v3_save_aprs,
|
||||
__KVM_HOST_SMCCC_FUNC___vgic_v3_restore_aprs,
|
||||
__KVM_HOST_SMCCC_FUNC___vgic_v3_save_vmcr_aprs,
|
||||
__KVM_HOST_SMCCC_FUNC___vgic_v3_restore_vmcr_aprs,
|
||||
__KVM_HOST_SMCCC_FUNC___pkvm_vcpu_init_traps,
|
||||
__KVM_HOST_SMCCC_FUNC___pkvm_init_vm,
|
||||
__KVM_HOST_SMCCC_FUNC___pkvm_init_vcpu,
|
||||
|
@ -241,8 +239,6 @@ extern int __kvm_vcpu_run(struct kvm_vcpu *vcpu);
|
|||
extern void __kvm_adjust_pc(struct kvm_vcpu *vcpu);
|
||||
|
||||
extern u64 __vgic_v3_get_gic_config(void);
|
||||
extern u64 __vgic_v3_read_vmcr(void);
|
||||
extern void __vgic_v3_write_vmcr(u32 vmcr);
|
||||
extern void __vgic_v3_init_lrs(void);
|
||||
|
||||
extern u64 __kvm_get_mdcr_el2(void);
|
||||
|
|
|
@ -577,16 +577,14 @@ static __always_inline u64 kvm_get_reset_cptr_el2(struct kvm_vcpu *vcpu)
|
|||
} else if (has_hvhe()) {
|
||||
val = (CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN);
|
||||
|
||||
if (!vcpu_has_sve(vcpu) ||
|
||||
(*host_data_ptr(fp_owner) != FP_STATE_GUEST_OWNED))
|
||||
if (!vcpu_has_sve(vcpu) || !guest_owns_fp_regs())
|
||||
val |= CPACR_EL1_ZEN_EL1EN | CPACR_EL1_ZEN_EL0EN;
|
||||
if (cpus_have_final_cap(ARM64_SME))
|
||||
val |= CPACR_EL1_SMEN_EL1EN | CPACR_EL1_SMEN_EL0EN;
|
||||
} else {
|
||||
val = CPTR_NVHE_EL2_RES1;
|
||||
|
||||
if (vcpu_has_sve(vcpu) &&
|
||||
(*host_data_ptr(fp_owner) == FP_STATE_GUEST_OWNED))
|
||||
if (vcpu_has_sve(vcpu) && guest_owns_fp_regs())
|
||||
val |= CPTR_EL2_TZ;
|
||||
if (cpus_have_final_cap(ARM64_SME))
|
||||
val &= ~CPTR_EL2_TSM;
|
||||
|
|
|
@ -211,6 +211,7 @@ typedef unsigned int pkvm_handle_t;
|
|||
struct kvm_protected_vm {
|
||||
pkvm_handle_t handle;
|
||||
struct kvm_hyp_memcache teardown_mc;
|
||||
bool enabled;
|
||||
};
|
||||
|
||||
struct kvm_mpidr_data {
|
||||
|
@ -663,8 +664,6 @@ struct kvm_vcpu_arch {
|
|||
struct kvm_guest_debug_arch vcpu_debug_state;
|
||||
struct kvm_guest_debug_arch external_debug_state;
|
||||
|
||||
struct task_struct *parent_task;
|
||||
|
||||
/* VGIC state */
|
||||
struct vgic_cpu vgic_cpu;
|
||||
struct arch_timer_cpu timer_cpu;
|
||||
|
@ -1212,6 +1211,18 @@ DECLARE_KVM_HYP_PER_CPU(struct kvm_host_data, kvm_host_data);
|
|||
&this_cpu_ptr_hyp_sym(kvm_host_data)->f)
|
||||
#endif
|
||||
|
||||
/* Check whether the FP regs are owned by the guest */
|
||||
static inline bool guest_owns_fp_regs(void)
|
||||
{
|
||||
return *host_data_ptr(fp_owner) == FP_STATE_GUEST_OWNED;
|
||||
}
|
||||
|
||||
/* Check whether the FP regs are owned by the host */
|
||||
static inline bool host_owns_fp_regs(void)
|
||||
{
|
||||
return *host_data_ptr(fp_owner) == FP_STATE_HOST_OWNED;
|
||||
}
|
||||
|
||||
static inline void kvm_init_host_cpu_context(struct kvm_cpu_context *cpu_ctxt)
|
||||
{
|
||||
/* The host's MPIDR is immutable, so let's set it up at boot time */
|
||||
|
@ -1255,7 +1266,6 @@ void kvm_arch_vcpu_load_fp(struct kvm_vcpu *vcpu);
|
|||
void kvm_arch_vcpu_ctxflush_fp(struct kvm_vcpu *vcpu);
|
||||
void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu);
|
||||
void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu);
|
||||
void kvm_vcpu_unshare_task_fp(struct kvm_vcpu *vcpu);
|
||||
|
||||
static inline bool kvm_pmu_counter_deferred(struct perf_event_attr *attr)
|
||||
{
|
||||
|
@ -1291,10 +1301,9 @@ struct kvm *kvm_arch_alloc_vm(void);
|
|||
|
||||
#define __KVM_HAVE_ARCH_FLUSH_REMOTE_TLBS_RANGE
|
||||
|
||||
static inline bool kvm_vm_is_protected(struct kvm *kvm)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
#define kvm_vm_is_protected(kvm) (is_protected_kvm_enabled() && (kvm)->arch.pkvm.enabled)
|
||||
|
||||
#define vcpu_is_protected(vcpu) kvm_vm_is_protected((vcpu)->kvm)
|
||||
|
||||
int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature);
|
||||
bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu);
|
||||
|
|
|
@ -80,8 +80,8 @@ void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if);
|
|||
void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if);
|
||||
void __vgic_v3_activate_traps(struct vgic_v3_cpu_if *cpu_if);
|
||||
void __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if);
|
||||
void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if);
|
||||
void __vgic_v3_restore_aprs(struct vgic_v3_cpu_if *cpu_if);
|
||||
void __vgic_v3_save_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if);
|
||||
void __vgic_v3_restore_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if);
|
||||
int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu);
|
||||
|
||||
#ifdef __KVM_NVHE_HYPERVISOR__
|
||||
|
|
|
@ -82,6 +82,12 @@ bool is_kvm_arm_initialised(void);
|
|||
|
||||
DECLARE_STATIC_KEY_FALSE(kvm_protected_mode_initialized);
|
||||
|
||||
static inline bool is_pkvm_initialized(void)
|
||||
{
|
||||
return IS_ENABLED(CONFIG_KVM) &&
|
||||
static_branch_likely(&kvm_protected_mode_initialized);
|
||||
}
|
||||
|
||||
/* Reports the availability of HYP mode */
|
||||
static inline bool is_hyp_mode_available(void)
|
||||
{
|
||||
|
@ -89,8 +95,7 @@ static inline bool is_hyp_mode_available(void)
|
|||
* If KVM protected mode is initialized, all CPUs must have been booted
|
||||
* in EL2. Avoid checking __boot_cpu_mode as CPUs now come up in EL1.
|
||||
*/
|
||||
if (IS_ENABLED(CONFIG_KVM) &&
|
||||
static_branch_likely(&kvm_protected_mode_initialized))
|
||||
if (is_pkvm_initialized())
|
||||
return true;
|
||||
|
||||
return (__boot_cpu_mode[0] == BOOT_CPU_MODE_EL2 &&
|
||||
|
@ -104,8 +109,7 @@ static inline bool is_hyp_mode_mismatched(void)
|
|||
* If KVM protected mode is initialized, all CPUs must have been booted
|
||||
* in EL2. Avoid checking __boot_cpu_mode as CPUs now come up in EL1.
|
||||
*/
|
||||
if (IS_ENABLED(CONFIG_KVM) &&
|
||||
static_branch_likely(&kvm_protected_mode_initialized))
|
||||
if (is_pkvm_initialized())
|
||||
return false;
|
||||
|
||||
return __boot_cpu_mode[0] != __boot_cpu_mode[1];
|
||||
|
|
|
@ -70,15 +70,42 @@ int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
|
|||
return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE;
|
||||
}
|
||||
|
||||
/*
|
||||
* This functions as an allow-list of protected VM capabilities.
|
||||
* Features not explicitly allowed by this function are denied.
|
||||
*/
|
||||
static bool pkvm_ext_allowed(struct kvm *kvm, long ext)
|
||||
{
|
||||
switch (ext) {
|
||||
case KVM_CAP_IRQCHIP:
|
||||
case KVM_CAP_ARM_PSCI:
|
||||
case KVM_CAP_ARM_PSCI_0_2:
|
||||
case KVM_CAP_NR_VCPUS:
|
||||
case KVM_CAP_MAX_VCPUS:
|
||||
case KVM_CAP_MAX_VCPU_ID:
|
||||
case KVM_CAP_MSI_DEVID:
|
||||
case KVM_CAP_ARM_VM_IPA_SIZE:
|
||||
case KVM_CAP_ARM_PMU_V3:
|
||||
case KVM_CAP_ARM_SVE:
|
||||
case KVM_CAP_ARM_PTRAUTH_ADDRESS:
|
||||
case KVM_CAP_ARM_PTRAUTH_GENERIC:
|
||||
return true;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
|
||||
struct kvm_enable_cap *cap)
|
||||
{
|
||||
int r;
|
||||
u64 new_cap;
|
||||
int r = -EINVAL;
|
||||
|
||||
if (cap->flags)
|
||||
return -EINVAL;
|
||||
|
||||
if (kvm_vm_is_protected(kvm) && !pkvm_ext_allowed(kvm, cap->cap))
|
||||
return -EINVAL;
|
||||
|
||||
switch (cap->cap) {
|
||||
case KVM_CAP_ARM_NISV_TO_USER:
|
||||
r = 0;
|
||||
|
@ -87,9 +114,7 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
|
|||
break;
|
||||
case KVM_CAP_ARM_MTE:
|
||||
mutex_lock(&kvm->lock);
|
||||
if (!system_supports_mte() || kvm->created_vcpus) {
|
||||
r = -EINVAL;
|
||||
} else {
|
||||
if (system_supports_mte() && !kvm->created_vcpus) {
|
||||
r = 0;
|
||||
set_bit(KVM_ARCH_FLAG_MTE_ENABLED, &kvm->arch.flags);
|
||||
}
|
||||
|
@ -100,25 +125,22 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
|
|||
set_bit(KVM_ARCH_FLAG_SYSTEM_SUSPEND_ENABLED, &kvm->arch.flags);
|
||||
break;
|
||||
case KVM_CAP_ARM_EAGER_SPLIT_CHUNK_SIZE:
|
||||
new_cap = cap->args[0];
|
||||
|
||||
mutex_lock(&kvm->slots_lock);
|
||||
/*
|
||||
* To keep things simple, allow changing the chunk
|
||||
* size only when no memory slots have been created.
|
||||
*/
|
||||
if (!kvm_are_all_memslots_empty(kvm)) {
|
||||
r = -EINVAL;
|
||||
} else if (new_cap && !kvm_is_block_size_supported(new_cap)) {
|
||||
r = -EINVAL;
|
||||
} else {
|
||||
r = 0;
|
||||
kvm->arch.mmu.split_page_chunk_size = new_cap;
|
||||
if (kvm_are_all_memslots_empty(kvm)) {
|
||||
u64 new_cap = cap->args[0];
|
||||
|
||||
if (!new_cap || kvm_is_block_size_supported(new_cap)) {
|
||||
r = 0;
|
||||
kvm->arch.mmu.split_page_chunk_size = new_cap;
|
||||
}
|
||||
}
|
||||
mutex_unlock(&kvm->slots_lock);
|
||||
break;
|
||||
default:
|
||||
r = -EINVAL;
|
||||
break;
|
||||
}
|
||||
|
||||
|
@ -256,6 +278,10 @@ static bool kvm_has_full_ptr_auth(void)
|
|||
int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
|
||||
{
|
||||
int r;
|
||||
|
||||
if (kvm && kvm_vm_is_protected(kvm) && !pkvm_ext_allowed(kvm, ext))
|
||||
return 0;
|
||||
|
||||
switch (ext) {
|
||||
case KVM_CAP_IRQCHIP:
|
||||
r = vgic_present;
|
||||
|
@ -857,9 +883,8 @@ void kvm_vcpu_wfi(struct kvm_vcpu *vcpu)
|
|||
* doorbells to be signalled, should an interrupt become pending.
|
||||
*/
|
||||
preempt_disable();
|
||||
kvm_vgic_vmcr_sync(vcpu);
|
||||
vcpu_set_flag(vcpu, IN_WFI);
|
||||
vgic_v4_put(vcpu);
|
||||
kvm_vgic_put(vcpu);
|
||||
preempt_enable();
|
||||
|
||||
kvm_vcpu_halt(vcpu);
|
||||
|
@ -867,7 +892,7 @@ void kvm_vcpu_wfi(struct kvm_vcpu *vcpu)
|
|||
|
||||
preempt_disable();
|
||||
vcpu_clear_flag(vcpu, IN_WFI);
|
||||
vgic_v4_load(vcpu);
|
||||
kvm_vgic_load(vcpu);
|
||||
preempt_enable();
|
||||
}
|
||||
|
||||
|
@ -1047,7 +1072,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
|
|||
|
||||
if (run->exit_reason == KVM_EXIT_MMIO) {
|
||||
ret = kvm_handle_mmio_return(vcpu);
|
||||
if (ret)
|
||||
if (ret <= 0)
|
||||
return ret;
|
||||
}
|
||||
|
||||
|
|
|
@ -14,19 +14,6 @@
|
|||
#include <asm/kvm_mmu.h>
|
||||
#include <asm/sysreg.h>
|
||||
|
||||
void kvm_vcpu_unshare_task_fp(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
struct task_struct *p = vcpu->arch.parent_task;
|
||||
struct user_fpsimd_state *fpsimd;
|
||||
|
||||
if (!is_protected_kvm_enabled() || !p)
|
||||
return;
|
||||
|
||||
fpsimd = &p->thread.uw.fpsimd_state;
|
||||
kvm_unshare_hyp(fpsimd, fpsimd + 1);
|
||||
put_task_struct(p);
|
||||
}
|
||||
|
||||
/*
|
||||
* Called on entry to KVM_RUN unless this vcpu previously ran at least
|
||||
* once and the most recent prior KVM_RUN for this vcpu was called from
|
||||
|
@ -38,28 +25,18 @@ void kvm_vcpu_unshare_task_fp(struct kvm_vcpu *vcpu)
|
|||
*/
|
||||
int kvm_arch_vcpu_run_map_fp(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
struct user_fpsimd_state *fpsimd = ¤t->thread.uw.fpsimd_state;
|
||||
int ret;
|
||||
|
||||
struct user_fpsimd_state *fpsimd = ¤t->thread.uw.fpsimd_state;
|
||||
|
||||
kvm_vcpu_unshare_task_fp(vcpu);
|
||||
/* pKVM has its own tracking of the host fpsimd state. */
|
||||
if (is_protected_kvm_enabled())
|
||||
return 0;
|
||||
|
||||
/* Make sure the host task fpsimd state is visible to hyp: */
|
||||
ret = kvm_share_hyp(fpsimd, fpsimd + 1);
|
||||
if (ret)
|
||||
return ret;
|
||||
|
||||
/*
|
||||
* We need to keep current's task_struct pinned until its data has been
|
||||
* unshared with the hypervisor to make sure it is not re-used by the
|
||||
* kernel and donated to someone else while already shared -- see
|
||||
* kvm_vcpu_unshare_task_fp() for the matching put_task_struct().
|
||||
*/
|
||||
if (is_protected_kvm_enabled()) {
|
||||
get_task_struct(current);
|
||||
vcpu->arch.parent_task = current;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
@ -141,8 +118,7 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu)
|
|||
|
||||
WARN_ON_ONCE(!irqs_disabled());
|
||||
|
||||
if (*host_data_ptr(fp_owner) == FP_STATE_GUEST_OWNED) {
|
||||
|
||||
if (guest_owns_fp_regs()) {
|
||||
/*
|
||||
* Currently we do not support SME guests so SVCR is
|
||||
* always 0 and we just need a variable to point to.
|
||||
|
@ -195,16 +171,38 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu)
|
|||
isb();
|
||||
}
|
||||
|
||||
if (*host_data_ptr(fp_owner) == FP_STATE_GUEST_OWNED) {
|
||||
if (guest_owns_fp_regs()) {
|
||||
if (vcpu_has_sve(vcpu)) {
|
||||
__vcpu_sys_reg(vcpu, ZCR_EL1) = read_sysreg_el1(SYS_ZCR);
|
||||
|
||||
/* Restore the VL that was saved when bound to the CPU */
|
||||
/*
|
||||
* Restore the VL that was saved when bound to the CPU,
|
||||
* which is the maximum VL for the guest. Because the
|
||||
* layout of the data when saving the sve state depends
|
||||
* on the VL, we need to use a consistent (i.e., the
|
||||
* maximum) VL.
|
||||
* Note that this means that at guest exit ZCR_EL1 is
|
||||
* not necessarily the same as on guest entry.
|
||||
*
|
||||
* Restoring the VL isn't needed in VHE mode since
|
||||
* ZCR_EL2 (accessed via ZCR_EL1) would fulfill the same
|
||||
* role when doing the save from EL2.
|
||||
*/
|
||||
if (!has_vhe())
|
||||
sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1,
|
||||
SYS_ZCR_EL1);
|
||||
}
|
||||
|
||||
/*
|
||||
* Flush (save and invalidate) the fpsimd/sve state so that if
|
||||
* the host tries to use fpsimd/sve, it's not using stale data
|
||||
* from the guest.
|
||||
*
|
||||
* Flushing the state sets the TIF_FOREIGN_FPSTATE bit for the
|
||||
* context unconditionally, in both nVHE and VHE. This allows
|
||||
* the kernel to restore the fpsimd/sve state, including ZCR_EL1
|
||||
* when needed.
|
||||
*/
|
||||
fpsimd_save_and_flush_cpu_state();
|
||||
} else if (has_vhe() && system_supports_sve()) {
|
||||
/*
|
||||
|
|
|
@ -40,12 +40,6 @@ struct kvm_exception_table_entry {
|
|||
extern struct kvm_exception_table_entry __start___kvm_ex_table;
|
||||
extern struct kvm_exception_table_entry __stop___kvm_ex_table;
|
||||
|
||||
/* Check whether the FP regs are owned by the guest */
|
||||
static inline bool guest_owns_fp_regs(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
return *host_data_ptr(fp_owner) == FP_STATE_GUEST_OWNED;
|
||||
}
|
||||
|
||||
/* Save the 32-bit only FPSIMD system register state */
|
||||
static inline void __fpsimd_save_fpexc32(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
|
@ -375,7 +369,7 @@ static bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code)
|
|||
isb();
|
||||
|
||||
/* Write out the host state if it's in the registers */
|
||||
if (*host_data_ptr(fp_owner) == FP_STATE_HOST_OWNED)
|
||||
if (host_owns_fp_regs())
|
||||
__fpsimd_save_state(*host_data_ptr(fpsimd_state));
|
||||
|
||||
/* Restore the guest state */
|
||||
|
|
|
@ -53,7 +53,13 @@ pkvm_hyp_vcpu_to_hyp_vm(struct pkvm_hyp_vcpu *hyp_vcpu)
|
|||
return container_of(hyp_vcpu->vcpu.kvm, struct pkvm_hyp_vm, kvm);
|
||||
}
|
||||
|
||||
static inline bool pkvm_hyp_vcpu_is_protected(struct pkvm_hyp_vcpu *hyp_vcpu)
|
||||
{
|
||||
return vcpu_is_protected(&hyp_vcpu->vcpu);
|
||||
}
|
||||
|
||||
void pkvm_hyp_vm_table_init(void *tbl);
|
||||
void pkvm_host_fpsimd_state_init(void);
|
||||
|
||||
int __pkvm_init_vm(struct kvm *host_kvm, unsigned long vm_hva,
|
||||
unsigned long pgd_hva);
|
||||
|
|
|
@ -175,16 +175,6 @@ static void handle___vgic_v3_get_gic_config(struct kvm_cpu_context *host_ctxt)
|
|||
cpu_reg(host_ctxt, 1) = __vgic_v3_get_gic_config();
|
||||
}
|
||||
|
||||
static void handle___vgic_v3_read_vmcr(struct kvm_cpu_context *host_ctxt)
|
||||
{
|
||||
cpu_reg(host_ctxt, 1) = __vgic_v3_read_vmcr();
|
||||
}
|
||||
|
||||
static void handle___vgic_v3_write_vmcr(struct kvm_cpu_context *host_ctxt)
|
||||
{
|
||||
__vgic_v3_write_vmcr(cpu_reg(host_ctxt, 1));
|
||||
}
|
||||
|
||||
static void handle___vgic_v3_init_lrs(struct kvm_cpu_context *host_ctxt)
|
||||
{
|
||||
__vgic_v3_init_lrs();
|
||||
|
@ -195,18 +185,18 @@ static void handle___kvm_get_mdcr_el2(struct kvm_cpu_context *host_ctxt)
|
|||
cpu_reg(host_ctxt, 1) = __kvm_get_mdcr_el2();
|
||||
}
|
||||
|
||||
static void handle___vgic_v3_save_aprs(struct kvm_cpu_context *host_ctxt)
|
||||
static void handle___vgic_v3_save_vmcr_aprs(struct kvm_cpu_context *host_ctxt)
|
||||
{
|
||||
DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
|
||||
|
||||
__vgic_v3_save_aprs(kern_hyp_va(cpu_if));
|
||||
__vgic_v3_save_vmcr_aprs(kern_hyp_va(cpu_if));
|
||||
}
|
||||
|
||||
static void handle___vgic_v3_restore_aprs(struct kvm_cpu_context *host_ctxt)
|
||||
static void handle___vgic_v3_restore_vmcr_aprs(struct kvm_cpu_context *host_ctxt)
|
||||
{
|
||||
DECLARE_REG(struct vgic_v3_cpu_if *, cpu_if, host_ctxt, 1);
|
||||
|
||||
__vgic_v3_restore_aprs(kern_hyp_va(cpu_if));
|
||||
__vgic_v3_restore_vmcr_aprs(kern_hyp_va(cpu_if));
|
||||
}
|
||||
|
||||
static void handle___pkvm_init(struct kvm_cpu_context *host_ctxt)
|
||||
|
@ -337,10 +327,8 @@ static const hcall_t host_hcall[] = {
|
|||
HANDLE_FUNC(__kvm_tlb_flush_vmid_range),
|
||||
HANDLE_FUNC(__kvm_flush_cpu_context),
|
||||
HANDLE_FUNC(__kvm_timer_set_cntvoff),
|
||||
HANDLE_FUNC(__vgic_v3_read_vmcr),
|
||||
HANDLE_FUNC(__vgic_v3_write_vmcr),
|
||||
HANDLE_FUNC(__vgic_v3_save_aprs),
|
||||
HANDLE_FUNC(__vgic_v3_restore_aprs),
|
||||
HANDLE_FUNC(__vgic_v3_save_vmcr_aprs),
|
||||
HANDLE_FUNC(__vgic_v3_restore_vmcr_aprs),
|
||||
HANDLE_FUNC(__pkvm_vcpu_init_traps),
|
||||
HANDLE_FUNC(__pkvm_init_vm),
|
||||
HANDLE_FUNC(__pkvm_init_vcpu),
|
||||
|
|
|
@ -533,7 +533,13 @@ void handle_host_mem_abort(struct kvm_cpu_context *host_ctxt)
|
|||
int ret = 0;
|
||||
|
||||
esr = read_sysreg_el2(SYS_ESR);
|
||||
BUG_ON(!__get_fault_info(esr, &fault));
|
||||
if (!__get_fault_info(esr, &fault)) {
|
||||
/*
|
||||
* We've presumably raced with a page-table change which caused
|
||||
* AT to fail, try again.
|
||||
*/
|
||||
return;
|
||||
}
|
||||
|
||||
addr = (fault.hpfar_el2 & HPFAR_MASK) << 8;
|
||||
ret = host_stage2_idmap(addr);
|
||||
|
|
|
@ -200,7 +200,7 @@ static void pvm_init_trap_regs(struct kvm_vcpu *vcpu)
|
|||
}
|
||||
|
||||
/*
|
||||
* Initialize trap register values for protected VMs.
|
||||
* Initialize trap register values in protected mode.
|
||||
*/
|
||||
void __pkvm_vcpu_init_traps(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
|
@ -247,6 +247,17 @@ void pkvm_hyp_vm_table_init(void *tbl)
|
|||
vm_table = tbl;
|
||||
}
|
||||
|
||||
void pkvm_host_fpsimd_state_init(void)
|
||||
{
|
||||
unsigned long i;
|
||||
|
||||
for (i = 0; i < hyp_nr_cpus; i++) {
|
||||
struct kvm_host_data *host_data = per_cpu_ptr(&kvm_host_data, i);
|
||||
|
||||
host_data->fpsimd_state = &host_data->host_ctxt.fp_regs;
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
* Return the hyp vm structure corresponding to the handle.
|
||||
*/
|
||||
|
@ -430,6 +441,7 @@ static void *map_donated_memory(unsigned long host_va, size_t size)
|
|||
|
||||
static void __unmap_donated_memory(void *va, size_t size)
|
||||
{
|
||||
kvm_flush_dcache_to_poc(va, size);
|
||||
WARN_ON(__pkvm_hyp_donate_host(hyp_virt_to_pfn(va),
|
||||
PAGE_ALIGN(size) >> PAGE_SHIFT));
|
||||
}
|
||||
|
|
|
@ -300,6 +300,7 @@ void __noreturn __pkvm_init_finalise(void)
|
|||
goto out;
|
||||
|
||||
pkvm_hyp_vm_table_init(vm_table_base);
|
||||
pkvm_host_fpsimd_state_init();
|
||||
out:
|
||||
/*
|
||||
* We tail-called to here from handle___pkvm_init() and will not return,
|
||||
|
|
|
@ -53,7 +53,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
|
|||
val |= CPTR_EL2_TSM;
|
||||
}
|
||||
|
||||
if (!guest_owns_fp_regs(vcpu)) {
|
||||
if (!guest_owns_fp_regs()) {
|
||||
if (has_hvhe())
|
||||
val &= ~(CPACR_EL1_FPEN_EL0EN | CPACR_EL1_FPEN_EL1EN |
|
||||
CPACR_EL1_ZEN_EL0EN | CPACR_EL1_ZEN_EL1EN);
|
||||
|
@ -207,7 +207,7 @@ static const exit_handler_fn pvm_exit_handlers[] = {
|
|||
|
||||
static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
if (unlikely(kvm_vm_is_protected(kern_hyp_va(vcpu->kvm))))
|
||||
if (unlikely(vcpu_is_protected(vcpu)))
|
||||
return pvm_exit_handlers;
|
||||
|
||||
return hyp_exit_handlers;
|
||||
|
@ -226,9 +226,7 @@ static const exit_handler_fn *kvm_get_exit_handler_array(struct kvm_vcpu *vcpu)
|
|||
*/
|
||||
static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code)
|
||||
{
|
||||
struct kvm *kvm = kern_hyp_va(vcpu->kvm);
|
||||
|
||||
if (kvm_vm_is_protected(kvm) && vcpu_mode_is_32bit(vcpu)) {
|
||||
if (unlikely(vcpu_is_protected(vcpu) && vcpu_mode_is_32bit(vcpu))) {
|
||||
/*
|
||||
* As we have caught the guest red-handed, decide that it isn't
|
||||
* fit for purpose anymore by making the vcpu invalid. The VMM
|
||||
|
@ -335,7 +333,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
|
|||
|
||||
__sysreg_restore_state_nvhe(host_ctxt);
|
||||
|
||||
if (*host_data_ptr(fp_owner) == FP_STATE_GUEST_OWNED)
|
||||
if (guest_owns_fp_regs())
|
||||
__fpsimd_save_fpexc32(vcpu);
|
||||
|
||||
__debug_switch_to_host(vcpu);
|
||||
|
|
|
@ -11,13 +11,23 @@
|
|||
#include <nvhe/mem_protect.h>
|
||||
|
||||
struct tlb_inv_context {
|
||||
u64 tcr;
|
||||
struct kvm_s2_mmu *mmu;
|
||||
u64 tcr;
|
||||
u64 sctlr;
|
||||
};
|
||||
|
||||
static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu,
|
||||
struct tlb_inv_context *cxt,
|
||||
bool nsh)
|
||||
static void enter_vmid_context(struct kvm_s2_mmu *mmu,
|
||||
struct tlb_inv_context *cxt,
|
||||
bool nsh)
|
||||
{
|
||||
struct kvm_s2_mmu *host_s2_mmu = &host_mmu.arch.mmu;
|
||||
struct kvm_cpu_context *host_ctxt;
|
||||
struct kvm_vcpu *vcpu;
|
||||
|
||||
host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
|
||||
vcpu = host_ctxt->__hyp_running_vcpu;
|
||||
cxt->mmu = NULL;
|
||||
|
||||
/*
|
||||
* We have two requirements:
|
||||
*
|
||||
|
@ -40,20 +50,55 @@ static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu,
|
|||
else
|
||||
dsb(ish);
|
||||
|
||||
/*
|
||||
* If we're already in the desired context, then there's nothing to do.
|
||||
*/
|
||||
if (vcpu) {
|
||||
/*
|
||||
* We're in guest context. However, for this to work, this needs
|
||||
* to be called from within __kvm_vcpu_run(), which ensures that
|
||||
* __hyp_running_vcpu is set to the current guest vcpu.
|
||||
*/
|
||||
if (mmu == vcpu->arch.hw_mmu || WARN_ON(mmu != host_s2_mmu))
|
||||
return;
|
||||
|
||||
cxt->mmu = vcpu->arch.hw_mmu;
|
||||
} else {
|
||||
/* We're in host context. */
|
||||
if (mmu == host_s2_mmu)
|
||||
return;
|
||||
|
||||
cxt->mmu = host_s2_mmu;
|
||||
}
|
||||
|
||||
if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
|
||||
u64 val;
|
||||
|
||||
/*
|
||||
* For CPUs that are affected by ARM 1319367, we need to
|
||||
* avoid a host Stage-1 walk while we have the guest's
|
||||
* VMID set in the VTTBR in order to invalidate TLBs.
|
||||
* We're guaranteed that the S1 MMU is enabled, so we can
|
||||
* simply set the EPD bits to avoid any further TLB fill.
|
||||
* avoid a Stage-1 walk with the old VMID while we have
|
||||
* the new VMID set in the VTTBR in order to invalidate TLBs.
|
||||
* We're guaranteed that the host S1 MMU is enabled, so
|
||||
* we can simply set the EPD bits to avoid any further
|
||||
* TLB fill. For guests, we ensure that the S1 MMU is
|
||||
* temporarily enabled in the next context.
|
||||
*/
|
||||
val = cxt->tcr = read_sysreg_el1(SYS_TCR);
|
||||
val |= TCR_EPD1_MASK | TCR_EPD0_MASK;
|
||||
write_sysreg_el1(val, SYS_TCR);
|
||||
isb();
|
||||
|
||||
if (vcpu) {
|
||||
val = cxt->sctlr = read_sysreg_el1(SYS_SCTLR);
|
||||
if (!(val & SCTLR_ELx_M)) {
|
||||
val |= SCTLR_ELx_M;
|
||||
write_sysreg_el1(val, SYS_SCTLR);
|
||||
isb();
|
||||
}
|
||||
} else {
|
||||
/* The host S1 MMU is always enabled. */
|
||||
cxt->sctlr = SCTLR_ELx_M;
|
||||
}
|
||||
}
|
||||
|
||||
/*
|
||||
|
@ -62,18 +107,40 @@ static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu,
|
|||
* ensuring that we always have an ISB, but not two ISBs back
|
||||
* to back.
|
||||
*/
|
||||
__load_stage2(mmu, kern_hyp_va(mmu->arch));
|
||||
if (vcpu)
|
||||
__load_host_stage2();
|
||||
else
|
||||
__load_stage2(mmu, kern_hyp_va(mmu->arch));
|
||||
|
||||
asm(ALTERNATIVE("isb", "nop", ARM64_WORKAROUND_SPECULATIVE_AT));
|
||||
}
|
||||
|
||||
static void __tlb_switch_to_host(struct tlb_inv_context *cxt)
|
||||
static void exit_vmid_context(struct tlb_inv_context *cxt)
|
||||
{
|
||||
__load_host_stage2();
|
||||
struct kvm_s2_mmu *mmu = cxt->mmu;
|
||||
struct kvm_cpu_context *host_ctxt;
|
||||
struct kvm_vcpu *vcpu;
|
||||
|
||||
host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
|
||||
vcpu = host_ctxt->__hyp_running_vcpu;
|
||||
|
||||
if (!mmu)
|
||||
return;
|
||||
|
||||
if (vcpu)
|
||||
__load_stage2(mmu, kern_hyp_va(mmu->arch));
|
||||
else
|
||||
__load_host_stage2();
|
||||
|
||||
if (cpus_have_final_cap(ARM64_WORKAROUND_SPECULATIVE_AT)) {
|
||||
/* Ensure write of the host VMID */
|
||||
/* Ensure write of the old VMID */
|
||||
isb();
|
||||
/* Restore the host's TCR_EL1 */
|
||||
|
||||
if (!(cxt->sctlr & SCTLR_ELx_M)) {
|
||||
write_sysreg_el1(cxt->sctlr, SYS_SCTLR);
|
||||
isb();
|
||||
}
|
||||
|
||||
write_sysreg_el1(cxt->tcr, SYS_TCR);
|
||||
}
|
||||
}
|
||||
|
@ -84,7 +151,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
|
|||
struct tlb_inv_context cxt;
|
||||
|
||||
/* Switch to requested VMID */
|
||||
__tlb_switch_to_guest(mmu, &cxt, false);
|
||||
enter_vmid_context(mmu, &cxt, false);
|
||||
|
||||
/*
|
||||
* We could do so much better if we had the VA as well.
|
||||
|
@ -105,7 +172,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
|
|||
dsb(ish);
|
||||
isb();
|
||||
|
||||
__tlb_switch_to_host(&cxt);
|
||||
exit_vmid_context(&cxt);
|
||||
}
|
||||
|
||||
void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
|
||||
|
@ -114,7 +181,7 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
|
|||
struct tlb_inv_context cxt;
|
||||
|
||||
/* Switch to requested VMID */
|
||||
__tlb_switch_to_guest(mmu, &cxt, true);
|
||||
enter_vmid_context(mmu, &cxt, true);
|
||||
|
||||
/*
|
||||
* We could do so much better if we had the VA as well.
|
||||
|
@ -135,7 +202,7 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
|
|||
dsb(nsh);
|
||||
isb();
|
||||
|
||||
__tlb_switch_to_host(&cxt);
|
||||
exit_vmid_context(&cxt);
|
||||
}
|
||||
|
||||
void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
|
||||
|
@ -152,7 +219,7 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
|
|||
start = round_down(start, stride);
|
||||
|
||||
/* Switch to requested VMID */
|
||||
__tlb_switch_to_guest(mmu, &cxt, false);
|
||||
enter_vmid_context(mmu, &cxt, false);
|
||||
|
||||
__flush_s2_tlb_range_op(ipas2e1is, start, pages, stride,
|
||||
TLBI_TTL_UNKNOWN);
|
||||
|
@ -162,7 +229,7 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
|
|||
dsb(ish);
|
||||
isb();
|
||||
|
||||
__tlb_switch_to_host(&cxt);
|
||||
exit_vmid_context(&cxt);
|
||||
}
|
||||
|
||||
void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
|
||||
|
@ -170,13 +237,13 @@ void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
|
|||
struct tlb_inv_context cxt;
|
||||
|
||||
/* Switch to requested VMID */
|
||||
__tlb_switch_to_guest(mmu, &cxt, false);
|
||||
enter_vmid_context(mmu, &cxt, false);
|
||||
|
||||
__tlbi(vmalls12e1is);
|
||||
dsb(ish);
|
||||
isb();
|
||||
|
||||
__tlb_switch_to_host(&cxt);
|
||||
exit_vmid_context(&cxt);
|
||||
}
|
||||
|
||||
void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu)
|
||||
|
@ -184,19 +251,19 @@ void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu)
|
|||
struct tlb_inv_context cxt;
|
||||
|
||||
/* Switch to requested VMID */
|
||||
__tlb_switch_to_guest(mmu, &cxt, false);
|
||||
enter_vmid_context(mmu, &cxt, false);
|
||||
|
||||
__tlbi(vmalle1);
|
||||
asm volatile("ic iallu");
|
||||
dsb(nsh);
|
||||
isb();
|
||||
|
||||
__tlb_switch_to_host(&cxt);
|
||||
exit_vmid_context(&cxt);
|
||||
}
|
||||
|
||||
void __kvm_flush_vm_context(void)
|
||||
{
|
||||
/* Same remark as in __tlb_switch_to_guest() */
|
||||
/* Same remark as in enter_vmid_context() */
|
||||
dsb(ish);
|
||||
__tlbi(alle1is);
|
||||
dsb(ish);
|
||||
|
|
|
@ -914,12 +914,12 @@ static void stage2_unmap_put_pte(const struct kvm_pgtable_visit_ctx *ctx,
|
|||
static bool stage2_pte_cacheable(struct kvm_pgtable *pgt, kvm_pte_t pte)
|
||||
{
|
||||
u64 memattr = pte & KVM_PTE_LEAF_ATTR_LO_S2_MEMATTR;
|
||||
return memattr == KVM_S2_MEMATTR(pgt, NORMAL);
|
||||
return kvm_pte_valid(pte) && memattr == KVM_S2_MEMATTR(pgt, NORMAL);
|
||||
}
|
||||
|
||||
static bool stage2_pte_executable(kvm_pte_t pte)
|
||||
{
|
||||
return !(pte & KVM_PTE_LEAF_ATTR_HI_S2_XN);
|
||||
return kvm_pte_valid(pte) && !(pte & KVM_PTE_LEAF_ATTR_HI_S2_XN);
|
||||
}
|
||||
|
||||
static u64 stage2_map_walker_phys_addr(const struct kvm_pgtable_visit_ctx *ctx,
|
||||
|
@ -979,6 +979,21 @@ static int stage2_map_walker_try_leaf(const struct kvm_pgtable_visit_ctx *ctx,
|
|||
if (!stage2_pte_needs_update(ctx->old, new))
|
||||
return -EAGAIN;
|
||||
|
||||
/* If we're only changing software bits, then store them and go! */
|
||||
if (!kvm_pgtable_walk_shared(ctx) &&
|
||||
!((ctx->old ^ new) & ~KVM_PTE_LEAF_ATTR_HI_SW)) {
|
||||
bool old_is_counted = stage2_pte_is_counted(ctx->old);
|
||||
|
||||
if (old_is_counted != stage2_pte_is_counted(new)) {
|
||||
if (old_is_counted)
|
||||
mm_ops->put_page(ctx->ptep);
|
||||
else
|
||||
mm_ops->get_page(ctx->ptep);
|
||||
}
|
||||
WARN_ON_ONCE(!stage2_try_set_pte(ctx, new));
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (!stage2_try_break_pte(ctx, data->mmu))
|
||||
return -EAGAIN;
|
||||
|
||||
|
@ -1370,7 +1385,7 @@ static int stage2_flush_walker(const struct kvm_pgtable_visit_ctx *ctx,
|
|||
struct kvm_pgtable *pgt = ctx->arg;
|
||||
struct kvm_pgtable_mm_ops *mm_ops = pgt->mm_ops;
|
||||
|
||||
if (!kvm_pte_valid(ctx->old) || !stage2_pte_cacheable(pgt, ctx->old))
|
||||
if (!stage2_pte_cacheable(pgt, ctx->old))
|
||||
return 0;
|
||||
|
||||
if (mm_ops->dcache_clean_inval_poc)
|
||||
|
|
|
@ -330,7 +330,7 @@ void __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if)
|
|||
write_gicreg(0, ICH_HCR_EL2);
|
||||
}
|
||||
|
||||
void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if)
|
||||
static void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if)
|
||||
{
|
||||
u64 val;
|
||||
u32 nr_pre_bits;
|
||||
|
@ -363,7 +363,7 @@ void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if)
|
|||
}
|
||||
}
|
||||
|
||||
void __vgic_v3_restore_aprs(struct vgic_v3_cpu_if *cpu_if)
|
||||
static void __vgic_v3_restore_aprs(struct vgic_v3_cpu_if *cpu_if)
|
||||
{
|
||||
u64 val;
|
||||
u32 nr_pre_bits;
|
||||
|
@ -455,16 +455,35 @@ u64 __vgic_v3_get_gic_config(void)
|
|||
return val;
|
||||
}
|
||||
|
||||
u64 __vgic_v3_read_vmcr(void)
|
||||
static u64 __vgic_v3_read_vmcr(void)
|
||||
{
|
||||
return read_gicreg(ICH_VMCR_EL2);
|
||||
}
|
||||
|
||||
void __vgic_v3_write_vmcr(u32 vmcr)
|
||||
static void __vgic_v3_write_vmcr(u32 vmcr)
|
||||
{
|
||||
write_gicreg(vmcr, ICH_VMCR_EL2);
|
||||
}
|
||||
|
||||
void __vgic_v3_save_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if)
|
||||
{
|
||||
__vgic_v3_save_aprs(cpu_if);
|
||||
if (cpu_if->vgic_sre)
|
||||
cpu_if->vgic_vmcr = __vgic_v3_read_vmcr();
|
||||
}
|
||||
|
||||
void __vgic_v3_restore_vmcr_aprs(struct vgic_v3_cpu_if *cpu_if)
|
||||
{
|
||||
/*
|
||||
* If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen
|
||||
* is dependent on ICC_SRE_EL1.SRE, and we have to perform the
|
||||
* VMCR_EL2 save/restore in the world switch.
|
||||
*/
|
||||
if (cpu_if->vgic_sre)
|
||||
__vgic_v3_write_vmcr(cpu_if->vgic_vmcr);
|
||||
__vgic_v3_restore_aprs(cpu_if);
|
||||
}
|
||||
|
||||
static int __vgic_v3_bpr_min(void)
|
||||
{
|
||||
/* See Pseudocode for VPriorityGroup */
|
||||
|
|
|
@ -107,7 +107,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu)
|
|||
|
||||
val |= CPTR_EL2_TAM;
|
||||
|
||||
if (guest_owns_fp_regs(vcpu)) {
|
||||
if (guest_owns_fp_regs()) {
|
||||
if (vcpu_has_sve(vcpu))
|
||||
val |= CPACR_EL1_ZEN_EL0EN | CPACR_EL1_ZEN_EL1EN;
|
||||
} else {
|
||||
|
@ -341,7 +341,7 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
|
|||
|
||||
sysreg_restore_host_state_vhe(host_ctxt);
|
||||
|
||||
if (*host_data_ptr(fp_owner) == FP_STATE_GUEST_OWNED)
|
||||
if (guest_owns_fp_regs())
|
||||
__fpsimd_save_fpexc32(vcpu);
|
||||
|
||||
__debug_switch_to_host(vcpu);
|
||||
|
|
|
@ -17,8 +17,8 @@ struct tlb_inv_context {
|
|||
u64 sctlr;
|
||||
};
|
||||
|
||||
static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu,
|
||||
struct tlb_inv_context *cxt)
|
||||
static void enter_vmid_context(struct kvm_s2_mmu *mmu,
|
||||
struct tlb_inv_context *cxt)
|
||||
{
|
||||
struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
|
||||
u64 val;
|
||||
|
@ -67,7 +67,7 @@ static void __tlb_switch_to_guest(struct kvm_s2_mmu *mmu,
|
|||
isb();
|
||||
}
|
||||
|
||||
static void __tlb_switch_to_host(struct tlb_inv_context *cxt)
|
||||
static void exit_vmid_context(struct tlb_inv_context *cxt)
|
||||
{
|
||||
/*
|
||||
* We're done with the TLB operation, let's restore the host's
|
||||
|
@ -97,7 +97,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
|
|||
dsb(ishst);
|
||||
|
||||
/* Switch to requested VMID */
|
||||
__tlb_switch_to_guest(mmu, &cxt);
|
||||
enter_vmid_context(mmu, &cxt);
|
||||
|
||||
/*
|
||||
* We could do so much better if we had the VA as well.
|
||||
|
@ -118,7 +118,7 @@ void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu,
|
|||
dsb(ish);
|
||||
isb();
|
||||
|
||||
__tlb_switch_to_host(&cxt);
|
||||
exit_vmid_context(&cxt);
|
||||
}
|
||||
|
||||
void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
|
||||
|
@ -129,7 +129,7 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
|
|||
dsb(nshst);
|
||||
|
||||
/* Switch to requested VMID */
|
||||
__tlb_switch_to_guest(mmu, &cxt);
|
||||
enter_vmid_context(mmu, &cxt);
|
||||
|
||||
/*
|
||||
* We could do so much better if we had the VA as well.
|
||||
|
@ -150,7 +150,7 @@ void __kvm_tlb_flush_vmid_ipa_nsh(struct kvm_s2_mmu *mmu,
|
|||
dsb(nsh);
|
||||
isb();
|
||||
|
||||
__tlb_switch_to_host(&cxt);
|
||||
exit_vmid_context(&cxt);
|
||||
}
|
||||
|
||||
void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
|
||||
|
@ -169,7 +169,7 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
|
|||
dsb(ishst);
|
||||
|
||||
/* Switch to requested VMID */
|
||||
__tlb_switch_to_guest(mmu, &cxt);
|
||||
enter_vmid_context(mmu, &cxt);
|
||||
|
||||
__flush_s2_tlb_range_op(ipas2e1is, start, pages, stride,
|
||||
TLBI_TTL_UNKNOWN);
|
||||
|
@ -179,7 +179,7 @@ void __kvm_tlb_flush_vmid_range(struct kvm_s2_mmu *mmu,
|
|||
dsb(ish);
|
||||
isb();
|
||||
|
||||
__tlb_switch_to_host(&cxt);
|
||||
exit_vmid_context(&cxt);
|
||||
}
|
||||
|
||||
void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
|
||||
|
@ -189,13 +189,13 @@ void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu)
|
|||
dsb(ishst);
|
||||
|
||||
/* Switch to requested VMID */
|
||||
__tlb_switch_to_guest(mmu, &cxt);
|
||||
enter_vmid_context(mmu, &cxt);
|
||||
|
||||
__tlbi(vmalls12e1is);
|
||||
dsb(ish);
|
||||
isb();
|
||||
|
||||
__tlb_switch_to_host(&cxt);
|
||||
exit_vmid_context(&cxt);
|
||||
}
|
||||
|
||||
void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu)
|
||||
|
@ -203,14 +203,14 @@ void __kvm_flush_cpu_context(struct kvm_s2_mmu *mmu)
|
|||
struct tlb_inv_context cxt;
|
||||
|
||||
/* Switch to requested VMID */
|
||||
__tlb_switch_to_guest(mmu, &cxt);
|
||||
enter_vmid_context(mmu, &cxt);
|
||||
|
||||
__tlbi(vmalle1);
|
||||
asm volatile("ic iallu");
|
||||
dsb(nsh);
|
||||
isb();
|
||||
|
||||
__tlb_switch_to_host(&cxt);
|
||||
exit_vmid_context(&cxt);
|
||||
}
|
||||
|
||||
void __kvm_flush_vm_context(void)
|
||||
|
|
|
@ -86,7 +86,7 @@ int kvm_handle_mmio_return(struct kvm_vcpu *vcpu)
|
|||
|
||||
/* Detect an already handled MMIO return */
|
||||
if (unlikely(!vcpu->mmio_needed))
|
||||
return 0;
|
||||
return 1;
|
||||
|
||||
vcpu->mmio_needed = 0;
|
||||
|
||||
|
@ -117,7 +117,7 @@ int kvm_handle_mmio_return(struct kvm_vcpu *vcpu)
|
|||
*/
|
||||
kvm_incr_pc(vcpu);
|
||||
|
||||
return 0;
|
||||
return 1;
|
||||
}
|
||||
|
||||
int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
|
||||
|
@ -133,11 +133,19 @@ int io_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa)
|
|||
/*
|
||||
* No valid syndrome? Ask userspace for help if it has
|
||||
* volunteered to do so, and bail out otherwise.
|
||||
*
|
||||
* In the protected VM case, there isn't much userspace can do
|
||||
* though, so directly deliver an exception to the guest.
|
||||
*/
|
||||
if (!kvm_vcpu_dabt_isvalid(vcpu)) {
|
||||
trace_kvm_mmio_nisv(*vcpu_pc(vcpu), kvm_vcpu_get_esr(vcpu),
|
||||
kvm_vcpu_get_hfar(vcpu), fault_ipa);
|
||||
|
||||
if (vcpu_is_protected(vcpu)) {
|
||||
kvm_inject_dabt(vcpu, kvm_vcpu_get_hfar(vcpu));
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (test_bit(KVM_ARCH_FLAG_RETURN_NISV_IO_ABORT_TO_USER,
|
||||
&vcpu->kvm->arch.flags)) {
|
||||
run->exit_reason = KVM_EXIT_ARM_NISV;
|
||||
|
|
|
@ -1522,8 +1522,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
|
|||
|
||||
read_lock(&kvm->mmu_lock);
|
||||
pgt = vcpu->arch.hw_mmu->pgt;
|
||||
if (mmu_invalidate_retry(kvm, mmu_seq))
|
||||
if (mmu_invalidate_retry(kvm, mmu_seq)) {
|
||||
ret = -EAGAIN;
|
||||
goto out_unlock;
|
||||
}
|
||||
|
||||
/*
|
||||
* If we are not forced to use page mapping, check if we are
|
||||
|
@ -1581,6 +1583,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
|
|||
memcache,
|
||||
KVM_PGTABLE_WALK_HANDLE_FAULT |
|
||||
KVM_PGTABLE_WALK_SHARED);
|
||||
out_unlock:
|
||||
read_unlock(&kvm->mmu_lock);
|
||||
|
||||
/* Mark the page dirty only if the fault is handled successfully */
|
||||
if (writable && !ret) {
|
||||
|
@ -1588,8 +1592,6 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
|
|||
mark_page_dirty_in_slot(kvm, memslot, gfn);
|
||||
}
|
||||
|
||||
out_unlock:
|
||||
read_unlock(&kvm->mmu_lock);
|
||||
kvm_release_pfn_clean(pfn);
|
||||
return ret != -EAGAIN ? ret : 0;
|
||||
}
|
||||
|
|
|
@ -222,7 +222,6 @@ void pkvm_destroy_hyp_vm(struct kvm *host_kvm)
|
|||
|
||||
int pkvm_init_host_vm(struct kvm *host_kvm)
|
||||
{
|
||||
mutex_init(&host_kvm->lock);
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
@ -259,6 +258,7 @@ static int __init finalize_pkvm(void)
|
|||
* at, which would end badly once inaccessible.
|
||||
*/
|
||||
kmemleak_free_part(__hyp_bss_start, __hyp_bss_end - __hyp_bss_start);
|
||||
kmemleak_free_part(__hyp_rodata_start, __hyp_rodata_end - __hyp_rodata_start);
|
||||
kmemleak_free_part_phys(hyp_mem_base, hyp_mem_size);
|
||||
|
||||
ret = pkvm_drop_host_privileges();
|
||||
|
|
|
@ -151,7 +151,6 @@ void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu)
|
|||
{
|
||||
void *sve_state = vcpu->arch.sve_state;
|
||||
|
||||
kvm_vcpu_unshare_task_fp(vcpu);
|
||||
kvm_unshare_hyp(vcpu, vcpu + 1);
|
||||
if (sve_state)
|
||||
kvm_unshare_hyp(sve_state, sve_state + vcpu_sve_state_size(vcpu));
|
||||
|
|
|
@ -464,17 +464,10 @@ void vgic_v2_load(struct kvm_vcpu *vcpu)
|
|||
kvm_vgic_global_state.vctrl_base + GICH_APR);
|
||||
}
|
||||
|
||||
void vgic_v2_vmcr_sync(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
|
||||
|
||||
cpu_if->vgic_vmcr = readl_relaxed(kvm_vgic_global_state.vctrl_base + GICH_VMCR);
|
||||
}
|
||||
|
||||
void vgic_v2_put(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
|
||||
|
||||
vgic_v2_vmcr_sync(vcpu);
|
||||
cpu_if->vgic_vmcr = readl_relaxed(kvm_vgic_global_state.vctrl_base + GICH_VMCR);
|
||||
cpu_if->vgic_apr = readl_relaxed(kvm_vgic_global_state.vctrl_base + GICH_APR);
|
||||
}
|
||||
|
|
|
@ -722,15 +722,7 @@ void vgic_v3_load(struct kvm_vcpu *vcpu)
|
|||
{
|
||||
struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
|
||||
|
||||
/*
|
||||
* If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen
|
||||
* is dependent on ICC_SRE_EL1.SRE, and we have to perform the
|
||||
* VMCR_EL2 save/restore in the world switch.
|
||||
*/
|
||||
if (likely(cpu_if->vgic_sre))
|
||||
kvm_call_hyp(__vgic_v3_write_vmcr, cpu_if->vgic_vmcr);
|
||||
|
||||
kvm_call_hyp(__vgic_v3_restore_aprs, cpu_if);
|
||||
kvm_call_hyp(__vgic_v3_restore_vmcr_aprs, cpu_if);
|
||||
|
||||
if (has_vhe())
|
||||
__vgic_v3_activate_traps(cpu_if);
|
||||
|
@ -738,24 +730,13 @@ void vgic_v3_load(struct kvm_vcpu *vcpu)
|
|||
WARN_ON(vgic_v4_load(vcpu));
|
||||
}
|
||||
|
||||
void vgic_v3_vmcr_sync(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
|
||||
|
||||
if (likely(cpu_if->vgic_sre))
|
||||
cpu_if->vgic_vmcr = kvm_call_hyp_ret(__vgic_v3_read_vmcr);
|
||||
}
|
||||
|
||||
void vgic_v3_put(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3;
|
||||
|
||||
kvm_call_hyp(__vgic_v3_save_vmcr_aprs, cpu_if);
|
||||
WARN_ON(vgic_v4_put(vcpu));
|
||||
|
||||
vgic_v3_vmcr_sync(vcpu);
|
||||
|
||||
kvm_call_hyp(__vgic_v3_save_aprs, cpu_if);
|
||||
|
||||
if (has_vhe())
|
||||
__vgic_v3_deactivate_traps(cpu_if);
|
||||
}
|
||||
|
|
|
@ -937,17 +937,6 @@ void kvm_vgic_put(struct kvm_vcpu *vcpu)
|
|||
vgic_v3_put(vcpu);
|
||||
}
|
||||
|
||||
void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
if (unlikely(!irqchip_in_kernel(vcpu->kvm)))
|
||||
return;
|
||||
|
||||
if (kvm_vgic_global_state.type == VGIC_V2)
|
||||
vgic_v2_vmcr_sync(vcpu);
|
||||
else
|
||||
vgic_v3_vmcr_sync(vcpu);
|
||||
}
|
||||
|
||||
int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu)
|
||||
{
|
||||
struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
|
||||
|
|
|
@ -215,7 +215,6 @@ int vgic_register_dist_iodev(struct kvm *kvm, gpa_t dist_base_address,
|
|||
void vgic_v2_init_lrs(void);
|
||||
void vgic_v2_load(struct kvm_vcpu *vcpu);
|
||||
void vgic_v2_put(struct kvm_vcpu *vcpu);
|
||||
void vgic_v2_vmcr_sync(struct kvm_vcpu *vcpu);
|
||||
|
||||
void vgic_v2_save_state(struct kvm_vcpu *vcpu);
|
||||
void vgic_v2_restore_state(struct kvm_vcpu *vcpu);
|
||||
|
@ -254,7 +253,6 @@ bool vgic_v3_check_base(struct kvm *kvm);
|
|||
|
||||
void vgic_v3_load(struct kvm_vcpu *vcpu);
|
||||
void vgic_v3_put(struct kvm_vcpu *vcpu);
|
||||
void vgic_v3_vmcr_sync(struct kvm_vcpu *vcpu);
|
||||
|
||||
bool vgic_has_its(struct kvm *kvm);
|
||||
int kvm_vgic_register_its_device(void);
|
||||
|
|
|
@ -389,7 +389,6 @@ int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu);
|
|||
|
||||
void kvm_vgic_load(struct kvm_vcpu *vcpu);
|
||||
void kvm_vgic_put(struct kvm_vcpu *vcpu);
|
||||
void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu);
|
||||
|
||||
#define irqchip_in_kernel(k) (!!((k)->arch.vgic.in_kernel))
|
||||
#define vgic_initialized(k) ((k)->arch.vgic.initialized)
|
||||
|
|
Loading…
Reference in a new issue