- Merged v6.11-rc3 into trace/ring-buffer/core
The v6.10 ring buffer pull request was not made due to Mathieu Desnoyers
making a comment to the pull request. Mathieu and I resolved it on IRC,
but we did not let Linus know that it was resolved. Linus did not do the
pull thinking it still had some unresolved issues.
The ring buffer work for 6.12 was dependent on both this pull request as
well as the reserve_mem kernel command line option that was going upstream
through the memory management tree. The ring buffer repo was being used by
others so it could not be rebased. In order to continue the work, the
v6.11-rc3 branch was pulled in to get access to the reserve_mem work.
This has the 6.11 pull request that did not make it into 6.11, which was:
tracing/ring-buffer: Have persistent buffer across reboots
This allows for the tracing instance ring buffer to stay persistent across
reboots. The way this is done is by adding to the kernel command line:
trace_instance=boot_map@0x285400000:12M
This will reserve 12 megabytes at the address 0x285400000, and then map
the tracing instance "boot_map" ring buffer to that memory. This will
appear as a normal instance in the tracefs system:
/sys/kernel/tracing/instances/boot_map
A user could enable tracing in that instance, and on reboot or kernel
crash, if the memory is not wiped by the firmware, it will recreate the
trace in that instance. For example, if one was debugging a shutdown of a
kernel reboot:
# cd /sys/kernel/tracing
# echo function > instances/boot_map/current_tracer
# reboot
[..]
# cd /sys/kernel/tracing
# tail instances/boot_map/trace
swapper/0-1 [000] d..1. 164.549800: restore_boot_irq_mode <-native_machine_shutdown
swapper/0-1 [000] d..1. 164.549801: native_restore_boot_irq_mode <-native_machine_shutdown
swapper/0-1 [000] d..1. 164.549802: disconnect_bsp_APIC <-native_machine_shutdown
swapper/0-1 [000] d..1. 164.549811: hpet_disable <-native_machine_shutdown
swapper/0-1 [000] d..1. 164.549812: iommu_shutdown_noop <-native_machine_restart
swapper/0-1 [000] d..1. 164.549813: native_machine_emergency_restart <-__do_sys_reboot
swapper/0-1 [000] d..1. 164.549813: tboot_shutdown <-native_machine_emergency_restart
swapper/0-1 [000] d..1. 164.549820: acpi_reboot <-native_machine_emergency_restart
swapper/0-1 [000] d..1. 164.549821: acpi_reset <-acpi_reboot
swapper/0-1 [000] d..1. 164.549822: acpi_os_write_port <-acpi_reboot
On reboot, the buffer is examined to make sure it is valid. The validation
check even steps through every event to make sure the meta data of the
event is correct. If any test fails, it will simply reset the buffer, and
the buffer will be empty on boot.
The new changes for 6.12 are:
- Allow the tracing persistent boot buffer to use the "reserve_mem" option
Instead of having the admin find a physical address to store the persistent
buffer, which can be very tedious if they have to administrate several
different machines, allow them to use the "reserve_mem" option that will
find a location for them. It is not as reliable because of KASLR, as the
loading of the kernel in different locations can cause the memory
allocated to be inconsistent. Booting with "nokaslr" can make reserve_mem
more reliable.
- Have function graph tracer handle offsets from a previous boot.
The ring buffer output from a previous boot may have different addresses
due to kaslr. Have the function graph tracer handle these by using the
delta from the previous boot to the new boot address space.
- Only reset the saved meta offset when the buffer is started or reset
In the persistent memory meta data, it holds the previous address space
information, so that it can calculate the delta to have function tracing
work. But this gets updated after being read to hold the new address
space. But if the buffer isn't used for that boot, on reboot, the delta is
now calculated from the previous boot and not the boot that holds the data
in the ring buffer. This causes the functions not to be shown. Do not save
the address space information of the current kernel until it is being
recorded.
- Add a magic variable to test the valid meta data
Add a magic variable in the meta data that can also be used for
validation. The validator of the previous buffer doesn't need this magic
data, but it can be used if the meta data is changed by a new kernel, which
may have the same format that passes the validator but is used
differently. This magic number can also be used as a "versioning" of the
meta data.
- Align user space mapped ring buffer sub buffers to improve TLB entries
Linus mentioned that the mapped ring buffer sub buffers were misaligned
between the meta page and the sub-buffers, so that if the sub-buffers were
bigger than PAGE_SIZE, it wouldn't allow the TLB to use bigger entries.
- Add new kernel command line "traceoff" to disable tracing on boot for instances
If tracing is enabled for a boot instance, there needs a way to be able to
disable it on boot so that new events do not get entered into the ring
buffer and be mixed with events from a previous boot, as that can be
confusing.
- Allow trace_printk() to go to other instances
Currently, trace_printk() can only go to the top level instance. When
debugging with a persistent buffer, it is really useful to be able to add
trace_printk() to go to that buffer, so that you have access to them after
a crash.
- Do not use "bin_printk()" for traces to a boot instance
The bin_printk() saves only a pointer to the printk format in the ring
buffer, as the reader of the buffer can still have access to it. But this
is not the case if the buffer is from a previous boot. If the
trace_printk() is going to a "persistent" buffer, it will use the slower
version that writes the printk format into the buffer.
- Add command line option to allow trace_printk() to go to an instance
Allow the kernel command line to define which instance the trace_printk()
goes to, instead of forcing the admin to set it for every boot via the
tracefs options.
- Start a document that explains how to use tracefs to debug the kernel
- Add some more kernel selftests to test user mapped ring buffer
-----BEGIN PGP SIGNATURE-----
iIoEABYIADIWIQRRSw7ePDh/lE+zeZMp5XQQmuv6qgUCZu/PxxQccm9zdGVkdEBn
b29kbWlzLm9yZwAKCRAp5XQQmuv6qowiAQCx86Nm48aCACjrvGWCFb+jgQZn8QdO
MeK15Fcc5C3b5gEAkJkDKqtul7ybI9+vq+3yNzdl7pO7Y7+pCNzz3PfVaQA=
=Ce81
-----END PGP SIGNATURE-----
Merge tag 'trace-ring-buffer-v6.12' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace
Pull ring-buffer updates from Steven Rostedt:
- tracing/ring-buffer: persistent buffer across reboots
This allows for the tracing instance ring buffer to stay persistent
across reboots. The way this is done is by adding to the kernel
command line:
trace_instance=boot_map@0x285400000:12M
This will reserve 12 megabytes at the address 0x285400000, and then
map the tracing instance "boot_map" ring buffer to that memory. This
will appear as a normal instance in the tracefs system:
/sys/kernel/tracing/instances/boot_map
A user could enable tracing in that instance, and on reboot or kernel
crash, if the memory is not wiped by the firmware, it will recreate
the trace in that instance. For example, if one was debugging a
shutdown of a kernel reboot:
# cd /sys/kernel/tracing
# echo function > instances/boot_map/current_tracer
# reboot
[..]
# cd /sys/kernel/tracing
# tail instances/boot_map/trace
swapper/0-1 [000] d..1. 164.549800: restore_boot_irq_mode <-native_machine_shutdown
swapper/0-1 [000] d..1. 164.549801: native_restore_boot_irq_mode <-native_machine_shutdown
swapper/0-1 [000] d..1. 164.549802: disconnect_bsp_APIC <-native_machine_shutdown
swapper/0-1 [000] d..1. 164.549811: hpet_disable <-native_machine_shutdown
swapper/0-1 [000] d..1. 164.549812: iommu_shutdown_noop <-native_machine_restart
swapper/0-1 [000] d..1. 164.549813: native_machine_emergency_restart <-__do_sys_reboot
swapper/0-1 [000] d..1. 164.549813: tboot_shutdown <-native_machine_emergency_restart
swapper/0-1 [000] d..1. 164.549820: acpi_reboot <-native_machine_emergency_restart
swapper/0-1 [000] d..1. 164.549821: acpi_reset <-acpi_reboot
swapper/0-1 [000] d..1. 164.549822: acpi_os_write_port <-acpi_reboot
On reboot, the buffer is examined to make sure it is valid. The
validation check even steps through every event to make sure the meta
data of the event is correct. If any test fails, it will simply reset
the buffer, and the buffer will be empty on boot.
- Allow the tracing persistent boot buffer to use the "reserve_mem"
option
Instead of having the admin find a physical address to store the
persistent buffer, which can be very tedious if they have to
administrate several different machines, allow them to use the
"reserve_mem" option that will find a location for them. It is not as
reliable because of KASLR, as the loading of the kernel in different
locations can cause the memory allocated to be inconsistent. Booting
with "nokaslr" can make reserve_mem more reliable.
- Have function graph tracer handle offsets from a previous boot.
The ring buffer output from a previous boot may have different
addresses due to kaslr. Have the function graph tracer handle these
by using the delta from the previous boot to the new boot address
space.
- Only reset the saved meta offset when the buffer is started or reset
In the persistent memory meta data, it holds the previous address
space information, so that it can calculate the delta to have
function tracing work. But this gets updated after being read to hold
the new address space. But if the buffer isn't used for that boot, on
reboot, the delta is now calculated from the previous boot and not
the boot that holds the data in the ring buffer. This causes the
functions not to be shown. Do not save the address space information
of the current kernel until it is being recorded.
- Add a magic variable to test the valid meta data
Add a magic variable in the meta data that can also be used for
validation. The validator of the previous buffer doesn't need this
magic data, but it can be used if the meta data is changed by a new
kernel, which may have the same format that passes the validator but
is used differently. This magic number can also be used as a
"versioning" of the meta data.
- Align user space mapped ring buffer sub buffers to improve TLB
entries
Linus mentioned that the mapped ring buffer sub buffers were
misaligned between the meta page and the sub-buffers, so that if the
sub-buffers were bigger than PAGE_SIZE, it wouldn't allow the TLB to
use bigger entries.
- Add new kernel command line "traceoff" to disable tracing on boot for
instances
If tracing is enabled for a boot instance, there needs a way to be
able to disable it on boot so that new events do not get entered into
the ring buffer and be mixed with events from a previous boot, as
that can be confusing.
- Allow trace_printk() to go to other instances
Currently, trace_printk() can only go to the top level instance. When
debugging with a persistent buffer, it is really useful to be able to
add trace_printk() to go to that buffer, so that you have access to
them after a crash.
- Do not use "bin_printk()" for traces to a boot instance
The bin_printk() saves only a pointer to the printk format in the
ring buffer, as the reader of the buffer can still have access to it.
But this is not the case if the buffer is from a previous boot. If
the trace_printk() is going to a "persistent" buffer, it will use the
slower version that writes the printk format into the buffer.
- Add command line option to allow trace_printk() to go to an instance
Allow the kernel command line to define which instance the
trace_printk() goes to, instead of forcing the admin to set it for
every boot via the tracefs options.
- Start a document that explains how to use tracefs to debug the kernel
- Add some more kernel selftests to test user mapped ring buffer
* tag 'trace-ring-buffer-v6.12' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace: (28 commits)
selftests/ring-buffer: Handle meta-page bigger than the system
selftests/ring-buffer: Verify the entire meta-page padding
tracing/Documentation: Start a document on how to debug with tracing
tracing: Add option to set an instance to be the trace_printk destination
tracing: Have trace_printk not use binary prints if boot buffer
tracing: Allow trace_printk() to go to other instance buffers
tracing: Add "traceoff" flag to boot time tracing instances
ring-buffer: Align meta-page to sub-buffers for improved TLB usage
ring-buffer: Add magic and struct size to boot up meta data
ring-buffer: Don't reset persistent ring-buffer meta saved addresses
tracing/fgraph: Have fgraph handle previous boot function addresses
tracing: Allow boot instances to use reserve_mem boot memory
tracing: Fix ifdef of snapshots to not prevent last_boot_info file
ring-buffer: Use vma_pages() helper function
tracing: Fix NULL vs IS_ERR() check in enable_instances()
tracing: Add last boot delta offset for stack traces
tracing: Update function tracing output for previous boot buffer
tracing: Handle old buffer mappings for event strings and functions
tracing/ring-buffer: Add last_boot_info file to boot instance
ring-buffer: Save text and data locations in mapped meta data
...
af9c191ac2