Paul E. McKenney bb94b12e45 srcu: Allow inlining of __srcu_read_{,un}lock_lite() ...

This commit moves __srcu_read_lock_lite() and __srcu_read_unlock_lite()
into include/linux/srcu.h and marks them "static inline" so that they
can be inlined into srcu_read_lock_lite() and srcu_read_unlock_lite(),
respectively.  They are not hand-inlined due to Tree SRCU and Tiny SRCU
having different implementations.

The earlier removal of smp_mb() combined with the inlining produce
significant single-percentage performance wins.

Link: https://lore.kernel.org/all/CAEf4BzYgiNmSb=ZKQ65tm6nJDi1UX2Gq26cdHSH1mPwXJYZj5g@mail.gmail.com/

Reported-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Kent Overstreet <kent.overstreet@linux.dev>
Cc: <bpf@vger.kernel.org>
Reviewed-by: Neeraj Upadhyay <Neeraj.Upadhyay@amd.com>
Signed-off-by: Frederic Weisbecker <frederic@kernel.org>

2024-11-12 15:44:03 +01:00

..

bpf

move asm/unaligned.h to linux/unaligned.h

2024-10-02 17:23:23 -04:00

cgroup

Merge tag 'pull-stable-struct_fd' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs

2024-09-23 09:35:36 -07:00

configs

tinyconfig: remove unnecessary 'is not set' for choice blocks

2024-09-01 20:34:38 +09:00

debug

move asm/unaligned.h to linux/unaligned.h

2024-10-02 17:23:23 -04:00

dma

dma-mapping: report unlimited DMA addressing in IOMMU DMA path

2024-09-23 08:38:56 +02:00

entry

treewide: context_tracking: Rename CONTEXT_* into CT_STATE_*

2024-07-29 07:33:10 +05:30

events

uprobes: fix kernel info leak via "[uprobes]" vma

2024-09-30 08:19:11 +09:00

futex

fault-inject: improve build for CONFIG_FAULT_INJECTION=n

2024-09-01 20:43:33 -07:00

gcov

gcov: add support for GCC 14

2024-06-15 10:43:06 -07:00

irq

Merge tag 'pci-v6.12-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/pci/pci

2024-09-23 12:47:06 -07:00

kcsan

kcsan: Use min() to fix Coccinelle warning

2024-08-01 16:40:44 -07:00

livepatch

livepatch: Replace snprintf() with sysfs_emit()

2024-07-02 16:56:18 +02:00

locking

Merge tag 'locking-urgent-2024-09-29' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2024-09-29 08:51:30 -07:00

module

Merge tag 'modules-6.12-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux

2024-09-28 09:06:15 -07:00

power

[tree-wide] finally take no_llseek out

2024-09-27 08:18:43 -07:00

printk

Merge tag 'drm-next-2024-09-19' of https://gitlab.freedesktop.org/drm/kernel

2024-09-19 10:18:15 +02:00

rcu

srcu: Allow inlining of __srcu_read_{,un}lock_lite()

2024-11-12 15:44:03 +01:00

sched

sched: psi: fix bogus pressure spikes from aggregation race

2024-10-03 16:03:16 -07:00

time

[tree-wide] finally take no_llseek out

2024-09-27 08:18:43 -07:00

trace

tracing/hwlat: Fix a race during cpuhp processing

2024-10-03 16:43:23 -04:00

.gitignore

…

acct.c

kernel misc: Remove the now superfluous sentinel elements from ctl_table array

2024-04-24 09:43:53 +02:00

async.c

async: Use a dedicated unbound workqueue with raised min_active

2024-02-09 11:13:59 -10:00

audit_fsnotify.c

…

audit_tree.c

fsnotify: create a wrapper fsnotify_find_inode_mark()

2024-04-04 16:24:16 +02:00

audit_watch.c

fsnotify: create a wrapper fsnotify_find_inode_mark()

2024-04-04 16:24:16 +02:00

audit.c

audit: Make use of str_enabled_disabled() helper

2024-09-03 16:35:16 -04:00

audit.h

…

auditfilter.c

audit: use task_tgid_nr() instead of task_pid_nr()

2024-08-28 16:48:28 -04:00

auditsc.c

audit: use task_tgid_nr() instead of task_pid_nr()

2024-08-28 16:48:28 -04:00

backtracetest.c

backtracetest: add MODULE_DESCRIPTION()

2024-06-24 22:24:55 -07:00

bounds.c

bounds: Use the right number of bits for power-of-two CONFIG_NR_CPUS

2024-04-29 08:29:29 -07:00

capability.c

…

cfi.c

…

compat.c

…

configs.c

…

context_tracking.c

context_tracking, rcu: Rename rcu_dyntick trace event into rcu_watching

2024-08-15 21:30:43 +05:30

cpu_pm.c

…

cpu.c

Merge tag 'timers-core-2024-09-16' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2024-09-17 07:25:37 +02:00

crash_core.c

Document/kexec: generalize crash hotplug description

2024-09-01 20:43:37 -07:00

crash_reserve.c

crash: fix crash memory reserve exceed system memory bug

2024-09-01 20:43:30 -07:00

cred.c

cred: Use KMEM_CACHE() instead of kmem_cache_create()

2024-02-23 17:33:31 -05:00

delayacct.c

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

dma.c

…

elfcorehdr.c

crash: remove dependency of FA_DUMP on CRASH_DUMP

2024-02-23 17:48:22 -08:00

exec_domain.c

…

exit.c

Merge tag 'mm-stable-2024-09-20-02-31' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

2024-09-21 07:29:05 -07:00

exit.h

…

extable.c

…

fail_function.c

…

fork.c

close_range(): fix the logics in descriptor table trimming

2024-09-29 21:52:29 -04:00

freezer.c

sched,freezer: Mark TASK_FROZEN special

2024-08-17 11:06:44 +02:00

gen_kheaders.sh

kheaders: use command -v to test for existence of cpio

2024-05-30 01:13:20 +09:00

groups.c

…

hung_task.c

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

iomem.c

…

irq_work.c

…

jump_label.c

jump_label: Fix static_key_slow_dec() yet again

2024-09-10 11:57:27 +02:00

kallsyms_internal.h

kallsyms: get rid of code for absolute kallsyms

2024-07-20 16:33:21 +09:00

kallsyms_selftest.c

kallsyms: Match symbols exactly with CONFIG_LTO_CLANG

2024-08-15 09:33:35 -07:00

kallsyms_selftest.h

…

kallsyms.c

kallsyms: Match symbols exactly with CONFIG_LTO_CLANG

2024-08-15 09:33:35 -07:00

kcmp.c

…

Kconfig.freezer

…

Kconfig.hz

…

Kconfig.kexec

crash: clean up kdump related config items

2024-02-23 17:48:22 -08:00

Kconfig.locks

…

Kconfig.preempt

sched_ext: Build fix on !CONFIG_STACKTRACE[_SUPPORT]

2024-08-01 07:08:01 -10:00

kcov.c

Merge tag 'x86-build-2024-09-17' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip

2024-09-17 12:40:34 +02:00

kexec_core.c

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

kexec_elf.c

…

kexec_file.c

kexec_file: fix elfcorehdr digest exclusion when CONFIG_CRASH_HOTPLUG=y

2024-09-01 17:59:01 -07:00

kexec_internal.h

kexec: use atomic_try_cmpxchg_acquire() in kexec_trylock()

2024-09-01 20:43:23 -07:00

kexec.c

crash: add a new kexec flag for hotplug support

2024-04-23 14:59:01 +10:00

kheaders.c

…

kprobes.c

kprobes: Fix to check symbol prefixes correctly

2024-08-05 14:04:03 +09:00

ksyms_common.c

…

ksysfs.c

profiling: remove prof_cpu_mask

2024-07-29 10:45:54 -07:00

kthread.c

kthread: Fix task state in kthread worker if being frozen

2024-09-10 09:51:14 +02:00

latencytop.c

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

Makefile

mm: move kernel/numa.c to mm/

2024-09-03 21:15:26 -07:00

module_signature.c

…

notifier.c

…

nsproxy.c

introduce fd_file(), convert all accessors to it.

2024-08-12 22:00:43 -04:00

padata.c

Merge tag 'v6.12-p1' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6

2024-09-16 06:28:28 +02:00

panic.c

Merge tag 'drm-next-2024-09-19' of https://gitlab.freedesktop.org/drm/kernel

2024-09-19 10:18:15 +02:00

params.c

params: Fix multi-line comment style

2023-12-01 09:51:44 -08:00

pid_namespace.c

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

pid_sysctl.h

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

pid.c

introduce fd_file(), convert all accessors to it.

2024-08-12 22:00:43 -04:00

profile.c

profiling: remove profile=sleep support

2024-08-04 13:36:28 -07:00

ptrace.c

ptrace_attach: shift send(SIGSTOP) into ptrace_set_stopped()

2024-02-22 15:38:52 -08:00

range.c

…

reboot.c

kernel misc: Remove the now superfluous sentinel elements from ctl_table array

2024-04-24 09:43:53 +02:00

regset.c

regset: use kvzalloc() for regset_get_alloc()

2024-04-25 21:07:03 -07:00

relay.c

[tree-wide] finally take no_llseek out

2024-09-27 08:18:43 -07:00

resource_kunit.c

resource, kunit: add test case for region_intersects()

2024-09-17 01:07:00 -07:00

resource.c

Merge tag 'mm-stable-2024-09-20-02-31' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

2024-09-21 07:29:05 -07:00

rseq.c

…

scftorture.c

scftorture: Make torture_type static

2024-05-30 15:31:51 -07:00

scs.c

…

seccomp.c

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

signal.c

Revert "binfmt_elf, coredump: Log the reason of the failed core dumps"

2024-09-26 11:39:02 -07:00

smp.c

smp: print only local CPU info when sched_clock goes backward

2024-08-15 00:06:48 +05:30

smpboot.c

…

smpboot.h

…

softirq.c

softirq: Remove unused 'action' parameter from action callback

2024-08-20 17:13:40 +02:00

stackleak.c

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

stacktrace.c

stacktrace: fix kernel-doc typo

2023-12-29 12:22:29 -08:00

static_call_inline.c

static_call: Replace pointless WARN_ON() in static_call_module_notify()

2024-09-06 16:29:22 +02:00

static_call.c

…

stop_machine.c

rcu: Rename rcu_momentary_dyntick_idle() into rcu_momentary_eqs()

2024-08-15 21:30:42 +05:30

sys_ni.c

Merge tag 'probes-v6.11' of git://git.kernel.org/pub/scm/linux/kernel/git/trace/linux-trace

2024-07-18 12:19:20 -07:00

sys.c

Merge tag 'pull-stable-struct_fd' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs

2024-09-23 09:35:36 -07:00

sysctl-test.c

sysctl: Add module description to sysctl-testing

2024-06-03 15:20:37 +02:00

sysctl.c

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

task_work.c

task_work: make TWA_NMI_CURRENT handling conditional on IRQ_WORK

2024-07-29 12:05:06 -07:00

taskstats.c

introduce fd_file(), convert all accessors to it.

2024-08-12 22:00:43 -04:00

torture.c

torture: Add MODULE_DESCRIPTION()

2024-05-30 15:31:38 -07:00

tracepoint.c

tracepoint: Support iterating tracepoints in a loading module

2024-09-25 23:23:44 +09:00

tsacct.c

tsacct: replace strncpy() with strscpy()

2024-07-12 16:39:53 -07:00

ucount.c

Merge tag 'sysctl-6.10-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/sysctl/sysctl

2024-05-17 17:31:24 -07:00

uid16.c

…

uid16.h

…

umh.c

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

up.c

…

user_namespace.c

user_namespace: use kmemdup_array() instead of kmemdup() for multiple allocation

2024-09-09 16:47:42 -07:00

user-return-notifier.c

…

user.c

uidgid: make sure we fit into one cacheline

2024-09-12 12:16:09 +02:00

usermode_driver.c

…

utsname_sysctl.c

sysctl: treewide: constify the ctl_table argument of proc_handlers

2024-07-24 20:59:29 +02:00

utsname.c

…

vhost_task.c

vhost_task: Handle SIGKILL by flushing work and exiting

2024-05-22 08:31:15 -04:00

vmcore_info.c

mm: support only one page_type per page

2024-09-03 21:15:43 -07:00

watch_queue.c

introduce fd_file(), convert all accessors to it.

2024-08-12 22:00:43 -04:00

watchdog_buddy.c

…

watchdog_perf.c

watchdog/perf: properly initialize the turbo mode timestamp and rearm counter

2024-07-17 21:11:34 -07:00

watchdog.c

watchdog: handle the ENODEV failure case of lockup_detector_delay_init() separately

2024-09-01 20:43:32 -07:00

workqueue_internal.h

…

workqueue.c

Merge tag 'wq-for-6.12' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq

2024-09-18 06:59:44 +02:00