Skip to content

Commit 9855e87

Browse files
torvaldstorvalds
committed
Merge tag 'rcu.2024.07.12a' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu
Pull RCU updates from Paul McKenney: - Update Tasks RCU and Tasks Rude RCU description in Requirements.rst and clarify rcu_assign_pointer() and rcu_dereference() ordering properties - Add lockdep assertions for RCU readers, limit inline wakeups for callback-bypass synchronize_rcu(), add an rcutree.nohz_full_patience_delay to reduce nohz_full OS jitter, add Uladzislau Rezki as RCU maintainer, and fix a subtle callback-migration memory-ordering issue - Remove a number of redundant memory barriers - Remove unnecessary bypass-list lock-contention mitigation, use parking API instead of open-coded ad-hoc equivalent, and upgrade obsolete comments - Revert avoidance of a deadlock that can no longer occur and properly synchronize Tasks Trace RCU checking of runqueues - Add tests for handling of double-call_rcu() bug, add missing MODULE_DESCRIPTION, and add a script that histograms the number of calls to RCU updaters - Fill out SRCU polled-grace-period API * tag 'rcu.2024.07.12a' of git://git.kernel.org/pub/scm/linux/kernel/git/paulmck/linux-rcu: (29 commits) rcu: Fix rcu_barrier() VS post CPUHP_TEARDOWN_CPU invocation rcu: Eliminate lockless accesses to rcu_sync->gp_count MAINTAINERS: Add Uladzislau Rezki as RCU maintainer rcu: Add rcutree.nohz_full_patience_delay to reduce nohz_full OS jitter rcu/exp: Remove redundant full memory barrier at the end of GP rcu: Remove full memory barrier on RCU stall printout rcu: Remove full memory barrier on boot time eqs sanity check rcu/exp: Remove superfluous full memory barrier upon first EQS snapshot rcu: Remove superfluous full memory barrier upon first EQS snapshot rcu: Remove full ordering on second EQS snapshot srcu: Fill out polled grace-period APIs srcu: Update cleanup_srcu_struct() comment srcu: Add NUM_ACTIVE_SRCU_POLL_OLDSTATE srcu: Disable interrupts directly in srcu_gp_end() rcu: Disable interrupts directly in rcu_gp_init() rcu/tree: Reduce wake up for synchronize_rcu() common case rcu/tasks: Fix stale task snaphot for Tasks Trace tools/rcu: Add rcu-updaters.sh script rcutorture: Add missing MODULE_DESCRIPTION() macros rcutorture: Fix rcu_torture_fwd_cb_cr() data race ...
2 parents 253e1e9 + 02219ca commit 9855e87

24 files changed

Lines changed: 439 additions & 278 deletions

‎Documentation/RCU/Design/Memory-Ordering/Tree-RCU-Memory-Ordering.rst‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -149,9 +149,9 @@ This case is handled by calls to the strongly ordered
149149
``atomic_add_return()`` read-modify-write atomic operation that
150150
is invoked within ``rcu_dynticks_eqs_enter()`` at idle-entry
151151
time and within ``rcu_dynticks_eqs_exit()`` at idle-exit time.
152-
The grace-period kthread invokes ``rcu_dynticks_snap()`` and
153-
``rcu_dynticks_in_eqs_since()`` (both of which invoke
154-
an ``atomic_add_return()`` of zero) to detect idle CPUs.
152+
The grace-period kthread invokes first ``ct_dynticks_cpu_acquire()``
153+
(preceded by a full memory barrier) and ``rcu_dynticks_in_eqs_since()``
154+
(both of which rely on acquire semantics) to detect idle CPUs.
155155

156156
+-----------------------------------------------------------------------+
157157
| **Quick Quiz**: |

‎Documentation/RCU/Design/Requirements/Requirements.rst‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2357,6 +2357,7 @@ section.
23572357
#. `Sched Flavor (Historical)`_
23582358
#. `Sleepable RCU`_
23592359
#. `Tasks RCU`_
2360+
#. `Tasks Trace RCU`_
23602361

23612362
Bottom-Half Flavor (Historical)
23622363
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -2610,6 +2611,16 @@ critical sections that are delimited by voluntary context switches, that
26102611
is, calls to schedule(), cond_resched(), and
26112612
synchronize_rcu_tasks(). In addition, transitions to and from
26122613
userspace execution also delimit tasks-RCU read-side critical sections.
2614+
Idle tasks are ignored by Tasks RCU, and Tasks Rude RCU may be used to
2615+
interact with them.
2616+
2617+
Note well that involuntary context switches are *not* Tasks-RCU quiescent
2618+
states. After all, in preemptible kernels, a task executing code in a
2619+
trampoline might be preempted. In this case, the Tasks-RCU grace period
2620+
clearly cannot end until that task resumes and its execution leaves that
2621+
trampoline. This means, among other things, that cond_resched() does
2622+
not provide a Tasks RCU quiescent state. (Instead, use rcu_softirq_qs()
2623+
from softirq or rcu_tasks_classic_qs() otherwise.)
26132624

26142625
The tasks-RCU API is quite compact, consisting only of
26152626
call_rcu_tasks(), synchronize_rcu_tasks(), and
@@ -2632,6 +2643,11 @@ moniker. And this operation is considered to be quite rude by real-time
26322643
workloads that don't want their ``nohz_full`` CPUs receiving IPIs and
26332644
by battery-powered systems that don't want their idle CPUs to be awakened.
26342645

2646+
Once kernel entry/exit and deep-idle functions have been properly tagged
2647+
``noinstr``, Tasks RCU can start paying attention to idle tasks (except
2648+
those that are idle from RCU's perspective) and then Tasks Rude RCU can
2649+
be removed from the kernel.
2650+
26352651
The tasks-rude-RCU API is also reader-marking-free and thus quite compact,
26362652
consisting of call_rcu_tasks_rude(), synchronize_rcu_tasks_rude(),
26372653
and rcu_barrier_tasks_rude().

‎Documentation/RCU/whatisRCU.rst‎

Lines changed: 19 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -250,21 +250,25 @@ rcu_assign_pointer()
250250
^^^^^^^^^^^^^^^^^^^^
251251
void rcu_assign_pointer(p, typeof(p) v);
252252

253-
Yes, rcu_assign_pointer() **is** implemented as a macro, though it
254-
would be cool to be able to declare a function in this manner.
255-
(Compiler experts will no doubt disagree.)
253+
Yes, rcu_assign_pointer() **is** implemented as a macro, though
254+
it would be cool to be able to declare a function in this manner.
255+
(And there has been some discussion of adding overloaded functions
256+
to the C language, so who knows?)
256257

257258
The updater uses this spatial macro to assign a new value to an
258259
RCU-protected pointer, in order to safely communicate the change
259260
in value from the updater to the reader. This is a spatial (as
260261
opposed to temporal) macro. It does not evaluate to an rvalue,
261-
but it does execute any memory-barrier instructions required
262-
for a given CPU architecture. Its ordering properties are that
263-
of a store-release operation.
264-
265-
Perhaps just as important, it serves to document (1) which
266-
pointers are protected by RCU and (2) the point at which a
267-
given structure becomes accessible to other CPUs. That said,
262+
but it does provide any compiler directives and memory-barrier
263+
instructions required for a given compile or CPU architecture.
264+
Its ordering properties are that of a store-release operation,
265+
that is, any prior loads and stores required to initialize the
266+
structure are ordered before the store that publishes the pointer
267+
to that structure.
268+
269+
Perhaps just as important, rcu_assign_pointer() serves to document
270+
(1) which pointers are protected by RCU and (2) the point at which
271+
a given structure becomes accessible to other CPUs. That said,
268272
rcu_assign_pointer() is most frequently used indirectly, via
269273
the _rcu list-manipulation primitives such as list_add_rcu().
270274

@@ -283,7 +287,11 @@ rcu_dereference()
283287
executes any needed memory-barrier instructions for a given
284288
CPU architecture. Currently, only Alpha needs memory barriers
285289
within rcu_dereference() -- on other CPUs, it compiles to a
286-
volatile load.
290+
volatile load. However, no mainstream C compilers respect
291+
address dependencies, so rcu_dereference() uses volatile casts,
292+
which, in combination with the coding guidelines listed in
293+
rcu_dereference.rst, prevent current compilers from breaking
294+
these dependencies.
287295

288296
Common coding practice uses rcu_dereference() to copy an
289297
RCU-protected pointer to a local variable, then dereferences

‎Documentation/admin-guide/kernel-parameters.txt‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5015,6 +5015,14 @@
50155015
the ->nocb_bypass queue. The definition of "too
50165016
many" is supplied by this kernel boot parameter.
50175017

5018+
rcutree.nohz_full_patience_delay= [KNL]
5019+
On callback-offloaded (rcu_nocbs) CPUs, avoid
5020+
disturbing RCU unless the grace period has
5021+
reached the specified age in milliseconds.
5022+
Defaults to zero. Large values will be capped
5023+
at five seconds. All values will be rounded down
5024+
to the nearest value representable by jiffies.
5025+
50185026
rcutree.qhimark= [KNL]
50195027
Set threshold of queued RCU callbacks beyond which
50205028
batch limiting is disabled.

‎MAINTAINERS‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18863,6 +18863,7 @@ M: Neeraj Upadhyay <neeraj.upadhyay@kernel.org> (kernel/rcu/tasks.h)
1886318863
M: Joel Fernandes <joel@joelfernandes.org>
1886418864
M: Josh Triplett <josh@joshtriplett.org>
1886518865
M: Boqun Feng <boqun.feng@gmail.com>
18866+
M: Uladzislau Rezki <urezki@gmail.com>
1886618867
R: Steven Rostedt <rostedt@goodmis.org>
1886718868
R: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
1886818869
R: Lai Jiangshan <jiangshanlai@gmail.com>

‎include/linux/rcu_segcblist.h‎

Lines changed: 38 additions & 50 deletions
Original file line numberDiff line numberDiff line change
@@ -80,36 +80,35 @@ struct rcu_cblist {
8080
* | SEGCBLIST_RCU_CORE | SEGCBLIST_LOCKING | SEGCBLIST_OFFLOADED |
8181
* | |
8282
* | Callbacks processed by rcu_core() from softirqs or local |
83-
* | rcuc kthread, while holding nocb_lock. Waking up CB and GP kthreads, |
84-
* | allowing nocb_timer to be armed. |
83+
* | rcuc kthread, while holding nocb_lock. Waking up CB and GP kthreads. |
8584
* ----------------------------------------------------------------------------
8685
* |
8786
* v
88-
* -----------------------------------
89-
* | |
90-
* v v
91-
* --------------------------------------- ----------------------------------|
92-
* | SEGCBLIST_RCU_CORE | | | SEGCBLIST_RCU_CORE | |
93-
* | SEGCBLIST_LOCKING | | | SEGCBLIST_LOCKING | |
94-
* | SEGCBLIST_OFFLOADED | | | SEGCBLIST_OFFLOADED | |
95-
* | SEGCBLIST_KTHREAD_CB | | SEGCBLIST_KTHREAD_GP |
96-
* | | | |
97-
* | | | |
98-
* | CB kthread woke up and | | GP kthread woke up and |
99-
* | acknowledged SEGCBLIST_OFFLOADED. | | acknowledged SEGCBLIST_OFFLOADED|
100-
* | Processes callbacks concurrently | | |
101-
* | with rcu_core(), holding | | |
102-
* | nocb_lock. | | |
103-
* --------------------------------------- -----------------------------------
104-
* | |
105-
* -----------------------------------
87+
* ----------------------------------------------------------------------------
88+
* | SEGCBLIST_RCU_CORE | SEGCBLIST_LOCKING | SEGCBLIST_OFFLOADED |
89+
* | + unparked CB kthread |
90+
* | |
91+
* | CB kthread got unparked and processes callbacks concurrently with |
92+
* | rcu_core(), holding nocb_lock. |
93+
* ---------------------------------------------------------------------------
94+
* |
95+
* v
96+
* ---------------------------------------------------------------------------|
97+
* | SEGCBLIST_RCU_CORE | |
98+
* | SEGCBLIST_LOCKING | |
99+
* | SEGCBLIST_OFFLOADED | |
100+
* | SEGCBLIST_KTHREAD_GP |
101+
* | + unparked CB kthread |
102+
* | |
103+
* | GP kthread woke up and acknowledged nocb_lock. |
104+
* ---------------------------------------- -----------------------------------
106105
* |
107106
* v
108107
* |--------------------------------------------------------------------------|
109-
* | SEGCBLIST_LOCKING | |
110-
* | SEGCBLIST_OFFLOADED | |
108+
* | SEGCBLIST_LOCKING | |
109+
* | SEGCBLIST_OFFLOADED | |
111110
* | SEGCBLIST_KTHREAD_GP | |
112-
* | SEGCBLIST_KTHREAD_CB |
111+
* | + unparked CB kthread |
113112
* | |
114113
* | Kthreads handle callbacks holding nocb_lock, local rcu_core() stops |
115114
* | handling callbacks. Enable bypass queueing. |
@@ -125,8 +124,8 @@ struct rcu_cblist {
125124
* |--------------------------------------------------------------------------|
126125
* | SEGCBLIST_LOCKING | |
127126
* | SEGCBLIST_OFFLOADED | |
128-
* | SEGCBLIST_KTHREAD_CB | |
129127
* | SEGCBLIST_KTHREAD_GP |
128+
* | + unparked CB kthread |
130129
* | |
131130
* | CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core() |
132131
* | ignores callbacks. Bypass enqueue is enabled. |
@@ -137,11 +136,11 @@ struct rcu_cblist {
137136
* | SEGCBLIST_RCU_CORE | |
138137
* | SEGCBLIST_LOCKING | |
139138
* | SEGCBLIST_OFFLOADED | |
140-
* | SEGCBLIST_KTHREAD_CB | |
141139
* | SEGCBLIST_KTHREAD_GP |
140+
* | + unparked CB kthread |
142141
* | |
143142
* | CB/GP kthreads handle callbacks holding nocb_lock, local rcu_core() |
144-
* | handles callbacks concurrently. Bypass enqueue is enabled. |
143+
* | handles callbacks concurrently. Bypass enqueue is disabled. |
145144
* | Invoke RCU core so we make sure not to preempt it in the middle with |
146145
* | leaving some urgent work unattended within a jiffy. |
147146
* ----------------------------------------------------------------------------
@@ -150,42 +149,31 @@ struct rcu_cblist {
150149
* |--------------------------------------------------------------------------|
151150
* | SEGCBLIST_RCU_CORE | |
152151
* | SEGCBLIST_LOCKING | |
153-
* | SEGCBLIST_KTHREAD_CB | |
154152
* | SEGCBLIST_KTHREAD_GP |
153+
* | + unparked CB kthread |
155154
* | |
156155
* | CB/GP kthreads and local rcu_core() handle callbacks concurrently |
157-
* | holding nocb_lock. Wake up CB and GP kthreads if necessary. Disable |
158-
* | bypass enqueue. |
156+
* | holding nocb_lock. Wake up GP kthread if necessary. |
159157
* ----------------------------------------------------------------------------
160158
* |
161159
* v
162-
* -----------------------------------
163-
* | |
164-
* v v
165-
* ---------------------------------------------------------------------------|
166-
* | | |
167-
* | SEGCBLIST_RCU_CORE | | SEGCBLIST_RCU_CORE | |
168-
* | SEGCBLIST_LOCKING | | SEGCBLIST_LOCKING | |
169-
* | SEGCBLIST_KTHREAD_CB | SEGCBLIST_KTHREAD_GP |
170-
* | | |
171-
* | GP kthread woke up and | CB kthread woke up and |
172-
* | acknowledged the fact that | acknowledged the fact that |
173-
* | SEGCBLIST_OFFLOADED got cleared. | SEGCBLIST_OFFLOADED got cleared. |
174-
* | | The CB kthread goes to sleep |
175-
* | The callbacks from the target CPU | until it ever gets re-offloaded. |
176-
* | will be ignored from the GP kthread | |
177-
* | loop. | |
160+
* |--------------------------------------------------------------------------|
161+
* | SEGCBLIST_RCU_CORE | |
162+
* | SEGCBLIST_LOCKING | |
163+
* | + unparked CB kthread |
164+
* | |
165+
* | GP kthread woke up and acknowledged the fact that SEGCBLIST_OFFLOADED |
166+
* | got cleared. The callbacks from the target CPU will be ignored from the|
167+
* | GP kthread loop. |
178168
* ----------------------------------------------------------------------------
179-
* | |
180-
* -----------------------------------
181169
* |
182170
* v
183171
* ----------------------------------------------------------------------------
184172
* | SEGCBLIST_RCU_CORE | SEGCBLIST_LOCKING |
173+
* | + parked CB kthread |
185174
* | |
186-
* | Callbacks processed by rcu_core() from softirqs or local |
187-
* | rcuc kthread, while holding nocb_lock. Forbid nocb_timer to be armed. |
188-
* | Flush pending nocb_timer. Flush nocb bypass callbacks. |
175+
* | CB kthread is parked. Callbacks processed by rcu_core() from softirqs or |
176+
* | local rcuc kthread, while holding nocb_lock. |
189177
* ----------------------------------------------------------------------------
190178
* |
191179
* v

‎include/linux/rcupdate.h‎

Lines changed: 60 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -209,7 +209,6 @@ void synchronize_rcu_tasks_rude(void);
209209

210210
#define rcu_note_voluntary_context_switch(t) rcu_tasks_qs(t, false)
211211
void exit_tasks_rcu_start(void);
212-
void exit_tasks_rcu_stop(void);
213212
void exit_tasks_rcu_finish(void);
214213
#else /* #ifdef CONFIG_TASKS_RCU_GENERIC */
215214
#define rcu_tasks_classic_qs(t, preempt) do { } while (0)
@@ -218,7 +217,6 @@ void exit_tasks_rcu_finish(void);
218217
#define call_rcu_tasks call_rcu
219218
#define synchronize_rcu_tasks synchronize_rcu
220219
static inline void exit_tasks_rcu_start(void) { }
221-
static inline void exit_tasks_rcu_stop(void) { }
222220
static inline void exit_tasks_rcu_finish(void) { }
223221
#endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */
224222

@@ -421,11 +419,71 @@ static inline void rcu_preempt_sleep_check(void) { }
421419
"Illegal context switch in RCU-sched read-side critical section"); \
422420
} while (0)
423421

422+
// See RCU_LOCKDEP_WARN() for an explanation of the double call to
423+
// debug_lockdep_rcu_enabled().
424+
static inline bool lockdep_assert_rcu_helper(bool c)
425+
{
426+
return debug_lockdep_rcu_enabled() &&
427+
(c || !rcu_is_watching() || !rcu_lockdep_current_cpu_online()) &&
428+
debug_lockdep_rcu_enabled();
429+
}
430+
431+
/**
432+
* lockdep_assert_in_rcu_read_lock - WARN if not protected by rcu_read_lock()
433+
*
434+
* Splats if lockdep is enabled and there is no rcu_read_lock() in effect.
435+
*/
436+
#define lockdep_assert_in_rcu_read_lock() \
437+
WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_lock_map)))
438+
439+
/**
440+
* lockdep_assert_in_rcu_read_lock_bh - WARN if not protected by rcu_read_lock_bh()
441+
*
442+
* Splats if lockdep is enabled and there is no rcu_read_lock_bh() in effect.
443+
* Note that local_bh_disable() and friends do not suffice here, instead an
444+
* actual rcu_read_lock_bh() is required.
445+
*/
446+
#define lockdep_assert_in_rcu_read_lock_bh() \
447+
WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_bh_lock_map)))
448+
449+
/**
450+
* lockdep_assert_in_rcu_read_lock_sched - WARN if not protected by rcu_read_lock_sched()
451+
*
452+
* Splats if lockdep is enabled and there is no rcu_read_lock_sched()
453+
* in effect. Note that preempt_disable() and friends do not suffice here,
454+
* instead an actual rcu_read_lock_sched() is required.
455+
*/
456+
#define lockdep_assert_in_rcu_read_lock_sched() \
457+
WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_sched_lock_map)))
458+
459+
/**
460+
* lockdep_assert_in_rcu_reader - WARN if not within some type of RCU reader
461+
*
462+
* Splats if lockdep is enabled and there is no RCU reader of any
463+
* type in effect. Note that regions of code protected by things like
464+
* preempt_disable, local_bh_disable(), and local_irq_disable() all qualify
465+
* as RCU readers.
466+
*
467+
* Note that this will never trigger in PREEMPT_NONE or PREEMPT_VOLUNTARY
468+
* kernels that are not also built with PREEMPT_COUNT. But if you have
469+
* lockdep enabled, you might as well also enable PREEMPT_COUNT.
470+
*/
471+
#define lockdep_assert_in_rcu_reader() \
472+
WARN_ON_ONCE(lockdep_assert_rcu_helper(!lock_is_held(&rcu_lock_map) && \
473+
!lock_is_held(&rcu_bh_lock_map) && \
474+
!lock_is_held(&rcu_sched_lock_map) && \
475+
preemptible()))
476+
424477
#else /* #ifdef CONFIG_PROVE_RCU */
425478

426479
#define RCU_LOCKDEP_WARN(c, s) do { } while (0 && (c))
427480
#define rcu_sleep_check() do { } while (0)
428481

482+
#define lockdep_assert_in_rcu_read_lock() do { } while (0)
483+
#define lockdep_assert_in_rcu_read_lock_bh() do { } while (0)
484+
#define lockdep_assert_in_rcu_read_lock_sched() do { } while (0)
485+
#define lockdep_assert_in_rcu_reader() do { } while (0)
486+
429487
#endif /* #else #ifdef CONFIG_PROVE_RCU */
430488

431489
/*

0 commit comments

Comments
 (0)