user/sven/linux.git/lib/percpu-refcount.c, branch v5.0.12

percpu-refcount: Replace call_rcu_sched() with call_rcu()

2018-11-27T17:21:45Z

Now that call_rcu()'s callback is not invoked until after all preempt-disable regions of code have completed (in addition to explicitly marked RCU read-side critical sections), call_rcu() can be used in place of call_rcu_sched(). This commit therefore makes that change. Signed-off-by: Paul E. McKenney Cc: Ming Lei Cc: Bart Van Assche Cc: Jens Axboe Acked-by: Tejun Heo

percpu-refcount: Introduce percpu_ref_resurrect()

2018-09-26T21:11:29Z

This function will be used in a later patch to switch the struct request_queue q_usage_counter from killed back to live. In contrast to percpu_ref_reinit(), this new function does not require that the refcount is zero. Signed-off-by: Bart Van Assche Acked-by: Tejun Heo Reviewed-by: Ming Lei Cc: Christoph Hellwig Cc: Jianchao Wang Cc: Hannes Reinecke Cc: Johannes Thumshirn Signed-off-by: Jens Axboe

percpu_ref: Update doc to dissuade users from depending on internal RCU grace periods

2018-03-19T17:09:44Z

percpu_ref internally uses sched-RCU to implement the percpu -> atomic mode switching and the documentation suggested that this could be depended upon. This doesn't seem like a good idea. * percpu_ref uses sched-RCU which has different grace periods regular RCU. Users may combine percpu_ref with regular RCU usage and incorrectly believe that regular RCU grace periods are performed by percpu_ref. This can lead to, for example, use-after-free due to premature freeing. * percpu_ref has a grace period when switching from percpu to atomic mode. It doesn't have one between the last put and release. This distinction is subtle and can lead to surprising bugs. * percpu_ref allows starting in and switching to atomic mode manually for debugging and other purposes. This means that there may not be any grace periods from kill to release. This patch makes it clear that the grace periods are percpu_ref's internal implementation detail and can't be depended upon by the users. Signed-off-by: Tejun Heo Cc: Kent Overstreet Cc: Linus Torvalds Signed-off-by: Tejun Heo

percpu: READ_ONCE() now implies smp_read_barrier_depends()

2017-12-04T18:52:53Z

Because READ_ONCE() now implies smp_read_barrier_depends(), this commit removes the now-redundant smp_read_barrier_depends() following the READ_ONCE() in __ref_is_percpu(). Signed-off-by: Paul E. McKenney Acked-by: Tejun Heo Cc: Christoph Lameter

percpu-refcount: support synchronous switch to atomic mode.

2017-03-23T02:18:43Z

percpu_ref_switch_to_atomic_sync() schedules the switch to atomic mode, then waits for it to complete. Also export percpu_ref_switch_to_* so they can be used from modules. This will be used in md/raid to count the number of pending write requests to an array. We occasionally need to check if the count is zero, but most often we don't care. We always want updates to the counter to be fast, as in some cases we count every 4K page. Signed-off-by: NeilBrown Acked-by: Tejun Heo Signed-off-by: Shaohua Li

percpu-refcount: init ->confirm_switch member properly

2016-08-11T17:52:23Z

This patch targets two things which are related to ->confirm_switch: 1. Init ->confirm_switch pointer with NULL on percpu_ref_init() or kernel frightfully complains with WARN_ON_ONCE(ref->confirm_switch) at __percpu_ref_switch_to_atomic if memory chunk was not properly zeroed. 2. Warn if RCU callback is still in progress on percpu_ref_exit(). The race still exists, because percpu_ref_call_confirm_rcu() drops ->confirm_switch to NULL early, but that is only a warning and still the caller is responsible that ref is no longer in active use. Hopefully that can help to catch incorrect usage of percpu-refcount. Signed-off-by: Roman Pen Cc: Tejun Heo Cc: linux-kernel@vger.kernel.org Signed-off-by: Tejun Heo

percpu_ref: allow operation mode switching operations to be called concurrently

2016-08-10T19:02:58Z

percpu_ref initially didn't have explicit mode switching operations. It started out in percpu mode and switched to atomic mode on kill and then released. Ensuring that kill operation is initiated only after init completes was naturally the caller's responsibility. percpu_ref_reinit() was introduced later but it didn't shift the synchronization responsibility. Reinit can't be performed until kill is confirmed, so there was nothing to worry about synchronization-wise. Also, as both reinit and kill manipulate the base reference, invocations of the same function couldn't be allowed to race each other. The latest additions of percpu_ref_switch_to_atomic/percpu() changed the situation. These two functions can be called any time as long as the percpu_ref is between init and exit and thus there are valid valid usage scenarios where these new functions race with each other or against reinit/kill. Mostly from inertia, f47ad4578461 ("percpu_ref: decouple switching to percpu mode and reinit") still left synchronization among percpu mode switching operations to its users. That the new switch functions can be freely mixed with kill/reinit but the operations themselves should be synchronized is too subtle a requirement and led to a very subtle race condition in blk-mq freezing path. This patch fixes the situation by introducing percpu_ref_switch_lock to protect mode switching operations. This ensures that percpu-ref users don't have to worry about mode changing operations racing against each other, e.g. switch_to_percpu against kill, as long as the sequence of operations is valid. Signed-off-by: Tejun Heo Reported-by: Akinobu Mita Link: http://lkml.kernel.org/g/1443287365-4244-7-git-send-email-akinobu.mita@gmail.com Fixes: f47ad4578461 ("percpu_ref: decouple switching to percpu mode and reinit")

percpu_ref: restructure operation mode switching

2016-08-10T19:02:58Z

Restructure atomic/percpu mode switching. * The users of __percpu_ref_switch_to_atomic/percpu() now call a new function __percpu_ref_switch_mode() which calls either of the original switching functions depending on the current state of ref->force_atomic and the __PERCPU_REF_DEAD flag. The callers no longer check whether switching is necessary but always invoke __percpu_ref_switch_mode(). * !ref->confirm_switch waiting is collected into __percpu_ref_switch_mode(). This patch doesn't cause any behavior differences. Signed-off-by: Tejun Heo

percpu_ref: unify staggered atomic switching wait behavior

2016-08-10T19:02:58Z

When an atomic or percpu switching starts before the previous atomic switching finishes, the taken behaviors are * If the new atomic switching has confirmation callback, it waits for the previous atomic switching to complete. * If the new percpu switching is the first percpu switching following the previous atomic switching, it waits the previous atomic switching to complete. No percpu_ref user depends on these subtleties. The only meaningful part is that, if the caller ensures that atomic switching isn't in progress, mode switching operations can be issued from any context. This patch pulls the wait logic to the top of both switching functions so that they always wait for the previous atomic switching to complete. This makes the behavior simpler and consistent for both directions and will help allowing concurrent invocations of mode switching functions. Signed-off-by: Tejun Heo

percpu_ref: reorganize __percpu_ref_switch_to_atomic() and relocate percpu_ref_switch_to_atomic()

2016-08-10T19:02:58Z

Reorganize __percpu_ref_switch_to_atomic() so that it looks structurally similar to __percpu_ref_switch_to_percpu() and relocate percpu_ref_switch_to_atomic so that the two internal functions are co-located. This patch doesn't introduce any functional differences. Signed-off-by: Tejun Heo