user/sven/linux.git/lib/percpu-refcount.c, branch v4.14.87

percpu-refcount: support synchronous switch to atomic mode.

2017-03-23T02:18:43Z

percpu_ref_switch_to_atomic_sync() schedules the switch to atomic mode, then waits for it to complete. Also export percpu_ref_switch_to_* so they can be used from modules. This will be used in md/raid to count the number of pending write requests to an array. We occasionally need to check if the count is zero, but most often we don't care. We always want updates to the counter to be fast, as in some cases we count every 4K page. Signed-off-by: NeilBrown Acked-by: Tejun Heo Signed-off-by: Shaohua Li

percpu-refcount: init ->confirm_switch member properly

2016-08-11T17:52:23Z

This patch targets two things which are related to ->confirm_switch: 1. Init ->confirm_switch pointer with NULL on percpu_ref_init() or kernel frightfully complains with WARN_ON_ONCE(ref->confirm_switch) at __percpu_ref_switch_to_atomic if memory chunk was not properly zeroed. 2. Warn if RCU callback is still in progress on percpu_ref_exit(). The race still exists, because percpu_ref_call_confirm_rcu() drops ->confirm_switch to NULL early, but that is only a warning and still the caller is responsible that ref is no longer in active use. Hopefully that can help to catch incorrect usage of percpu-refcount. Signed-off-by: Roman Pen Cc: Tejun Heo Cc: linux-kernel@vger.kernel.org Signed-off-by: Tejun Heo

percpu_ref: allow operation mode switching operations to be called concurrently

2016-08-10T19:02:58Z

percpu_ref initially didn't have explicit mode switching operations. It started out in percpu mode and switched to atomic mode on kill and then released. Ensuring that kill operation is initiated only after init completes was naturally the caller's responsibility. percpu_ref_reinit() was introduced later but it didn't shift the synchronization responsibility. Reinit can't be performed until kill is confirmed, so there was nothing to worry about synchronization-wise. Also, as both reinit and kill manipulate the base reference, invocations of the same function couldn't be allowed to race each other. The latest additions of percpu_ref_switch_to_atomic/percpu() changed the situation. These two functions can be called any time as long as the percpu_ref is between init and exit and thus there are valid valid usage scenarios where these new functions race with each other or against reinit/kill. Mostly from inertia, f47ad4578461 ("percpu_ref: decouple switching to percpu mode and reinit") still left synchronization among percpu mode switching operations to its users. That the new switch functions can be freely mixed with kill/reinit but the operations themselves should be synchronized is too subtle a requirement and led to a very subtle race condition in blk-mq freezing path. This patch fixes the situation by introducing percpu_ref_switch_lock to protect mode switching operations. This ensures that percpu-ref users don't have to worry about mode changing operations racing against each other, e.g. switch_to_percpu against kill, as long as the sequence of operations is valid. Signed-off-by: Tejun Heo Reported-by: Akinobu Mita Link: http://lkml.kernel.org/g/1443287365-4244-7-git-send-email-akinobu.mita@gmail.com Fixes: f47ad4578461 ("percpu_ref: decouple switching to percpu mode and reinit")

percpu_ref: restructure operation mode switching

2016-08-10T19:02:58Z

Restructure atomic/percpu mode switching. * The users of __percpu_ref_switch_to_atomic/percpu() now call a new function __percpu_ref_switch_mode() which calls either of the original switching functions depending on the current state of ref->force_atomic and the __PERCPU_REF_DEAD flag. The callers no longer check whether switching is necessary but always invoke __percpu_ref_switch_mode(). * !ref->confirm_switch waiting is collected into __percpu_ref_switch_mode(). This patch doesn't cause any behavior differences. Signed-off-by: Tejun Heo

percpu_ref: unify staggered atomic switching wait behavior

2016-08-10T19:02:58Z

When an atomic or percpu switching starts before the previous atomic switching finishes, the taken behaviors are * If the new atomic switching has confirmation callback, it waits for the previous atomic switching to complete. * If the new percpu switching is the first percpu switching following the previous atomic switching, it waits the previous atomic switching to complete. No percpu_ref user depends on these subtleties. The only meaningful part is that, if the caller ensures that atomic switching isn't in progress, mode switching operations can be issued from any context. This patch pulls the wait logic to the top of both switching functions so that they always wait for the previous atomic switching to complete. This makes the behavior simpler and consistent for both directions and will help allowing concurrent invocations of mode switching functions. Signed-off-by: Tejun Heo

percpu_ref: reorganize __percpu_ref_switch_to_atomic() and relocate percpu_ref_switch_to_atomic()

2016-08-10T19:02:58Z

Reorganize __percpu_ref_switch_to_atomic() so that it looks structurally similar to __percpu_ref_switch_to_percpu() and relocate percpu_ref_switch_to_atomic so that the two internal functions are co-located. This patch doesn't introduce any functional differences. Signed-off-by: Tejun Heo

percpu_ref: remove unnecessary RCU grace period for staggered atomic switching confirmation

2016-08-10T19:02:58Z

At the beginning, percpu_ref guaranteed a RCU grace period between a call to percpu_ref_kill_and_confirm() and the invocation of the confirmation callback. This guarantee exposed internal implementation details and got rescinded while switching over to sched RCU; however, __percpu_ref_switch_to_atomic() still inserts a full sched RCU grace period even when it can simply wait for the previous attempt. Remove the unnecessary grace period and perform the confirmation synchronously for staggered atomic switching attempts. Update comments accordingly. Signed-off-by: Tejun Heo

lib+mm: fix few spelling mistakes

2016-02-15T10:18:23Z

All are in comments. Signed-off-by: Bogdan Sikora Cc: Cc: Rafael Aquini Cc: Kent Overstreet Cc: Jan Kara [jkosina@suse.cz: more fixup] Acked-by: Rafael Aquini Signed-off-by: Jiri Slaby Signed-off-by: Jiri Kosina

percpu_ref: make INIT_ATOMIC and switch_to_atomic() sticky

2014-09-24T17:31:50Z

Currently, a percpu_ref which is initialized with PERPCU_REF_INIT_ATOMIC or switched to atomic mode via switch_to_atomic() automatically reverts to percpu mode on the first percpu_ref_reinit(). This makes the atomic mode difficult to use for cases where a percpu_ref is used as a persistent on/off switch which may be cycled multiple times. This patch makes such atomic state sticky so that it survives through kill/reinit cycles. After this patch, atomic state is cleared only by an explicit percpu_ref_switch_to_percpu() call. Signed-off-by: Tejun Heo Reviewed-by: Kent Overstreet Cc: Jens Axboe Cc: Christoph Hellwig Cc: Johannes Weiner

percpu_ref: add PERCPU_REF_INIT_* flags

2014-09-24T17:31:50Z

With the recent addition of percpu_ref_reinit(), percpu_ref now can be used as a persistent switch which can be turned on and off repeatedly where turning off maps to killing the ref and waiting for it to drain; however, there currently isn't a way to initialize a percpu_ref in its off (killed and drained) state, which can be inconvenient for certain persistent switch use cases. Similarly, percpu_ref_switch_to_atomic/percpu() allow dynamic selection of operation mode; however, currently a newly initialized percpu_ref is always in percpu mode making it impossible to avoid the latency overhead of switching to atomic mode. This patch adds @flags to percpu_ref_init() and implements the following flags. * PERCPU_REF_INIT_ATOMIC : start ref in atomic mode * PERCPU_REF_INIT_DEAD : start ref killed and drained These flags should be able to serve the above two use cases. v2: target_core_tpg.c conversion was missing. Fixed. Signed-off-by: Tejun Heo Reviewed-by: Kent Overstreet Cc: Jens Axboe Cc: Christoph Hellwig Cc: Johannes Weiner