user/sven/linux.git/include/linux/percpu.h, branch v3.18.22

percpu: implement asynchronous chunk population

2014-09-02T18:46:05Z

The percpu allocator now supports atomic allocations by only allocating from already populated areas but the mechanism to ensure that there's adequate amount of populated areas was missing. This patch expands pcpu_balance_work so that in addition to freeing excess free chunks it also populates chunks to maintain an adequate level of populated areas. pcpu_alloc() schedules pcpu_balance_work if the amount of free populated areas is too low or after an atomic allocation failure. * PERPCU_DYNAMIC_RESERVE is increased by two pages to account for PCPU_EMPTY_POP_PAGES_LOW. * pcpu_async_enabled is added to gate both async jobs - chunk->map_extend_work and pcpu_balance_work - so that we don't end up scheduling them while the needed subsystems aren't up yet. Signed-off-by: Tejun Heo

percpu: implement [__]alloc_percpu_gfp()

2014-09-02T18:46:04Z

Now that pcpu_alloc_area() can allocate only from populated areas, it's easy to add atomic allocation support to [__]alloc_percpu(). Update pcpu_alloc() so that it accepts @gfp and skips all the blocking operations and allocates only from the populated areas if @gfp doesn't contain GFP_KERNEL. New interface functions [__]alloc_percpu_gfp() are added. While this means that atomic allocations are possible, this isn't complete yet as there's no mechanism to ensure that certain amount of populated areas is kept available and atomic allocations may keep failing under certain conditions. Signed-off-by: Tejun Heo

percpu: move {raw|this}_cpu_*() definitions to include/linux/percpu-defs.h

2014-06-17T23:12:39Z

We're in the process of moving all percpu accessors and operations to include/linux/percpu-defs.h so that they're available to arch headers without having to include full include/linux/percpu.h which may cause cyclic inclusion dependency. This patch moves {raw|this}_cpu_*() definitions from include/linux/percpu.h to include/linux/percpu-defs.h. The code is moved mostly verbatim; however, raw_cpu_*() are placed above this_cpu_*() which is more conventional as the raw operations may be used to defined other variants. This is pure reorganization. Signed-off-by: Tejun Heo Acked-by: Christoph Lameter

percpu: move generic {raw|this}_cpu_*_N() definitions to include/asm-generic/percpu.h

2014-06-17T23:12:39Z

{raw|this}_cpu_*_N() operations are expected to be provided by archs and the generic definitions are provided as fallbacks. As such, these firmly belong to include/asm-generic/percpu.h. Move the generic definitions to include/asm-generic/percpu.h. The code is moved mostly verbatim; however, raw_cpu_*_N() are placed above this_cpu_*_N() which is more conventional as the raw operations may be used to defined other variants. This is pure reorganization. Signed-off-by: Tejun Heo Acked-by: Christoph Lameter

percpu: only allow sized arch overrides for {raw|this}_cpu_*() ops

2014-06-17T23:12:39Z

Currently, percpu allows two separate methods for overriding {raw|this}_cpu_*() ops - for a given operation, an arch can provide whole replacement or sized sub operations to override specific parts of it. e.g. arch either can provide this_cpu_add() or this_cpu_add_4() to override only the 4 byte operation. While quite flexible on a glance, the dual-overriding scheme complicates the code path for no actual gain. It compilcates the already complex operation definitions and if an arch wants to override all sizes, it can easily provide all variants anyway. In fact, no arch is actually making use of whole operation override. Another oddity is that __this_cpu_*() operations are defined in the same way as raw_cpu_*() but ignores full overrides of the raw_cpu_*() and doesn't allow full operation override, so if an arch provides whole overrides for raw_cpu_*() operations __this_cpu_*() ends up using the generic implementations. More importantly, it takes away the layering between arch-specific and generic parts making it impossible for the generic part to implement arch-independent features on top of arch-specific overrides. This patch removes the support for whole operation overrides. As no arch is using it, this doesn't cause any actual difference. Signed-off-by: Tejun Heo Acked-by: Christoph Lameter

percpu: move accessors from include/linux/percpu.h to percpu-defs.h

2014-06-17T23:12:34Z

include/linux/percpu-defs.h is gonna host all accessors and operations so that arch headers can make use of them too without worrying about circular dependency through include/linux/percpu.h. This patch moves the following accessors from include/linux/percpu.h to include/linux/percpu-defs.h. * get/put_cpu_var() * get/put_cpu_ptr() * per_cpu_ptr() This is pure reorgniazation. Signed-off-by: Tejun Heo Acked-by: Christoph Lameter

Merge branch 'for-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu

2014-06-09T21:56:07Z

Pull percpu updates from Tejun Heo: "Nothing too exciting. percpu_ref is going through some interface changes and getting new features with more changes in the pipeline but given its young age and few users, it's very low impact" * 'for-3.16' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/percpu: percpu-refcount: implement percpu_ref_tryget() percpu-refcount: rename percpu_ref_tryget() to percpu_ref_tryget_live() percpu: Replace __get_cpu_var with this_cpu_ptr

percpu: Fix raw_cpu_inc_return()

2014-05-14T16:46:10Z

The definition for raw_cpu_add_return() uses the operation prefix "raw_add_return_", but the definitions in the various percpu.h files expect "raw_cpu_add_return_". This commit therefore appropriately adjusts the definition of raw_cpu_add_return(). Signed-off-by: Paul E. McKenney Acked-by: Christoph Lameter Reviewed-by: Josh Triplett

percpu: Replace __get_cpu_var with this_cpu_ptr

2014-04-15T18:21:37Z

__this_cpu_ptr is being phased out. Use raw_cpu_ptr instead which was introduced in 3.15-rc1. One case of using __get_cpu_var in the get_cpu_var macro for address calculation was remaining in include/linux/percpu.h. tj: Updated patch description. Signed-off-by: Christoph Lameter Signed-off-by: Tejun Heo

percpu: add preemption checks to __this_cpu ops

2014-04-07T23:36:14Z

We define a check function in order to avoid trouble with the include files. Then the higher level __this_cpu macros are modified to invoke the preemption check. [akpm@linux-foundation.org: coding-style fixes] Signed-off-by: Christoph Lameter Acked-by: Ingo Molnar Cc: Tejun Heo Tested-by: Grygorii Strashko Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds