user/sven/linux.git/kernel/spinlock.c, branch v2.6.12.2

[PATCH] spin_unlock_bh() and preempt_check_resched()

2005-05-21T17:46:48Z

In _spin_unlock_bh(lock): do { \ _raw_spin_unlock(lock); \ preempt_enable(); \ local_bh_enable(); \ __release(lock); \ } while (0) there is no reason for using preempt_enable() instead of a simple preempt_enable_no_resched() Since we know bottom halves are disabled, preempt_schedule() will always return at once (preempt_count!=0), and hence preempt_check_resched() is useless here... This fixes it by using "preempt_enable_no_resched()" instead of the "preempt_enable()", and thus avoids the useless preempt_check_resched() just before re-enabling bottom halves. Signed-off-by: Samuel Thibault Signed-off-by: Linus Torvalds

[PATCH] break_lock fix

2005-03-28T11:51:53Z

lock->break_lock is set when a lock is contended, but cleared only in cond_resched_lock. Users of need_lockbreak (journal_commit_transaction, copy_pte_range, unmap_vmas) don't necessarily use cond_resched_lock on it. So, if the lock has been contended at some time in the past, break_lock remains set thereafter, and the fastpath keeps dropping lock unnecessarily. Hanging the system if you make a change like I did, forever restarting a loop before making any progress. And even users of cond_resched_lock may well suffer an initial unnecessary lockbreak. There seems to be no point at which break_lock can be cleared when unlocking, any point being either too early or too late; but that's okay, it's only of interest while the lock is held. So clear it whenever the lock is acquired - and any waiting contenders will quickly set it again. Additional locking overhead? well, this is only when CONFIG_PREEMPT is on. Since cond_resched_lock's spin_lock clears break_lock, no need to clear it itself; and use need_lockbreak there too, preferring optimizer to #ifdefs. Signed-off-by: Hugh Dickins Signed-off-by: Ingo Molnar Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds

[PATCH] nonintrusive spin-polling loop in kernel/spinlock.c

2005-01-20T07:28:52Z

This re-implements the nonintrusive spin-polling loop for the SMP+PREEMPT spinlock/rwlock variants, using the new *_can_lock() primitives. (The patch also adds *_can_lock() to the UP branch of spinlock.h, for completeness.) build- and boot-tested on x86 SMP+PREEMPT and SMP+!PREEMPT. Signed-off-by: Ingo Molnar Signed-off-by: Linus Torvalds

Revert "Don't busy-lock-loop in preemptable spinlocks" patch

2005-01-19T23:59:25Z

The locking tests were wrong, and the fixes are up in the air. In the meantime, the get-it-working patch is to just not do this. Cset exclude: mingo@elte.hu[torvalds]|ChangeSet|20050115174045|30241

[PATCH] Don't busy-lock-loop in preemptable spinlocks

2005-01-15T01:40:45Z

Paul Mackerras points out that doing the _raw_spin_trylock each time through the loop will generate tons of unnecessary bus traffic. Instead, after we fail to get the lock we should poll it with simple loads until we see that it is clear and then retry the atomic op. Assuming a reasonable cache design, the loads won't generate any bus traffic until another cpu writes to the cacheline containing the lock. Agreed. Signed-off-by: Ingo Molnar Signed-off-by: Linus Torvalds

[PATCH] improve preemption on SMP

2005-01-08T05:49:02Z

SMP locking latencies are one of the last architectural problems that cause millisec-category scheduling delays. CONFIG_PREEMPT tries to solve some of the SMP issues but there are still lots of problems remaining: spinlocks nested at multiple levels, spinning with irqs turned off, and non-nested spinning with preemption turned off permanently. The nesting problem goes like this: if a piece of kernel code (e.g. the MM or ext3's journalling code) does the following: spin_lock(&spinlock_1); ... spin_lock(&spinlock_2); ... then even with CONFIG_PREEMPT enabled, current kernels may spin on spinlock_2 indefinitely. A number of critical sections break their long paths by using cond_resched_lock(), but this does not break the path on SMP, because need_resched() *of the other CPU* is not set so cond_resched_lock() doesnt notice that a reschedule is due. to solve this problem i've introduced a new spinlock field, lock->break_lock, which signals towards the holding CPU that a spinlock-break is requested by another CPU. This field is only set if a CPU is spinning in a spinlock function [at any locking depth], so the default overhead is zero. I've extended cond_resched_lock() to check for this flag - in this case we can also save a reschedule. I've added the lock_need_resched(lock) and need_lockbreak(lock) methods to check for the need to break out of a critical section. Another latency problem was that the stock kernel, even with CONFIG_PREEMPT enabled, didnt have any spin-nicely preemption logic for the following, commonly used SMP locking primitives: read_lock(), spin_lock_irqsave(), spin_lock_irq(), spin_lock_bh(), read_lock_irqsave(), read_lock_irq(), read_lock_bh(), write_lock_irqsave(), write_lock_irq(), write_lock_bh(). Only spin_lock() and write_lock() [the two simplest cases] where covered. In addition to the preemption latency problems, the _irq() variants in the above list didnt do any IRQ-enabling while spinning - possibly resulting in excessive irqs-off sections of code! preempt-smp.patch fixes all these latency problems by spinning irq-nicely (if possible) and by requesting lock-breaks if needed. Two architecture-level changes were necessary for this: the addition of the break_lock field to spinlock_t and rwlock_t, and the addition of the _raw_read_trylock() function. Testing done by Mark H Johnson and myself indicate SMP latencies comparable to the UP kernel - while they were basically indefinitely high without this patch. i successfully test-compiled and test-booted this patch ontop of BK-curr using the following .config combinations: SMP && PREEMPT, !SMP && PREEMPT, SMP && !PREEMPT and !SMP && !PREEMPT on x86, !SMP && !PREEMPT and SMP && PREEMPT on x64. I also test-booted x86 with the generic_read_trylock function to check that it works fine. Essentially the same patch has been in testing as part of the voluntary-preempt patches for some time already. NOTE to architecture maintainers: generic_raw_read_trylock() is a crude version that should be replaced with the proper arch-optimized version ASAP. From: Hugh Dickins The i386 and x86_64 _raw_read_trylocks in preempt-smp.patch are too successful: atomic_read() returns a signed integer. Signed-off-by: Ingo Molnar Signed-off-by: Hugh Dickins Signed-off-by: Andrew Morton Signed-off-by: Linus Torvalds

[PATCH] Move __preempt_*lock into kernel_spinlock, clean up.

2004-09-10T00:13:40Z

- create in_lock_functions() to match in_sched_functions(). Export it for use in oprofile. - use char __lock_text_start[] instead of long __lock_text_start when declaring linker symbols. Rusty fixed a number of these a while ago based on advice from rth. - Move __preempt_*_lock into kernel/spinlock.c and make it inline. This means locks are only one deep. - Make in_sched_functions() check in_lock_functions() Signed-off-by: Anton Blanchard Signed-off-by: Linus Torvalds

[PATCH] out-of-line locks / generic

2004-09-04T09:18:26Z

This patch achieves out of line spinlocks by creating kernel/spinlock.c and using the _raw_* inline locking functions. Now, as much as this is supposed to be arch agnostic, there was still a fair amount of rummaging about in archs, mostly for the cases where the arch already has out of line locks and i wanted to avoid the extra call, saving that extra call also makes lock profiling easier. PPC32/64 was an example of such an arch and i have added the necessary profile_pc() function as an example. Size differences are with CONFIG_PREEMPT enabled since we wanted to determine how much could be saved by moving that lot out of line too. ppc64 = 259897 bytes: text data bss dec hex filename 5489808 1962724 709064 8161596 7c893c vmlinux-after 5749577 1962852 709064 8421493 808075 vmlinux-before sparc64 = 193368 bytes: text data bss dec hex filename 3472037 633712 308920 4414669 435ccd vmlinux-after 3665285 633832 308920 4608037 465025 vmlinux-before i386 = 416075 bytes text data bss dec hex filename 5808371 867442 326864 7002677 6ada35 vmlinux-after 6221254 870634 326864 7418752 713380 vmlinux-before x86-64 = 282446 bytes text data bss dec hex filename 4598025 1450644 523632 6572301 64490d vmlinux-after 4881679 1449436 523632 6854747 68985b vmlinux-before It has been compile tested (UP, SMP, PREEMPT) on i386, x86-64, sparc, sparc64, ppc64, ppc32 and runtime tested on i386, x86-64 and sparc64. Signed-off-by: Zwane Mwaikambo Signed-off-by: Linus Torvalds