Add "wait morphing" to improve condvar wake performance

[nyh]

The existing codvar_wake_one()/all() code wakes up one/all of the threads sleeping on the mutex, and they all wake up and try to take the user mutex associated with the condvar.

This creates two performance problems:

1. The "thundering herd" problem - When there are many threads waiting on the condvar and they are all woken with condvar_wake_all(), all these threads start to run (requiring a lot of work on scheduling, IPIs, etc.) but all of them except one will immediately go back to sleep waiting to take the user mutex. See a description of this problem and how it was solved in Linux in https://lwn.net/Articles/32746/.
2. The "locked wakeup" problem - Often, condvar_wake is done with the user mutex locked (see a good discussion in http://www.domaigne.com/blog/computing/condvars-signal-with-mutex-locked-or-not/). Most of the BSD code we use does this, for example. When condvar_wake() wakes the thread, if it wakes up quickly enough, it can try to take the mutex before the waker released it, and need to go back to sleep. This problem is especially noticable when both threads run on a single CPU, and the scheduler decides to context-switch to the wakee when it is woken.

Both problems can be solved by a technique called wait morphing: when the condvar has an associated mutex, condvar_wake_*() does not wake up the threads, but rather move them from waiting on the condition variable, to waiting on the mutex. Each of these threads will later be woken (separately) when the mutex becomes available for it, and when it wakes up it doesn't need to take the mutex, because it was already taken for it.

It is not clear that this issue actually has a lot of performance impact on any of our use cases (once the scheduler bugs are eliminated), but it seems there's popular demand for solving it.

[nyh]

Here are some tentative implementation ideas for the convar/mutex wait morphing:

1. Probably we should no longer support the old spin-based mutex implementation. Doing the following changes to both it and the newer lockfree::mutex will be a waste of time.
2. Condvar and mutex each uses a "wait record" structure that contain a link to the next record, and a thread pointer. In mutex it is linked_itemsched::thread*, in condvar it is struct ccondvar_waiter. Both of them already have the same content, but we now need them to be the same type, so convert them both to use linked_itemsched::thread* (and create an alias for it, "struct waitrrecord", pointers to which we need to use in the C-language condvar.h).
3. Add to the waitrecord type a method wake(), which does wr->value->wake_with([&] { wr->value = nullptr; }). This is the idiom we always use to wake up a waiter (in both mutex and condvar) and it will be nice to encapsulate it in a method.
4. Add the assumption that when a condvar is waited on concurrently, all uses must wait with the same mutex or the results will be undefined. pthread_cond_wait(3) also makes this assumption. This way, it will be enough to remember the last mutex associated with the condvar, and we won't need to remember the mutex associate with each waitrecord. This assumption can be relatively easily lifted, but I think it's not worth the extra effort.
5. Add a new mutex operation, mutex->lockfor(waitrecord *wr). It will be similar to lock(), but rather than taking the mutex for the current thread it will take it for a different thread which is already waiting on the given waitrecord. lockfor(wr) will attempt to take the lock immediately, and if successful will wake up the given waitrecord (wr->wake() explained above), but if can't get the lock it will add the given wr (not some local wr like in lock()) to the mutex's queue.
6. Now, condvar->wake_one/wake_all, for each wr it pops from the condvar's wait queue, instead of waking up the waitrecord as we do now, will first check if the condvar has an associate mutex. If it does, it will call mutex->lockfor(wr) and not wake up wr - it will be woken either now or later when it gets the mutex. Note how in a condvar_wake_all() the first popped wr might have a chance to be woken up immediately, but the rest will all just be queued in the mutex's wait queue and not cause any context switches.
7. When the condvar_wait() is woken up from its wait-record (the wait_until() in condvar_wait) it now knows that it already has the user mutex! So it doesn't need to get it again.
8. The case of timeout is somewhat special - a timeout that occurs while still waiting on the condition variable will work just as it does now. But the mutex grabbing cannot time out - a condvar_timed_wait will wait forever for the mutex (just like any mutex->lock()) does. I think this is fine, but we need to consider this.

[nyh]

Done, in commit aa3a624 and adjacent commits.