This is difficult to break into pieces, so one big fat commit it is.
A few trivial bits
- include epoch in PGQueueable.
- PGQueuable operator<<
- remove op_wq ref from OSDService; use simple set of queue methods instead
The big stuff:
- Fast dispatch now passes messages directly to the queue based on an
spg_t. The exception is MOSDOp's from legacy clients. We add a
waiting_for_map mechanism on the front-side that is similar to but simpler
than the previous one so that we can map those legacy requests to an
accurate spg_t.
- The dequeue path now has a waiting_for_pg mechanism. It also uses a
much simpler set of data structures that should make it much faster than
the previous incarnation.
- Shutdown works a bit differently; we drain the queue instead of trying
to remove work for individual PGs. This lets us remove the dequeue_pg
machinery.