common: resolve config proxy deadlock using refcounted pointers
7e8c683 introduced some gymnastics with a "CallGate" to maintain a count for
each observer we may be "calling into" with a config change (namely:
handle_conf_change). This was to prevent remove_observer coming in and deleting
the observer in the middle of the call. More importantly, it was to avoid
holding the lock while traversing the observers so that the config_proxy lock
can be dropped while calling handle_conf_change. This is important as e.g. the
MDS may attempt to acquire the config_proxy lock in its
MDSRank::handle_conf_change method (what prompted the change).
However, this introduces a new deadlock:
- Thread 2 acquires the config_proxy lock and then removes an observer. It blocks
waiting for the observer's CallGate to close.
- Thread 1 had dropped the config_proxy lock while traversing the observers to call each
observer's handle_conf_change method. Those methods may attempt to reacquire the
config_proxy lock. This creates the deadlock as it's waiting for Thread 2 to drop the lock
while Thread 1 cannot release the CallGate.
The solution, I believe, is to properly refcount "uses" of the observers for the purposes
of flushing these changes. Use std::shared_ptr to effect this.
Reproducing this is fairly simply with several parallel calls to `config set`.
During the course of executing `config set`, the Objecter may receive config
updates that will be flushed and potentially race with cleanup of observers
during shutdown.
Fixes: https://tracker.ceph.com/issues/62832
Partial-revert:
7e8c683
Partial-revert:
4458a72
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit
0c70dd8e39cc3d0cdef8bbcc8a0c6f214e54c770)