crimson/osd: fix Watch::connect() behaviour on reconnect.

It's perfectly legal for a client to reconnect to particular `Watch`
using different socket / `Connection` than original one. This shall
include proper handling of the watch timer which is currently broken
as, when reconnecting, we don't cancel the timer. This leaded to the
following crash at Sepia:

```
rzarzynski@teuthology:/home/teuthworker/archive/rzarzynski-2021-09-02_07:44:51-rados-master-distro-basic-smithi/6372357$ less ./remote/smithi183/log/ceph-osd.4.log.gz
...
DEBUG 2021-09-02 08:10:45,462 [shard 0] osd - client_request(id=12, detail=m=[osd_op(client.5087.0:93 7.1e 7:7c7084bd:::repobj:head {watch reconnect cookie 94478891024832 gen 1} snapc 0={} ondisk+write+know
n_if_redirected e40) v8]): got obc lock
...
DEBUG 2021-09-02 08:10:45,462 [shard 0] osd - do_op_watch
INFO  2021-09-02 08:10:45,462 [shard 0] osd - found existing watch by client.5087
DEBUG 2021-09-02 08:10:45,462 [shard 0] osd - do_op_watch_subop_watch
INFO  2021-09-02 08:10:45,462 [shard 0] osd - found existing watch watch(cookie 94478891024832 30s 172.21.15.150:0/3544196211) by client.5087
...
INFO  2021-09-02 08:10:45,462 [shard 0] osd - op_effect: found existing watcher: 94478891024832,client.5087
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-7406-g9d30203c/rpm/el8/BUILD/ceph-
17.0.0-7406-g9d30203c/src/seastar/include/seastar/core/timer.hh:95: void seastar::timer<Clock>::arm_state(seastar::timer<Clock>::time_point, std::optional<typename Clock::duration>) [with Clock = seastar::l
owres_clock; seastar::timer<Clock>::time_point = std::chrono::time_point<seastar::lowres_clock, std::chrono::duration<long int, std::ratio<1, 1000> > >; typename Clock::duration = std::chrono::duration<long
 int, std::ratio<1, 1000> >]: Assertion `!_armed' failed.
Aborting on shard 0.
Backtrace:
 0# 0x000055CC052CF0B6 in ceph-osd
 1# FatalSignal::signaled(int, siginfo_t const&) in ceph-osd
 2# FatalSignal::install_oneshot_signal_handler<6>()::{lambda(int, siginfo_t*, void*)#1}::_FUN(int, siginfo_t*, void*) in ceph-osd
 3# 0x00007FA58349FB20 in /lib64/libpthread.so.0
 4# gsignal in /lib64/libc.so.6
 5# abort in /lib64/libc.so.6
 6# 0x00007FA581A98C89 in /lib64/libc.so.6
 7# 0x00007FA581AA6A76 in /lib64/libc.so.6
 8# 0x000055CC0BEEE9DD in ceph-osd
 9# crimson::osd::Watch::connect(seastar::shared_ptr<crimson::net::Connection>, bool) in ceph-osd
10# 0x000055CC00B1D246 in ceph-osd
11# 0x000055CBFFEF01AE in ceph-osd
...
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42942 from zdover23/wip-doc-mgr-progress

doc/mgr: add progress module documentation

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42948 from llamerada-jp/fix-radosgw-admin-document

rgw: fix document

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42995 from rhcs-dashboard/fix-kcli-dashboard-e2e-doc

doc/dashboard: Fix kcli pool and network requirements documentation

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43004 from gregsfortytwo/wip-nodaemon

vstart: make "--crimson" imply "--nodaemon"

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43003 from gregsfortytwo/wip-error-code-value

crimson: osd: do not capture std::error_code by reference

Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

vstart: make "--crimson" imply "--nodaemon"

If you want to run multiple crimson-osd instances, you need nodaemon. Default
to nodaemon when using crimson.

Add a new "--crimson-foreground" which preserves the old behavior, in case
it's needed.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

crimson: osd: do not capture std::error_code by reference

I can't find any reason to assume it has a longer lifetime than
the future we're creating has.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #42831 from ifed01/wip-ifed-fix-52311

os/bluestore: fix using incomplete bluefs log when dumping it.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41912 from trociny/wip-crushdiff

tools/crushdiff: new tool to test crushmap change

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42985 from cyx1231st/wip-seastore-lba-metrics

crimson/os/seastore: misc improvements to metrics

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/dashboard: Fix kcli pool and network requirements documentation

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #42874 from cbodley/wip-51491

rgw: use existing s->bucket in s3 website retarget()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

crimson/os/seastore: drop Transaction::src_t::INIT category from metrics

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: measure insert/erase operations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: fix logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42973 from liewegas/doc-foundation-osnexus

doc/foundation: add osnexus

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42966 from cyx1231st/wip-seastore-lba-cleanup

crimson/os/seastore/lba_manager: misc cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42970 from sebastian-philipp/monitoring-custom-webhook

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

doc/foundation: add osnexus

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephadm: monitoring: Further Reading

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

.github/labeler: Add monitoring

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: Add "Setting up Grafana"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: move "deploying w/o" up

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: default placements

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42848 from ronen-fr/wip-ronenf-argv2vec-users

common: modify 'main()s' to use new argv_to_vec() signature

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42959 from idryomov/wip-krbd-escape-match-sysattr

krbd: escape udev_enumerate_add_match_sysattr values

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #42913 from yaarith/update-sudoers-smartctl

rpm, debian: move smartmontools and nvme-cli to ceph-base

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42770 from ktdreyer/no-lsb-release

Remove dependency on lsb_release

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

crimson/os/seastore/lba_manager: drop the unecessary lookup in leaf

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: drop the impossible case that the insert key exists after handle_split

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: end iterator should be pointing to the leaf-node size already

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: misc improvements on debugging

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: check depth when load lba node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: mark iterator(depth_t) private

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: cleanup LBABtree::iterate_repeat()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: drop unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42961 from tchaikov/wip-docker-test

test/docker-test.sh: add support for fedora 34 and debian stretch

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

test/docker-test.sh: add support for fedora 34 and debian strech

and drop fedora-31, fedora32, debian-jessie and ubuntu-16.04 since
they are not maintained anymore. see

- https://fedoraproject.org/wiki/End_of_life
- https://wiki.debian.org/LTS
- https://wiki.ubuntu.com/Releases

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

krbd: escape udev_enumerate_add_match_sysattr values

libudev uses fnmatch(3) for matching attributes, meaning that shell
glob pattern matching is employed instead of literal string matching.
Escape glob metacharacters to suppress pattern matching.

Fixes: https://tracker.ceph.com/issues/52425
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42957 from athanatos/sjust/wip-lba-rewrite-working

crimson: add object_info, log support to crimson-store-nbd, fix lba btree bugs

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore/lba_manager/btree: don't assert deltas are empty

We use the contents to update counters when transactions conflict,
deltas may validly be empty there.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: retire_extent_addr: add addr asserts

retire_extent_addr can only be called on absolute or record-relative
addrs.  Record-relative addrs are only valid on extents allocated as
part of the current transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: fix get_val() paddr value from iterator

This was causing a stray RETIRED_PLACEHOLDER to be created resulting in
a segment cleaner segfault in release on an invalid segment and a crash
upon adding a duplicate lba btree pin since the returned addr didn't
match the addr of the INITIAL_PENDING extent on the transaction.

Fixes: https://tracker.ceph.com/issues/52434
Fixes: https://tracker.ceph.com/issues/52435
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd: add log emulation to fs_driver

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/mgr: add progress module documentation

This PR ingests the work added to the documentation
in PR#29335.

The technical information in this PR concerns the
installation and use of the "Progress Module".

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Signed-off-by: kamoltat <ksirivad@redhat.com>

Merge PR #42899 into master

* refs/pull/42899/head:
src/client: fix dump mds twice

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42751 into master

* refs/pull/42751/head:
cephfs-mirror: shutdown ClusterWatcher on termination

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #42687 into master

* refs/pull/42687/head:
qa: test the "ms_mode" options in kclient workflows

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42620 into master

* refs/pull/42620/head:
mds: switch mds_lock to fair mutex
common/Timer: make SafeTimer a template
common/fair_mutex: add is_locked_by_me support
common: do not compile condition_variable_debug in none debug mode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38481 into master

* refs/pull/38481/head:
qa/vstart_runner: inherit methods instead of duplicating them
qa/ceph_manager: make it possible to reuse few methods
qa/vstart_runner: don't use "shell=False" in run_ceph_w()
qa/ceph_manager: minor refactor

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42955 from tchaikov/wip-fio-tag

cmake: use ceph repo with tag

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/tools/store_nbd: add object_info emulation to fs_driver

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: move int future refcounting to trace

We probably want to add a dedicated subsys for this, but for now it's just
too much for debug level.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd/fs_driver: collid should be the msb

This way, objects in a collection will sort together.  Probably
a typo.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39377 from ifed01/wip-ifed-fix-huge-omap-rename

os/bluestore: cap omap naming scheme upgrade transaction

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #42918 from ifed01/wip-ifed-fix-52398

test/store_test: fix broken test case

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42926 from zdover23/wip-doc-2021-08-26-nomad-cleanup-cherry-picker

doc: clean rbd-nomad.rst

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cmake: use ceph repo with tag

to address following warning:

fatal: reference is not a tree: 7f02f21f53ccd5e2448086f8e9015489693dd2dc
CMake Error at /home/jenkins-build/build/workspace/ceph-pull-requests/build/fio_ext-prefix/tmp/fio_ext-gitclone.cmake:40 (message):
  Failed to checkout tag: '7f02f21f53ccd5e2448086f8e9015489693dd2dc'

it seems that the shallow option does not work with a sha1 tag option,
let's continue using the ceph repo with a tag.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mailmap, organizationmap: update Mykola Golub affiliation

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>

qa/suites/rados: add crushdiff test

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>

Merge pull request #42944 from dengchl01/dengchl01

mgr/restful/api/crush.py: delete unused reference

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #39933 from linuxbox2/wip-bl-opslog

rgw: permit logging of list-bucket (and any other no-bucket op)

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42934 from tchaikov/wip-fio-cxx

cmake: use upstream repo for fio

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/rbd/rbd-nomad: fix indentation in config snippets

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: clean rbd-nomad.rst

This PR improves the rbd-nomad.rst file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw: fix document

add the description of --show-config option in radosgw-admin

Signed-off-by: Ushitora Anqou <ushitora@anqou.net>

Merge pull request #42901 from athanatos/sjust/wip-lba-rewrite

crimson/os/seastore: replace lba btree implementation with simpler, more correct iterator based implementation

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/restful/api/crush.py: delete unused reference

This commit delete unused reference of the crush page.

Signed-off-by: Charlie Developer <dengchl01@inspur.com>

Merge PR #42530 into master

* refs/pull/42530/head:
qa: ignore expected metadata cluster log error

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

crimson/os/seastore/lba_manager: replace btree implementation

Replace previous implementation with one based around an internal
iterator interface.  Besides simplifying the implementation and
removing duplicate lookups in the allocation pathway, this implementation
should correct a design problem in the prior implementation wherein
LBALeafNode::find_hole couldn't see the first element of the subsequent
node and therefore assumed that there was one at get_meta().end.

This patch removes the btree logic from lba_btree_node_impl.* leaving
the LBAInternalNode and LBALeafNode layout in lba_btree_node.*.
lba_btree.h/cc now have the main btree update/query logic.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #42941 from kamoltat/ksirivad-add-release-notes-quincy

PendingReleaseNotes: add autoscaler profile

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/os/seastore/seastore_types: add lba_root_t::set_depth

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: fix comment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add iterator operator-- and misc

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: remove unneeded TransactionManager::find_hole

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd/tm_driver: reset segment_manager in mkfs()

Otherwise, instantiating segment_manager in mount() results in two
instances and a duplicate metric exception.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: fix formatting in get_extent

In particular, this->invalidate and *ref.get() are superfluous.

Signed-off-by: Samuel Just <sjust@redhat.com>

key_value_store/kv_flat_btree_async: add using namespace std

Not worth annotating the specific elements of std used.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: get_next_dirty_extents: record in transaction read set

Record the extents in the read set after wait_io() as in get_extent.  This
should ensure that the interruptible_future machinery will handle the
event that one of them gets invalidated prior to beging rewritten.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager/block: add segment_size to make_superblock debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: add better merge test

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: update find_hole_assert_trigger

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/seastore:: add find_hole/add_pin assert test case

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/os/seastore: fix some clang capture warnings

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: fix update_usage call

Signed-off-by: Samuel Just <sjust@redhat.com>

PendingReleaseNotes: add autoscaler profile

Added autoscaler profile release notes for
>=17.0.0

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #42843 from majianpeng/pwl-ssd-restart-failed

librbd/cache/pwl/ssd: fix first_valid_entry calculation in retire_entries()

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42741 from sosodev/master

mgr/pg_autoscaler: add threshold module option

Merge pull request #42569 from ljflores/perf-histograms

mgr, mgr/telemetry: add access to osd commands in mgr and workload metrics to telemetry

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

cmake: use upstream repo for fio

this change partially reverts 10baab3fc8293b8c30ca90a4acd76f70d011f1b5,
but since the fix for C++ build is not included by any tag or branche so
far. let's just use the sha1 for now.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42858 from josephsawaya/wip-mgr-rook-osd-removal

mgr/rook: implementing OSD removal in the rook orchestrator

rgw: permit logging of list-bucket (and any other no-bucket op)

This limitation is from inception (2011), but just recently reported by
a site using ops-log for system audit.

Fixes: https://tracker.ceph.com/issues/49313
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #42869 from cbodley/wip-49955

qa/rgw: move mixed-version multisite tests out of upgrade suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42922 from dang/wip-dang-zipper-cleanup

RGW Zipper Cleanups

Reviewed-by: Soumya Kadouri <skoduri@redhat.com>

Merge pull request #42893 from jan--f/snap-sched-fix-multi-retention

snap-schedule: count retained snapshots per retention policy

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #42895 from ifed01/wip-ifed-improve-alloc-replay

test/allocator_replay_test: implement "try_alloc" command.

This tool has just limited manual  application when troubleshooting allocator's issues hence omitting QA run...

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

test/store_test: fix broken test case

After merging NCB stuff missed shared blob detection doesn't reveal
extent leak any more - which is expected. Hence we need to adjust the
relevant UT.

Fixes:https://tracker.ceph.com/issues/52398

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42916 from cyx1231st/wip-seastar-tree-metrics2

crimson/seastore: improve metrics about onode and lba trees

Reviewed-by: Samuel Just <sjust@redhat.com>