admin/build-doc: use venv module again

Commit 690ca1c5628af629a25a80611c20e56095f13352 accidentally
re-introduced virtualenv. Switch back to the venv module in stdlib.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #42942 from zdover23/wip-doc-mgr-progress

doc/mgr: add progress module documentation

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42948 from llamerada-jp/fix-radosgw-admin-document

rgw: fix document

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42995 from rhcs-dashboard/fix-kcli-dashboard-e2e-doc

doc/dashboard: Fix kcli pool and network requirements documentation

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43004 from gregsfortytwo/wip-nodaemon

vstart: make "--crimson" imply "--nodaemon"

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43003 from gregsfortytwo/wip-error-code-value

crimson: osd: do not capture std::error_code by reference

Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

vstart: make "--crimson" imply "--nodaemon"

If you want to run multiple crimson-osd instances, you need nodaemon. Default
to nodaemon when using crimson.

Add a new "--crimson-foreground" which preserves the old behavior, in case
it's needed.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

crimson: osd: do not capture std::error_code by reference

I can't find any reason to assume it has a longer lifetime than
the future we're creating has.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #42831 from ifed01/wip-ifed-fix-52311

os/bluestore: fix using incomplete bluefs log when dumping it.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41912 from trociny/wip-crushdiff

tools/crushdiff: new tool to test crushmap change

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42985 from cyx1231st/wip-seastore-lba-metrics

crimson/os/seastore: misc improvements to metrics

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/dashboard: Fix kcli pool and network requirements documentation

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #42874 from cbodley/wip-51491

rgw: use existing s->bucket in s3 website retarget()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

crimson/os/seastore: drop Transaction::src_t::INIT category from metrics

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: measure insert/erase operations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: fix logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42973 from liewegas/doc-foundation-osnexus

doc/foundation: add osnexus

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42966 from cyx1231st/wip-seastore-lba-cleanup

crimson/os/seastore/lba_manager: misc cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42970 from sebastian-philipp/monitoring-custom-webhook

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

doc/foundation: add osnexus

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephadm: monitoring: Further Reading

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

.github/labeler: Add monitoring

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: Add "Setting up Grafana"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: move "deploying w/o" up

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: default placements

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42848 from ronen-fr/wip-ronenf-argv2vec-users

common: modify 'main()s' to use new argv_to_vec() signature

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42959 from idryomov/wip-krbd-escape-match-sysattr

krbd: escape udev_enumerate_add_match_sysattr values

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #42913 from yaarith/update-sudoers-smartctl

rpm, debian: move smartmontools and nvme-cli to ceph-base

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42770 from ktdreyer/no-lsb-release

Remove dependency on lsb_release

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

crimson/os/seastore/lba_manager: drop the unecessary lookup in leaf

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: drop the impossible case that the insert key exists after handle_split

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: end iterator should be pointing to the leaf-node size already

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: misc improvements on debugging

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: check depth when load lba node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: mark iterator(depth_t) private

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: cleanup LBABtree::iterate_repeat()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: drop unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42961 from tchaikov/wip-docker-test

test/docker-test.sh: add support for fedora 34 and debian stretch

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

test/docker-test.sh: add support for fedora 34 and debian strech

and drop fedora-31, fedora32, debian-jessie and ubuntu-16.04 since
they are not maintained anymore. see

- https://fedoraproject.org/wiki/End_of_life
- https://wiki.debian.org/LTS
- https://wiki.ubuntu.com/Releases

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

krbd: escape udev_enumerate_add_match_sysattr values

libudev uses fnmatch(3) for matching attributes, meaning that shell
glob pattern matching is employed instead of literal string matching.
Escape glob metacharacters to suppress pattern matching.

Fixes: https://tracker.ceph.com/issues/52425
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42957 from athanatos/sjust/wip-lba-rewrite-working

crimson: add object_info, log support to crimson-store-nbd, fix lba btree bugs

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore/lba_manager/btree: don't assert deltas are empty

We use the contents to update counters when transactions conflict,
deltas may validly be empty there.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: retire_extent_addr: add addr asserts

retire_extent_addr can only be called on absolute or record-relative
addrs.  Record-relative addrs are only valid on extents allocated as
part of the current transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: fix get_val() paddr value from iterator

This was causing a stray RETIRED_PLACEHOLDER to be created resulting in
a segment cleaner segfault in release on an invalid segment and a crash
upon adding a duplicate lba btree pin since the returned addr didn't
match the addr of the INITIAL_PENDING extent on the transaction.

Fixes: https://tracker.ceph.com/issues/52434
Fixes: https://tracker.ceph.com/issues/52435
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd: add log emulation to fs_driver

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/mgr: add progress module documentation

This PR ingests the work added to the documentation
in PR#29335.

The technical information in this PR concerns the
installation and use of the "Progress Module".

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Signed-off-by: kamoltat <ksirivad@redhat.com>

Merge PR #42899 into master

* refs/pull/42899/head:
src/client: fix dump mds twice

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42751 into master

* refs/pull/42751/head:
cephfs-mirror: shutdown ClusterWatcher on termination

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #42687 into master

* refs/pull/42687/head:
qa: test the "ms_mode" options in kclient workflows

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42620 into master

* refs/pull/42620/head:
mds: switch mds_lock to fair mutex
common/Timer: make SafeTimer a template
common/fair_mutex: add is_locked_by_me support
common: do not compile condition_variable_debug in none debug mode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38481 into master

* refs/pull/38481/head:
qa/vstart_runner: inherit methods instead of duplicating them
qa/ceph_manager: make it possible to reuse few methods
qa/vstart_runner: don't use "shell=False" in run_ceph_w()
qa/ceph_manager: minor refactor

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42955 from tchaikov/wip-fio-tag

cmake: use ceph repo with tag

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/tools/store_nbd: add object_info emulation to fs_driver

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: move int future refcounting to trace

We probably want to add a dedicated subsys for this, but for now it's just
too much for debug level.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd/fs_driver: collid should be the msb

This way, objects in a collection will sort together.  Probably
a typo.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39377 from ifed01/wip-ifed-fix-huge-omap-rename

os/bluestore: cap omap naming scheme upgrade transaction

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #42918 from ifed01/wip-ifed-fix-52398

test/store_test: fix broken test case

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42926 from zdover23/wip-doc-2021-08-26-nomad-cleanup-cherry-picker

doc: clean rbd-nomad.rst

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cmake: use ceph repo with tag

to address following warning:

fatal: reference is not a tree: 7f02f21f53ccd5e2448086f8e9015489693dd2dc
CMake Error at /home/jenkins-build/build/workspace/ceph-pull-requests/build/fio_ext-prefix/tmp/fio_ext-gitclone.cmake:40 (message):
  Failed to checkout tag: '7f02f21f53ccd5e2448086f8e9015489693dd2dc'

it seems that the shallow option does not work with a sha1 tag option,
let's continue using the ceph repo with a tag.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mailmap, organizationmap: update Mykola Golub affiliation

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>

qa/suites/rados: add crushdiff test

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>

Merge pull request #42944 from dengchl01/dengchl01

mgr/restful/api/crush.py: delete unused reference

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #39933 from linuxbox2/wip-bl-opslog

rgw: permit logging of list-bucket (and any other no-bucket op)

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42934 from tchaikov/wip-fio-cxx

cmake: use upstream repo for fio

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/rbd/rbd-nomad: fix indentation in config snippets

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: clean rbd-nomad.rst

This PR improves the rbd-nomad.rst file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw: fix document

add the description of --show-config option in radosgw-admin

Signed-off-by: Ushitora Anqou <ushitora@anqou.net>

Merge pull request #42901 from athanatos/sjust/wip-lba-rewrite

crimson/os/seastore: replace lba btree implementation with simpler, more correct iterator based implementation

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/restful/api/crush.py: delete unused reference

This commit delete unused reference of the crush page.

Signed-off-by: Charlie Developer <dengchl01@inspur.com>

Merge PR #42530 into master

* refs/pull/42530/head:
qa: ignore expected metadata cluster log error

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

crimson/os/seastore/lba_manager: replace btree implementation

Replace previous implementation with one based around an internal
iterator interface.  Besides simplifying the implementation and
removing duplicate lookups in the allocation pathway, this implementation
should correct a design problem in the prior implementation wherein
LBALeafNode::find_hole couldn't see the first element of the subsequent
node and therefore assumed that there was one at get_meta().end.

This patch removes the btree logic from lba_btree_node_impl.* leaving
the LBAInternalNode and LBALeafNode layout in lba_btree_node.*.
lba_btree.h/cc now have the main btree update/query logic.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #42941 from kamoltat/ksirivad-add-release-notes-quincy

PendingReleaseNotes: add autoscaler profile

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/os/seastore/seastore_types: add lba_root_t::set_depth

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: fix comment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add iterator operator-- and misc

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: remove unneeded TransactionManager::find_hole

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd/tm_driver: reset segment_manager in mkfs()

Otherwise, instantiating segment_manager in mount() results in two
instances and a duplicate metric exception.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: fix formatting in get_extent

In particular, this->invalidate and *ref.get() are superfluous.

Signed-off-by: Samuel Just <sjust@redhat.com>

key_value_store/kv_flat_btree_async: add using namespace std

Not worth annotating the specific elements of std used.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: get_next_dirty_extents: record in transaction read set

Record the extents in the read set after wait_io() as in get_extent.  This
should ensure that the interruptible_future machinery will handle the
event that one of them gets invalidated prior to beging rewritten.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager/block: add segment_size to make_superblock debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: add better merge test

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: update find_hole_assert_trigger

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/seastore:: add find_hole/add_pin assert test case

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/os/seastore: fix some clang capture warnings

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: fix update_usage call

Signed-off-by: Samuel Just <sjust@redhat.com>

PendingReleaseNotes: add autoscaler profile

Added autoscaler profile release notes for
>=17.0.0

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #42843 from majianpeng/pwl-ssd-restart-failed

librbd/cache/pwl/ssd: fix first_valid_entry calculation in retire_entries()

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42741 from sosodev/master

mgr/pg_autoscaler: add threshold module option

Merge pull request #42569 from ljflores/perf-histograms

mgr, mgr/telemetry: add access to osd commands in mgr and workload metrics to telemetry

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

cmake: use upstream repo for fio

this change partially reverts 10baab3fc8293b8c30ca90a4acd76f70d011f1b5,
but since the fix for C++ build is not included by any tag or branche so
far. let's just use the sha1 for now.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42858 from josephsawaya/wip-mgr-rook-osd-removal

mgr/rook: implementing OSD removal in the rook orchestrator

rgw: permit logging of list-bucket (and any other no-bucket op)

This limitation is from inception (2011), but just recently reported by
a site using ops-log for system audit.

Fixes: https://tracker.ceph.com/issues/49313
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #42869 from cbodley/wip-49955

qa/rgw: move mixed-version multisite tests out of upgrade suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42922 from dang/wip-dang-zipper-cleanup

RGW Zipper Cleanups

Reviewed-by: Soumya Kadouri <skoduri@redhat.com>

Merge pull request #42893 from jan--f/snap-sched-fix-multi-retention

snap-schedule: count retained snapshots per retention policy

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #42895 from ifed01/wip-ifed-improve-alloc-replay

test/allocator_replay_test: implement "try_alloc" command.

This tool has just limited manual  application when troubleshooting allocator's issues hence omitting QA run...

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

test/store_test: fix broken test case

After merging NCB stuff missed shared blob detection doesn't reveal
extent leak any more - which is expected. Hence we need to adjust the
relevant UT.

Fixes:https://tracker.ceph.com/issues/52398

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42916 from cyx1231st/wip-seastar-tree-metrics2

crimson/seastore: improve metrics about onode and lba trees

Reviewed-by: Samuel Just <sjust@redhat.com>