crimson/os/seastore/lba_manager/btree/lba_btree: fix FTBFS on gcc 9

gcc-9 doesn't seem to consider interator nothrow move constructible with
the default move constructor implementation yielding the following build
failure:

m/el8/BUILD/ceph-17.0.0-7373-gfc349212/src/seastar/include/seastar/core/future.hh:584:58: error: static assertion failed: Types must be no-throw move constructible
  584 |     static_assert(std::is_nothrow_move_constructible<T>::value,

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #43004 from gregsfortytwo/wip-nodaemon

vstart: make "--crimson" imply "--nodaemon"

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43003 from gregsfortytwo/wip-error-code-value

crimson: osd: do not capture std::error_code by reference

Reviewed-by: Xuehan Xu <xuxuehan@360.cn>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

vstart: make "--crimson" imply "--nodaemon"

If you want to run multiple crimson-osd instances, you need nodaemon. Default
to nodaemon when using crimson.

Add a new "--crimson-foreground" which preserves the old behavior, in case
it's needed.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

crimson: osd: do not capture std::error_code by reference

I can't find any reason to assume it has a longer lifetime than
the future we're creating has.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #42831 from ifed01/wip-ifed-fix-52311

os/bluestore: fix using incomplete bluefs log when dumping it.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41912 from trociny/wip-crushdiff

tools/crushdiff: new tool to test crushmap change

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42985 from cyx1231st/wip-seastore-lba-metrics

crimson/os/seastore: misc improvements to metrics

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42874 from cbodley/wip-51491

rgw: use existing s->bucket in s3 website retarget()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

crimson/os/seastore: drop Transaction::src_t::INIT category from metrics

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: measure insert/erase operations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: fix logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42973 from liewegas/doc-foundation-osnexus

doc/foundation: add osnexus

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42966 from cyx1231st/wip-seastore-lba-cleanup

crimson/os/seastore/lba_manager: misc cleanup

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42970 from sebastian-philipp/monitoring-custom-webhook

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

doc/foundation: add osnexus

Signed-off-by: Sage Weil <sage@newdream.net>

doc/cephadm: monitoring: Further Reading

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

.github/labeler: Add monitoring

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: Add "Adding Alertmanager webhooks"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: Add "Setting up Grafana"

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: move "deploying w/o" up

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: monitoring: default placements

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42848 from ronen-fr/wip-ronenf-argv2vec-users

common: modify 'main()s' to use new argv_to_vec() signature

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42959 from idryomov/wip-krbd-escape-match-sysattr

krbd: escape udev_enumerate_add_match_sysattr values

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #42913 from yaarith/update-sudoers-smartctl

rpm, debian: move smartmontools and nvme-cli to ceph-base

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42770 from ktdreyer/no-lsb-release

Remove dependency on lsb_release

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

crimson/os/seastore/lba_manager: drop the unecessary lookup in leaf

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: drop the impossible case that the insert key exists after handle_split

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: end iterator should be pointing to the leaf-node size already

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: misc improvements on debugging

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: check depth when load lba node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: mark iterator(depth_t) private

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: cleanup LBABtree::iterate_repeat()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: drop unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42961 from tchaikov/wip-docker-test

test/docker-test.sh: add support for fedora 34 and debian stretch

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

test/docker-test.sh: add support for fedora 34 and debian strech

and drop fedora-31, fedora32, debian-jessie and ubuntu-16.04 since
they are not maintained anymore. see

- https://fedoraproject.org/wiki/End_of_life
- https://wiki.debian.org/LTS
- https://wiki.ubuntu.com/Releases

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

krbd: escape udev_enumerate_add_match_sysattr values

libudev uses fnmatch(3) for matching attributes, meaning that shell
glob pattern matching is employed instead of literal string matching.
Escape glob metacharacters to suppress pattern matching.

Fixes: https://tracker.ceph.com/issues/52425
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42957 from athanatos/sjust/wip-lba-rewrite-working

crimson: add object_info, log support to crimson-store-nbd, fix lba btree bugs

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore/lba_manager/btree: don't assert deltas are empty

We use the contents to update counters when transactions conflict,
deltas may validly be empty there.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: retire_extent_addr: add addr asserts

retire_extent_addr can only be called on absolute or record-relative
addrs.  Record-relative addrs are only valid on extents allocated as
part of the current transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: fix get_val() paddr value from iterator

This was causing a stray RETIRED_PLACEHOLDER to be created resulting in
a segment cleaner segfault in release on an invalid segment and a crash
upon adding a duplicate lba btree pin since the returned addr didn't
match the addr of the INITIAL_PENDING extent on the transaction.

Fixes: https://tracker.ceph.com/issues/52434
Fixes: https://tracker.ceph.com/issues/52435
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd: add log emulation to fs_driver

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #42899 into master

* refs/pull/42899/head:
src/client: fix dump mds twice

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42751 into master

* refs/pull/42751/head:
cephfs-mirror: shutdown ClusterWatcher on termination

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #42687 into master

* refs/pull/42687/head:
qa: test the "ms_mode" options in kclient workflows

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #42620 into master

* refs/pull/42620/head:
mds: switch mds_lock to fair mutex
common/Timer: make SafeTimer a template
common/fair_mutex: add is_locked_by_me support
common: do not compile condition_variable_debug in none debug mode

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38481 into master

* refs/pull/38481/head:
qa/vstart_runner: inherit methods instead of duplicating them
qa/ceph_manager: make it possible to reuse few methods
qa/vstart_runner: don't use "shell=False" in run_ceph_w()
qa/ceph_manager: minor refactor

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42955 from tchaikov/wip-fio-tag

cmake: use ceph repo with tag

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/tools/store_nbd: add object_info emulation to fs_driver

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: move int future refcounting to trace

We probably want to add a dedicated subsys for this, but for now it's just
too much for debug level.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd/fs_driver: collid should be the msb

This way, objects in a collection will sort together.  Probably
a typo.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39377 from ifed01/wip-ifed-fix-huge-omap-rename

os/bluestore: cap omap naming scheme upgrade transaction

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #42918 from ifed01/wip-ifed-fix-52398

test/store_test: fix broken test case

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42926 from zdover23/wip-doc-2021-08-26-nomad-cleanup-cherry-picker

doc: clean rbd-nomad.rst

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cmake: use ceph repo with tag

to address following warning:

fatal: reference is not a tree: 7f02f21f53ccd5e2448086f8e9015489693dd2dc
CMake Error at /home/jenkins-build/build/workspace/ceph-pull-requests/build/fio_ext-prefix/tmp/fio_ext-gitclone.cmake:40 (message):
  Failed to checkout tag: '7f02f21f53ccd5e2448086f8e9015489693dd2dc'

it seems that the shallow option does not work with a sha1 tag option,
let's continue using the ceph repo with a tag.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mailmap, organizationmap: update Mykola Golub affiliation

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>

qa/suites/rados: add crushdiff test

Signed-off-by: Mykola Golub <mykola.golub@clyso.com>

Merge pull request #42944 from dengchl01/dengchl01

mgr/restful/api/crush.py: delete unused reference

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #39933 from linuxbox2/wip-bl-opslog

rgw: permit logging of list-bucket (and any other no-bucket op)

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42934 from tchaikov/wip-fio-cxx

cmake: use upstream repo for fio

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/rbd/rbd-nomad: fix indentation in config snippets

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc: clean rbd-nomad.rst

This PR improves the rbd-nomad.rst file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42901 from athanatos/sjust/wip-lba-rewrite

crimson/os/seastore: replace lba btree implementation with simpler, more correct iterator based implementation

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/restful/api/crush.py: delete unused reference

This commit delete unused reference of the crush page.

Signed-off-by: Charlie Developer <dengchl01@inspur.com>

Merge PR #42530 into master

* refs/pull/42530/head:
qa: ignore expected metadata cluster log error

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

crimson/os/seastore/lba_manager: replace btree implementation

Replace previous implementation with one based around an internal
iterator interface.  Besides simplifying the implementation and
removing duplicate lookups in the allocation pathway, this implementation
should correct a design problem in the prior implementation wherein
LBALeafNode::find_hole couldn't see the first element of the subsequent
node and therefore assumed that there was one at get_meta().end.

This patch removes the btree logic from lba_btree_node_impl.* leaving
the LBAInternalNode and LBALeafNode layout in lba_btree_node.*.
lba_btree.h/cc now have the main btree update/query logic.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #42941 from kamoltat/ksirivad-add-release-notes-quincy

PendingReleaseNotes: add autoscaler profile

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/os/seastore/seastore_types: add lba_root_t::set_depth

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: fix comment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add iterator operator-- and misc

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: remove unneeded TransactionManager::find_hole

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd/tm_driver: reset segment_manager in mkfs()

Otherwise, instantiating segment_manager in mount() results in two
instances and a duplicate metric exception.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: fix formatting in get_extent

In particular, this->invalidate and *ref.get() are superfluous.

Signed-off-by: Samuel Just <sjust@redhat.com>

key_value_store/kv_flat_btree_async: add using namespace std

Not worth annotating the specific elements of std used.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: get_next_dirty_extents: record in transaction read set

Record the extents in the read set after wait_io() as in get_extent.  This
should ensure that the interruptible_future machinery will handle the
event that one of them gets invalidated prior to beging rewritten.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager/block: add segment_size to make_superblock debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: add better merge test

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: update find_hole_assert_trigger

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/seastore:: add find_hole/add_pin assert test case

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/os/seastore: fix some clang capture warnings

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: fix update_usage call

Signed-off-by: Samuel Just <sjust@redhat.com>

PendingReleaseNotes: add autoscaler profile

Added autoscaler profile release notes for
>=17.0.0

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #42843 from majianpeng/pwl-ssd-restart-failed

librbd/cache/pwl/ssd: fix first_valid_entry calculation in retire_entries()

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42741 from sosodev/master

mgr/pg_autoscaler: add threshold module option

Merge pull request #42569 from ljflores/perf-histograms

mgr, mgr/telemetry: add access to osd commands in mgr and workload metrics to telemetry

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

cmake: use upstream repo for fio

this change partially reverts 10baab3fc8293b8c30ca90a4acd76f70d011f1b5,
but since the fix for C++ build is not included by any tag or branche so
far. let's just use the sha1 for now.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42858 from josephsawaya/wip-mgr-rook-osd-removal

mgr/rook: implementing OSD removal in the rook orchestrator

rgw: permit logging of list-bucket (and any other no-bucket op)

This limitation is from inception (2011), but just recently reported by
a site using ops-log for system audit.

Fixes: https://tracker.ceph.com/issues/49313
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #42869 from cbodley/wip-49955

qa/rgw: move mixed-version multisite tests out of upgrade suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42922 from dang/wip-dang-zipper-cleanup

RGW Zipper Cleanups

Reviewed-by: Soumya Kadouri <skoduri@redhat.com>

Merge pull request #42893 from jan--f/snap-sched-fix-multi-retention

snap-schedule: count retained snapshots per retention policy

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #42895 from ifed01/wip-ifed-improve-alloc-replay

test/allocator_replay_test: implement "try_alloc" command.

This tool has just limited manual  application when troubleshooting allocator's issues hence omitting QA run...

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

test/store_test: fix broken test case

After merging NCB stuff missed shared blob detection doesn't reveal
extent leak any more - which is expected. Hence we need to adjust the
relevant UT.

Fixes:https://tracker.ceph.com/issues/52398

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #42916 from cyx1231st/wip-seastar-tree-metrics2

crimson/seastore: improve metrics about onode and lba trees

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42917 from rzarzynski/wip-crimson-watch-list

crimson/osd: implement CEPH_OSD_OP_LIST_WATCHERS.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

RGW Zipper - chunk/stripe size are RadosStore only

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42886 from sebastian-philipp/prometheus-url-missing-slash

mgr/{prometheus,restful}: Fix url generation again

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #42921 from tchaikov/wip-crimson-cleanup

crimson/os: use structured binding in loop

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: use structured binding in loop

also avoid using `map[key] = val` for setting an item in map, as, if
the key does not exist in map, `map[key]` would have to create a value
using its default ctor, and then call the `operator=(bufferlist&&)` to
set it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rook: add better error handling to remove

This commit adds better error handling to the remove method
in the DefaultRemover by hiding stack traces from the user
and displaying meaningful error messages.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>