cmake: use ceph repo with tag

to address following warning:

fatal: reference is not a tree: 7f02f21f53ccd5e2448086f8e9015489693dd2dc
CMake Error at /home/jenkins-build/build/workspace/ceph-pull-requests/build/fio_ext-prefix/tmp/fio_ext-gitclone.cmake:40 (message):
  Failed to checkout tag: '7f02f21f53ccd5e2448086f8e9015489693dd2dc'

it seems that the shallow option does not work with a sha1 tag option,
let's continue using the ceph repo with a tag.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42944 from dengchl01/dengchl01

mgr/restful/api/crush.py: delete unused reference

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #39933 from linuxbox2/wip-bl-opslog

rgw: permit logging of list-bucket (and any other no-bucket op)

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42934 from tchaikov/wip-fio-cxx

cmake: use upstream repo for fio

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42901 from athanatos/sjust/wip-lba-rewrite

crimson/os/seastore: replace lba btree implementation with simpler, more correct iterator based implementation

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/restful/api/crush.py: delete unused reference

This commit delete unused reference of the crush page.

Signed-off-by: Charlie Developer <dengchl01@inspur.com>

Merge PR #42530 into master

* refs/pull/42530/head:
qa: ignore expected metadata cluster log error

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

crimson/os/seastore/lba_manager: replace btree implementation

Replace previous implementation with one based around an internal
iterator interface.  Besides simplifying the implementation and
removing duplicate lookups in the allocation pathway, this implementation
should correct a design problem in the prior implementation wherein
LBALeafNode::find_hole couldn't see the first element of the subsequent
node and therefore assumed that there was one at get_meta().end.

This patch removes the btree logic from lba_btree_node_impl.* leaving
the LBAInternalNode and LBALeafNode layout in lba_btree_node.*.
lba_btree.h/cc now have the main btree update/query logic.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #42941 from kamoltat/ksirivad-add-release-notes-quincy

PendingReleaseNotes: add autoscaler profile

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/os/seastore/seastore_types: add lba_root_t::set_depth

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: fix comment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add iterator operator-- and misc

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: remove unneeded TransactionManager::find_hole

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd/tm_driver: reset segment_manager in mkfs()

Otherwise, instantiating segment_manager in mount() results in two
instances and a duplicate metric exception.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: fix formatting in get_extent

In particular, this->invalidate and *ref.get() are superfluous.

Signed-off-by: Samuel Just <sjust@redhat.com>

key_value_store/kv_flat_btree_async: add using namespace std

Not worth annotating the specific elements of std used.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: get_next_dirty_extents: record in transaction read set

Record the extents in the read set after wait_io() as in get_extent.  This
should ensure that the interruptible_future machinery will handle the
event that one of them gets invalidated prior to beging rewritten.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager/block: add segment_size to make_superblock debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: add better merge test

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: update find_hole_assert_trigger

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/seastore:: add find_hole/add_pin assert test case

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/os/seastore: fix some clang capture warnings

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: fix update_usage call

Signed-off-by: Samuel Just <sjust@redhat.com>

PendingReleaseNotes: add autoscaler profile

Added autoscaler profile release notes for
>=17.0.0

Signed-off-by: Kamoltat <ksirivad@redhat.com>

Merge pull request #42843 from majianpeng/pwl-ssd-restart-failed

librbd/cache/pwl/ssd: fix first_valid_entry calculation in retire_entries()

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42741 from sosodev/master

mgr/pg_autoscaler: add threshold module option

Merge pull request #42569 from ljflores/perf-histograms

mgr, mgr/telemetry: add access to osd commands in mgr and workload metrics to telemetry

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

cmake: use upstream repo for fio

this change partially reverts 10baab3fc8293b8c30ca90a4acd76f70d011f1b5,
but since the fix for C++ build is not included by any tag or branche so
far. let's just use the sha1 for now.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42858 from josephsawaya/wip-mgr-rook-osd-removal

mgr/rook: implementing OSD removal in the rook orchestrator

rgw: permit logging of list-bucket (and any other no-bucket op)

This limitation is from inception (2011), but just recently reported by
a site using ops-log for system audit.

Fixes: https://tracker.ceph.com/issues/49313
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #42869 from cbodley/wip-49955

qa/rgw: move mixed-version multisite tests out of upgrade suite

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42922 from dang/wip-dang-zipper-cleanup

RGW Zipper Cleanups

Reviewed-by: Soumya Kadouri <skoduri@redhat.com>

Merge pull request #42893 from jan--f/snap-sched-fix-multi-retention

snap-schedule: count retained snapshots per retention policy

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #42895 from ifed01/wip-ifed-improve-alloc-replay

test/allocator_replay_test: implement "try_alloc" command.

This tool has just limited manual  application when troubleshooting allocator's issues hence omitting QA run...

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #42916 from cyx1231st/wip-seastar-tree-metrics2

crimson/seastore: improve metrics about onode and lba trees

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42917 from rzarzynski/wip-crimson-watch-list

crimson/osd: implement CEPH_OSD_OP_LIST_WATCHERS.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

RGW Zipper - chunk/stripe size are RadosStore only

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42886 from sebastian-philipp/prometheus-url-missing-slash

mgr/{prometheus,restful}: Fix url generation again

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #42921 from tchaikov/wip-crimson-cleanup

crimson/os: use structured binding in loop

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: use structured binding in loop

also avoid using `map[key] = val` for setting an item in map, as, if
the key does not exist in map, `map[key]` would have to create a value
using its default ctor, and then call the `operator=(bufferlist&&)` to
set it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rook: add better error handling to remove

This commit adds better error handling to the remove method
in the DefaultRemover by hiding stack traces from the user
and displaying meaningful error messages.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: use label selector to fetch pods and jobs in remover

This commit adds a label_selector to the fetching of the osd pods and osd prepare
jobs in the DefaultRemover.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

Merge pull request #42915 from AmnonHanuhov/wip-fix_omap_set_values

crimson/os: Use operator[] in CyanStore when inserting values to omap

librbd/cache/pwl/ssd: fix first_valid_entry calculation in retire_entries()

Consider one control_block which cotain multi encode(WriteLogCacheEntry):
Log1: WriteLogEntry
Log2: WriteLogEntry
Log3: Non-WriteLogEntry
For this case, currently calc method is: control_block_pos + sizeof(control_block).
But in fact, it should: control_block_pos + sizeof(control_block) +
data_length(Log1 + Log2).

Wrong first_valid_entry will persist to superblock and restart to read.
This cause read wrong position and when decode(WriteLogCacheEntry) it
will report bug.

Fixes: https://tracker.ceph.com/issues/52323
Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

crimson/osd: implement CEPH_OSD_OP_LIST_WATCHERS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42712 from idryomov/wip-malignant-noexcept

common: drop noexcept on thread entry points

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #42090 from cypherean/feedback

mgr/dashboard: report ceph tracker bug/feature through CLI/API

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #41918 from SudhanAruna/feature_parse

behave_tests: Implemented basic behave test scenario's for cephadm, ceph shell and OSD commands

crimson/seastore: improve metrics about onode and lba trees

Absolute value:
* tree_depth
By transaction source:
* tree_inserts_committed
* tree_inserts_invalidated
* tree_erases_committed
* tree_erase_invalidated

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os: Use operator[] in CyanStore when inserting values to object omap

std::map::insert() doesn't update the value associated with an existing
key, therefore we should use operator[] instead

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #42885 from cyx1231st/wip-seastar-tree-metrics

crimson/seastore: add metrics about onode and lba trees

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42907 from rzarzynski/wip-crimson-smphack

 crimson/osd: implicitly append '--smp 1' when invoked without it

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implicitly append '--smp 1' when invoked without it

This commit is basically a hack supposed to fulfil the obligation
of crimson being a drop-in replacement for the classical OSD
we already made by packaging it under `/usr/bin/ceph-osd`.
The discussion whether the interface-exactness should be continued
or not is out of scope of the commit; it's supposed just to handle
the issue unveiled by the Rook integration effort: `crimson-osd`
is unable to `--mkfs` because Seastar, if not restricted by passing
`--smp N`, considers all CPU cores available in the system when
allocating resources. This leads to the following error:

```
ERROR 2021-08-24 14:17:32,105 [shard 5] seastar - Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application
```

This hack will need to be dropped when integrating multi-reactor
support in crimson.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42910 from zdover23/doc-2021-08-25-doc-dev-config-verb-agreement-typo

doc/dev/config: fix verb disagreement

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42713 from ronen-fr/wip-ronenf-explicit-config

common: removing the explicit attribute from a public copy constructor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/dev/config: fix verb disagreement

This PR changes the following sentence:

   The other is to register an observer
   that will be called every time the
   relevant configuration values changes.

This sentence has been rewritten so that the
verb agrees with its subject, like this:

   The other is to register an observer
   that will be called every time the
   relevant configuration values change.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/dashboard: report ceph tracker bug/feature through CLI/API

Fixes: https://tracker.ceph.com/issues/44851
Signed-off-by: Shreya Sharma <shreyasharma.ss305@gmail.com>

Merge pull request #42806 from zdover23/wip-doc-2021-08-17-rbd-nomad-3-of-x

doc: rewrite configure ceph-csi to "conf. nomad"

Reviewed-by: Etienne Menguy <etienne.menguy@croit.io>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42845 from tchaikov/wip-51483

common/options: validate see-also

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

doc: rewrite configure ceph-csi to "conf. nomad"

This PR rewrites the sections
     - Configure Ceph-CSI
     - Configure Nomad

in the rbd-nomad.rst Chapter of the RBD
Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42857 from dsavineau/cephadm_tcmalloc_ceph_daemons_only

cephadm: Set tcmalloc env var for Ceph daemons

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42833 from oleeander/cephadm_openntpd

cephadm: check for openntpd.service as time sync service

Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42824 from pcuzner/cephadm-add-ports

cephadm:Add listening ports to gather-facts output

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #42699 from sebastian-philipp/cephadm-duplicate-ips

cephadm: list-networks: Avoid duplicated IPs

Reviewed-by: Francesco Pantano <fpantano@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #42405 from Daniel-Pivonka/cephadm_rm_offline_host

mgr/cephadm: add ability to remove offline host

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42805 from zdover23/wip-doc-2021-08-17-rbd-nomad-2-of-x

doc: improve "Create a Pool" in rbd-nomad.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cmake: s/Python_EXECUTABLE/Python3_EXECUTABLE/

as FindPython3.cmake only sets Python3_EXECUTABLE for us, we should
stick with Python3_EXECUTABLE instead of Python_EXECUTABLE.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: validate see-also

y2c.py is like a compiler which translates .yaml to .cc and .h files,
it does not have access to all .yaml files. to validate the dangling
see-also issue, we need to do this with a "linker".

in this change, validate-options.py is introduced to check if any of
option name included by the see-also property is valid.

Fixes: https://tracker.ceph.com/issues/51483
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: improve "Create a Pool" in rbd-nomad.rst

This PR improves the English in the "Create
a Pool" section of the "RBD & Nomad Integration"
chapter of the RBD Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42826 from tchaikov/wip-test-fair-mutex

test/common/test_fair_mutex: increase NR_ROUNDS from 256 to 512

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42881 from wjwithagen/wjw-fix-vstart-mgr

src/vstart.sh: put modules option for MGR in the [mgr] block.

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #42882 from cyx1231st/wip-seastar-cleanup

crimson/common: minor cleanups to futures

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42802 from Rethan/fix-rbd-bench-unit

rbd: avoid overflow of ios and clarify io-size limit for bench

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

snap-schedule: add test for multiple retention set calculation

Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

snap-schedule: count retained snapshots per retention policy

Before this commit the length of the already-retained snapshots
terminated the loop. This fails when muliple retention policies are set.

Fixes: https://tracker.ceph.com/issues/52388
Signed-off-by: Jan Fajerski <jfajersk@redhat.com>

behave_test: Implemented basic bahave test scenario's

Fixes: https://tracker.ceph.com/issues/52371
This commit includes the basic implementation of behave test scenario's
(for cephadm, ceph shell and OSD commands) and python implementations for
interacting with kcli and behave test cases. The test scenarios can be executed
using behave command. The files are created under src/test/behave_tests directory.

Signed-off-by: Arunagirinadan Sudharshan <sudhan.aruna.97@gmail.com>

Merge pull request #42841 from xxhdx1985126/wip-52305

crimson/common: keep ref count of crimson::interruptible::interrupt_cond

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

rbd: avoid overflow of ios and clarify io-size limit for bench

When doing rbd bench, we record done ios to print progress, current it's unsigned.
Suppose we do a bench of io-size 512B and io-total 4T, that means a total number of
8G ios which causes an overflow.
And we don't support io-size greater than 4G, so change help message.

Signed-off-by: haoyixing <haoyixing@kuaishou.com>

crimson/interruptible_future: remove unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/common: remove unused futurize::apply()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #42371 into master

* refs/pull/42371/head:
mgr/volumes: Fix a race during clone cancel
mgr/volumes: Fail subvolume removal if it's in progress

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #42884 from ideepika/wip-remove-unused-submodules

.gitmodules: remove thrift submodule

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41335 from noonedeadpunk/doc/stretch_mon

doc/rados/operations/stretch-mode: Adjust description of mon.e placement

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #42053 from TRYTOBE8TME/wip-rgw-dpp-4

rgw: DPP addition 5

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #42439 from ifed01/wip-ifed-more-errors-shared-blob-repair

os/bluestore: fix additional errors during missed shared blob repair.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

test/allocator_replay_test: implement "try_alloc" command.

This permits trying multiple allocations of chosen size on allocator
being at specific state (which is determined by free-dump report replay)

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #41569 from MrFreezeex/rbd-mirror-snap-perf-dump

rbd-mirror: add perf counters to snapshot replayer

Merge pull request #42871 from tchaikov/wip-cmake-exclude-grafonnet-lib

cmake: exclude "grafonnet-lib" target from "all"

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #42434 from zenomri/wip-omri-rgw-tracing

[WIP] -  RGW tracing implementation

Merge pull request #41380 from Devp00l/wip-rgw-multisite-doc

doc: Improve RGW multisite documentation

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Alex Settle <asettle@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42864 from aaryanporwal/vrt-doc

doc/dev: mgr/dashboard: Visual regression tests for ceph dashboard

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

src/vstart.sh: put modules option for MGR in the [mgr] block.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #42739 from Matan-B/lua-install-packages-fix

rgw: fix install packages function

Merge pull request #42865 from rhcs-dashboard/fix-rgw-bucket-list-api

mgr/dashboard: stats=false not working when listing buckets

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #42051 from melissa-kun-li/asyncssh

mgr/cephadm: replace execnet and remoto with asyncssh

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42804 from zdover23/wip-doc-2021-08-17-rbd-nomad-1-of-x

doc: improve RBD "Block Devices and Nomad"

Reviewed-by: Etienne Menguy <etienne.menguy@croit.io>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42879 from tchaikov/wip-seastar

seastar: pick up change silencing gcc warning

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: stats=false not working when listing buckets

Fixes: https://tracker.ceph.com/issues/51154
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

crimson/common: all interruptible future invocations should have already had interrupt condition in place

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: correct comment error about interruptible future

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>