Merge PR #41821 into master

* refs/pull/41821/head:
qa: specify distro for fs:bugs

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #41773 into master

* refs/pull/41773/head:
test/pybind/test_ceph_argparse: add a test where args contains comma
cmake: run nose test using "python3 -m nose"
test/pybind/test_ceph_argparse: check for expected argument
test/pybind/test_ceph_argparse: decorate helper method with @nottest
test/pybind: do not test obsoleted command
test/pybind: drop forward and readforward from test
test/pybind: do not use tab for indent
pybind/ceph_argparse: validate csv if desc.N
test/pybind: pep8 cleanups

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #41771 into master

* refs/pull/41771/head:
qa: update scrub start code to use comma sep scrubopts

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41864 from rzarzynski/wip-crimson-iointrcond-fix-debug

crimson/osd: fix debug in IOInterruptCondition::new_interval_created().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41865 from rzarzynski/wip-crimson-opseq-fix-dropped-abort

crimson/osd: fix missed OpSequencer::abort() on primary change.

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix missed OpSequencer::abort() on primary change.

Commit b5efdc6f1c9563357d7dfd33a8f379053592a215 has unified
the interruption handling among `InternalClientRequest` and
`ClientRequest`. Unfortunately, a call to `abort()` of
`OpSequencer` has been overlooked and dropped. This was causing
crashes at Sepia after after fixing the ordering issues in
`ClientRequest`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix debug in IOInterruptCondition::new_interval_created().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41801 from liu-chunmei/seastore-fix-get-retired-extent

crimson/seastore: fix cache::get_extent got retired extent

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41854 from sebastian-philipp/fix-mypy-yaml-representation

mgr/orch: fix mypy errors

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #41846 from batrick/i51214

doc: fix typo in option

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>

pyhton-common: fix mypy errors

Fixes:

```
py3 run-test: commands[2] | mypy --config-file=../mypy.ini -p ceph
ceph/deployment/service_spec.py: note: In member "yaml_representer" of class "ServiceSpec":
ceph/deployment/service_spec.py:659: error: Argument 1 to "represent_dict" of "SafeRepresenter" has incompatible type "_OrderedDictItemsView[str, Any]"; expected "Mapping[Any, Any]"
```

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/orch: fix mypy errors

Fixes:

```
orchestrator/__init__.py:6: note: In module imported here:
orchestrator/_interface.py: note: In member "yaml_representer" of class "DaemonDescription":
orchestrator/_interface.py:1039: error: Argument 1 to "represent_dict" of "SafeRepresenter" has incompatible type "ItemsView[Any, Any]"; expected "Mapping[Any, Any]"
orchestrator/_interface.py: note: In member "yaml_representer" of class "ServiceDescription":
orchestrator/_interface.py:1178: error: Argument 1 to "represent_dict" of "SafeRepresenter" has incompatible type "ItemsView[Any, Any]"; expected "Mapping[Any, Any]"
orchestrator/_interface.py: note: At top level:
orchestrator/_interface.py:1181: error: Argument 2 to "add_representer" has incompatible type "Callable[[SafeDumper, DaemonDescription], Any]"; expected "Callable[[SafeDumper, ServiceDescription], Node]"
Found 3 errors in 1 file (checked 29 source files)
```

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41841 from ljflores/patch-1

doc: update testing.rst and build-integration test to say ./run-make-check.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/seastore: fix cache::get_extent got retired extent

one transaction got an extent whose state is MUTATION_PENDINGat that time.
but another transaction do split and set the extent state to RETIRED.
when the first transaction resume and do continuation, the state of the extent
has been changed to RETIRED. So need eagain to try again.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

doc: fix typo in option

Fixes: https://tracker.ceph.com/issues/51214
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

script/build-integration-branch: update print statement based on update to documentation

The print statement originally suggested to the user to run "make && ctest -j12", but that is outdated. The user should execute "./run-make-check.sh" instead.

Signed-off-by: Laura Flores <lflores@redhat.com>

doc/dev: should be ./run-make-check.sh

Updating the initial pull request. The smoke test line should read "./run-make-check.sh" instead of "make && ctest -j12".

Signed-off-by: Laura Flores <lflores@redhat.com>

doc/dev: update testing.rst to say ninja

The current version of Ceph uses "ninja" rather than "make," so the testing documentation needs to be updated.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge PR #41799 into master

* refs/pull/41799/head:
cephadm: validate `--fsid` during bootstrap

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #41805 into master

* refs/pull/41805/head:
cephadm: set TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=134217728

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #39624 from sebastian-philipp/mypy-812

src,qa: Upgrade to mypy 0.901

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41632 from jmolmo/fix_osd_rm_help_msg

mgr/cephadm: fix errors in command <ceph orch osd> help and documentation

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41556 from adk3798/daemon-count

mgr/cephadm: fix --service-type flag in orch ls when service has id

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #41827 into master

* refs/pull/41827/head:
qa: move dashboard e2e from cephadm -> rados suite

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41796 from rzarzynski/wip-crimson-cltreq-dbgs

crimson/osd: improve debugs in ClientRequest

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: improve debugs in OpSequencer.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: should_abort_request() dumps Operation's details.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: print prev_op_id in ClientRequest's debugs.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41832 from inspur-wyq/patch-4

os/ObjectStore: fix typo in comments

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41786 into master

* refs/pull/41786/head:
mon/MonCommands: remove obsolete mds commands
mon/MonCommands: remove obsolete mon commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41580 into master

* refs/pull/41580/head:
cephfs: make ceph_fallocate() platform independent

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41482 into master

* refs/pull/41482/head:
qa: remove obsolete deactivate routines

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #41481 into master

* refs/pull/41481/head:
client: Add ceph.caps vxattr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #41431 into master

* refs/pull/41431/head:
mds: progress the recover queue immediately after the inode is enqueued
mds: switch to mdr->get_client()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41422 into master

* refs/pull/41422/head:
qa/tasks/cephfs/test_sessionmap: reap connections immediately
msg/async: configurable threshold for reaping dead connections

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41403 into master

* refs/pull/41403/head:
mgr/volumes: Add config to insert delay at the beginning of the clone

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

os/ObjectStore: fix typo in comments

Signed-off-by: wangyunqing <wangyunqing@inspur.com>

test/pybind/test_ceph_argparse: add a test where args contains comma

to ensure the support for the new syntax of "prefix --name bon,jour"
does not break existing behavior of "prefix hello cruel,world" where value2
contains reads "hello,world", and the parsed result should be

prefix="prefix"
value=["hello", "cruel,world"]

instead of something like

prefix="prefix"
value=["hello", "cruel", "world"]

or

prefix="prefix"
value=["cruel", "world"]

the above test only applies to the test where "value" is a CephString.
if "value" is a CephChoices, the parsed argument should be

prefix="prefix"
value=["hello", "cruel", "world"]

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: run nose test using "python3 -m nose"

otherwise the tests are not performed at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind/test_ceph_argparse: check for expected argument

instead of comparing the validated argument with empty dict, check for
its content for better coverage of testing.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind/test_ceph_argparse: decorate helper method with @nottest

otherwise nose tries to run it as a test as its name contains "test", like

E.
======================================================================
ERROR: Runs validate in different arg/kargs ways.
----------------------------------------------------------------------
TypeError: arg_kwarg_test() missing 3 required positional arguments: 'prefix', 'args', and 'sig'

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: do not test obsoleted command

"scrub" command was marked obsoleted in
e9a5ce0897efc6126caeebea9900bf05ec3d2174, so the test_ceph_argparse
cannot retrieve its command description using "get_command_descriptions"
cli tool anymore, let's drop the related test accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: drop forward and readforward from test

these two write modes were dropped in
a0a3ed324a130a9934735927600f01d1fa65bc0d, so update the test
accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: do not use tab for indent

the python source file is idented using spaces.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/ceph_argparse: validate csv if desc.N

if desc.N is not None, we should take the argument as a comma separated
values, and validate the values individually.

restructure the validate() function and its helpers to pass the
validated args if desc.N explicitly, as desc.instance.val should only
hold a single value of desc.instance type, otherwise we need to reset
it after collecting all the argument in a CSV string value is parsed.

this change would be less consistent and hackish.

and update a test to verify this behavior.

Fixes: https://tracker.ceph.com/issues/51145
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/pybind: pep8 cleanups

* do not import unused symbols
* add two empty lines before method
* aligned indents

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40740 from batrick/i50278

pybind/mgr: add support for common sqlite3 databases

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41809 from linuxbox2/wip-returnvec-64

rados: increase osd_max_write_op_reply_len default to 64 bytes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41746 from strenuous-life/wip-cephadm-osd-create

mgr/cephadm: When create osd which include db, _create_daemon will be…

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #41781 into master

* refs/pull/41781/head:
mgr/mgr_module:delete invalid judgment

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41699 into master

* refs/pull/41699/head:
mgr/telemetry: pass leaderboard flag even w/o ident

Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

qa: move dashboard e2e from cephadm -> rados suite

This test fails ~20% of the time.

Signed-off-by: Sage Weil <sage@newdream.net>

qa: choose victim pg from rbd pool

Right now scrub_test picks any pg in ceph. Unfortunately, it picked the
.mgr pool's only pg in [1]:

2021-05-16T11:36:35.035 DEBUG:teuthology.orchestra.run.smithi049:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool rbd setomapval main.db-journal.0000000000000000 key val

Instead, only pick a pg in the rbd pool.

[1] /ceph/teuthology-archive/kchai-2021-05-16_11:19:39-rados-wip-kefu-testing-2021-05-16-1043-distro-basic-smithi/6117396/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa,pybind/mgr: allow disabling .mgr pool

This is mostly for testing: a lot of tests assume that there are no
existing pools. These tests relied on a config to turn off creating the
"device_health_metrics" pool which generally exists for any new Ceph
cluster. It would be better to make these tests tolerant of the new .mgr
pool but clearly there's a lot of these. So just convert the config to
make it work.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: document new .mgr pool

And that pool names beginning with `.` are reserved names.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add upgrade test for devicehealth

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix api test failures

"device_health_metrics" pool is gone -- .mgr pool is in.

I don't think the pool removal code in some test cases is necessary any
longer with recent changes to remove those warnings; so that code is
gone too.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/devicehealth: load legacy device metrics pool

Fixes: https://tracker.ceph.com/issues/50278
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/devicehealth: update to store metrics in sqlite

This commit just modifies the module to use the sqlite3 database for
storing health metrics. A followup commit loads the legacy pool's data.

Fixes: https://tracker.ceph.com/issues/50278
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr: add sqlite3 db module helpers

This creates a new '.mgr' pool for storing a default sqlite3 database
for each mgr module. Each module's database is stored in:

        file:///.mgr:<mgr module name>/main.db?vfs=ceph

The "main.db" is the only one used presently but perhaps a module may
want extra databases for some reason. The module name is used for the
RADOS namespace.

Databases are versioned in a common table called MgrModuleKV using the
"__version" key. A mechanism is in place (SCHEMA_VERSIONED) to allow
modules to upgrade their databases over time in a consistent way.

Fixes: https://tracker.ceph.com/issues/50278
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40480 from kamoltat/wip-ksirivad-fix-bug-49988

pybind/mgr/progress: Disregard unreported pgs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41772 from cyx1231st/wip-seastore-onode-tree-node-size

crimson/onode-staged-tree: extend tree node sizes to fit insert upper-bounds

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #39505 into master

* refs/pull/39505/head:
qa: test nowsync option in kernel client workflows
qa: deep merge top level overrides for fuse/kclient

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #40952 from tchaikov/wip-deb-build-profile

debian: use build-profile to manage build deps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41683 into master

* refs/pull/41683/head:
qa: update RHEL to 8.4

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: specify distro for fs:bugs

Fixes: https://tracker.ceph.com/issues/51184
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41550 from aaryanporwal/navbar-fix

mgr/dashboard: fix for right sidebar nav icon not clickable

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: aaryanporwal <NOT@FOUND>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/onode-staged-tree: convert ghobject_t to key_hobj_t to print

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: merge with single-value tail internal node

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: switch to and test larger node sizes

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: support unaligned read to leaf_sub_items_t::num_keys

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add the missing retired validation

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: encode/decode stage_size instead

end-offset may risk overflow the node_offset_t, but stage_size will not.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: use extent_len_t and node_offset_t correctly

extent_len_t represents a value that may include the node size, but
node_offset_t cannot and may overflow. Also add validations when
try to cast a larger type to node_offset_t.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: remove unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: validate node size where possible

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: remove compile-time NODE_BLOCK_SIZE

Switch to run-time node_size.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: pass node_size to lower node stages

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: remove compile-time node SIZE

Switch to run-time node_extent_t::node_size instead.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup internal_fields_3_t template

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: pass node_size into node_extent_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: extend num_keys_t to support larger nodes

Also adjust the accroding layout related unit tests.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: remove hard-coded node_stage_t::EXTENT_SIZE

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: check the maximum tree height

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: validate safe-split from tree_conf_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add and use node sizes in tree_conf_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement size upper-bounds to value

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: implement size upper-bounds to ns/oid

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: introduce compile-time tree_conf_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup, decouple test_item_t from TestValue

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: introduce TransactionManager::read_extent(t, offset)

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: correct the node size equation

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: try merge peer if node has one value

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup and introduce has_single_value()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41814 from tchaikov/wip-crimson-fmt-seq

crimson/osd: specialize fmt::formatter<> for OpSequencer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41759 from rhcs-dashboard/wip-api_label-master

.github/labeler: add api-change label

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41242 from zdover23/wip-doc-cephadm-monitoring-deploying-monitoring-second-half-2021-05-10

doc/cephadm: rewrite "deploy. mon. w/cephadm" 2/2

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41241 from zdover23/wip-doc-cephadm-monitoring-deploying-monitoring-first-half-2021-05-10

doc/cephadm: rewrite "deploy. mon. w/cephadm"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

crimson/osd: specialize fmt::formatter<> for OpSequencer

so we can print it like fmt::format("{}", sequencer)

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: When test osd active, it need to add "mock call" to make sure execute correctly

Signed-off-by: jianglong01 <jianglong01@qianxin.com>