Merge pull request #41808 from rhcs-dashboard/51164-show-only-days-in-bucket-details

mgr/dashboard: bucket details: show lock retention period only in days

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41758 from rhcs-dashboard/support-multiple-crush-trees

mgr/dashboard: crushmap tree doesn't display crush type other than root

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #35903 from agayev/fix-deployment-guide

doc: Add a missing instruction to manual deployment guide.

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: Add a missing instruction to manual deployment guide.

Following the instructions as is results in the following error at step 15:

$ sudo -u ceph ceph-mon --mkfs -i node1 --monmap /tmp/monmap --keyring /tmp/ceph.mon.keyring
global_init: error reading config file.

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>

Merge pull request #41911 from tchaikov/wip-crimson-nbd-cleanup

crimson/tools/store_nbd: better cleanup

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd: replace wait_pending with seastar::gate

the inc_pending + promise<> solution is pratically identical to
seastar::gate, so let's use the prepackaged solution instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: drop unnecessary seastar::now()

the body of handle_exception() is a synchronous operation, there is no
need to return seastar::now() here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: better cleanup

* remove unix domain socket file when cleanup
  so we don't need to remove it manually after each run.
* shutdown input and output streams when cleanup
  so reactor does not watch them anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: s/socket/server_socket/

to prepare for the next commit, which will keep track of the
connected_socket as another member variable.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41916 from AmnonHanuhov/wip-Refactor_test_messenger

test/crimson: Use crimson's make_message in test_messenger

test/crimson: Use crimson's make_message in test_messenger

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #41905 from ivancich/wip-improve-cls-rgw-tracing

rgw: clean-up logging of function entering to make thorough and consistent

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: clean-up logging of function entering to make thorough and consistent

This provides more thorough and consistent function tracing in CLS/RGW
when logging is set to 10 or higher.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #41903 from liewegas/update-rook-client

rook-client-python: update to update-june-21

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41835 from TRYTOBE8TME/wip-rgw-keycloak-failure-fix

qa/tasks: Keycloak failure fix

Reviewed-by: Ali Maredia <amaredia@redhat.com>

mgr/rook: comment out osd creation functions

This commit comments out the OSD creation functions in rook_cluster.py
and module.py, since the submodule update has broken them.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: fix some mypy typing errors in rook_cluster.py

This commit fixes some errors caught by mypy in rook_cluster.py. Most of the
errors were caused by the update of the rook-client-python submodule in a previous
commit.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: pass zone attribute to CephObjectStore CR when creating rgw

This commit passes the zone attribute to the CephObjectStore CR when
creating a RGW instance using the Rook Orchestrator backend:
`ceph orch apply rgw <rgw-name> --realm=<realm-name> --zone=<zone-name>`

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

Merge pull request #41856 from rhcs-dashboard/maintenance-bug-fix

mgr/dashboard: Fix 500 error while exiting out of maintenance

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>

Merge pull request #41694 from jmolmo/kcli_cephadm_doc

doc: Add kcli utilization for development environments

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41848 from xxhdx1985126/wip-errorator-parallel_for_each

crimson: errorator parallel_for_each

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41894 from tchaikov/wip-crimson-sigint

crimson/{osd,store_nbd}: handle SIGINT

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41895 from ceph/wip-cacephcom

MIRRORS: Add ca.ceph.com

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix 500 error while exiting out of maintenance

When you add a host in maintenance mode and then exit the maintenance
mode, a 500 server error will popup which will interrupt the whole
exit maintenance process and leave the host in an unknown/offline state.
It happened when I was setting the status of the host through the
HostSpec(). With this change, I am using the enter_maintenance api of
the orch to enable the maintenance.

Fixes: https://tracker.ceph.com/issues/51218
Signed-off-by: Nizamudeen A <nia@redhat.com>

rook-client-python: update to update-june-21

Signed-off-by: Sage Weil <sage@newdream.net>

Merge PR #41890 into master

* refs/pull/41890/head:
doc/cephadm: removing "Octopus" from procedure

Reviewed-by: Sage Weil <sage@redhat.com>

MIRRORS: Add ca.ceph.com

Signed-off-by: David Galloway <dgallowa@redhat.com>

crimson/osd: use stop_signal from seastar

and disable app_cfg.auto_handle_sigint_sigterm, otherwise app template
handles SIGINT and SIGTERM by itself, and calls app.stop(). but we don't
use this mechinary at all. we use seastar::defer() instead of
seastar::at_exit() for doing graceful shutdown and cleanup.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: update example usage in comment

--total-device-size is not supported any more.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: add graceful shutdown support

we could have more sophisticate mechinary for interrupting fio job,
but so far it is able to stop itself if it idle by handling ctrl-C.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/seastore/nbd: destruct the store before create

Otherwise the store will register the conflicting metrics and result in
double_registration.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/cephadm: removing "Octopus" from procedure

This PR removes "Octopus" from the curl-based installation
procedure.

After we moved on to Pacific, referring to Octopus looks wrong.
It looks wrong because it now is wrong.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41882 from tchaikov/wip-crimson-int-safty

crimson/osd: guard non-pg-op handling with with_sequencer()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41885 from tchaikov/wip-crimson-os-cleanups

crimson/os: cleanups and reformat

Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

crimson/osd: reindent

for less indent, hence better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: wrap line at 80

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: guard non-pg-op handling with with_sequencer()

because we should only ensure the ordering of the requests touching
the objects, the other requests like pgls should not be ordered along
with them. so as the second step, guard the non-pg-op handling with
with_sequencer().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use reference for loop variable

for better performance, also silences the warning like:

../src/crimson/os/seastore/random_block_manager/nvme_manager.cc:444:23: warning: loop variable ‘b’ creates a copy from type ‘const crimson::os::seastore::rbm_alloc_delta_t’ [-Wrange-loop-construct]
  444 |       for (const auto b : alloc_blocks) {
      |                       ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41859 from sebastian-philipp/mypy-constrains.txt

global,tox.ini: add mypy-constrains.txt

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

crimson/os: return seastar::now() in "finally()" block

so finally() is able to identify the return is a future, and discard it
manually.
otherwise the return value will be discarded even the future is marked
[[nodiscard]], hence the C++ compiler warns.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: remove unnecessary now()

the previous continuation in the chain already returns a future, no need
to hook up another now().

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: remove unnecessary parentheses

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: fix the indent

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

global,tox.ini: add mypy-constrains.txt

let's avoid getting new versions of those packages by accident.
Unfortunately this means we have to manually update those
packages regurarly.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

crimson/osd: extract with_sequencer() method out

because we should only ensure the ordering of the requests touching
the objects, the other requests like pgls should not be ordered along
with them, so as the first step, let's extract with_sequencer() method
out, so we can differentiate these two cases with minimal repeatings.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41881 from tchaikov/wip-crimson-alien

crimson: adapt to the new alien API

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: adapt to the new alien API

since seastar now deprecates the old alien::submit_to() API, and
replaced it with a variant which requires a new parameter of "alien"
instance, let's pick it up, to amortize the cost to catch up with
the seastar master HEAD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: add test case for errorator parallel_for_each

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson: add parallel_for_each to errorator

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #41825 from tchaikov/wip-avl-cleanup

os/bluestore/AvlAllocator: specialize _block_picker() and cleanups

Reviewed-by: Igor Fedotov <ifedotov@suse,com>

Merge pull request #41346 from adk3798/enough-mds-upgrade

mgr/cephadm: skip ok-to-stop for mds in upgrade if not enough mds daemons

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41813 from adk3798/repo

cephadm: Use gpg rather than asc key for add-repo

Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #41820 from mgfritch/cephadm-not-a-mon-ip

cephadm: raise Error() when unable to bind to an ip

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41041 from myoungwon/wip-randomblockmanager-part1

seastore: RandomBlockManager part1

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #27465 from tchaikov/wip-38219

ceph-monstore-tool: use a large enough paxos/{first,last}_committed

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41873 from bobbyphilip/patch-1

doc/start/hardware-recommendations.rst: Corrected typo

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #40972 into master

* refs/pull/40972/head:
doc: add missing note about restoring allow_standby_replay
doc: clarify distributed ephemeral pins new mechanism
doc: update mds_autoscaler documentation to reflect recent changes

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/start/hardware-recommendations.rst: Corrected typo

129GB - >128GB
Signed-off-by: Bobby Philip bobbyphilip@gmail.com

Merge PR #41844 into master

* refs/pull/41844/head:
qa/suites/orch/cephadm/dashboard: remove remaining bits

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #41840 into master

* refs/pull/41840/head:
qa: update cli syntax to conventional

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #41821 into master

* refs/pull/41821/head:
qa: specify distro for fs:bugs

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #41773 into master

* refs/pull/41773/head:
test/pybind/test_ceph_argparse: add a test where args contains comma
cmake: run nose test using "python3 -m nose"
test/pybind/test_ceph_argparse: check for expected argument
test/pybind/test_ceph_argparse: decorate helper method with @nottest
test/pybind: do not test obsoleted command
test/pybind: drop forward and readforward from test
test/pybind: do not use tab for indent
pybind/ceph_argparse: validate csv if desc.N
test/pybind: pep8 cleanups

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge PR #41771 into master

* refs/pull/41771/head:
qa: update scrub start code to use comma sep scrubopts

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

seastore/rbm: remove changes related to transaction/cache

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #41864 from rzarzynski/wip-crimson-iointrcond-fix-debug

crimson/osd: fix debug in IOInterruptCondition::new_interval_created().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41865 from rzarzynski/wip-crimson-opseq-fix-dropped-abort

crimson/osd: fix missed OpSequencer::abort() on primary change.

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix missed OpSequencer::abort() on primary change.

Commit b5efdc6f1c9563357d7dfd33a8f379053592a215 has unified
the interruption handling among `InternalClientRequest` and
`ClientRequest`. Unfortunately, a call to `abort()` of
`OpSequencer` has been overlooked and dropped. This was causing
crashes at Sepia after after fixing the ordering issues in
`ClientRequest`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: fix debug in IOInterruptCondition::new_interval_created().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41801 from liu-chunmei/seastore-fix-get-retired-extent

crimson/seastore: fix cache::get_extent got retired extent

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41854 from sebastian-philipp/fix-mypy-yaml-representation

mgr/orch: fix mypy errors

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #41846 from batrick/i51214

doc: fix typo in option

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>

pyhton-common: fix mypy errors

Fixes:

```
py3 run-test: commands[2] | mypy --config-file=../mypy.ini -p ceph
ceph/deployment/service_spec.py: note: In member "yaml_representer" of class "ServiceSpec":
ceph/deployment/service_spec.py:659: error: Argument 1 to "represent_dict" of "SafeRepresenter" has incompatible type "_OrderedDictItemsView[str, Any]"; expected "Mapping[Any, Any]"
```

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/orch: fix mypy errors

Fixes:

```
orchestrator/__init__.py:6: note: In module imported here:
orchestrator/_interface.py: note: In member "yaml_representer" of class "DaemonDescription":
orchestrator/_interface.py:1039: error: Argument 1 to "represent_dict" of "SafeRepresenter" has incompatible type "ItemsView[Any, Any]"; expected "Mapping[Any, Any]"
orchestrator/_interface.py: note: In member "yaml_representer" of class "ServiceDescription":
orchestrator/_interface.py:1178: error: Argument 1 to "represent_dict" of "SafeRepresenter" has incompatible type "ItemsView[Any, Any]"; expected "Mapping[Any, Any]"
orchestrator/_interface.py: note: At top level:
orchestrator/_interface.py:1181: error: Argument 2 to "add_representer" has incompatible type "Callable[[SafeDumper, DaemonDescription], Any]"; expected "Callable[[SafeDumper, ServiceDescription], Node]"
Found 3 errors in 1 file (checked 29 source files)
```

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41841 from ljflores/patch-1

doc: update testing.rst and build-integration test to say ./run-make-check.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

seastore: add protected_read interface to nvmedevice

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore/rbm: use const

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: return std::list in get_data_health()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: remove unused bits in nvmedevice.h

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: replace blk_paddr_t with size_t in free_extent()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: relocate file layout like SegmentManager

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: fix wrong loop condition

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: fix wrong statement

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: remove newline before function name

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: add check_bitmap_blocks when open()

This commits adds check_bitmap_blocks to check free_block_count
is correct whenever open() is called

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: fix wrong statement

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/rbm: use move()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore/nvmedevice: use the necessary capture value and fix typo

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: move implementation of TestMemory to nvmeblockdevice.cc

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: utilize errorator as a return type of discard()

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: utilize unique_ptr as a return value of create()

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: change class name NormalNBD to PosixNVMeDevice

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: replace POSIX call to seastar::file in NormalNBD

Signed-off-by: Jinyong Ha <jy200.ha@samsung.com>

seastore: add abort_allocation

when aborting uncommitted transaction, abort_allocation
should be called to cancel in-flight allocations in
in-memory allocator.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test: add more tests

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

test: modify existing unit test to cover range based operations

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: fix wrong operator

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: make multiple bitmap block initialization to work correctly

when initializing bitmap area, this commit corrects multiple
block initialization.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: continous range based free_extent()

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

seastore: continous range based complete_allocation()

To work with a range of allocated blocks as a input,
this commit modify complete_allocation().

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>