debian,spec: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

.github: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: add prettytable to requirmenets.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix doc urls to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Use CLICommand, except it's global variable

`CLICommand.COMMANDS` is a global varialbe that prevents
anyone from importing other modules, as the `COMMANS` are then
merged together. Let's use a meta class instead of a global variable.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: orchestrator_cli rename: fix imports

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestsrator: make parse_host_specs a classmethod

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: rename to mgr/orchestrator

* Move `mgr/orchestrator.py` to `orchestrator/_interface.py`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33361 from bk201/wip-44164

ceph.spec.in: fix python coverage dependency for non-rhel distros

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33358 from howard0su/nvme1

os/bluestore: allocate Task on stack

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: fix python coverage dependency for non-rhel distros

The coverage package under openSUSE (and other distros) are named as
python{major_version}-coverage (without minor version).

Fixes: https://tracker.ceph.com/issues/44164
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #33342 into master

* refs/pull/33342/head:
cephadm: expose `timeout` for `is_available` check
cephadm: add `--retry` arg

Reviewed-by: Sage Weil <sage@redhat.com>

osd/bluestore: Avoid allocate Task on Heap

When the I/O is synced, we can allocate the task object
on stack to avoid malloc calls.

Signed-off-by: Jun Su <howard0su@gmail.com>

Merge pull request #33357 from tchaikov/wip-crimson-asok

crimson: clean up and refactor asok

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/admin: no need to check for '\n'

as we don't need to mimic the behavior of classic OSD, what we need to
to fulfill the needs of ceph cli. see `admin_socket()` in
`src/pybind/ceph_daemon.py`, which sends a `\0` to indicate the end of a
command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/asok: disconnect client when shutdown

track the established connection as well, please note, the current asok
implementation only allows a single connection at the same time, even
though unix domain socket allows multiple concurrent clients. so there
is no need to track multiple clients at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/asok: do not assume the order of param eval

* do not assume the order of parameter evaluation, before this change,
  we have `do_with(cn.input(), cn.output(), std::move(cn) ...)`, see
  https://en.cppreference.com/w/cpp/language/eval_order,
> side effects of the initialization of every parameter are
> indeterminately sequenced with respect to value computations and side
> effects of any other parameter.
  we cannot move `cn` out and then call its member functions. so
  introduce a struct for capturing its input and output.
* move `do_until_gate()` into `start()`, no need to check if
  gate is stopped in `safe_action`, as `sestar::do_until()` will do
  this for us.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: register commands separately

so we can do command registration in the same place, in future, we can
move all of them into another place if necessary

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: refactor asok command

* do not define another iterator type, use `map::const_iterator`
  directly
* do not register hooks/commands with server block, register them
  one by one, much simpler this way.
* encapsulate the hook metadata in `AdminSocketHook`, so each
  `AdminSocketHook` instance is self-contained in the sense that
  we don't need to use an extra type for keeping track of them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg_map: add 'PGMap::get_pgs() const'

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: send beacon only if active

mimic the behavior of classic osd, and this behavior does make sense.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add OSD::dump_status()

so it can be used by asock command

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: refactor OSD::stop_asok_admin()

the comment does not apply anymore, since `admin` and `asok` are
created in the constructor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/thread/Threads::Threads/

Threads::Threads is cmake's way to present thread library. see
https://cmake.org/cmake/help/v3.1/module/FindThreads.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: remove unused include and forward decl

and add those used

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33339 into master

* refs/pull/33339/head:
osd/PeeringState: require SERVER_OCTOPUS to respond to RenewLease

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33289 into master

* refs/pull/33289/head:
qa/tasks/cephadm: deploy rgw daemons too

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32775 into master

* refs/pull/32775/head:
ceph.spec.in: fix python3 dependencies in centos7

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33129 into master

* refs/pull/33129/head:
osd/PeeringState: do not start renewing leases until PG is activated

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #33163 into master

* refs/pull/33163/head:
msg/Policy: limit unregistered anon connections to mon

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33350 from rzarzynski/wip-crimson-clang-watchnotify

crimson/osd: fix the Clang build in create_watch_info().

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33347 from tchaikov/wip-orch-addr

mgr/orchestrator: "addr" is optional for constructing InventoryNode

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33349 from ronen-fr/clang-5

crimson/osd: remove unneeded captures - pg.cc

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix the Clang build in create_watch_info().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33325 from tchaikov/wip-super-setup

qa/tasks: call super class's setUp()

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33348 from tchaikov/wip-rbd-mirror-test

mgr/dashboard: s/fsid/mirror_uuid/

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/orchestrator: "addr" is optional for constructing InventoryNode

this addresses a regression introduced by 5276871e15

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/orch: s/service ls/ps/

to fix the broken test of "test_load_data". it's a regression introduced
by aacc9a650f052fd5be543e9265ec94833b8e8bb3

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: remove unneeded captures - pg.cc

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #33243 from tchaikov/wip-43795

ceph_argparse: put args from env before existing ones

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks: call super class's setUp()

to address the regression introduced by
87292811215f6ded9a784d3216a910faaef648e2

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: s/fsid/mirror_uuid/

to fix the broken rbd-mirror test. it's a regression introduced by
7b07e3c9dcf1eda325fc4fe7960765c019243076

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33345 from athanatos/sjust/wip-fix-crimson-build

Crimson build fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33343 into master

* refs/pull/33343/head:
qa/suites/rados/cephadm/upgrade: add simple upgrade test
qa/tasks/cephadm: improve shell subcommand

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/tasks/cephadm: deploy rgw daemons too

Signed-off-by: Sage Weil <sage@redhat.com>

test_alien_echo: convert Condition to use readable_eventfd

Should have been included in 5f05a50bae8bb4889dba0d249ed5fc3a2fcdcfa5.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/CMakeLists.txt: link pthread to libcrimson for setting thread affinity

Signed-off-by: Samuel Just <sjust@redhat.com>

cephadm: expose `timeout` for `is_available` check

allow for the `--timeout` arg to override the default 30sec timeout
while waiting for a service to become available

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add `--retry` arg

enables overriding the the default number of retries when waiting for a
service to become available

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/cephadm/upgrade: add simple upgrade test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: improve shell subcommand

- pass teuth job params through environment
- run commands via bash -c

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: require SERVER_OCTOPUS to respond to RenewLease

To avoid sending pg_lease to pre-octopus OSDs during upgrades.

Fixes: https://tracker.ceph.com/issues/44156
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #33073 into master

* refs/pull/33073/head:
qa/suites/rados/cephadm: deploy prometheus.a
mgr/cephadm: implement prometheus add/update
mgr/cephadm: teach _create_daemon how to provision prometheus
mgr/orch: add prom hooks

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #33332 from jan--f/c-v-filestore-zap-fix

ceph-volume: don't remove vg twice when zapping filestore

Merge pull request #33320 from jan--f/c-v-fix-filestore-journal-size

ceph-volume: pass journal_size as Size not string

ceph-volume: don't remove vg twice when zapping filestore

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Fixes: https://tracker.ceph.com/issues/44149

Merge pull request #32679 from rzarzynski/wip-crimson-watchnotify_part1

crimson: add support for watch / notify, part 1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33296 from tchaikov/wip-crimson-cmake

cmake: compile crimson-auth with crimson::cflags

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33097 from dillaman/wip-43933

librbd: tweak deep-copy to avoid creating last snapshot until sync is complete

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33298 from sebastian-philipp/debian-rook-jsonpatch

debian: add python3-jsonpatch as dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: pass journal_size as Size not string

Fixes: https://tracker.ceph.com/issues/44148
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #32174 from ronen-fr/admin_commands_3

common,crimson: supporting admin-socket commands

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

debian: add python3-jsonpatch as dependency

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33283 from jan--f/c-v-zap-on-vg-without-lv

ceph-volume: avoid calling zap_lv with a LV-less VG

Merge pull request #32690 from rosinL/wip-timeevent

msg/async/dpdk: destroy fd in do_request

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: compile crimson-auth with crimson::cflags

* move auth related stuff into crimson/CMakeLists.txt, so we can
  link them against crimson::cflags, which populates the necessary
  definitions and other cxx flags when building these source files.
* expose crimson::cflags as a public library of crimson, as
  crimson-osd links against crimson. and the cflags should be populated
  to crimson-osd, otherwise they are compiled with different compiler
  options.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33084 from tchaikov/wip-crimson-pg-coll

crimson/os: init PG with pg coll not meta coll

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #33241 from nagpavan-chilakam/patch-1

doc/architecture.rst: fix a typo in EC section

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33194 into master

* refs/pull/33194/head:
qa: add tests for mds_join_fs cluster affinity
qa: update cluster warning message for removed MDS
doc: add section on new mds_join_fs behavior
mon/MDSMonitor: enforce mds_join_fs cluster affinity
mon/MDSMonitor: use type of info.rank or mds_rank_t
qa: accept operation on current fs status
qa: add method to enable multifs
qa: fix nested generator use
qa: manage config changes through mons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix the lifetime management of OpsExecuter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: dispatch MWatchNotify on watch removal.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: improve debugs in OpsExecuter::do_op_notify_ack.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement the complete phase of Notify propagation.

Q: How to test?
A:  * `${CEPHBIN:-bin}/rados -p rbd touch moj_watch`
    * `${CEPHBIN:-bin}/rados -p rbd watch moj_watch`
    * `${CEPHBIN:-bin}/rados -p rbd notify moj_watch test2`

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: dispatch MWatchNotify on CEPH_OSD_OP_NOTIFY.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

messages: MWatchNotify takes notifier_gid during construction.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: create_n_propagate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: pass real connection in OpsExecuter::do_op_watch_subop_watch()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: minor fix formatting in Watch.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: clean-up raw connection checks.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: unify connected() and is_connected() of Watch.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop the legacy path for undecodable notify acks.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: implement CEPH_OSD_OP_NOTIFY_ACK.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: implement CEPH_OSD_OP_NOTIFY.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: implement CEPH_OSD_WATCH_OP_PING.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: implement CEPH_OSD_WATCH_OP_RECONNECT.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: add execution units for WATCH and UNWATCH.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: introduce watchers to ObjectContext.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #33244 into master

* refs/pull/33244/head:
doc/mgr/orchestrator_modules: update docs
mgr/orch: break out apply_$type() in orchestrator.py interface
qa/tasks/cephadm: adapt to new cli
mgr/orch: add --force arg to 'daemon rm'
mgr/orch: fix host set-addr prefix
mgr/orch: service action cleanup
mgr/orch: prevent rm mon|mgr
mgr/orch: service rm -> rm
mgr/cephadm: add mon
mgr/orch: add mon hook
mgr/cephadm: add mgr
mgr/orch: add 'daemon add mgr'
mgr/orch: orch * add -> orch daemon add *
mgr/volumes: use apply_mds
mgr/orch: move to service_apply; remove update_foo()
mgr/rook: move to remove_service()
mgr/orch: remove old remove_{osds,rgw,mds,nfs,...} methods
mgr/orch: test remove_daemons and remove_service

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/suites/rados/cephadm: deploy prometheus.a

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: implement prometheus add/update

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: teach _create_daemon how to provision prometheus

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add prom hooks

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: do not start renewing leases until PG is activated

The activate() work renews the lease so that we can send lease info out
to the peers immediately.  However, these messages may get delayed.  Since
we immediately start scheduling renewals, it's possible for the renewal
to go out before the PG is active, crashing the replicas.

Fix by not scheduling renewals until the PG is really active.

Also, renew aggressively at that point in time, since it may have been a
while since we first started the activation.

Fixes: https://tracker.ceph.com/issues/44041
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30862 from zhangsw/fix-rgw-lc-repeated

rgw: set bucket attr twice when delete lifecycle config.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #33248 into master

* refs/pull/33248/head:
cephadm: fix deploy crash when no `args.fsid`

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31088 from joke-lee/rgw-storage-class-apend-obj

rgw: set correct storage class for append

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #31661 from joke-lee/sts-args

rgw: sts: add all http args to req_info

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #33053 from tpsilva/add_max_connections

rgw: make max_connections configurable in beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33165 from clwluvw/policy-subuser

rgw: Add support bucket policy for subuser

Reviewed-By: Adam Emersen <aemerson@redhat.com>

Merge pull request #32636 from ivancich/wip-avoid-0-length-ordered-list-result

rgw: address 0-length listing results when non-vis entries dominate

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>