mgr/orchestrator_cli: sort host list

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32817 from sebastian-philipp/rename-orchestrator_cli-orchestrator

pybind/mgr: Rename orchestrator_cli to orchestrator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33167 from tchaikov/wip-formatter-string_view

common,mgr,osd: pass string_view as "name"

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge PR #33227 into master

* refs/pull/33227/head:
mds: remove unused CDir members

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33197 into master

* refs/pull/33197/head:
mount.ceph: fix incorrect options parsing

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge PR #33180 into master

* refs/pull/33180/head:
mds: add scrub_info_t into mempool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33104 into master

* refs/pull/33104/head:
client: Fixes for missing consts SEEK_DATA and SEEK_HOLE on alpine linux

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33005 into master

* refs/pull/33005/head:
mds: fix 'can wrlock' check in Locker::acquire_locks()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32435 into master

* refs/pull/32435/head:
mds: Reorganize structure and class members in mdstypes header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33344 from athanatos/sjust/wip-errorator-handlers

errorator: improve general error handlers

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31001 from rosinL/wip-set-radosgw-cpu-affinity

rgw/rgw_main: auto set radosgw's cpu affinity according to numa_node configuration

Merge pull request #33206 from rhcs-dashboard/44075-rgw-user-system-field

mgr/dashboard: show correct RGW user 'system' info

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33125 from yaarith/telemetry-add-last-report-to-status

mgr/telemetry: add 'last_upload' to status

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33013 from s0nea/wip-dashboard-43912

mgr/dashboard: wait for PG unknown state to be cleared

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #33107 from votdev/cleanup_code

mgr/dashboard: Cleanup code

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31901 from matthewoliver/rgw-admin-user-add-modify

 rgw: make radosgw-admin user create and modify distinct

Merge pull request #32632 from cyx1231st/rfc-seastar-test-socket-nevermove

crimson/net: configure seastar to accept on a fixed core

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add pyyaml to requirements.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: rename orchestrator_cli -> orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: add prettytable to admin/doc-requirements.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mon: always_on_modules: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

debian,spec: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

.github: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: add prettytable to requirmenets.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix doc urls to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Use CLICommand, except it's global variable

`CLICommand.COMMANDS` is a global varialbe that prevents
anyone from importing other modules, as the `COMMANS` are then
merged together. Let's use a meta class instead of a global variable.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: orchestrator_cli rename: fix imports

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestsrator: make parse_host_specs a classmethod

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: rename to mgr/orchestrator

* Move `mgr/orchestrator.py` to `orchestrator/_interface.py`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33361 from bk201/wip-44164

ceph.spec.in: fix python coverage dependency for non-rhel distros

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: remove duplicated error codes and conditions

The duplicated error codes and conditions were originally introduced to
match connection errors with both system category (thrown by seastar)
and generic category (thrown by standard library).  Since error_code
with system category can be matched by error_condition with generic
category (see std::errc and
system_error_category::default_error_condition(int)), our duplicated
counterparts are not needed actually.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #33358 from howard0su/nvme1

os/bluestore: allocate Task on stack

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: fix python coverage dependency for non-rhel distros

The coverage package under openSUSE (and other distros) are named as
python{major_version}-coverage (without minor version).

Fixes: https://tracker.ceph.com/issues/44164
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

rgw: make radosgw-admin user create and modify distinct

Currently if you run 'radosgw-admin user create ..' when the user
already exists and you happen to specify, at least,  '--uid' and
'--display-name' that match the existing user, radowgw-admin will
actaully go modify the existing user.

This behaviour is a little confusing, hence the bug this patch is
fixing. This patch instead simplifies the tool to make
'create' create and 'modify' modify.
Meaning when you go 'create' a user that already exists, you'll get an
error, as expected. If you want to modify a user, you actually have to
use 'modify'.

For exapmle, now:

  $ radosgw-admin user create --uid="test-user"  --display-name="test user"
  could not create user: unable to create user, user: test-user exists

Signed-off-by: Matthew Oliver <moliver@suse.com>
Fixes: https://tracker.ceph.com/issues/38619

Merge PR #33342 into master

* refs/pull/33342/head:
cephadm: expose `timeout` for `is_available` check
cephadm: add `--retry` arg

Reviewed-by: Sage Weil <sage@redhat.com>

osd/bluestore: Avoid allocate Task on Heap

When the I/O is synced, we can allocate the task object
on stack to avoid malloc calls.

Signed-off-by: Jun Su <howard0su@gmail.com>

Merge pull request #33357 from tchaikov/wip-crimson-asok

crimson: clean up and refactor asok

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/admin: no need to check for '\n'

as we don't need to mimic the behavior of classic OSD, what we need to
to fulfill the needs of ceph cli. see `admin_socket()` in
`src/pybind/ceph_daemon.py`, which sends a `\0` to indicate the end of a
command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/asok: disconnect client when shutdown

track the established connection as well, please note, the current asok
implementation only allows a single connection at the same time, even
though unix domain socket allows multiple concurrent clients. so there
is no need to track multiple clients at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/asok: do not assume the order of param eval

* do not assume the order of parameter evaluation, before this change,
  we have `do_with(cn.input(), cn.output(), std::move(cn) ...)`, see
  https://en.cppreference.com/w/cpp/language/eval_order,
> side effects of the initialization of every parameter are
> indeterminately sequenced with respect to value computations and side
> effects of any other parameter.
  we cannot move `cn` out and then call its member functions. so
  introduce a struct for capturing its input and output.
* move `do_until_gate()` into `start()`, no need to check if
  gate is stopped in `safe_action`, as `sestar::do_until()` will do
  this for us.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: register commands separately

so we can do command registration in the same place, in future, we can
move all of them into another place if necessary

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: refactor asok command

* do not define another iterator type, use `map::const_iterator`
  directly
* do not register hooks/commands with server block, register them
  one by one, much simpler this way.
* encapsulate the hook metadata in `AdminSocketHook`, so each
  `AdminSocketHook` instance is self-contained in the sense that
  we don't need to use an extra type for keeping track of them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg_map: add 'PGMap::get_pgs() const'

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: send beacon only if active

mimic the behavior of classic osd, and this behavior does make sense.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add OSD::dump_status()

so it can be used by asock command

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: refactor OSD::stop_asok_admin()

the comment does not apply anymore, since `admin` and `asok` are
created in the constructor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/thread/Threads::Threads/

Threads::Threads is cmake's way to present thread library. see
https://cmake.org/cmake/help/v3.1/module/FindThreads.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: remove unused include and forward decl

and add those used

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33339 into master

* refs/pull/33339/head:
osd/PeeringState: require SERVER_OCTOPUS to respond to RenewLease

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33289 into master

* refs/pull/33289/head:
qa/tasks/cephadm: deploy rgw daemons too

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32775 into master

* refs/pull/32775/head:
ceph.spec.in: fix python3 dependencies in centos7

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33129 into master

* refs/pull/33129/head:
osd/PeeringState: do not start renewing leases until PG is activated

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #33163 into master

* refs/pull/33163/head:
msg/Policy: limit unregistered anon connections to mon

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33350 from rzarzynski/wip-crimson-clang-watchnotify

crimson/osd: fix the Clang build in create_watch_info().

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33347 from tchaikov/wip-orch-addr

mgr/orchestrator: "addr" is optional for constructing InventoryNode

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33349 from ronen-fr/clang-5

crimson/osd: remove unneeded captures - pg.cc

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix the Clang build in create_watch_info().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33325 from tchaikov/wip-super-setup

qa/tasks: call super class's setUp()

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33348 from tchaikov/wip-rbd-mirror-test

mgr/dashboard: s/fsid/mirror_uuid/

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/orchestrator: "addr" is optional for constructing InventoryNode

this addresses a regression introduced by 5276871e15

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/orch: s/service ls/ps/

to fix the broken test of "test_load_data". it's a regression introduced
by aacc9a650f052fd5be543e9265ec94833b8e8bb3

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: remove unneeded captures - pg.cc

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #33243 from tchaikov/wip-43795

ceph_argparse: put args from env before existing ones

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks: call super class's setUp()

to address the regression introduced by
87292811215f6ded9a784d3216a910faaef648e2

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: s/fsid/mirror_uuid/

to fix the broken rbd-mirror test. it's a regression introduced by
7b07e3c9dcf1eda325fc4fe7960765c019243076

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33345 from athanatos/sjust/wip-fix-crimson-build

Crimson build fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33343 into master

* refs/pull/33343/head:
qa/suites/rados/cephadm/upgrade: add simple upgrade test
qa/tasks/cephadm: improve shell subcommand

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/tasks/cephadm: deploy rgw daemons too

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/common/errorator: restrict all_same_way to valid types

As with pass_further/discard_all, we don't want the returned handler
to work on types outside of the errorator at all.  Otherwise, the
handler will transparently apply to any error.

Signed-off-by: Samuel Just <sjust@redhat.com>

common/crimson/errorator: add universal pass_further_all, discard_all, all_same_way

In many cases, we simply want to add catch-all handling.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator: fix errorator::pass_further and discard_all

Previously, both of these were invocable on all errors, but would
static_assert on invalid ones.  What we actually want is for them
to only be invocable on valid errors.  That way, we can do, for
instance:

  }).handle_error(
    roll_journal_segment_ertr::pass_further{},
    SegmentManager::open_ertr::all_same_way([this](auto &&e) {
      logger().error(
"error {} in close segment {}",
e,
current_journal_segment_id);
      ceph_assert(0 == "error in close");
      return;
    })

to explicitely propogate any errors in roll_journal_segment_ertr
while asserting on anything else.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator: add pass_further and discard to unthrowable_wrapper

This lets us use, for instance, ct_error::enoent::pass_further{} to
explicitely propogate enoent.

Signed-off-by: Samuel Just <sjust@redhat.com>

test_alien_echo: convert Condition to use readable_eventfd

Should have been included in 5f05a50bae8bb4889dba0d249ed5fc3a2fcdcfa5.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/CMakeLists.txt: link pthread to libcrimson for setting thread affinity

Signed-off-by: Samuel Just <sjust@redhat.com>

cephadm: expose `timeout` for `is_available` check

allow for the `--timeout` arg to override the default 30sec timeout
while waiting for a service to become available

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add `--retry` arg

enables overriding the the default number of retries when waiting for a
service to become available

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/cephadm/upgrade: add simple upgrade test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: improve shell subcommand

- pass teuth job params through environment
- run commands via bash -c

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: require SERVER_OCTOPUS to respond to RenewLease

To avoid sending pg_lease to pre-octopus OSDs during upgrades.

Fixes: https://tracker.ceph.com/issues/44156
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #33073 into master

* refs/pull/33073/head:
qa/suites/rados/cephadm: deploy prometheus.a
mgr/cephadm: implement prometheus add/update
mgr/cephadm: teach _create_daemon how to provision prometheus
mgr/orch: add prom hooks

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #33332 from jan--f/c-v-filestore-zap-fix

ceph-volume: don't remove vg twice when zapping filestore

Merge pull request #33320 from jan--f/c-v-fix-filestore-journal-size

ceph-volume: pass journal_size as Size not string

ceph-volume: don't remove vg twice when zapping filestore

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Fixes: https://tracker.ceph.com/issues/44149

Merge pull request #32679 from rzarzynski/wip-crimson-watchnotify_part1

crimson: add support for watch / notify, part 1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33296 from tchaikov/wip-crimson-cmake

cmake: compile crimson-auth with crimson::cflags

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33097 from dillaman/wip-43933

librbd: tweak deep-copy to avoid creating last snapshot until sync is complete

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33298 from sebastian-philipp/debian-rook-jsonpatch

debian: add python3-jsonpatch as dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: pass journal_size as Size not string

Fixes: https://tracker.ceph.com/issues/44148
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #32174 from ronen-fr/admin_commands_3

common,crimson: supporting admin-socket commands

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

debian: add python3-jsonpatch as dependency

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33283 from jan--f/c-v-zap-on-vg-without-lv

ceph-volume: avoid calling zap_lv with a LV-less VG

Merge pull request #32690 from rosinL/wip-timeevent

msg/async/dpdk: destroy fd in do_request

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: compile crimson-auth with crimson::cflags

* move auth related stuff into crimson/CMakeLists.txt, so we can
  link them against crimson::cflags, which populates the necessary
  definitions and other cxx flags when building these source files.
* expose crimson::cflags as a public library of crimson, as
  crimson-osd links against crimson. and the cflags should be populated
  to crimson-osd, otherwise they are compiled with different compiler
  options.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33084 from tchaikov/wip-crimson-pg-coll

crimson/os: init PG with pg coll not meta coll

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #33241 from nagpavan-chilakam/patch-1

doc/architecture.rst: fix a typo in EC section

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33194 into master

* refs/pull/33194/head:
qa: add tests for mds_join_fs cluster affinity
qa: update cluster warning message for removed MDS
doc: add section on new mds_join_fs behavior
mon/MDSMonitor: enforce mds_join_fs cluster affinity
mon/MDSMonitor: use type of info.rank or mds_rank_t
qa: accept operation on current fs status
qa: add method to enable multifs
qa: fix nested generator use
qa: manage config changes through mons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix the lifetime management of OpsExecuter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: dispatch MWatchNotify on watch removal.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: improve debugs in OpsExecuter::do_op_notify_ack.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement the complete phase of Notify propagation.

Q: How to test?
A:  * `${CEPHBIN:-bin}/rados -p rbd touch moj_watch`
    * `${CEPHBIN:-bin}/rados -p rbd watch moj_watch`
    * `${CEPHBIN:-bin}/rados -p rbd notify moj_watch test2`

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>