mgr/cephadm: move cutoff calc inside helper

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: clean up service_action, remove_service

Standardize on service_name argument that looks like 'mgr', 'mds.fsname',
'mds', or some other daemon name prefix.  Avoid the ambiguously-named
service_type+service_name combinations.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move into DaemonCache class

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix tests

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: implete hacky 'refresh=True' path

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: raise health alert when scrape fails

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: persist cached daemon state

- load cached state on startup
- persist state after a scrape only
- scrape everything after startup

Note that we modify our in-memory cache when we add or remove a service
and then immediately trigger a scrape, but we do not invalidate the
persisted state, since it's simpler (and presumably a good idea) to
simply re-scrape everything after a mgr restart.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: serialize DaemonDescription last_refresh

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move _get_daemons() impl into list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace remaining _get_daemons() with daemon cache

New function _get_daemons_by_type() returns immediately with a result
from the cache.

Remove the with_daemons decorator.

Push daemon list fetch into _add_new_daemon.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use daemon map for service|daemon removal

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: avoid _get_daemons for service|daemon actions

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace daemon_cache with an explicit set of dicts

- Cache DaemonDescription explicitly
- explicit timestamp for the host
- serve() scrapes inventory based on that timestamp

For the moment, persistence is broken, and --refresh is broken.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: move DaemonDescription construction into helper

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33390 into master

* refs/pull/33390/head:
mds: remove dead get_commands code

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #33336 into master

* refs/pull/33336/head:
osd: fix racy accesses to OSD::osdmap.

Reviewed-by: Sage Weil <sage@redhat.com>

mds: remove dead get_commands code

Left over from #31255, d8c0bde04b88cb3ad96855bd0948ae10072c9da7.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33369 into master

* refs/pull/33369/head:
cephadm: check for both chrony service names

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33360 from liewegas/fix-prom

qa/suites/rados/cephadm/smoke: disable rgw role for now

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32817 from sebastian-philipp/rename-orchestrator_cli-orchestrator

pybind/mgr: Rename orchestrator_cli to orchestrator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33167 from tchaikov/wip-formatter-string_view

common,mgr,osd: pass string_view as "name"

Reviewed-by: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge PR #33227 into master

* refs/pull/33227/head:
mds: remove unused CDir members

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33197 into master

* refs/pull/33197/head:
mount.ceph: fix incorrect options parsing

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge PR #33180 into master

* refs/pull/33180/head:
mds: add scrub_info_t into mempool

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33104 into master

* refs/pull/33104/head:
client: Fixes for missing consts SEEK_DATA and SEEK_HOLE on alpine linux

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33005 into master

* refs/pull/33005/head:
mds: fix 'can wrlock' check in Locker::acquire_locks()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32435 into master

* refs/pull/32435/head:
mds: Reorganize structure and class members in mdstypes header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33344 from athanatos/sjust/wip-errorator-handlers

errorator: improve general error handlers

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31001 from rosinL/wip-set-radosgw-cpu-affinity

rgw/rgw_main: auto set radosgw's cpu affinity according to numa_node configuration

qa/suites/rados/cephadm/smoke: remove rgw

Fixes: https://tracker.ceph.com/issues/44168
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33206 from rhcs-dashboard/44075-rgw-user-system-field

mgr/dashboard: show correct RGW user 'system' info

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33125 from yaarith/telemetry-add-last-report-to-status

mgr/telemetry: add 'last_upload' to status

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33013 from s0nea/wip-dashboard-43912

mgr/dashboard: wait for PG unknown state to be cleared

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #33107 from votdev/cleanup_code

mgr/dashboard: Cleanup code

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

cephadm: check for both chrony service names

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31901 from matthewoliver/rgw-admin-user-add-modify

 rgw: make radosgw-admin user create and modify distinct

Merge pull request #32632 from cyx1231st/rfc-seastar-test-socket-nevermove

crimson/net: configure seastar to accept on a fixed core

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: add pyyaml to requirements.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: rename orchestrator_cli -> orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: add prettytable to admin/doc-requirements.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mon: always_on_modules: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

debian,spec: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

.github: Rename orchestrator_cli to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: add prettytable to requirmenets.txt

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix doc urls to orchestrator

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Use CLICommand, except it's global variable

`CLICommand.COMMANDS` is a global varialbe that prevents
anyone from importing other modules, as the `COMMANS` are then
merged together. Let's use a meta class instead of a global variable.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/mgr: orchestrator_cli rename: fix imports

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestsrator: make parse_host_specs a classmethod

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator_cli: rename to mgr/orchestrator

* Move `mgr/orchestrator.py` to `orchestrator/_interface.py`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33361 from bk201/wip-44164

ceph.spec.in: fix python coverage dependency for non-rhel distros

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: remove duplicated error codes and conditions

The duplicated error codes and conditions were originally introduced to
match connection errors with both system category (thrown by seastar)
and generic category (thrown by standard library).  Since error_code
with system category can be matched by error_condition with generic
category (see std::errc and
system_error_category::default_error_condition(int)), our duplicated
counterparts are not needed actually.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #33358 from howard0su/nvme1

os/bluestore: allocate Task on stack

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: fix python coverage dependency for non-rhel distros

The coverage package under openSUSE (and other distros) are named as
python{major_version}-coverage (without minor version).

Fixes: https://tracker.ceph.com/issues/44164
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

rgw: make radosgw-admin user create and modify distinct

Currently if you run 'radosgw-admin user create ..' when the user
already exists and you happen to specify, at least,  '--uid' and
'--display-name' that match the existing user, radowgw-admin will
actaully go modify the existing user.

This behaviour is a little confusing, hence the bug this patch is
fixing. This patch instead simplifies the tool to make
'create' create and 'modify' modify.
Meaning when you go 'create' a user that already exists, you'll get an
error, as expected. If you want to modify a user, you actually have to
use 'modify'.

For exapmle, now:

  $ radosgw-admin user create --uid="test-user"  --display-name="test user"
  could not create user: unable to create user, user: test-user exists

Signed-off-by: Matthew Oliver <moliver@suse.com>
Fixes: https://tracker.ceph.com/issues/38619

Merge PR #33342 into master

* refs/pull/33342/head:
cephadm: expose `timeout` for `is_available` check
cephadm: add `--retry` arg

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: fix prometheus shutdown

Signed-off-by: Sage Weil <sage@redhat.com>

osd/bluestore: Avoid allocate Task on Heap

When the I/O is synced, we can allocate the task object
on stack to avoid malloc calls.

Signed-off-by: Jun Su <howard0su@gmail.com>

Merge pull request #33357 from tchaikov/wip-crimson-asok

crimson: clean up and refactor asok

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

crimson/admin: no need to check for '\n'

as we don't need to mimic the behavior of classic OSD, what we need to
to fulfill the needs of ceph cli. see `admin_socket()` in
`src/pybind/ceph_daemon.py`, which sends a `\0` to indicate the end of a
command.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/asok: disconnect client when shutdown

track the established connection as well, please note, the current asok
implementation only allows a single connection at the same time, even
though unix domain socket allows multiple concurrent clients. so there
is no need to track multiple clients at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/asok: do not assume the order of param eval

* do not assume the order of parameter evaluation, before this change,
  we have `do_with(cn.input(), cn.output(), std::move(cn) ...)`, see
  https://en.cppreference.com/w/cpp/language/eval_order,
> side effects of the initialization of every parameter are
> indeterminately sequenced with respect to value computations and side
> effects of any other parameter.
  we cannot move `cn` out and then call its member functions. so
  introduce a struct for capturing its input and output.
* move `do_until_gate()` into `start()`, no need to check if
  gate is stopped in `safe_action`, as `sestar::do_until()` will do
  this for us.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: register commands separately

so we can do command registration in the same place, in future, we can
move all of them into another place if necessary

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: refactor asok command

* do not define another iterator type, use `map::const_iterator`
  directly
* do not register hooks/commands with server block, register them
  one by one, much simpler this way.
* encapsulate the hook metadata in `AdminSocketHook`, so each
  `AdminSocketHook` instance is self-contained in the sense that
  we don't need to use an extra type for keeping track of them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd/pg_map: add 'PGMap::get_pgs() const'

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: send beacon only if active

mimic the behavior of classic osd, and this behavior does make sense.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add OSD::dump_status()

so it can be used by asock command

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: refactor OSD::stop_asok_admin()

the comment does not apply anymore, since `admin` and `asok` are
created in the constructor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: s/thread/Threads::Threads/

Threads::Threads is cmake's way to present thread library. see
https://cmake.org/cmake/help/v3.1/module/FindThreads.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: remove unused include and forward decl

and add those used

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33339 into master

* refs/pull/33339/head:
osd/PeeringState: require SERVER_OCTOPUS to respond to RenewLease

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33289 into master

* refs/pull/33289/head:
qa/tasks/cephadm: deploy rgw daemons too

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32775 into master

* refs/pull/32775/head:
ceph.spec.in: fix python3 dependencies in centos7

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33129 into master

* refs/pull/33129/head:
osd/PeeringState: do not start renewing leases until PG is activated

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #33163 into master

* refs/pull/33163/head:
msg/Policy: limit unregistered anon connections to mon

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33350 from rzarzynski/wip-crimson-clang-watchnotify

crimson/osd: fix the Clang build in create_watch_info().

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33347 from tchaikov/wip-orch-addr

mgr/orchestrator: "addr" is optional for constructing InventoryNode

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33349 from ronen-fr/clang-5

crimson/osd: remove unneeded captures - pg.cc

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix the Clang build in create_watch_info().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33325 from tchaikov/wip-super-setup

qa/tasks: call super class's setUp()

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33348 from tchaikov/wip-rbd-mirror-test

mgr/dashboard: s/fsid/mirror_uuid/

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/orchestrator: "addr" is optional for constructing InventoryNode

this addresses a regression introduced by 5276871e15

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/orch: s/service ls/ps/

to fix the broken test of "test_load_data". it's a regression introduced
by aacc9a650f052fd5be543e9265ec94833b8e8bb3

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: remove unneeded captures - pg.cc

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #33243 from tchaikov/wip-43795

ceph_argparse: put args from env before existing ones

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks: call super class's setUp()

to address the regression introduced by
87292811215f6ded9a784d3216a910faaef648e2

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: s/fsid/mirror_uuid/

to fix the broken rbd-mirror test. it's a regression introduced by
7b07e3c9dcf1eda325fc4fe7960765c019243076

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33345 from athanatos/sjust/wip-fix-crimson-build

Crimson build fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33343 into master

* refs/pull/33343/head:
qa/suites/rados/cephadm/upgrade: add simple upgrade test
qa/tasks/cephadm: improve shell subcommand

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/tasks/cephadm: deploy rgw daemons too

Signed-off-by: Sage Weil <sage@redhat.com>

crimson/common/errorator: restrict all_same_way to valid types

As with pass_further/discard_all, we don't want the returned handler
to work on types outside of the errorator at all.  Otherwise, the
handler will transparently apply to any error.

Signed-off-by: Samuel Just <sjust@redhat.com>

common/crimson/errorator: add universal pass_further_all, discard_all, all_same_way

In many cases, we simply want to add catch-all handling.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator: fix errorator::pass_further and discard_all

Previously, both of these were invocable on all errors, but would
static_assert on invalid ones.  What we actually want is for them
to only be invocable on valid errors.  That way, we can do, for
instance:

  }).handle_error(
    roll_journal_segment_ertr::pass_further{},
    SegmentManager::open_ertr::all_same_way([this](auto &&e) {
      logger().error(
"error {} in close segment {}",
e,
current_journal_segment_id);
      ceph_assert(0 == "error in close");
      return;
    })

to explicitely propogate any errors in roll_journal_segment_ertr
while asserting on anything else.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator: add pass_further and discard to unthrowable_wrapper

This lets us use, for instance, ct_error::enoent::pass_further{} to
explicitely propogate enoent.

Signed-off-by: Samuel Just <sjust@redhat.com>

test_alien_echo: convert Condition to use readable_eventfd

Should have been included in 5f05a50bae8bb4889dba0d249ed5fc3a2fcdcfa5.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/CMakeLists.txt: link pthread to libcrimson for setting thread affinity

Signed-off-by: Samuel Just <sjust@redhat.com>

cephadm: expose `timeout` for `is_available` check

allow for the `--timeout` arg to override the default 30sec timeout
while waiting for a service to become available

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: add `--retry` arg

enables overriding the the default number of retries when waiting for a
service to become available

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/cephadm/upgrade: add simple upgrade test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: improve shell subcommand

- pass teuth job params through environment
- run commands via bash -c

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PeeringState: require SERVER_OCTOPUS to respond to RenewLease

To avoid sending pg_lease to pre-octopus OSDs during upgrades.

Fixes: https://tracker.ceph.com/issues/44156
Signed-off-by: Neha Ojha <nojha@redhat.com>