qa/tasks/mgr/test_orchestrator_cli: fix test_ps* tests

The ps output names daemons like 'type.foo', e.g., 'mgr.x'.  Now that
the test_orchestrator impl is less bonkers this needs to be adjusted to
match reality.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: refactor listing services/daemons

- Refactor list_daemons: return ceph-xxx processes on host.
- Refactor describe_service: return services by grouping ceph-xxx
  processes on host.
- Add dummy data.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: list services and daemons

- Display services and daemons in the cluster/services page.
- Display daemons in the cluster/hosts/host-detail page (Daemons tab).

This PR also partially addresses https://tracker.ceph.com/issues/43165:
The endpoint `/api/orchestrator/service` is removed.

Create new endpoints:
  - `/api/service`: listing all services in the Ceph cluster.
  - `/api/service/<service_name>/daemons`: listing daemons for a
    service. e.g. daemons of OSD.
  - `/api/host/<hostname>/daemons`: listing daemons of a host.

Fixes: https://tracker.ceph.com/issues/44221
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #33512 into master

* refs/pull/33512/head:
os/bluestore/BlueFS: less verbose about alloc adjustments

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #33575 into master

* refs/pull/33575/head:
mgr/telemetry: fix and document proxy usage

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge PR #33644 into master

* refs/pull/33644/head:
qa/suites/rados/verity/tasks/mon_recovery: whitelist SLOW_OPS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33541 into master

* refs/pull/33541/head:
mgr/test_orchestrator: add daemon_action
qa/tasksmgr/test_orchestrator_cli: fix test_service_action
qa/tasks/test_orchestrator_cli: disable test_osd_create exception check
mgr/test_orchestrator: update_foo -> apply_foo
mgr/test_orchestrator: add force flag to remove_daemons
qa/tasks/mgr/test_orchestrator_cli: update
qa/tasks/mgr/test_orchestrator_cli: fix test_load_data
mgr/test_orchestrator: fix list_daemons

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33627 into master

* refs/pull/33627/head:
doc/cephadm/administration: clarify log gathering

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33553 into master

* refs/pull/33553/head:
mgr/cephadm: orch ls: include specs with no daemons
mgr/cephadm: orch ls: show spec size
mgr/orch: remove unused fields in RGWSpec
mgr/orch: fix ServiceSpec fields
mgr/cephadm: simplify spec apply
pybind/mgr/mgr_module: revert PersistentStoreDict seperator
mgr/cephadm: apply services after refreshing inventory
mgr/cephadm: catch and log exceptions from apply
mgr/orch: no extra whitespace in stored json specs
mgr/cephadm: drop daemon_type arg to _apply_service
mgr/cephadm: use _apply() helper for all apply_ methods
mgr/cephadm: replace PersistentStoreDict with SpecStore
mgr/cephadm: do not remove service spec when removing a daemon
mgr/cephadm: rename completion variables&cleanup
mgr/cephadm: leverage service specs

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: add daemon_action

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasksmgr/test_orchestrator_cli: fix test_service_action

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/test_orchestrator_cli: disable test_osd_create exception check

I'm not sure what behavior we want here...

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: orch ls: include specs with no daemons

This way you see entries like

NAME           RUNNING  REFRESHED  IMAGE NAME                                      IMAGE ID        SPEC
mds.foo            0/1  -          <unknown>                                       <unknown>     present

before any daemons have been created.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: orch ls: show spec size

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: remove unused fields in RGWSpec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: fix ServiceSpec fields

- service_type is required.  Make it the first position arg to the ctor.
- service_id is the id *only* and optional.
- service_name() is the full service name (no change)

The old 'name' was previously used as the id only, so it was poorly named,
and optional, but in this series was changed to include the type, breaking
naming for a bunch of things (e.g., daemons called mds.mds.fsname.xyz).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: simplify spec apply

- Teach _apply_service how to pick the create (and config) functions, so
  that we don't need any weird wrappers in the callers.
- Replace trigger_deploy() and _apply_services() with a simpler
  _apply_all_services()
- Drop all of the per-type _apply_foo() methods.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/mgr_module: revert PersistentStoreDict seperator

This fixes the self-test.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: apply services after refreshing inventory

Ensure we have an inventory to apply against.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: catch and log exceptions from apply

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: no extra whitespace in stored json specs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: drop daemon_type arg to _apply_service

This is part of ServiceSpec now.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use _apply() helper for all apply_ methods

Clean up a few that were missed (due to racing PRs, presumably).

'creation' -> 'update', since an apply doesn't necessarily create (it may
destroy!).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace PersistentStoreDict with SpecStore

Explicit implementation of the dict of specs.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not remove service spec when removing a daemon

This makes our behavior similar to kube: if you kill a pod, the operator
or controller will come along and create a new one (probably somewhere
else).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: rename completion variables&cleanup

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: leverage service specs

Fixes: https://tracker.ceph.com/issues/44205
This does a couple of things:

* Change the way apply_$service() works:

Instead of triggering the deployment mechanism it will rather
transform the already passed ServiceSpec into a json representation
and save it in a persistent mon_store section.

`mgr/cephadm/service_spec/$service|daemon_type/service_name`

These locations will be periodically checked in the serve() thread.
This works since all the apply_$service_type functions are idempotent.

* Allow to save a config-like specification in the mon_store.

`ceph orch apply -i <service_spec_file.yaml>`

will read the specified services and save them in the mon store
section like mentioned above. The same serve() mechanism like above
also applies to deployment.

Signed-off-by: Joshua Schmid <jschmid@suse.de>

qa/suites/rados/verity/tasks/mon_recovery: whitelist SLOW_OPS

The mon can see slow ops when thrashing.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: update_foo -> apply_foo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33639 into master

* refs/pull/33639/head:
pybind/mgr/mgr_module: fix standby module logging options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31109 from liupengs/wip-msg-async-fix-event-center-block

msg/async/rdma: unblock event center if the peer is down when connecting

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: move C_handle_connection to RDMAConnectionSocketImpl.cc

Signed-off-by: Peng Liu <liupeng37@baidu.com>

Merge pull request #33591 from badone/wip-install-deps-set-gpgcheck-for-reals

install-deps.sh: Actually set gpgcheck to false

Reviewed-by: Dan Mick <dan.mick@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: fix bug event center is blocked by rdma construct connection for transport ib sync msg

We construct a tcp connection to transport ib sync msg, if the
remote node is shutdown (shutdown by accident), the net.connect will be blocked until timeout
is reached, which cause the event center be blocked.

This bug may cause mon probe timeout and osd not reply, and so on.

Signed-off-by: Peng Liu <liupeng37@baidu.com>

mgr/test_orchestrator: add force flag to remove_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: update

Most of these were broken due to CLI changes weeks ago.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33634 into master

* refs/pull/33634/head:
qa/workunits/cephadm/test_cephadm.sh: dump logs on exit
qa/workunits/cephadm/test_cephadm.sh: add `cleanup` function

Reviewed-by: Sage Weil <sage@redhat.com>

pybind/mgr/mgr_module: fix standby module logging options

We need to define the module options and their default so that
_configure_logging can succeed.

Broken by 8ec3b3d3cc7d4c2d2678f460de27ecc8cb07d8fe

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33438 into master

* refs/pull/33438/head:
cephadm: add prometheus adopt

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33433 into master

* refs/pull/33433/head:
cephadm: also return JSON decode error.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33596 from badone/wip-serve-doc-python3

admin/serve-doc: Switch to python3 only

Reviewed-by: Kefu Chai <kchai@redhat.com>

admin/serve-doc: Switch to python3 only

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: dump logs on exit

dumps the last few lines from each of the surviving daemon logs

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/workunits/cephadm/test_cephadm.sh: add `cleanup` function

moves logic for clean-up during `trap EXIT` into a function

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #33624 from trociny/wip-dateutil-dep

rpm,deb: fix python dateutil module dependency

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc/cephadm/administration: clarify log gathering

This is an attempt to bring the current state of the documentation more
into line with the current state of the cephadm code.

However, when I try to grab logs from a daemon on a host other than the
one where the daemon is running, I get an empty log...

References: https://tracker.ceph.com/issues/44354
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm,deb: fix python dateutil module dependency

(needed for mgr/rbd_support)

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33063 from theanalyst/boto3-ext-append

examples: rgw: add boto3 append & get usage api extensions

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #33527 from alimaredia/wip-rgw-realm-pull-log-msg

rgw: increase log level for same or older period pull msg

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #33600 from ceph/doc_config_set

doc/cephadm: adjust syntax for config set

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: adjust syntax for config set

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge PR #33226 into master

* refs/pull/33226/head:
unittest_auth: update for new ms_*_mode semantics
auth: treat mgr the same as mon when selecting auth mode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33542 into master

* refs/pull/33542/head:
mon/Monitor: fail forwarded tell commands

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #33558 into master

* refs/pull/33558/head:
common/config: fix lack of normalize_key_name() apply.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33566 from dzafman/wip-44296

test: Expect being off by up to 2 and make sure all PGs are active+clean

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33592 from tchaikov/wip-doc-search

doc,admin: fix the builtin search

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

install-deps.sh: Actually set gpgcheck to false

The current command appears to do nothing.

Fixes: https://tracker.ceph.com/issues/44344
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

doc: drop http://ayni.ceph.com/public/js/ceph.js

seems we will never have this file in foreseeable future.

See-also: https://tracker.ceph.com/issues/11490
See-also: https://tracker.ceph.com/issues/18075
Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Expect being off by up to 2 and make sure all PGs are active+clean

Fixes: https://tracker.ceph.com/issues/44296
Signed-off-by: David Zafman <dzafman@redhat.com>

doc/_static/js/ceph.js: avoid the "//" in URI

it's just annoying.

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/doc-requirements: bump sphinx version

to pick up the fix to address the search when using dirhtml builder

see https://github.com/sphinx-doc/sphinx/pull/6741

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33554 into master

* refs/pull/33554/head:
mgr/cephadm: Configure Alertmanagers in Prometheus configuration
mgr/cephadm: Create configuration for alertmanager
mgr/cephadm: Provision alertmanager
mgr/orchestrator: Add alertmanager hooks

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33585 into master

* refs/pull/33585/head:
mgr/orchestrator: fix 'orch host label rm' help text

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33586 into master

* refs/pull/33586/head:
mgr/orch: fix 'orch ls' table spacing

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33587 into master

* refs/pull/33587/head:
remove cephadm-adoption-corpus as submodule

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33488 into master

* refs/pull/33488/head:
doc/mgr/cephadm: document how to see the logs
ceph: -W as shorthand for --watch-channel
pybind/mgr_module: make mgr log message prefix a bit less verbose
mgr/cephadm: silence remoto logger info messages
mgr/cephadm: add misc info messages for major events
mgr/cephadm: adjust logging
mgr/cephadm: log to cluster channel by default
pybind/mgr_module: infrastructure to allow logging to cluster log
mgr/mgr_module: log to mgr log even when log_to_file
mon: accept cephadm channel to 'ceph log last'
ceph: allow watching channels aside from cluster and audit
mon/LogMonitor: fix debug output for logmonitor output file parsing
mgr: allow logging to any channel

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #32983 into master

* refs/pull/32983/head:
mgr/cephadm: remove/replace osds

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33389 from trociny/wip-43619

rbd: schedule for running trash purge operations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks/mgr/test_orchestrator_cli: fix test_load_data

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: fix list_daemons

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32554 from dzafman/wip-dump-perf

Improve internal python to c++ interface

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #33493 from zhengchengyao/read

librbd: improved support for balanced and localized reads

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33563 from dillaman/wip-rbd-op-threads

librbd: fix potential race conditions

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: Disable self-test of diskprediction_cloud since it isn't loaded

See qa/packages/packages.yaml

Signed-off-by: David Zafman <dzafman@redhat.com>

mon mgr osd: Add dump_osd_times interface for python

The dump_osd_times interface is here for future use of a manager module.
The osd_stats and pg_dump python interfaces don't return network ping
information.

User facing ping time information is formatted 3 decimal places
Use dump_float() (low overhead) for network ping times
   used by python dump_osd_times and dump commands like
   "ceph --formet=json pg dump osds" which for now yields output like
   >>>> "1min": 0.61599999999999999,

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr/progress: Add pg_ready interface for python to get pgmap_ready state

Fixes: https://tracker.ceph.com/issues/43557
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #33559 from liewegas/fix-42328-more

msg/async/ProtocolV2: fix typo in register_lossy_clients fix

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33460 into master

* refs/pull/33460/head:
doc/cephadm: Add Troubleshooting

Reviewed-by: Sage Weil <sage@redhat.com>

doc/cephadm: Add Troubleshooting

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33518 into master

* refs/pull/33518/head:
doc/foundation: add Samsung

Reviewed-by: Myougnwon Oh <myoungwon.oh@samsung.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Myoungwon Oh <omwmw@sk.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33044 from jschmid1/drive_group_docs

doc/drivegroups: add docs for DriveGroups with excessive examples

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

doc/mgr/cephadm: document how to see the logs

Signed-off-by: Sage Weil <sage@redhat.com>

ceph: -W as shorthand for --watch-channel

'ceph -w' to watch default cluster channel
'ceph -W foo' to watch channel 'foo'

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr_module: make mgr log message prefix a bit less verbose

2020-02-25T16:22:48.484-0600 7f664dcec700  0 [cephadm DEBUG root] code: 0

instead of

2020-02-25T16:19:48.359-0600 7f0c0d776700  0 [cephadm] [DEBUG] [root] code: 0

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: silence remoto logger info messages

Remoto issues events like

 Running command: /usr/bin/python3 -u

at level 'info' which are below the threshold of other 'info' messages in
cephadm.  So, silence info and below.  We still want to see warning and
above.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: add misc info messages for major events

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: adjust logging

- drop a bunch of stale debug messages
- single site that logs full cephadm out/err/code

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: log to cluster channel by default

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr_module: infrastructure to allow logging to cluster log

This is incomplete--a module that wants to opt-in needs to call
_enable_cluster_log() and manage the level themselves.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/mgr_module: log to mgr log even when log_to_file

Signed-off-by: Sage Weil <sage@redhat.com>

mon: accept cephadm channel to 'ceph log last'

Signed-off-by: Sage Weil <sage@redhat.com>

ceph: allow watching channels aside from cluster and audit

Signed-off-by: Sage Weil <sage@redhat.com>

mon/LogMonitor: fix debug output for logmonitor output file parsing

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: allow logging to any channel

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33218 from yuvalif/add_cls_queue_ut

cls/queue: add unit tests

Reviewed-by: Pritha Srivastava <prsivas@redhat.com>

remove cephadm-adoption-corpus as submodule

The test_adoption.sh just clones from master--no need to make this a
submodule.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: fix 'orch ls' table spacing

This command crossed paths with the PR that changed the spacing.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: fix 'orch host label rm' help text

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: increase log level for same or older period pull msg

Signed-off-by: Ali Maredia <amaredia@redhat.com>

mgr/rbd_support: wait for latest osdmap on init

The main goal is to initialize the librados instance before
starting any thread that may use it.

Signed-off-by: Mykola Golub <mgolub@suse.com>