mgr/cephadm: Make sure we don't co-locate the same daemon

Fixes: https://tracker.ceph.com/issues/44397
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #33825 into octopus

* refs/pull/33825/head:
cephadm: bootstrap: tolerate error return from -h
ceph.in: only shut down rados on clean exit

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33811 into octopus

* refs/pull/33811/head:
mgr/cephadm: fix upgrade order

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/cephadm: fix upgrade order

Create two variables, CEPH_TYPES and CEPH_UPGRADE_ORDER.  In reality they
are both the same, but this way the meaning is clear, and they lists
won't get out of sync (they should always have the same elements).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33801 into octopus

* refs/pull/33801/head:
qa/suites/rados/ceph: drop opensuse for now

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge PR #33822 into octopus

* refs/pull/33822/head:
cephadm: use `sh` instead of `bash` during enter

Reviewed-by: Sage Weil <sage@redhat.com>

cephadm: bootstrap: tolerate error return from -h

Sometimes we time out connecting to the mon to get commands and return
an error code.

See https://tracker.ceph.com/issues/44526

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33809 into octopus

* refs/pull/33809/head:
qa/standalone/scrub/osd-scrub-repair: force osdmap prop to osds
qa/standalone/scrub/osd-scrub-test: wait longer for update

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #32678 into octopus

* refs/pull/32678/head:
mgr/dashboard: support multiple DriveGroups when creating OSDs

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

cephadm: use `sh` instead of `bash` during enter

not all container images use bash (e.g. node-exporter etc)

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph.in: only shut down rados on clean exit

If we exit due to a timeout, then calling rados shutdown can lead to all
sorts of problems, because we may still have another thread that is
trying to call rados_connect and/or do some work, and rados_connect
and rados_shutdown don't (and can't!) really behave well when racing
against each other.

Note that shutdown here isn't that important--the process is about to
exit anyway.  It's only useful to exercise the shutdown code path more
often.

Fixes: https://tracker.ceph.com/issues/44526
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33793 into master

* refs/pull/33793/head:
qa/suites/rados/cephadm/upgrade: new start point
qa/tasks/cephadm: put bootstrap config etc directly in /etc/ceph
cephadm: shell: default to config and keyring in /etc/ceph, if present

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge PR #33808 into master

* refs/pull/33808/head:
mgr/cephadm: apply: fill in default placement if none is provided
mgr/cephadm: make placement truly optional (default to count=1)
mgr/cephadm: allow count == 0
mgr/cephadm: remove magic labels

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #33756 from tspmelo/wip-remove-ngx-store

mgr/dashboard: Remove ngx-store

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #33691 from rhcs-dashboard/vstart-enable-nfs-ganesa-mgmt-dashboard

vstart.sh: enable nfs-ganesha mgmt. in dashboard.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33797 from tchaikov/wip-crimson-cleanups

crimson: cleanups

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #32440 from rosinL/wip-spdk

os/bluestore/spdk: Fix the overflow error of parsing spdk coremask

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/mgr: close() in background

as per Yingxin,

application code is not required to wait for the `close()` future, it
would be safe to ignore it, because:
- `close()` will shutdown its socket synchronously;
- `close()` will create an internal `ConnectionRef` when it's closing;
- `Messenger` will wait for all connections closed during `shutdown()`;

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/buffer.cc: silence ASan warning

silences following warning
```
../src/common/buffer.cc:472:9: runtime error: member access within null pointer of type 'struct raw'
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: reorder includes

to follow
https://google.github.io/styleguide/cppguide.html#Names_and_Order_of_Includes

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mgr: use periodical timer for report

* always rearm the timer when handling MMgrConfigure
* remove `mgr::Client::tick_period`

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33776 into master

* refs/pull/33776/head:
test: Add flush_pg_stats to avoid race with getting num_shards_repaired

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: apply: fill in default placement if none is provided

Most stateless daemons get 2x (so there is a standby).  Monitoring items
get just 1x.

By default we do 5 monitors, which will gracefully degrade to one per host
if the cluster has <5 hosts.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/spdk: Fix the overflow error of parsing spdk coremask

coremask supports up to 256 bits in DPDK19.05, but the use of stoll in
NVMEManager::try_get limits the maximum use to 64 bits. Parse coremask by
hex character from low to high.

Fixes: https://tracker.ceph.com/issues/43044
Signed-off-by: Hu Ye <yehu5@huawei.com>
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>
Signed-off-by: luo rixin <luorixin@huawei.com>

Merge PR #33804 into master

* refs/pull/33804/head:
cephadm: ls: warn if daemon type (version) is not supported
cephadm: report grafana version
cephadm: report prometheus, node-exporter, alertmanager versions
cephadm: use None (not '<no value>') for monitoring daemon version

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33792 into master

* refs/pull/33792/head:
doc/cephadm: fix formatting for osd section
doc/cephadm: update 'adding mons' section to suggest/prefer 'apply'
doc/cephadm: fix formatting, typos
mgr/cephadm: implement apply_mon
mgr/cephadm: allow mon creation without explicit ip or addr
mgr/cephadm: allow _apply_service to delete mon daemon's data
mgr/cephadm: remove mon from monmap before removing mon
mgr/cephadm: do not remove mon if it breaks quorum

Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm: ls: warn if daemon type (version) is not supported

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: report grafana version

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33802 into master

* refs/pull/33802/head:
mgr/cephadm: sanity check upgrade version
mgr/cephadm: only need to invalidate once here
mgr/cephadm: upgrade requires root mode for now

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33800 into master

* refs/pull/33800/head:
mgr/cephadm: fix prom config generation when hosts have no labels or addrs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33795 into master

* refs/pull/33795/head:
mgr/orch: collapse 'SPEC' and 'PLACEMENT' columns in 'orch ls'

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/standalone/scrub/osd-scrub-repair: force osdmap prop to osds

flush_pg_stats isn't sufficient to ensure that OSDs have the latest
OSDMap.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/scrub/osd-scrub-test: wait longer for update

Fixes: https://tracker.ceph.com/issues/43865
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33788 from ajarr/wip-44438

test_volumes: fix _verify_clone_attrs call

Reviewed-by: Venky Shankar <vshankar@redhat.com>

mgr/cephadm: make placement truly optional (default to count=1)

If no placement information is provided at all, assume 1 daemon over any
host.

This could perhaps be improved with a default placement that varies by
daemon type...

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: allow count == 0

Scale a service down to 0 without removing the spec.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove magic labels

Remove the magic label behavior.  It makes the code confusing, it
makes the overall behavior hard to explain, and it makes the PlacementSpec
meaning different than what Rook is doing.

Instead, if you want mons on hosts with label 'mon', then say 'label:mon'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33686 from yuvalif/fix_data_corruption_in_cls_queue_head

cls/queue: fix data corruption in urgent data

mgr/cephadm: fix prom config generation when hosts have no labels or addrs

The inventory for a host might be {}, which evaluates as false.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33742 into master

* refs/pull/33742/head:
msg/async: s/nowait/always_async/ in EventCenter::submit_to().
msg/async: perform recv reset immediately if called inside EC.

Reviewed-by: Sage Weil <sage@redhat.com>

cephadm: report prometheus, node-exporter, alertmanager versions

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: use None (not '<no value>') for monitoring daemon version

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm/upgrade: new start point

The starting cephadm needs to look for default ceph.conf etc in /etc/ceph
for cephadm.py to be happy.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: put bootstrap config etc directly in /etc/ceph

This puts the conf and keyring in /etc/ceph earlier rather than later,
making them useful for debugging a live system *during* bootstrap.  It's
also less code.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: shell: default to config and keyring in /etc/ceph, if present

This just makes things painless for humans: they can usually run
'cephadm shell' and have a working environment.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33706 into master

* refs/pull/33706/head:
qa/suites/rados/cephadm/upgrade: adjust starting version
mgr/orch: from_strings -> from_string; do not accept a list
mgr/volumes: pass placement as string, not list
qa/tasks/mgr/test_orchestrator_cli: adjust placement args
qa/tasks/cephadm: pass apply placement as a single arg
mgr/orch: PlacementSpec: allow 'count:123'
mgr/orch: PlacementSpec: may pretty_str() match input
mgr/orch: take single placement argument
mgr/orch: PlacementSpec.from_strings: take a string *or* a list

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33625 from sebastian-philipp/python-common-drive-groups-and

python-common: Make Drive Group filter by AND, instead of OR

Reviewed-by: Joshua Schmid <jschmid@suse.de>

doc/cephadm: fix formatting for osd section

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: update 'adding mons' section to suggest/prefer 'apply'

It's nicer for users to specify the cluster/mon subnet once and let
cephadm scale mons.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm: fix formatting, typos

No need for [monitor 1] when accessing the CLI--this can happy from
any node or container that has a working CLI.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: implement apply_mon

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: allow mon creation without explicit ip or addr

Allow mons to be created if the public_network option is defined in the
config database.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: allow _apply_service to delete mon daemon's data

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: remove mon from monmap before removing mon

Check for force flag early so we don't update the monmap if the daemon
remove is going to fail anyway.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not remove mon if it breaks quorum

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: sanity check upgrade version

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: only need to invalidate once here

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: upgrade requires root mode for now

See https://tracker.ceph.com/issues/44429

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33794 into master

* refs/pull/33794/head:
pybind/mgr/mgr_util: fix pretty time delta

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33752 into master

* refs/pull/33752/head:
mgr: update metadata if an osd just joins
mgr: update "hostname" when we already have the daemon state from the same entity

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/ceph: drop opensuse for now

Until https://tracker.ceph.com/issues/44501 is resolved and the builders
are able to keep up.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33747 into master

* refs/pull/33747/head:
doc/mgr/telemetry: add 'send' command
mgr/telemetry: force --license when sending while opted-out

Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm/upgrade: adjust starting version

This version understands how to apply a mgr spec like '2;host=x' with a
semicolon.  This particular test build does.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: from_strings -> from_string; do not accept a list

Normalize all callers to pass a single string.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/volumes: pass placement as string, not list

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33780 from xxhdx1985126/wip-crimson-osd-pg-creation-issue

crimson: solve the problem that crimson-osd's created pgs stuck in "unknown" state

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: fix the problem that crimson-osd can't establish stable connection with mgr

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

crimson: decouple mgr client reconnect and connect reset handling

As of now, the following invocation sequence triggers deadlock when
closing crimson-osd's connection with mgr:
  ProtocolV2::dispatch_reset() --> crimson::mgr::Client::ms_handle_reset
--> crimson::mgr::Client::reconnect --> crimson::net::SocketConnection::close
--> crimson::net::Protocol::close()

In the above invocation sequence, ProtocalV2::dispatch_reset() enters the gate
"pending_dispatch" the leaving of which would wait for the complete of crimson::\
net::Protocal::close() which further wait for the complete of the gate's close().

This commit decouples this waiting chain.

Signed-off-by: Xuehan Xu <xxhdx1985126@163.com>

qa/tasks/mgr/test_orchestrator_cli: adjust placement args

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: pass apply placement as a single arg

Use ';' as a separator, although ' ' would also work.  ',' would not
when msgr v2 addrs are in use.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: PlacementSpec: allow 'count:123'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: PlacementSpec: may pretty_str() match input

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: take single placement argument

This is maybe a wash on the 'ceph orch ...' portion of the CLI.  However,
it means that elsewhere, like 'ceph fs volume ...', we can be consistent
and have placement be (1) optional and (2) a single arg so that it is
easier to use both positionally and as a flag (--placement=all:true).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: PlacementSpec.from_strings: take a string *or* a list

The string can be ' ', ';', or ',' separated.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: collapse 'SPEC' and 'PLACEMENT' columns in 'orch ls'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33700 into master

* refs/pull/33700/head:
mgr/cephadm: point dashboard at grafana automatically
doc/cephadm/monitoring: document process to set up monitoring with cephadm

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #33785 into master

* refs/pull/33785/head:
qa/suites/rados/cephadm: deploy all monitoring components
qa/tasks/cephadm: deploy other monitoring components

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33602 into master

* refs/pull/33602/head:
mgr/cephadm/osd: call synchronous _remove_daemon()
mgr/cephadm/osd: Add pytest for osd removal
mgr/cephadm/osd: delete transitional attributes
mgr/cephadm: Move remove osd code to osd.py

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

pybind/mgr/mgr_util: fix pretty time delta

We were off by a factor of 10 with teh 'hours' values.  Oops!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm: deploy all monitoring components

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/osd: call synchronous _remove_daemon()

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/osd: Add pytest for osd removal

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm/osd: delete transitional attributes

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: Move remove osd code to osd.py

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: point dashboard at grafana automatically

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephadm/monitoring: document process to set up monitoring with cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33772 into master

* refs/pull/33772/head:
mgr/cephadm: upgrade all ceph daemon types
mgr/cephadm: do not specify --image arg for non-ceph daemons; fix upgrade
mgr/cephadm: fix error-path return values for _run_cephadm
mgr/orch: fix RGWSpec loading
mgr/cephadm: fix 'orch rm'

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge PR #33779 into master

* refs/pull/33779/head:
mgr/orch: fix ServiceSpec deserialization error

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33614 into master

* refs/pull/33614/head:
mgr/cephadm: enable custom TLS certificates for grafana
mgr: enable verification of TLS certs without files
mgr/cephadm: dump config to JSON only once when creating daemons

Reviewed-by: Sage Weil <sage@redhat.com>

test_volumes: fix _verify_clone_attrs call

... to verify the attributes of clone and source subvolume belonging
to different subvolume groups.

Introduced in e22d546beb0472bf7e9bb038aa5e781d90a52349

Fixes: https://tracker.ceph.com/issues/44438
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #33696 from cbodley/wip-common-spliterator

common: add iterator-based string splitter

Reviewed-by: Adam Emersen <aemerson@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33754 from cbodley/wip-rgw-formatter-plain-segfault

rgw: fix string_view formatting in RGWFormatter_Plain

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge PR #33784 into master

* refs/pull/33784/head:
mgr/cephadm: update type annotation

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: update type annotation

the method signature was changed in
ea987a0e56db106f7c76d11f86b3e602257f365e

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/cephadm: deploy other monitoring components

Grafana and alertmanager.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33757 into master

* refs/pull/33757/head:
qa/valgrind.supp: abstract from ceph::buffer's symbol versioning.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33755 into master

* refs/pull/33755/head:
mgr/cephadm: make osd create on an existing LV idempotent

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/orch: fix ServiceSpec deserialization error

cephadm fails to load Service spec data when deserializing
HostPlacementSpec resource. Fix the issue by loading it properly.

Renaming the call from_dict to from_json for consistency.

This change also makes `spec` filed visible in JSON output of
ServiceSpec list.

Fixes: https://tracker.ceph.com/issues/44491
Fixes: https://tracker.ceph.com/issues/44392
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

vstart.sh: enable nfs-ganesha mgmt. in dashboard.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #33081 from bk201/wip-43166

mgr/dashboard: improve device selection modal for creating OSDs

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>