Merge PR #33672 into master

* refs/pull/33672/head:
Refactor, prepare for other adoptions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33664 into master

* refs/pull/33664/head:
doc: change case from `apis` to `APIs`

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33655 from p-se/pse-fix-prom-alert-pool-full

monitoring: restore lost fix for `pool full` alert

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #33636 into master

* refs/pull/33636/head:
qa: add upgrade test for volume upgrade from legacy

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #33694 into master

* refs/pull/33694/head:
cephadm: fix `alertmanager not implemented yet`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33679 into master

* refs/pull/33679/head:
cephadm: fix failure when getting keyring for deploying daemons

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33588 from dillaman/wip-rbd-snapshot-mirror

rbd-mirror: initial end-to-end test and associated bug fixes

Reviewed-by: Mykola Golub <mgolub@suse.com>

cephadm: fix `alertmanager not implemented yet`

```
Traceback (most recent call last):
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 391, in do_work
    res = self._on_complete_(*args, **kwargs)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 455, in call_self
    return f(self, *inner_args)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 2576, in _create_alertmanager
    custom_config=self._generate_alertmanager_config)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 2051, in _create_daemon
    stdin=json.dumps(cephadm_config))
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 1459, in _run_cephadm
    code, '\n'.join(err)))
RuntimeError: cephadm exited with an error code: 1, stderr:ERROR: alertmanager not implemented yet
```

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #33120 from vshankar/wip-mgr-volumes-clone-sync-attrs

mgr/volumes: sync inode attributes for cloned subvolumes

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #32423 from bk201/wip-43420

mgr/dashboard: allow refreshing inventory page

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: change case from `apis` to `APIs`

use APIs instead of apis to be consistent throughout.

fixes: https://tracker.ceph.com/issues/44374
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #33680 from tchaikov/wip-crimson-cleanups

crimson: clang related cleanups

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #31910 from rosinL/wip-fix-dpdk-test

test/msgr: fix ComplexTest fail when using DPDK protocal stack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33661 from bk203/patch-1

doc: update Zabbix template reference

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: add upgrade test for volume upgrade from legacy

This tests that volumes created using the ceph_volume_client.py library
continue to be accessible/function via the Nautilus/Octopus ceph-mgr
volumes plugin.

Fixes: https://tracker.ceph.com/issues/42723
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cmake: build used source

to address the link failures when compiling using clang

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not capture unused variable

this silences the warning of:
```
../src/crimson/osd/osdmap_gate.cc:48:38: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
  std::for_each(first, last, [epoch, this](auto& blocked_requests) {
                                   ~~^~~~
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: fix failure when getting keyring for deploying daemons

Fixes: https://tracker.ceph.com/issues/44390
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

doc: update Zabbix template reference

The old link references a 2017 version of the template, I experienced problems using this version of the template with the latest version of Ceph. Ceph would report "Failed to send data to Zabbix", by importing the newer 2019 version of the template within Zabbix Ceph could again send data (due to changed Zabbix Trapper item keys). Propose to replace the link for a link referencing the master branch of the template so the newest version is always referenced in the docs.

Signed-off-by: Mathijs Smit <m.smit@goldenvalue.nl>

Merge pull request #33621 from aclamk/fix_mempool_btree

test/test_mempool: test accounting for btree_map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33363 into master

* refs/pull/33363/head:
mds: add perf counters for openfiletable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33491 into master

* refs/pull/33491/head:
mount.ceph: add "fs=<fs_name>" mount options support

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33506 into master

* refs/pull/33506/head:
client: add client_fs mount option support

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33538 into master

* refs/pull/33538/head:
mds: just delete MDSIOContextBase during shutdown
mds: add comment explaining unusual locking for MDSIOContextBase

Reviewed-by: Zheng Yan <zyan@redhat.com>

Refactor, prepare for other adoptions

Move the prometheus adopt out of command_adopt. refactor skip pull,
legacy check and fsid lock.

Signed-off-by: Eric Jackson <swiftgist@gmail.com>

rbd-mirror: move resetting of snapshot replayer rescan variable

The `m_image_updated` boolean should be reset at the start of the
state checking loop now that we scan the local image meta and check
for forced-promotion of the local image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33601 into master

* refs/pull/33601/head:
qa/tasks/thrashosds-health: disable osd_max_markdown behavior
osd: derr on do_shutdown cases

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #33647 into master

* refs/pull/33647/head:
vstart.sh: fix install of cephadm ssh keys from ~/.ssh

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33648 into master

* refs/pull/33648/head:
qa/suites/cephadm: move orchestrator_cli test into rados/cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33658 into master

* refs/pull/33658/head:
cephadm: remove remaining crash_keyring cruft
vstart.sh: deploy crash containers
cephadm: bootstrap: deploy crash on every host
mgr/cephadm: deploy crash daemons
mgr/orch: add crash hooks
cephadm: do not magically deploy 'crash' container

Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm: remove remaining crash_keyring cruft

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33633 into master

* refs/pull/33633/head:
msg: add get_{pid,random}_nonce() helpers
msg/Policy: make stateless_server default to anon (again)
osd: drop broken 'poison pill'

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33642 from trociny/wip-run-rbd-tests-pool-init

test/run-rbd-tests: properly initialize newly created rbd pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/cephadm: move orchestrator_cli test into rados/cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: deploy crash containers

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: bootstrap: deploy crash on every host

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: deploy crash daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add crash hooks

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not magically deploy 'crash' container

We'll soon do this explicitly with a persistent spec instead.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33523 into master

* refs/pull/33523/head:
mgr/orch: ServiceSpec: drop 'count'
mgr/rook: use spec.placement.count (instead of spec.count)
mgr/cephadm: make HostAssignment make sense
mgr/orch: PlacementSpec: do not combine all_hosts with anything else
mgr/orch: use PlacementSpec.from_strings() for all CLI commands

Reviewed-by: Joshua Schmid <jschmid@suse.de>

librbd: clean-up special casing for enabling mirroring on creating image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa: test cases for snapshot-based mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: simple snapshot split-brain detection

If the local image has a primary demotion and the remote does not
have a matching non-primary demotion snapshot, we should throw a
split-brain error.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refresh image after creating primary mirror snapshot

If RPC was used to create the snapshot, the local image context will
not yet have the snapshot id and will therefore return CEPH_NOSNAP.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: acquire exclusive lock from peer when removing

This solves an issue with snapshot-based mirroring when the
rbd-mirror daemon is the exclusive lock owner. For other cases,
it still checks for watchers before proceeding.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fixed issue with deep-copy incorrectly setting parent spec

When not syncing the full image, the destination image would receive
the incorrect source parent spec instead of the expected destination
parent spec.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: do not show mirroring mode if not enabled

The 'info' action was incorrectly showing the mirroring mode
when mirroring was not enabled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allow force promotions to create necessary orphan snapshot

The rbd-mirror daemon needs to accept the RPC request to create an
orphan snapshot so that it knows to shut-down the replayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: forced promotion of snapshot mirrored image needs to create orphan

Tweak the state machine to create an orphan record if the current
state is not demoted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: properly handle promoted/demoted snapshot-based images

We should return -EREMOTEIO if we don't have any primary images to sync to
since we want display a warning. Additionally, don't attempt to sync a
remote snapshot against a primary (demoted) local snapshot since it would
have an invalid primary snapshot id.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: improve debug logging on primary mirror snapshot creation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: fixed typo in mirror image state ostream operator

The non-primary demoted state was incorrectly outputting demoted
twice.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: do not re-register the ASOK if the image replayer isn't running

Avoid the possibility of a race when shutting down and post-unregister a status
update causes the ASOK to become re-registered.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32660 from cbodley/wip-rgw-admin-zone-shards

rgw: Increase the default number of RGW bucket shards

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #33628 from cbodley/wip-44353

rgw: fix sharded bucket listing with prefix/delimiter

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #31922 from rhcs-dashboard/42094-bucket-mfa-delete

mgr/dashboard: enable/disable MFA Delete on RGW bucket

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33178 from rhcs-dashboard/42475-rgw-api-keys-allowed

mgr/dashboard: do not show RGW API keys if only read-only privileges

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

common/mempool: Add test for mempool in btree_map

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #33638 from tchaikov/wip-btree

include/cpp-btree: use the same type when allocate/deallocate

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #33645 from liewegas/fix-mon-down-cephtool-test

qa/suites/rados/singleton-bluestore/cephtool: whitelist MON_DOWN

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33536 from yuvalif/sync_object_write_with_return_value_op

librados: allow passing flags to operate sync APIs

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #33583 from rzarzynski/wip-recovery-cleanup1

osd: tiny clean-ups around the backfill

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33593 from dzafman/wip-cot-fix

test: Fix failing ceph_objectstore_tool.py test

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

monitoring: restore lost `pool full` alert

Fixes: https://tracker.ceph.com/issues/44366
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #33373 from rzarzynski/wip-bl-bumpup_api_verion

common, include: bump the version of ceph::buffer's C++ API.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33572 from p-se/wip-pse-grafana-unit-issue

mgr/dashboard: fix stray requests/error in Grafana unit test

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #31041 from liu-chunmei/ceph_seastar_alien_blue_store

crimson:: add alien blue store

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33595 from cyx1231st/wip-crimson-osd-fixes

crimson/osd: misc fixes for OSD reboot-ability

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33597 from optimistyzy/228_improve_sync_read

NVMEDevice: Remove the unnecessary aio_wait in sync read

Reviewed-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: use the same type when allocate/deallocate

btree_set<> by default uses `std::allocator<Key>`, and btree_map by
default uses `std::allocator<std::pair<Key, Value>>`.

before this change, btree uses the allocator directly for allocating n
elements where element is `Key` or `std::pair<Key, Value>` respectively,
while "n" is actually supposed to be the number of bytes used by each
node which is being allocated.
but, what we need to allocate is actually a "node_type" for holding
multiple slots, and each slot holds an element. in addition to the
slots, a node also keeps track of metadata for btree itself. in short,
what we allocate now is (in bytes):

  alignof(sizeof(node_type)) * sizeof(element)

but what we should allocate is (in bytes):

  alignof(sizeof(node_type))

in this change:

* always rebind the allocator to the correct aligned type with given
  alignment
* extract the allocator related helpers into a template class

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33650 from joscollin/wip-unused-nvme

test: silence warning unused variable nvme

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: silence warning unused variable nvme

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #33512 into master

* refs/pull/33512/head:
os/bluestore/BlueFS: less verbose about alloc adjustments

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #33575 into master

* refs/pull/33575/head:
mgr/telemetry: fix and document proxy usage

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Yaarit Hatuka <yaarithatuka@gmail.com>

Merge PR #33644 into master

* refs/pull/33644/head:
qa/suites/rados/verity/tasks/mon_recovery: whitelist SLOW_OPS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33541 into master

* refs/pull/33541/head:
mgr/test_orchestrator: add daemon_action
qa/tasksmgr/test_orchestrator_cli: fix test_service_action
qa/tasks/test_orchestrator_cli: disable test_osd_create exception check
mgr/test_orchestrator: update_foo -> apply_foo
mgr/test_orchestrator: add force flag to remove_daemons
qa/tasks/mgr/test_orchestrator_cli: update
qa/tasks/mgr/test_orchestrator_cli: fix test_load_data
mgr/test_orchestrator: fix list_daemons

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #33627 into master

* refs/pull/33627/head:
doc/cephadm/administration: clarify log gathering

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/orch: ServiceSpec: drop 'count'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/rook: use spec.placement.count (instead of spec.count)

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make HostAssignment make sense

- if hosts are passed, use those.
- if all_hosts=true, use all hosts.

Otherwise, build a set of hosts based on the labels--either explicit or
implied.  If there's no label, use all hosts.

If there is a count, use a subset of candidate hosts.  If there was no
label and there is no count, fail.

If count and hosts are both provided, then we either (1) use the hosts
as the candidate list and place among them (if len(hosts) >= count), or
(2) ensure that any result includes *at least* the provided hosts.

Add tests that (I'm pretty sure) cover cases!

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: PlacementSpec: do not combine all_hosts with anything else

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: use PlacementSpec.from_strings() for all CLI commands

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33553 into master

* refs/pull/33553/head:
mgr/cephadm: orch ls: include specs with no daemons
mgr/cephadm: orch ls: show spec size
mgr/orch: remove unused fields in RGWSpec
mgr/orch: fix ServiceSpec fields
mgr/cephadm: simplify spec apply
pybind/mgr/mgr_module: revert PersistentStoreDict seperator
mgr/cephadm: apply services after refreshing inventory
mgr/cephadm: catch and log exceptions from apply
mgr/orch: no extra whitespace in stored json specs
mgr/cephadm: drop daemon_type arg to _apply_service
mgr/cephadm: use _apply() helper for all apply_ methods
mgr/cephadm: replace PersistentStoreDict with SpecStore
mgr/cephadm: do not remove service spec when removing a daemon
mgr/cephadm: rename completion variables&cleanup
mgr/cephadm: leverage service specs

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/test_orchestrator: add daemon_action

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasksmgr/test_orchestrator_cli: fix test_service_action

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: fix install of cephadm ssh keys from ~/.ssh

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/test_orchestrator_cli: disable test_osd_create exception check

I'm not sure what behavior we want here...

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: orch ls: include specs with no daemons

This way you see entries like

NAME           RUNNING  REFRESHED  IMAGE NAME                                      IMAGE ID        SPEC
mds.foo            0/1  -          <unknown>                                       <unknown>     present

before any daemons have been created.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: orch ls: show spec size

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: remove unused fields in RGWSpec

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: fix ServiceSpec fields

- service_type is required.  Make it the first position arg to the ctor.
- service_id is the id *only* and optional.
- service_name() is the full service name (no change)

The old 'name' was previously used as the id only, so it was poorly named,
and optional, but in this series was changed to include the type, breaking
naming for a bunch of things (e.g., daemons called mds.mds.fsname.xyz).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: simplify spec apply

- Teach _apply_service how to pick the create (and config) functions, so
  that we don't need any weird wrappers in the callers.
- Replace trigger_deploy() and _apply_services() with a simpler
  _apply_all_services()
- Drop all of the per-type _apply_foo() methods.

Signed-off-by: Sage Weil <sage@redhat.com>

pybind/mgr/mgr_module: revert PersistentStoreDict seperator

This fixes the self-test.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: apply services after refreshing inventory

Ensure we have an inventory to apply against.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: catch and log exceptions from apply

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: no extra whitespace in stored json specs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: drop daemon_type arg to _apply_service

This is part of ServiceSpec now.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: use _apply() helper for all apply_ methods

Clean up a few that were missed (due to racing PRs, presumably).

'creation' -> 'update', since an apply doesn't necessarily create (it may
destroy!).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: replace PersistentStoreDict with SpecStore

Explicit implementation of the dict of specs.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: do not remove service spec when removing a daemon

This makes our behavior similar to kube: if you kill a pod, the operator
or controller will come along and create a new one (probably somewhere
else).

Signed-off-by: Sage Weil <sage@redhat.com>