Merge pull request #33690 from rhcs-dashboard/44237-remove-config-opt-read-from-system-roles

mgr/dashboard: remove 'config-opt: read' perm. from system roles.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #33699 from liewegas/fix-44078

common/ceph_time: tolerate mono time going backwards

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #33701 from tchaikov/wip-mgr-orch-exception

mgr/orch: try harder when pickle fails to marshal an exception

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #33682 from thinkercui/bugfix_monsession_map_lock

mon: Get session_map_lock before remove_session

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33703 from smithfarm/wip-doc-mon_status

doc: adjust for mon_status changes in octopus

Reviewed-by: Sage Weil <sage@redhat.com>

mon: Get session_map_lock before remove_session

Get session_map_lock to prevent race.

Fixes: https://tracker.ceph.com/issues/44407
Signed-off-by: Xiaofei Cui <cuixiaofei@sangfor.com.cn>

Merge pull request #33482 from rosinL/wip-fix-unitest-seastar-socket

test/test_socket: fix dispatch_sockets() unexpected exception

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #33584 from liewegas/fix-39525

compressor/lz4: work around bug in liblz4 versions <1.8.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32444 from majianpeng/simple-debug-message

osd/PrimaryLogPG: simple debug message.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33659 from rzarzynski/wip-osd-ps-plco-via-setter

osd: peering updates peer_last_complete_ondisk via setter.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33660 from rzarzynski/wip-osd-clean-pgtrim-includes

osd: drop unnecessary includes of messages/MOSDPGTrim.h.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33673 from cbodley/wip-denc-enum

denc: support enums wider than 8 bits

Reviewed-By: Adam Emersen <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33689 from Songweibin/wip-fix-typos

doc/rbd: fix small typos

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #33667 into master

* refs/pull/33667/head:
mgr/orch: show placement in 'orch ls'
mgr/orch: fix SPEC alignment in 'orch ls'
mgr/orch: include spec ref in ServiceDescription

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33669 into master

* refs/pull/33669/head:
mgr/cephadm: reduce log level for NodeAssignment
mgr/cephadm: make NodeAssignment return a simple host list

Reviewed-by: Joshua Schmid <jschmid@suse.de>

doc: adjust for mon_status changes in octopus

In octopus, we no longer have "ceph mon_status", but we do have
"ceph tell mon.<id> mon_status", so reflect this in the docs.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #33678 into master

* refs/pull/33678/head:
mgr/cephadm: fix upgrade when daemon is stopped

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33671 into master

* refs/pull/33671/head:
mgr/cephadm: refresh configs when dependencies change
mgr/cephadm: track last config time and dependencies

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33695 into master

* refs/pull/33695/head:
mgr/orch: add apply alertmanager to CLI
mgr/orch: add apply grafana to CLI

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33672 into master

* refs/pull/33672/head:
Refactor, prepare for other adoptions

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/test_orchestrator_cli: enable test_osd_create exception check

this reverts 9639acfefe09c87adc821bb5c5cc41974685331d, as the test does
make sense. what fails this test is the machinary to marshal/unmarshal
exception fails to handle un-picklable exceptions. the previous commit
is supposed to use a fallback to handle them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: try harder when pickle fails to marshal an exception

pickle cannot marshal instances of class not defined in the top level of
a module. for instance, `DriveGroupValidationError` is defined in a
submodule of `ceph` python module, that's why we cannot capture it.

this prevent the `ceph` command line from getting a proper error when
the command fails if the command is implemented using cross python
module call(s).

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33664 into master

* refs/pull/33664/head:
doc: change case from `apis` to `APIs`

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33655 from p-se/pse-fix-prom-alert-pool-full

monitoring: restore lost fix for `pool full` alert

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #33636 into master

* refs/pull/33636/head:
qa: add upgrade test for volume upgrade from legacy

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

common/ceph_time: tolerate mono time going backwards

Some kernels (and possibly some hardware?) can trigger a monotonic clock
that goes back in time.  That, in turn, can lead to a negative monotonic
time span.  This would trigger an assert.

This this problem seems to be widespread, tolerate the case and interpret
it as a 0-length interval (vs something negative).

Fixes: https://tracker.ceph.com/issues/44078
Fixes: https://tracker.ceph.com/issues/43365
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33694 into master

* refs/pull/33694/head:
cephadm: fix `alertmanager not implemented yet`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33679 into master

* refs/pull/33679/head:
cephadm: fix failure when getting keyring for deploying daemons

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/orch: add apply alertmanager to CLI

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add apply grafana to CLI

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refresh configs when dependencies change

If a daemon config (e.g., prometheus) depends on other daemons'
existence, refresh the config if that list changes (e.g., new
node-exporter, mgr removed, etc.).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33588 from dillaman/wip-rbd-snapshot-mirror

rbd-mirror: initial end-to-end test and associated bug fixes

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/orch: show placement in 'orch ls'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix upgrade when daemon is stopped

Tried to fix this in 5649d5571c14 but didn't remove the early image_id
check.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix `alertmanager not implemented yet`

```
Traceback (most recent call last):
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 391, in do_work
    res = self._on_complete_(*args, **kwargs)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 455, in call_self
    return f(self, *inner_args)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 2576, in _create_alertmanager
    custom_config=self._generate_alertmanager_config)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 2051, in _create_daemon
    stdin=json.dumps(cephadm_config))
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 1459, in _run_cephadm
    code, '\n'.join(err)))
RuntimeError: cephadm exited with an error code: 1, stderr:ERROR: alertmanager not implemented yet
```

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: remove 'config-opt: read' perm. from system roles.

Fixes: https://tracker.ceph.com/issues/44237
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

doc/rbd: fix small typos

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #33120 from vshankar/wip-mgr-volumes-clone-sync-attrs

mgr/volumes: sync inode attributes for cloned subvolumes

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #32423 from bk201/wip-43420

mgr/dashboard: allow refreshing inventory page

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: change case from `apis` to `APIs`

use APIs instead of apis to be consistent throughout.

fixes: https://tracker.ceph.com/issues/44374
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #33680 from tchaikov/wip-crimson-cleanups

crimson: clang related cleanups

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #31910 from rosinL/wip-fix-dpdk-test

test/msgr: fix ComplexTest fail when using DPDK protocal stack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33661 from bk203/patch-1

doc: update Zabbix template reference

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: add upgrade test for volume upgrade from legacy

This tests that volumes created using the ceph_volume_client.py library
continue to be accessible/function via the Nautilus/Octopus ceph-mgr
volumes plugin.

Fixes: https://tracker.ceph.com/issues/42723
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cmake: build used source

to address the link failures when compiling using clang

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not capture unused variable

this silences the warning of:
```
../src/crimson/osd/osdmap_gate.cc:48:38: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
  std::for_each(first, last, [epoch, this](auto& blocked_requests) {
                                   ~~^~~~
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: fix failure when getting keyring for deploying daemons

Fixes: https://tracker.ceph.com/issues/44390
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

doc: update Zabbix template reference

The old link references a 2017 version of the template, I experienced problems using this version of the template with the latest version of Ceph. Ceph would report "Failed to send data to Zabbix", by importing the newer 2019 version of the template within Zabbix Ceph could again send data (due to changed Zabbix Trapper item keys). Propose to replace the link for a link referencing the master branch of the template so the newest version is always referenced in the docs.

Signed-off-by: Mathijs Smit <m.smit@goldenvalue.nl>

Merge pull request #33621 from aclamk/fix_mempool_btree

test/test_mempool: test accounting for btree_map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33363 into master

* refs/pull/33363/head:
mds: add perf counters for openfiletable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33491 into master

* refs/pull/33491/head:
mount.ceph: add "fs=<fs_name>" mount options support

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33506 into master

* refs/pull/33506/head:
client: add client_fs mount option support

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33538 into master

* refs/pull/33538/head:
mds: just delete MDSIOContextBase during shutdown
mds: add comment explaining unusual locking for MDSIOContextBase

Reviewed-by: Zheng Yan <zyan@redhat.com>

mgr/orch: fix SPEC alignment in 'orch ls'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: include spec ref in ServiceDescription

Signed-off-by: Sage Weil <sage@redhat.com>

denc: add denc_traits for enum class

enum traits cast to/from underlying type, and use the underlying type's
traits for encode/decode

Signed-off-by: Casey Bodley <cbodley@redhat.com>

denc: remove conditional underlying_type_t from ceph_le* traits

an enum's underlying type must be integral, so it's not possible for
ceph_le<T> to match for enums. for example, this:

  enum class enum_class_16 : ceph_le16 {};

fails to compile with:
underlying type ‘ceph_le16’ {aka ‘ceph_le<short unsigned int>’} of
‘enum_class_16’ must be an integral type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Refactor, prepare for other adoptions

Move the prometheus adopt out of command_adopt. refactor skip pull,
legacy check and fsid lock.

Signed-off-by: Eric Jackson <swiftgist@gmail.com>

test: add failing denc tests for enums

all but the 8-bit enums fail to compile because they don't match the
traits specialization for underlying_type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rbd-mirror: move resetting of snapshot replayer rescan variable

The `m_image_updated` boolean should be reset at the start of the
state checking loop now that we scan the local image meta and check
for forced-promotion of the local image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33601 into master

* refs/pull/33601/head:
qa/tasks/thrashosds-health: disable osd_max_markdown behavior
osd: derr on do_shutdown cases

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: reduce log level for NodeAssignment

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make NodeAssignment return a simple host list

We shouldn't modify the PlacementSpec in place (or at all).  Instead,
simply return a list of the hosts where we want to locate daemons.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: track last config time and dependencies

Every time we deploy or reconfig a service, record the dependencies
and service dependency list.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33647 into master

* refs/pull/33647/head:
vstart.sh: fix install of cephadm ssh keys from ~/.ssh

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33648 into master

* refs/pull/33648/head:
qa/suites/cephadm: move orchestrator_cli test into rados/cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33658 into master

* refs/pull/33658/head:
cephadm: remove remaining crash_keyring cruft
vstart.sh: deploy crash containers
cephadm: bootstrap: deploy crash on every host
mgr/cephadm: deploy crash daemons
mgr/orch: add crash hooks
cephadm: do not magically deploy 'crash' container

Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm: remove remaining crash_keyring cruft

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/lz4: rebuild if buffer is not contiguous

In older versions of lz4 (specifically < 1.8.2) bit errors
can be introduced when compressing from fragmented memory.  The lz4
bug was fixed by this lz4 commit:

commit af127334670a5e7b710bbd6adb71aa7c3ef0cd72
Author: Yann Collet <cyan@fb.com>
Date:   Sat May 5 18:24:11 2018 -0700

    fixed frametest error

    The error can be reproduced using following command :
    ./frametest -v -i100000000 -s1659 -t31096808

    It's actually a bug in the stream LZ4 API,
    when starting a new stream
    and providing a first chunk to complete with size < MINMATCH.
    In which case, the chunk becomes a dictionary.
    No hash was generated and stored,
    but the chunk is accessible as default position 0 points to dictStart,
    and position 0 is still within MAX_DISTANCE.
    Then, next attempt to read 32-bits from position 0 fails.

    The issue would have been mitigated by starting from index 64 KB,
    effectively eliminating position 0 as too far away.

    The proper fix is to eliminate such "dictionary" as too small.
    Which is what this patch does.

This is a workaround to rebuild our input buffer into a continguos buffer
if it is not already contiguous.

Fixes: https://tracker.ceph.com/issues/39525
Signed-off-by: Sage Weil <sage@redhat.com>

qa/objectstore: test all bluestore compression algorithms

Signed-off-by: Sage Weil <sage@redhat.com>

test/compressor: test round trip of an osdmap

Check if the compressors can compress/decompress a bufferlist which is not word
aligned, such as a freshly-encoded osdmap.

Related-to: https://tracker.ceph.com/issues/39525
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge PR #33633 into master

* refs/pull/33633/head:
msg: add get_{pid,random}_nonce() helpers
msg/Policy: make stateless_server default to anon (again)
osd: drop broken 'poison pill'

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33642 from trociny/wip-run-rbd-tests-pool-init

test/run-rbd-tests: properly initialize newly created rbd pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/cephadm: move orchestrator_cli test into rados/cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: deploy crash containers

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: bootstrap: deploy crash on every host

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: deploy crash daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add crash hooks

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not magically deploy 'crash' container

We'll soon do this explicitly with a persistent spec instead.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33523 into master

* refs/pull/33523/head:
mgr/orch: ServiceSpec: drop 'count'
mgr/rook: use spec.placement.count (instead of spec.count)
mgr/cephadm: make HostAssignment make sense
mgr/orch: PlacementSpec: do not combine all_hosts with anything else
mgr/orch: use PlacementSpec.from_strings() for all CLI commands

Reviewed-by: Joshua Schmid <jschmid@suse.de>

librbd: clean-up special casing for enabling mirroring on creating image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa: test cases for snapshot-based mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: simple snapshot split-brain detection

If the local image has a primary demotion and the remote does not
have a matching non-primary demotion snapshot, we should throw a
split-brain error.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refresh image after creating primary mirror snapshot

If RPC was used to create the snapshot, the local image context will
not yet have the snapshot id and will therefore return CEPH_NOSNAP.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: acquire exclusive lock from peer when removing

This solves an issue with snapshot-based mirroring when the
rbd-mirror daemon is the exclusive lock owner. For other cases,
it still checks for watchers before proceeding.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fixed issue with deep-copy incorrectly setting parent spec

When not syncing the full image, the destination image would receive
the incorrect source parent spec instead of the expected destination
parent spec.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: do not show mirroring mode if not enabled

The 'info' action was incorrectly showing the mirroring mode
when mirroring was not enabled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allow force promotions to create necessary orphan snapshot

The rbd-mirror daemon needs to accept the RPC request to create an
orphan snapshot so that it knows to shut-down the replayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: forced promotion of snapshot mirrored image needs to create orphan

Tweak the state machine to create an orphan record if the current
state is not demoted.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: properly handle promoted/demoted snapshot-based images

We should return -EREMOTEIO if we don't have any primary images to sync to
since we want display a warning. Additionally, don't attempt to sync a
remote snapshot against a primary (demoted) local snapshot since it would
have an invalid primary snapshot id.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: improve debug logging on primary mirror snapshot creation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: fixed typo in mirror image state ostream operator

The non-primary demoted state was incorrectly outputting demoted
twice.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: do not re-register the ASOK if the image replayer isn't running

Avoid the possibility of a race when shutting down and post-unregister a status
update causes the ASOK to become re-registered.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32660 from cbodley/wip-rgw-admin-zone-shards

rgw: Increase the default number of RGW bucket shards

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #33628 from cbodley/wip-44353

rgw: fix sharded bucket listing with prefix/delimiter

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #31922 from rhcs-dashboard/42094-bucket-mfa-delete

mgr/dashboard: enable/disable MFA Delete on RGW bucket

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33178 from rhcs-dashboard/42475-rgw-api-keys-allowed

mgr/dashboard: do not show RGW API keys if only read-only privileges

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

osd: drop unnecessary includes of messages/MOSDPGTrim.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: peering updates peer_last_complete_ondisk via setter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

test/test_socket: fix dispatch_sockets() unexpected exception

In the test test_unexpected_down(), there is client running on CPU0
and server runing on CPU1, the server maybe finished before the client
(client and other workloads share CPU0), so client couldn't read data
and throws "dispatch_sockets(): cb_client() got unexpected exception
std::system_error (error crimson::net:4, read eof)"

Signed-off-by: luo rixin <luorixin@huawei.com>