qa/tasks/ceph_deploy: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/manypools: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/vstart_runner: use io.BytesIO for py3 compat

Use io.BytesIO instead of StringIO for py3 compatibility.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_manager: ensure str for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_objectstore_tool: get rid of cStringIO for py3

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/divergent_priors2: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/radosbenchsweep: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/reg11184: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/netem: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/scrub_test: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/systemd: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/admin_socket: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/cephadm: get rid of cStringIO for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/omapbench: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/mgr: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/watch_notify_stress: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/osd_max_pg_per_osd: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/radosbench.py: get rid of itervalues for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/rados: get rid of itervalues for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/daemonwatchdog: py3 compat

Address error:
        RuntimeError: dictionary changed size during iteration

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/workunit: py3 compat

Use six.ensure_str for `get_file()` output in order to
make code py3 compatible

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: fix import module path for py3 compat

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/thrashosds: fix imports for py3

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph: py3 compatibility

Addresses:
  TypeError: 'dict_values' object is not subscriptable

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks: fix imports for py3 compatibility

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph_manager: get rid of CStringIO for py3

Use io.BytesIO instead cStringIO.StringIO for py3 compatibility

Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

qa/tasks/ceph: get rid of cStringIO for py3 compat

Use io.BytesIO instead of cStringIO.StringIO
Use six.ensure_str whenever it needs to convert binary to str.

Signed-off-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

Merge pull request #33584 from liewegas/fix-39525

compressor/lz4: work around bug in liblz4 versions <1.8.2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32444 from majianpeng/simple-debug-message

osd/PrimaryLogPG: simple debug message.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33659 from rzarzynski/wip-osd-ps-plco-via-setter

osd: peering updates peer_last_complete_ondisk via setter.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33660 from rzarzynski/wip-osd-clean-pgtrim-includes

osd: drop unnecessary includes of messages/MOSDPGTrim.h.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33673 from cbodley/wip-denc-enum

denc: support enums wider than 8 bits

Reviewed-By: Adam Emersen <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33689 from Songweibin/wip-fix-typos

doc/rbd: fix small typos

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #33667 into master

* refs/pull/33667/head:
mgr/orch: show placement in 'orch ls'
mgr/orch: fix SPEC alignment in 'orch ls'
mgr/orch: include spec ref in ServiceDescription

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33669 into master

* refs/pull/33669/head:
mgr/cephadm: reduce log level for NodeAssignment
mgr/cephadm: make NodeAssignment return a simple host list

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge PR #33678 into master

* refs/pull/33678/head:
mgr/cephadm: fix upgrade when daemon is stopped

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33671 into master

* refs/pull/33671/head:
mgr/cephadm: refresh configs when dependencies change
mgr/cephadm: track last config time and dependencies

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33695 into master

* refs/pull/33695/head:
mgr/orch: add apply alertmanager to CLI
mgr/orch: add apply grafana to CLI

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33672 into master

* refs/pull/33672/head:
Refactor, prepare for other adoptions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33664 into master

* refs/pull/33664/head:
doc: change case from `apis` to `APIs`

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33655 from p-se/pse-fix-prom-alert-pool-full

monitoring: restore lost fix for `pool full` alert

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #33636 into master

* refs/pull/33636/head:
qa: add upgrade test for volume upgrade from legacy

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #33694 into master

* refs/pull/33694/head:
cephadm: fix `alertmanager not implemented yet`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #33679 into master

* refs/pull/33679/head:
cephadm: fix failure when getting keyring for deploying daemons

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/orch: add apply alertmanager to CLI

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add apply grafana to CLI

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: refresh configs when dependencies change

If a daemon config (e.g., prometheus) depends on other daemons'
existence, refresh the config if that list changes (e.g., new
node-exporter, mgr removed, etc.).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33588 from dillaman/wip-rbd-snapshot-mirror

rbd-mirror: initial end-to-end test and associated bug fixes

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/orch: show placement in 'orch ls'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix upgrade when daemon is stopped

Tried to fix this in 5649d5571c14 but didn't remove the early image_id
check.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix `alertmanager not implemented yet`

```
Traceback (most recent call last):
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 391, in do_work
    res = self._on_complete_(*args, **kwargs)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 455, in call_self
    return f(self, *inner_args)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 2576, in _create_alertmanager
    custom_config=self._generate_alertmanager_config)
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 2051, in _create_daemon
    stdin=json.dumps(cephadm_config))
  File "/home/user/src/ceph/src/pybind/mgr/cephadm/module.py", line 1459, in _run_cephadm
    code, '\n'.join(err)))
RuntimeError: cephadm exited with an error code: 1, stderr:ERROR: alertmanager not implemented yet
```

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

doc/rbd: fix small typos

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #33120 from vshankar/wip-mgr-volumes-clone-sync-attrs

mgr/volumes: sync inode attributes for cloned subvolumes

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #32423 from bk201/wip-43420

mgr/dashboard: allow refreshing inventory page

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

doc: change case from `apis` to `APIs`

use APIs instead of apis to be consistent throughout.

fixes: https://tracker.ceph.com/issues/44374
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #33680 from tchaikov/wip-crimson-cleanups

crimson: clang related cleanups

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #31910 from rosinL/wip-fix-dpdk-test

test/msgr: fix ComplexTest fail when using DPDK protocal stack

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33661 from bk203/patch-1

doc: update Zabbix template reference

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: add upgrade test for volume upgrade from legacy

This tests that volumes created using the ceph_volume_client.py library
continue to be accessible/function via the Nautilus/Octopus ceph-mgr
volumes plugin.

Fixes: https://tracker.ceph.com/issues/42723
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cmake: build used source

to address the link failures when compiling using clang

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not capture unused variable

this silences the warning of:
```
../src/crimson/osd/osdmap_gate.cc:48:38: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
  std::for_each(first, last, [epoch, this](auto& blocked_requests) {
                                   ~~^~~~
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: fix failure when getting keyring for deploying daemons

Fixes: https://tracker.ceph.com/issues/44390
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

doc: update Zabbix template reference

The old link references a 2017 version of the template, I experienced problems using this version of the template with the latest version of Ceph. Ceph would report "Failed to send data to Zabbix", by importing the newer 2019 version of the template within Zabbix Ceph could again send data (due to changed Zabbix Trapper item keys). Propose to replace the link for a link referencing the master branch of the template so the newest version is always referenced in the docs.

Signed-off-by: Mathijs Smit <m.smit@goldenvalue.nl>

Merge pull request #33621 from aclamk/fix_mempool_btree

test/test_mempool: test accounting for btree_map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33363 into master

* refs/pull/33363/head:
mds: add perf counters for openfiletable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33491 into master

* refs/pull/33491/head:
mount.ceph: add "fs=<fs_name>" mount options support

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33506 into master

* refs/pull/33506/head:
client: add client_fs mount option support

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #33538 into master

* refs/pull/33538/head:
mds: just delete MDSIOContextBase during shutdown
mds: add comment explaining unusual locking for MDSIOContextBase

Reviewed-by: Zheng Yan <zyan@redhat.com>

mgr/orch: fix SPEC alignment in 'orch ls'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: include spec ref in ServiceDescription

Signed-off-by: Sage Weil <sage@redhat.com>

denc: add denc_traits for enum class

enum traits cast to/from underlying type, and use the underlying type's
traits for encode/decode

Signed-off-by: Casey Bodley <cbodley@redhat.com>

denc: remove conditional underlying_type_t from ceph_le* traits

an enum's underlying type must be integral, so it's not possible for
ceph_le<T> to match for enums. for example, this:

  enum class enum_class_16 : ceph_le16 {};

fails to compile with:
underlying type ‘ceph_le16’ {aka ‘ceph_le<short unsigned int>’} of
‘enum_class_16’ must be an integral type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Refactor, prepare for other adoptions

Move the prometheus adopt out of command_adopt. refactor skip pull,
legacy check and fsid lock.

Signed-off-by: Eric Jackson <swiftgist@gmail.com>

test: add failing denc tests for enums

all but the 8-bit enums fail to compile because they don't match the
traits specialization for underlying_type

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rbd-mirror: move resetting of snapshot replayer rescan variable

The `m_image_updated` boolean should be reset at the start of the
state checking loop now that we scan the local image meta and check
for forced-promotion of the local image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #33601 into master

* refs/pull/33601/head:
qa/tasks/thrashosds-health: disable osd_max_markdown behavior
osd: derr on do_shutdown cases

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: reduce log level for NodeAssignment

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: make NodeAssignment return a simple host list

We shouldn't modify the PlacementSpec in place (or at all).  Instead,
simply return a list of the hosts where we want to locate daemons.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: track last config time and dependencies

Every time we deploy or reconfig a service, record the dependencies
and service dependency list.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33647 into master

* refs/pull/33647/head:
vstart.sh: fix install of cephadm ssh keys from ~/.ssh

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33648 into master

* refs/pull/33648/head:
qa/suites/cephadm: move orchestrator_cli test into rados/cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #33658 into master

* refs/pull/33658/head:
cephadm: remove remaining crash_keyring cruft
vstart.sh: deploy crash containers
cephadm: bootstrap: deploy crash on every host
mgr/cephadm: deploy crash daemons
mgr/orch: add crash hooks
cephadm: do not magically deploy 'crash' container

Reviewed-by: Michael Fritch <mfritch@suse.com>

cephadm: remove remaining crash_keyring cruft

Signed-off-by: Sage Weil <sage@redhat.com>

compressor/lz4: rebuild if buffer is not contiguous

In older versions of lz4 (specifically < 1.8.2) bit errors
can be introduced when compressing from fragmented memory.  The lz4
bug was fixed by this lz4 commit:

commit af127334670a5e7b710bbd6adb71aa7c3ef0cd72
Author: Yann Collet <cyan@fb.com>
Date:   Sat May 5 18:24:11 2018 -0700

    fixed frametest error

    The error can be reproduced using following command :
    ./frametest -v -i100000000 -s1659 -t31096808

    It's actually a bug in the stream LZ4 API,
    when starting a new stream
    and providing a first chunk to complete with size < MINMATCH.
    In which case, the chunk becomes a dictionary.
    No hash was generated and stored,
    but the chunk is accessible as default position 0 points to dictStart,
    and position 0 is still within MAX_DISTANCE.
    Then, next attempt to read 32-bits from position 0 fails.

    The issue would have been mitigated by starting from index 64 KB,
    effectively eliminating position 0 as too far away.

    The proper fix is to eliminate such "dictionary" as too small.
    Which is what this patch does.

This is a workaround to rebuild our input buffer into a continguos buffer
if it is not already contiguous.

Fixes: https://tracker.ceph.com/issues/39525
Signed-off-by: Sage Weil <sage@redhat.com>

qa/objectstore: test all bluestore compression algorithms

Signed-off-by: Sage Weil <sage@redhat.com>

test/compressor: test round trip of an osdmap

Check if the compressors can compress/decompress a bufferlist which is not word
aligned, such as a freshly-encoded osdmap.

Related-to: https://tracker.ceph.com/issues/39525
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge PR #33633 into master

* refs/pull/33633/head:
msg: add get_{pid,random}_nonce() helpers
msg/Policy: make stateless_server default to anon (again)
osd: drop broken 'poison pill'

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #33642 from trociny/wip-run-rbd-tests-pool-init

test/run-rbd-tests: properly initialize newly created rbd pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/cephadm: move orchestrator_cli test into rados/cephadm

Signed-off-by: Sage Weil <sage@redhat.com>

vstart.sh: deploy crash containers

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: bootstrap: deploy crash on every host

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: deploy crash daemons

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add crash hooks

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not magically deploy 'crash' container

We'll soon do this explicitly with a persistent spec instead.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33523 into master

* refs/pull/33523/head:
mgr/orch: ServiceSpec: drop 'count'
mgr/rook: use spec.placement.count (instead of spec.count)
mgr/cephadm: make HostAssignment make sense
mgr/orch: PlacementSpec: do not combine all_hosts with anything else
mgr/orch: use PlacementSpec.from_strings() for all CLI commands

Reviewed-by: Joshua Schmid <jschmid@suse.de>

librbd: clean-up special casing for enabling mirroring on creating image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa: test cases for snapshot-based mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: simple snapshot split-brain detection

If the local image has a primary demotion and the remote does not
have a matching non-primary demotion snapshot, we should throw a
split-brain error.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>