ceph-daemon: deal with lack of TemporaryDirectory on py2

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make StringIO usage py2 compatible

Otherwise,

Traceback (most recent call last):
  File "../src/ceph-daemon", line 1698, in <module>
    r = args.func()
  File "../src/ceph-daemon", line 767, in command_bootstrap
    with StringIO() as f:
AttributeError: StringIO instance has no attribute '__exit__'

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: make ConfigParser py2 compatible

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Allow removal of single notification (#31048)

mgr/dashboard: Allow removal of single notification

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Use checkbox for multi selection (#30495)

mgr/dashboard: Use checkbox for multi selection

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #31193 from rhcs-dashboard/wip-42507-master

mgr/dashboard: update dashboard CODEOWNERShip

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30988 from smithfarm/wip-bri-multi

backport-resolve-issue: resolve multiple backport issues

Merge pull request #31163 from callithea/wip-issue-42489

doc: Small update of SubmittingPatches-backports

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30879 from smithfarm/wip-cbs-check-missing-milestone

ceph-backport.sh: implement --milestones feature and more-careful vetting

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29930 from IlsooByun/rocksdb_ver

cmake: require RocksDB 5.14 or higher

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30369 into master

* refs/pull/30369/head:
doc/cephfs: add doc for cephfs io path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31201 into master

* refs/pull/31201/head:
doc: document rank option for journal reset

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #31181 into master

* refs/pull/31181/head:
ceph-daemon: only pass podman -it if need an interactive shell

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31159 from alfredodeza/wip-rm42484

ceph-volume tests remove xenial from functional testing

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31222 from sebastian-philipp/python-common-avoid-setup_requires

python-common: avoid using setup_requires in setup.py

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Allow removal of single notification

Signed-off-by: Tiago Melo <tmelo@suse.com>

ceph-daemon: only pass podman -it if need an interactive shell

Otherwise, we get errors like

2019-10-26T17:29:42.004 INFO:tasks.workunit.client.0.mira109.stderr:+ sudo /usr/sbin/ceph-daemon shell -- ceph -v
2019-10-26T17:29:42.004 INFO:tasks.workunit.client.0.mira109.stderr:+ grep 'ceph version'
2019-10-26T17:29:42.149 INFO:tasks.workunit.client.0.mira109.stderr:the input device is not a TTY

Fixes: https://tracker.ceph.com/issues/42499
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31059 into master

* refs/pull/31059/head:
mgr/rook: implement update_mds
mgr/ssh: implement 'mds update'
mgr/orchestrator_cli: add 'mds update' command
mgr/ssh: unique names for mds instances
mgr/ssh: name mgrs with random string id; add removal support
mgr/ssh: provide get_unique_name helper
mgr/orchestrator_cli: rename 'mds rm' arg
mgr/orchestrator_cli: rename 'mds add' arg
mgr/ssh: add 'mds rm'
mgr/ssh: (broken) mds rm
mgr/ssh: make mds add work
mgr/orchestrator: add count, hosts args to 'mds add'
mgr/ssh: be consistently verbose
mgr/ActivePyModules: expose 'mds_metadata' to modules

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31197 from jan--f/c-v-fix-lvm-mokeypatch

ceph-volume: mokeypatch calls to lvm related binaries

python-common: avoid using setup_requires in setup.py

Fixes: https://tracker.ceph.com/issues/42528
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31023 from idryomov/wip-krbd-udev-enumerate-retry

krbd: retry on transient errors from udev_enumerate_scan_devices()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

cmake: require RocksDB 5.14 or higher

error: ‘class rocksdb::Env’ has no member named ‘SetAllowNonOwnerAccess’
       opt.env->SetAllowNonOwnerAccess(false);
                    ^~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Ilsoo Byun <ilsoobyun@linecorp.com>

Merge pull request #31072 from sebastian-philipp/ceph-volume-inventory-device-id

ceph-volume: add Ceph's device id to inventory

Merge PR #31063 into master

* refs/pull/31063/head:
qa: disable too few PG warning during Mimic deploy

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31178 from tchaikov/wip/rpm/pyyaml

ceph.spec.in: add missing python-yaml dependency for mgr-k8sevents

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge PR #31168 into master

* refs/pull/31168/head:
ceph-daemon: try py2 import before py3
qa/suites/rados/singleton-nomsgr/ceph-daemon: make sure python3 is installed
qa/standalone/test_ceph_damon.sh: test with python2 and python3
mgr/ssh: python, not python3
ceph-daemon: python, not python3
ceph-daemon: os.makedirs
ceph-daemon: configparser is ConfigParser on py2
ceph-daemon: avoid py3-isms

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/rook: implement update_mds

Note that this still doesn't provide control over active vs standby MDS
counts.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: implement 'mds update'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: add 'mds update' command

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: unique names for mds instances

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: name mgrs with random string id; add removal support

- stop naming mgrs after the host, since that doesn't scale well
- add mgr removal support.  prefer mgrs that aren't recognized by the
  cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: provide get_unique_name helper

Signed-off-by: Sage Weil <sage@redhat.com>

doc: document rank option for journal reset

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-daemon: try py2 import before py3

See https://github.com/ceph/ceph-medic/commit/f499a89b15b72a5bb842e1ebbc7cafc4daf58c1c

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/singleton-nomsgr/ceph-daemon: make sure python3 is installed

Centos7 doesn't have it by default.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_damon.sh: test with python2 and python3

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: python, not python3

python 2.x is okay too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: python, not python3

python2 is okay too.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: os.makedirs

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: configparser is ConfigParser on py2

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: avoid py3-isms

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: rename 'mds rm' arg

'name' can be either a group prefix or an instance name, so use a general
variable name.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: rename 'mds add' arg

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add 'mds rm'

This can take *either* an mds instance name *or* a group prefix.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: (broken) mds rm

This wrong because it takes a daemon name, not a group name, and because
it relies on mds_metadata--connected daemons, not deployed daemons.  Must
fix once we have better inventory.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: make mds add work

This is clearly primitive and not where we want to end up, but an initial
step.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add count, hosts args to 'mds add'

Rook already uses the count; the hosts hint is not relevant.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: be consistently verbose

Print args and stdin in _run_ceph_daemon; drop various debug lines for
printing keyrings and configs along the way.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ActivePyModules: expose 'mds_metadata' to modules

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31180 from liewegas/revert-bluestore-pinned-list

os/bluestore: revert cache pinned list

Reviewed-by: Mark Nelson <mnelson@redhat.com>

ceph-volume: add Ceph's device id to inventory

This will benefit the orchestrator and dashboard to show a unified view of devices with SMART data

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-volume: mokeypatch calls to lvm related binaries

Otherwise these tests fail occasionally when run on a system that does
not have lvm installed.

Fixes: https://tracker.ceph.com/issues/42045
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31071 from sebastian-philipp/python-common-mypy

python-common: Add mypy testing

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30995 from wjwithagen/wjw-wip-volume-zfs-inventory

ceph-volume-zfs: add the inventory command

Merge pull request #31014 from tchaikov/wip-38378

mgr/dashboard: accept expected exception when SSL handshaking

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31092 from liewegas/wip-revert-42026-test

Revert "test: librados startup/shutdown racer test"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #31153 from ceph/wip-restful-node-items

restful: Query nodes_by_id for items

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31176 from tchaikov/wip/mgr/diskprediction_cloud/cleanup

mgr/diskprediction_cloud: refactor timeout() decorator

Reviewed-by: Rick Chen <rick.chen@prophetstor.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31175 from tchaikov/wip/mon/OSDMonitor/cleanup

mon/OSDMonitor: use initializer_list<> for {si,iec}_options

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: update dashboard CODEOWNERShip

Fixes: https://tracker.ceph.com/issues/42507
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #30279 from runsisi/wip-fix-strerror_r

common: fix compat of strerror_r

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29836 from majianpeng/fix-async-msgr-perfcounter-bug

msg/async: Don't dec(msgr_active_connections) if conn still in accept…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #31156 from cyx1231st/wip-seastar-msgr-testpeer

test/crimson: implement a remote async TestPeer for crimson msgr tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: add nonstop option in async TestPeer

If set, TestPeer won't shutdown if all tests are done, and will wait for
another round of tests.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: add delay to markdown_peer()

Let test sleep awhile for peer markdown propagated.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: async msgr TestPeer for v2 failover tests

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: allow v2 failover tests to connect to remote TestPeer

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: extract msgr test commands to header

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #30610 from ivancich/wip-no-schedule-reshard-during-reshard

rgw: prevent bucket reshard scheduling if bucket is resharding

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30795 from ivancich/wip-dynamic-reshard-adjustments

rgw: adjust allowable bucket index shard counts for dynamic resharding

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30845 from croit/rgw-bucket-stats-add-num-shards

rgw: add num_shards to radosgw-admin bucket stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30956 from joke-lee/rgw-storage-class-post-obj

rgw: set correct storage class for post object upload

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30970 from theanalyst/rgw/list-common-prefix

rgw: url encode common prefixes for List Objects response

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31179 from liewegas/cleanup-exec-bit

os/bluestore,comon,erasure-code: chmod -x source files

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31171 from liewegas/bug-42496

qa/tasks/cbt: run stop-all.sh while shutting down

Reviewed-by: Kefu Chai <kchai@redhat.com>

Revert "Merge pull request #30964 from markhpc/wip-bs-cache-trim-pinned"

This reverts commit 304c37f52165363cad4aaf31386d62ea5a573425, reversing
changes made to 7b4f9a083f0c8cc015f7b0cc12a3bc88db632610.

This causes some bluestore test failures due to an ENOENT right after a
new object is created.  The simplest reproducer is the
ObjectStore/StoreTest.FiemapEmpty/2 test.

Fixes: https://tracker.ceph.com/issues/42495
Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore,comon,erasure-code: chmod -x source files

Fixes

*** WARNING: ./usr/src/debug/ceph-15.0.0-6512.g62bd825.el8.x86_64/src/common/obj_bencher.cc is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/src/debug/ceph-15.0.0-6512.g62bd825.el8.x86_64/src/erasure-code/shec/determinant.c is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/src/debug/ceph-15.0.0-6512.g62bd825.el8.x86_64/src/os/bluestore/AvlAllocator.cc is executable but has empty or no shebang, removing executable bit
*** WARNING: ./usr/src/debug/ceph-15.0.0-6512.g62bd825.el8.x86_64/src/os/bluestore/AvlAllocator.h is executable but has empty or no shebang, removing executable bit

in rpm build.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31170 into master

* refs/pull/31170/head:
src/telemetry: remove, now lives in ceph-telemetry.git

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31131 into master

* refs/pull/31131/head:
crush: remove invalid upmap items

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #31154 into master

* refs/pull/31154/head:
auth/cephx: always initialize local variables

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31167 into master

* refs/pull/31167/head:
os/bluestore: do not mark per_pool_omap updated unless we fixed it

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: accept socket error 0

see
https://github.com/cherrypy/cheroot/commit/9d73226ab28a07a38cbb6816edf9ff46be4f9de3

Fixes: https://tracker.ceph.com/issues/38378
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: accept exceptions from builtin SSL

see also https://github.com/cherrypy/cheroot/pull/4

so we don't panic when client is trying to talk with us with an
unsupported protocol, the exception should be accepted, and the
client can fallback to supported protocol.

Fixes: https://tracker.ceph.com/issues/38378
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: extract monkey patches for cherrypy out

so they are less distracting to the dashboard developers. and probably
less scaring.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: add missing python-yaml dependency for mgr-k8sevents

otherwise we might have:

```
ceph/src/pybind/mgr/k8sevents/__init__.py", line 1, in <module>
    from .module import Module
  File "/home/kchai/ceph/src/pybind/mgr/k8sevents/module.py", line 28, in <module>
    import yaml
ImportError: No module named yaml
```

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/diskprediction_cloud: refactor timeout() decorator

* timeout() is never passed any parameter when being called, so let's
  remove the parameters list of "seconds" and "error_message"
* use `getattr()` instead of `hasattr()` for retrieving the
  member variable of `self`
* pass `self` to wrapper function explicitly.
* return `func()` right away.
* hardwire the error message of `TimeoutError` to "Timer expired",
  because
  - as neither errno.ETIME nor errno.ETIMEOUT is portable
  - the only caller of `TimeoutError` is `timeout()`, so there is
    no need to have the flexibility to pass a different error message
* use `wraps()` as a decorator, simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/OSDMonitor: use initializer_list<> for {si,iec}_options

* use initializer_list<> for {si,iec}_options, no need to uset set<>
* remove the comments, the variable names are self-documented.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31010 from pdvian/wip-fix-target-max

mon/OSDMonitor.cc: Allow pool set target_max_(objects/bytes) with SI/IEC units

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30434 from smithfarm/wip-41820

qa: enable dashboard tests to be run with "--suite rados/dashboard"

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31005 from tchaikov/wip/qa/tasks/ceph/cleanup

qa/tasks/ceph.py: remove unused variables

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/cbt: run stop-all.sh when finishing up

stop-all.sh will work if the right deps are there (currently we lack 'nc')

also killall -9 java to be sure.

Fixes: https://tracker.ceph.com/issues/42496
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31086 into master

* refs/pull/31086/head:
doc/bootstrap.rst: fix github's url

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31169 into master

* refs/pull/31169/head:
mgr/ssh: implement (synchronous) describe_service
ceph-daemon: ls: replace 'active' bool with 'state' enum
ceph-daemon: include container_id and version in 'ls' output
ceph-daemon: tolerate systemctl is-* exit codes
mgr/ssh: modernize timeout config option

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc/bootstrap.rst: fix github's url

Signed-off-by: Alexandre Bruyelles <jack@jack.fr.eu.org>

Merge pull request #31126 from wjwithagen/wjw-fix-std-bitset

rgw: Select the std::bitset to resolv ambiguity

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

src/telemetry: remove, now lives in ceph-telemetry.git

Signed-off-by: Dan Mick <dan.mick@redhat.com>

mgr/ssh: implement (synchronous) describe_service

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: ls: replace 'active' bool with 'state' enum

('running', 'inactive', 'error', 'unknown')

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: include container_id and version in 'ls' output

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: tolerate systemctl is-* exit codes

Signed-off-by: Sage Weil <sage@redhat.com>