ceph.spec.in: always depends on python3.6-pyOpenSSL

since we've moved to py3-only world, there is no need to depend on
pyOpenSSL anymore.

also, _python_buildid was removed in
3bf8b4d7d6012458fc761edd759a4fb3070a30fe .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31527 into master

* refs/pull/31527/head:
vstart: Add note for nfs-ganesha dev packages
stop.sh: Add option to kill Ganesha daemons
vstart: Add option to deploy ganesha daemons

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #32303 from tchaikov/wip-build-integration-branch

build-integration-branch: s/prefix/postfix/

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #32291 into master

* refs/pull/32291/head:
doc: update 'adding an MDS'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32252 into master

* refs/pull/32252/head:
qa/cephfs/begin: libaio-devel on el8
qa/tasks: nosetests -> python -m nose
qa/tasks/rbd_fio: fio 2.21 -> 3.16
src/test/cli-integration/rbd/snap-diff.t: python -> python
qa/workunits: use nose 3
qa/tasks/cbt: install python3 deps
qa/tasks/ceph_manager.py: do not use python to write a file
test/pybind/test_rados: execute takes a bytes (not str) payload
qa/packages/packages: python[3]-ceph is no more
qa: use python3 for venvs etc
packaging: remove python3-ipaddres, as it is part of the stdlib in py3
qa/packages: python-ceph -> python3-ceph
qa/distros: centos7 -> centos8, rhel7 -> rhel8
spec: remove _python_buildid in favor of python3_pkgversion macro
spec: remove python2 packages and conditions
debian: remove python >= 2.7 requirement
debian: add mgr python versions
debian: explicitly set PYTHON2=OFF to prevent picking up python2 interpreter
debian: update control file to use python3 dependency names
debian: remove all python2 overrides and declarations
debian: remove all python2 install files

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #32146 into master

* refs/pull/32146/head:
mgr/ssh: Fix remove_osds()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32149 into master

* refs/pull/32149/head:
mgr/orchestrator: Rename PlacementSpec.nodes -> PlacementSpec.hosts
mgr/ansible: Added missing type annotations
mgr/orchestrator: Add __str__ to HostSpec
mgr/ansible: Integrate mgr/ansible/tox into mgr/tox

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32278 into master

* refs/pull/32278/head:
qa/suites/rgw: disable valgrind for tests that require py2/ubuntu
qa/suites/rgw: disable remaining ragweed test
qa/suites/rgw: pin swift tests to py2/ubuntu
qa/suites/rgw: ragweed on ubuntu
qa/suites: run s3tests on ubuntu

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/cephfs/begin: libaio-devel on el8

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32300 into master

* refs/pull/32300/head:
cephadm: drop 'sha256:' prefix on container id

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32302 into master

* refs/pull/32302/head:
cephadm: check for both ntp.service and ntpd.service

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/tasks: nosetests -> python -m nose

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw: disable valgrind for tests that require py2/ubuntu

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32280 from dillaman/wip-librbd-qos-test

qa/workunit/rbd: fixed QoS throughput unit parsing

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #32279 into master

* refs/pull/32279/head:
mgr/orchestrator_cli: combine all host into one 'device ls' table
python-common: 'ssd/nvme' -> 'ssd'
mgr/orchestrator_cli: make 'device ls' table formatting consistent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32282 into master

* refs/pull/32282/head:
cephadm: clean up shell containers on exit
cephadm: do not force container removal on unit stop

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/suites/rgw: disable remaining ragweed test

Needs to be py3 since it relies on the python bindings.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32284 into master

* refs/pull/32284/head:
qa/suites/rbd/openstack: use 18.04, not 16.04

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: drop 'sha256:' prefix on container id

Podman doesn't include the prefix--only the (bionic) docker as far as I
can tell.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32273 from changchengx/doc_libvirt

doc/rbd: refine rbd/libvirt usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

build-integration-branch: s/prefix/postfix/

prefix is not defined.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32296 from rzarzynski/wip-seastar-pickup-futureopt

seastar: pickup the recent future optimizations.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: Rename PlacementSpec.nodes -> PlacementSpec.hosts

By definition in the orchestrator's docs, we're using `host` insted of `node`

Fixes wrong usage of PlacementSpec in mgr/ansible

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32301 from tchaikov/wip-build-doc

admin/doc-requirements.txt: bump up Sphinx and breathe

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32285 from liewegas/wip-py3-docs

admin/build-doc: do not use system site-packages

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: check for both ntp.service and ntpd.service

Sigh...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32263 from rosinL/fix-vcluster-stop

stop.sh: use bash shell to solve syntax error

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: do not use system site-packages

Don't use site-packages, since the host might have the same version, and
it won't have the ditaa and breathe.

Require Cython to make the venv sufficient.

Signed-off-by: Sage Weil <sage@redhat.com>

admin/doc-requirements.txt: bump Sphinx and breathe

* use the latest Sphinx
* use the latest breathe. also, since there is no need to be compatible
  with python2. we can move to 4.14.0 which is python3 only.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunit/rbd: fixed QoS throughput unit parsing

The 'rbd bench' command was recently modified to print IEC units
instead of bytes/sec. This broke the handling for QoS throughput
tests since it was incorrectly evaluating the available RBD
throughput. Additionally, the QoS tests should use a "<="
comparison operator since the QoS is the upper-bound limit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32293 into master

* refs/pull/32293/head:
script/build-integration-branch: Add usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32295 from tchaikov/wip-stop.sh

stop.sh: do not try to contact mon unless cluster is up

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

cephadm: clean up shell containers on exit

The run_cmd() was already passing --rm; do it for shell_cmd too so that
we don't leave behind a mess from shell commands, ceph-volume, etc.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not force container removal on unit stop

The -f arg for 'podman rm' will remove the container without actually
stopping it, leaving the running containers and processes (conmon and
daemon) running and generally making a mess of things.

Doing 'podman rm' without the -f on a running container will complain
that the container is still running.

So, first stop, then remove.

Fixes: f1de7060762d31e567a277c02b0d8d533fbb4661
Signed-off-by: Sage Weil <sage@redhat.com>

stop.sh: use ${CEPH_BIN} and -c ${conf_fn}

to make sure ${CEPH_BIN} and ${CEPH_CONF_PATH} are respected

Signed-off-by: Kefu Chai <kchai@redhat.com>

stop.sh: do not try to contact mon unless cluster is up

otherwise "ceph mon metadata" will hang, when we try to stop an
already-stopped cluster. and `vstart_runner.py` always calls `stop.sh`
before setting up a cluster.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup the recent future optimizations.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

vstart: Add note for nfs-ganesha dev packages

Signed-off-by: Varsha Rao <varao@redhat.com>

script/build-integration-branch: Add usage

Also add a --no-date to make it possible to re-use an existing branch name

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: update 'adding an MDS'

Fixes: https://tracker.ceph.com/issues/43162
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #32207 from bk201/wip-43136

mgr/dashboard: Fix OSD IDs are not displayed when using cephadm backend

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32266 from xiexingguo/wip-balancer-efficiency-1

mgr/balancer: avoid pulling pg_dump twice

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/ssh: Fix remove_osds()

Changed signature from `(str) -> ...` to `(List[str]) -> ...`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32237 from tchaikov/wip-cmake-rdkafka

cmake: check version of librdkafka

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

stop.sh: Add option to kill Ganesha daemons

Signed-off-by: Varsha Rao <varao@redhat.com>

vstart: Add option to deploy ganesha daemons

To create Ganesha daemons with vstart following packages needs to be installed:
nfs-ganesha nfs-ganesha-ceph nfs-ganesha-rados-grace

Version used for these packages 2.7.6-2 on fedora 30 system.

This is an experimental option, it just deploys specified number of ganesha
daemons.

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #31144 from badone/wip-use-dnf-on-rhel-or-centos-8

install-deps.sh: Use dnf for rhel/centos 8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31884 into master

* refs/pull/31884/head:
qa/tasks: Fix the volume ls in test_volume_rm
qa/tasks: tests for 'fs volume create' and 'fs volume ls'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29059 into master

* refs/pull/29059/head:
mds: complete all the replay op when mds is restarted ,no matter the session is closed or not.

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31662 into master

* refs/pull/31662/head:
mds: fix revoking caps after after stale->resume circle
client: fix incorrect debug message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32162 into master

* refs/pull/32162/head:
qa: note timeout in debug message

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31118 into master

* refs/pull/31118/head:
mds: Reorganize class members in MDSMap header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31123 into master

* refs/pull/31123/head:
mds : optimization functions,get_dirfrags_under, to speed up processing directories with tens of millions of files

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31595 into master

* refs/pull/31595/head:
mds: optimize function, fragset_t::simplify, to improve the efficiency of  merging fragment

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #32020 into master

* refs/pull/32020/head:
mds: fix assert(omap_num_objs <= MAX_OBJECTS) of OpenFileTable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: note timeout in debug message

Some tests set this to a dynamic value, it'd be helpful to know how long
a test is planning to wait.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32042 into master

* refs/pull/32042/head:
mds: update default MDS cache size memory limit
common/options: note runtime effect for some MDS configs

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

qa/suites/rbd/openstack: use 18.04, not 16.04

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/rbd_fio: fio 2.21 -> 3.16

2.21 doesn't build on el8; 3.16 does.

Signed-off-by: Sage Weil <sage@redhat.com>

src/test/cli-integration/rbd/snap-diff.t: python -> python

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits: use nose 3

Ubuntu is /usr/bin/nosetests3, el is /usr/bin/nosetests-3, so do neither.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw: pin swift tests to py2/ubuntu

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32275 into master

* refs/pull/32275/head:
qa/suites/upgrade: all upgrades to octopus on ubuntu only

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32274 from dillaman/wip-rbd-mirror-test

rbd-mirror: ensure deterministic ordering of method calls

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #32212 into master

* refs/pull/32212/head:
rpm/cephadm: start description and summary with capital letter
rpm/cephadm: move HOMEDIR to /var/lib and make scriptlets idempotent on SUSE

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge PR #32277 into master

* refs/pull/32277/head:
lvm/deactivate: add unit tests, remove --all

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: combine all host into one 'device ls' table

Signed-off-by: Sage Weil <sage@redhat.com>

python-common: 'ssd/nvme' -> 'ssd'

NVMe's are generally 'solid state disks'.  More importantly, NVMe is an
interface, not a device type, and there is talk of HDDs with NVMe
interfaces.

But mostly this is just makes for a more compact table.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: make 'device ls' table formatting consistent

- all caps headers
- align left by default
- reorder columns

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade: all upgrades to octopus on ubuntu only

We cannot do a traditional upgrade (install old package, start cluster,
install new package, ...) because nautilus is el7-only and octopus is
el8-only.

So, do these tests on ubuntu.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32217 from trociny/wip-41557-2

librbd: explicitly specify mode on mirror image enable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rgw: ragweed on ubuntu

ragweed uses nosetests; stick to py2 for now.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites: run s3tests on ubuntu

Note that this does not capture the rgw/website test on valgrind, since
valgrind is centos only.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cbt: install python3 deps

Signed-off-by: Sage Weil <sage@redhat.com>

lvm/deactivate: add unit tests, remove --all

Remove the --all flag until its actually implemented.

Fixes: https://tracker.ceph.com/issues/43330
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #32198 into master

* refs/pull/32198/head:
cephadm: Fix Python2.7 isssue
cephadm: fix how uid/gid is determined
cephadm: add sample configuration for prometheus
cephadm: support deployment of prometheus

Reviewed-by: Sebastian Wagner <swagner@suse.com>

rpm/cephadm: start description and summary with capital letter

This commit fixes RPMLINT issue:

cephadm.x86_64: E: summary-not-capitalized (Badness: 20) C cephadm utility to bootstrap Ceph clusters
Summary doesn't begin with a capital letter.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm/cephadm: move HOMEDIR to /var/lib and make scriptlets idempotent on SUSE

Also introduce a cephadm group. Since the cephadm package does not
require ceph-common, the ceph group is not available to be used.

Drop the -r option to useradd, because cephadm does not qualify as a system
user.

Finally, fix the SUSE RPM build. SUSE RPM builds using "osc" implement a check
that compares the cephadm package's "before" and "after" file lists, to enforce
idempotency of the scriptlets. The "userdel cephadm" in the %postun breaks this
check, so omit it from the SUSE RPM builds.

Fixes: https://tracker.ceph.com/issues/43285
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd-mirror: ensure deterministic ordering of method calls

The mock tests will require method calls in a known order. Previously
different environments could evaluate the parameters in different
orders resulting in out-of-order method calls.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32211 from rhcs-dashboard/43284-fix-frontend-e2e-script

mgr/dashboard: run-frontend-e2e-tests.sh: allow user defined BASE_URL…

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31819 from ricardoasmarques/wip-42687

mgr/dashboard: iSCSI targets not available if any gateway is down

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #32246 into master

* refs/pull/32246/head:
mgr/cephadm: remove redundant /dev when blinking device light

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32257 from dillaman/wip-41556

rbd-mirror: extract journal replaying logic from image replayer

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/rbd: remind Selinux/AppArmor effect on libvirtd

If libvird is under Selinux/Apparmor enforced mode, libvirtd
may not be enable to do some operatoins on the rbd image. It
need remind this in document.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

doc/rbd: correct software stack of VM with Ceph

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cephadm: Fix Python2.7 isssue

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: CephFS quota management (#31811)

mgr/dashboard: CephFS quota management

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: add password expiration date (#30939)

mgr/dashboard: add password expiration date

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Display OSD device class in OSD list (#32092)

mgr/dashboard: Display OSD device class in OSD list

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31815 from tchaikov/wip-mgr-dashboard-gevent

mgr/dashboard: install teuthology using pip

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>

rbd: show mirror mode in image info

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: improve mirror snapshot list formatting for status command

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: allow to specify mode on mirror image enable

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: explicitly specify mode on mirror image enable

- don't use journaling feature to chose the mode;
- provide new API function mirror_image_enable2;
- return back the old behavior to automatically enable/disable
  journaling feature on enabling/disabling image mirroring.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/balancer: avoid pulling pg_dump twice

For large cluster with thousands of osds/pgs, it may take
seconds even mitutes to refresh the relevant stats.
Since we already have pulling them down at initialization,
make a direct use of them whenever possible.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

stop.sh: use bash shell to solve syntax error

When running on ubuntu 18.04 for aarch64, there is syntax error in
line 52, caused by the /bin/sh is defalut to dash, which is not
support array type.

Signed-off-by: luo rixin <luorixin@huawei.com>

rbd-mirror: switch image replayer to new standalone journal replayer

Remove all the original journal replaying code embedded in the image
replayer and instead rely on the new journal replayer class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: extract journal replay logic to its own class

This will help to greatly reduce the journal-specific code in the
current image replayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved image asok/perf counter name helper to utils

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved event pre-processor to sub-namespace

The current implementation is tied to journal-based replaying so
move it down into the journal sub-namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>