osd/: maintain PeeringState::min_last_complete_ondisk on replica

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend,MOSDRepOp: replace pg_roll_forward_to with mlcod

New clients will ignore the field on header.version <= 2.  New
primaries will send the old variant if there are peers expecting
the old variant.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend: always rollforward on repop to version

The primary always sets this field to be at_version, and we're
going to reuse this field to mean min_last_complete_ondisk.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PGBackend: rename submit_transaction field roll_forward_to

This field is actually just a lower bound on committed, client
visible log versions.

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_test_rados: use - not _ in args

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash[-erasure-code]: add misc -{localized,balanced}.yaml jobs

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/rados: add {localize,balance}_reads field

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados: add --localize_reads option

Same as balance_reads, but sets the LOCALIZE_READS op flag on read ops.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32300 into master

* refs/pull/32300/head:
cephadm: drop 'sha256:' prefix on container id

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32302 into master

* refs/pull/32302/head:
cephadm: check for both ntp.service and ntpd.service

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32280 from dillaman/wip-librbd-qos-test

qa/workunit/rbd: fixed QoS throughput unit parsing

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #32279 into master

* refs/pull/32279/head:
mgr/orchestrator_cli: combine all host into one 'device ls' table
python-common: 'ssd/nvme' -> 'ssd'
mgr/orchestrator_cli: make 'device ls' table formatting consistent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32282 into master

* refs/pull/32282/head:
cephadm: clean up shell containers on exit
cephadm: do not force container removal on unit stop

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32284 into master

* refs/pull/32284/head:
qa/suites/rbd/openstack: use 18.04, not 16.04

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: drop 'sha256:' prefix on container id

Podman doesn't include the prefix--only the (bionic) docker as far as I
can tell.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32273 from changchengx/doc_libvirt

doc/rbd: refine rbd/libvirt usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32296 from rzarzynski/wip-seastar-pickup-futureopt

seastar: pickup the recent future optimizations.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32301 from tchaikov/wip-build-doc

admin/doc-requirements.txt: bump up Sphinx and breathe

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32285 from liewegas/wip-py3-docs

admin/build-doc: do not use system site-packages

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: check for both ntp.service and ntpd.service

Sigh...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32263 from rosinL/fix-vcluster-stop

stop.sh: use bash shell to solve syntax error

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: do not use system site-packages

Don't use site-packages, since the host might have the same version, and
it won't have the ditaa and breathe.

Require Cython to make the venv sufficient.

Signed-off-by: Sage Weil <sage@redhat.com>

admin/doc-requirements.txt: bump Sphinx and breathe

* use the latest Sphinx
* use the latest breathe. also, since there is no need to be compatible
  with python2. we can move to 4.14.0 which is python3 only.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunit/rbd: fixed QoS throughput unit parsing

The 'rbd bench' command was recently modified to print IEC units
instead of bytes/sec. This broke the handling for QoS throughput
tests since it was incorrectly evaluating the available RBD
throughput. Additionally, the QoS tests should use a "<="
comparison operator since the QoS is the upper-bound limit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32293 into master

* refs/pull/32293/head:
script/build-integration-branch: Add usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32295 from tchaikov/wip-stop.sh

stop.sh: do not try to contact mon unless cluster is up

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

cephadm: clean up shell containers on exit

The run_cmd() was already passing --rm; do it for shell_cmd too so that
we don't leave behind a mess from shell commands, ceph-volume, etc.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not force container removal on unit stop

The -f arg for 'podman rm' will remove the container without actually
stopping it, leaving the running containers and processes (conmon and
daemon) running and generally making a mess of things.

Doing 'podman rm' without the -f on a running container will complain
that the container is still running.

So, first stop, then remove.

Fixes: f1de7060762d31e567a277c02b0d8d533fbb4661
Signed-off-by: Sage Weil <sage@redhat.com>

stop.sh: use ${CEPH_BIN} and -c ${conf_fn}

to make sure ${CEPH_BIN} and ${CEPH_CONF_PATH} are respected

Signed-off-by: Kefu Chai <kchai@redhat.com>

stop.sh: do not try to contact mon unless cluster is up

otherwise "ceph mon metadata" will hang, when we try to stop an
already-stopped cluster. and `vstart_runner.py` always calls `stop.sh`
before setting up a cluster.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup the recent future optimizations.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

script/build-integration-branch: Add usage

Also add a --no-date to make it possible to re-use an existing branch name

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32207 from bk201/wip-43136

mgr/dashboard: Fix OSD IDs are not displayed when using cephadm backend

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32266 from xiexingguo/wip-balancer-efficiency-1

mgr/balancer: avoid pulling pg_dump twice

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32237 from tchaikov/wip-cmake-rdkafka

cmake: check version of librdkafka

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #31144 from badone/wip-use-dnf-on-rhel-or-centos-8

install-deps.sh: Use dnf for rhel/centos 8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31884 into master

* refs/pull/31884/head:
qa/tasks: Fix the volume ls in test_volume_rm
qa/tasks: tests for 'fs volume create' and 'fs volume ls'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29059 into master

* refs/pull/29059/head:
mds: complete all the replay op when mds is restarted ,no matter the session is closed or not.

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31662 into master

* refs/pull/31662/head:
mds: fix revoking caps after after stale->resume circle
client: fix incorrect debug message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32162 into master

* refs/pull/32162/head:
qa: note timeout in debug message

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31118 into master

* refs/pull/31118/head:
mds: Reorganize class members in MDSMap header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31123 into master

* refs/pull/31123/head:
mds : optimization functions,get_dirfrags_under, to speed up processing directories with tens of millions of files

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31595 into master

* refs/pull/31595/head:
mds: optimize function, fragset_t::simplify, to improve the efficiency of  merging fragment

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #32020 into master

* refs/pull/32020/head:
mds: fix assert(omap_num_objs <= MAX_OBJECTS) of OpenFileTable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: note timeout in debug message

Some tests set this to a dynamic value, it'd be helpful to know how long
a test is planning to wait.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32042 into master

* refs/pull/32042/head:
mds: update default MDS cache size memory limit
common/options: note runtime effect for some MDS configs

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

qa/suites/rbd/openstack: use 18.04, not 16.04

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32275 into master

* refs/pull/32275/head:
qa/suites/upgrade: all upgrades to octopus on ubuntu only

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32274 from dillaman/wip-rbd-mirror-test

rbd-mirror: ensure deterministic ordering of method calls

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #32212 into master

* refs/pull/32212/head:
rpm/cephadm: start description and summary with capital letter
rpm/cephadm: move HOMEDIR to /var/lib and make scriptlets idempotent on SUSE

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge PR #32277 into master

* refs/pull/32277/head:
lvm/deactivate: add unit tests, remove --all

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: combine all host into one 'device ls' table

Signed-off-by: Sage Weil <sage@redhat.com>

python-common: 'ssd/nvme' -> 'ssd'

NVMe's are generally 'solid state disks'.  More importantly, NVMe is an
interface, not a device type, and there is talk of HDDs with NVMe
interfaces.

But mostly this is just makes for a more compact table.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: make 'device ls' table formatting consistent

- all caps headers
- align left by default
- reorder columns

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade: all upgrades to octopus on ubuntu only

We cannot do a traditional upgrade (install old package, start cluster,
install new package, ...) because nautilus is el7-only and octopus is
el8-only.

So, do these tests on ubuntu.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32217 from trociny/wip-41557-2

librbd: explicitly specify mode on mirror image enable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

lvm/deactivate: add unit tests, remove --all

Remove the --all flag until its actually implemented.

Fixes: https://tracker.ceph.com/issues/43330
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #32198 into master

* refs/pull/32198/head:
cephadm: Fix Python2.7 isssue
cephadm: fix how uid/gid is determined
cephadm: add sample configuration for prometheus
cephadm: support deployment of prometheus

Reviewed-by: Sebastian Wagner <swagner@suse.com>

rpm/cephadm: start description and summary with capital letter

This commit fixes RPMLINT issue:

cephadm.x86_64: E: summary-not-capitalized (Badness: 20) C cephadm utility to bootstrap Ceph clusters
Summary doesn't begin with a capital letter.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm/cephadm: move HOMEDIR to /var/lib and make scriptlets idempotent on SUSE

Also introduce a cephadm group. Since the cephadm package does not
require ceph-common, the ceph group is not available to be used.

Drop the -r option to useradd, because cephadm does not qualify as a system
user.

Finally, fix the SUSE RPM build. SUSE RPM builds using "osc" implement a check
that compares the cephadm package's "before" and "after" file lists, to enforce
idempotency of the scriptlets. The "userdel cephadm" in the %postun breaks this
check, so omit it from the SUSE RPM builds.

Fixes: https://tracker.ceph.com/issues/43285
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd-mirror: ensure deterministic ordering of method calls

The mock tests will require method calls in a known order. Previously
different environments could evaluate the parameters in different
orders resulting in out-of-order method calls.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32211 from rhcs-dashboard/43284-fix-frontend-e2e-script

mgr/dashboard: run-frontend-e2e-tests.sh: allow user defined BASE_URL…

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31819 from ricardoasmarques/wip-42687

mgr/dashboard: iSCSI targets not available if any gateway is down

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #32246 into master

* refs/pull/32246/head:
mgr/cephadm: remove redundant /dev when blinking device light

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32257 from dillaman/wip-41556

rbd-mirror: extract journal replaying logic from image replayer

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/rbd: remind Selinux/AppArmor effect on libvirtd

If libvird is under Selinux/Apparmor enforced mode, libvirtd
may not be enable to do some operatoins on the rbd image. It
need remind this in document.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

doc/rbd: correct software stack of VM with Ceph

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cephadm: Fix Python2.7 isssue

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: CephFS quota management (#31811)

mgr/dashboard: CephFS quota management

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: add password expiration date (#30939)

mgr/dashboard: add password expiration date

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Display OSD device class in OSD list (#32092)

mgr/dashboard: Display OSD device class in OSD list

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31815 from tchaikov/wip-mgr-dashboard-gevent

mgr/dashboard: install teuthology using pip

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>

rbd: show mirror mode in image info

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: improve mirror snapshot list formatting for status command

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: allow to specify mode on mirror image enable

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: explicitly specify mode on mirror image enable

- don't use journaling feature to chose the mode;
- provide new API function mirror_image_enable2;
- return back the old behavior to automatically enable/disable
  journaling feature on enabling/disabling image mirroring.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/balancer: avoid pulling pg_dump twice

For large cluster with thousands of osds/pgs, it may take
seconds even mitutes to refresh the relevant stats.
Since we already have pulling them down at initialization,
make a direct use of them whenever possible.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

stop.sh: use bash shell to solve syntax error

When running on ubuntu 18.04 for aarch64, there is syntax error in
line 52, caused by the /bin/sh is defalut to dash, which is not
support array type.

Signed-off-by: luo rixin <luorixin@huawei.com>

rbd-mirror: switch image replayer to new standalone journal replayer

Remove all the original journal replaying code embedded in the image
replayer and instead rely on the new journal replayer class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: extract journal replay logic to its own class

This will help to greatly reduce the journal-specific code in the
current image replayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved image asok/perf counter name helper to utils

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved event pre-processor to sub-namespace

The current implementation is tied to journal-based replaying so
move it down into the journal sub-namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved replay status formatter to sub-namespace

The current implementation is tied to journal-based replaying so
move it down into a new journal sub-namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: drop 'using namespace' in image replayer state machine

This is going to cause conflicts w/ the '::journal' namespace and
future 'rbd::mirror::image_replayer::journal' namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32142 from changchengx/doc_libvirt

doc/rbd: simplify libvirt usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32243 from trociny/wip-43289

rbd-mirror: fix race on namespace replayer initialization failure

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32184 from trociny/wip-43004

rbd-mirror: rename per-image replication perf counters

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32113 from lixiaoy1/master

librbd: finish write request early

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32251 into master

* refs/pull/32251/head:
mds/MDSDaemon: fix asok 'exit' and 'respawn' commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32260 into master

* refs/pull/32260/head:
.organizationmap: Wido 42on -> 42on

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: fix how uid/gid is determined

The uid/gid is determined by looking at file ownership. It
was looking at /etc/ceph, but on current builds this is owned by root so switched to using /var/lib/ceph which
is the homedir of the ceph account.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: add sample configuration for prometheus

Provides an example definition that sets up an
initial prometheus server with a scrape of itself.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: support deployment of prometheus

Initial support to enable cephadm to deploy
monitoring containers like prometheus. This
patch adds support for prometheus. It is the callers
responsibility to provide a valid prometheus.yml
file passed within the --config-json parameter.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

.organizationmap: Wido 42on -> 42on

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: check version of librdkafka

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32158 into master

* refs/pull/32158/head:
cephadm: fix exception when no time sync is running
cephadm: properly extract osd fsid during adoption
cephadm: do ceph-volume activate+deactivate as part of systemd unit
ceph-daemon: behave with no /etc/ceph/ceph.conf

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: fix exception when no time sync is running

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: properly extract osd fsid during adoption

This needs to be passed to the deploy_daemon_units method so that we can
set up the pre and post c-v calls properly.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do ceph-volume activate+deactivate as part of systemd unit

1- Do the activate as part of the unit on start, since it may include
setup that disappears on a reboot, like starting up the dmcrypted volume.

2- Do the matching deactivate step when the container stops, like shutting
down the dmcrypt volume.

Right now the OSD is the only thing that needs this special behavior.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: behave with no /etc/ceph/ceph.conf

Signed-off-by: Sage Weil <sage@redhat.com>