osd/: bounce writes on objects written since mlcod back to primary

If there are any writes since mlcod on an object with the same head,
our ondisk state may be unstable or may reflect uncommitted state.
Bounce back to primary.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: maintain PeeringState::min_last_complete_ondisk on replica

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend,MOSDRepOp: replace pg_roll_forward_to with mlcod

New clients will ignore the field on header.version <= 2.  New
primaries will send the old variant if there are peers expecting
the old variant.

Signed-off-by: Samuel Just <sjust@redhat.com>

ReplicatedBackend: always rollforward on repop to version

The primary always sets this field to be at_version, and we're
going to reuse this field to mean min_last_complete_ondisk.

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PGBackend: rename submit_transaction field roll_forward_to

This field is actually just a lower bound on committed, client
visible log versions.

Signed-off-by: Samuel Just <sjust@redhat.com>

ceph_test_rados: use - not _ in args

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash[-erasure-code]: add misc -{localized,balanced}.yaml jobs

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/rados: add {localize,balance}_reads field

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_test_rados: add --localize_reads option

Same as balance_reads, but sets the LOCALIZE_READS op flag on read ops.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32300 into master

* refs/pull/32300/head:
cephadm: drop 'sha256:' prefix on container id

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32302 into master

* refs/pull/32302/head:
cephadm: check for both ntp.service and ntpd.service

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #32280 from dillaman/wip-librbd-qos-test

qa/workunit/rbd: fixed QoS throughput unit parsing

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #32279 into master

* refs/pull/32279/head:
mgr/orchestrator_cli: combine all host into one 'device ls' table
python-common: 'ssd/nvme' -> 'ssd'
mgr/orchestrator_cli: make 'device ls' table formatting consistent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32282 into master

* refs/pull/32282/head:
cephadm: clean up shell containers on exit
cephadm: do not force container removal on unit stop

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #32284 into master

* refs/pull/32284/head:
qa/suites/rbd/openstack: use 18.04, not 16.04

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cephadm: drop 'sha256:' prefix on container id

Podman doesn't include the prefix--only the (bionic) docker as far as I
can tell.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32273 from changchengx/doc_libvirt

doc/rbd: refine rbd/libvirt usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32296 from rzarzynski/wip-seastar-pickup-futureopt

seastar: pickup the recent future optimizations.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32301 from tchaikov/wip-build-doc

admin/doc-requirements.txt: bump up Sphinx and breathe

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32285 from liewegas/wip-py3-docs

admin/build-doc: do not use system site-packages

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: check for both ntp.service and ntpd.service

Sigh...

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32263 from rosinL/fix-vcluster-stop

stop.sh: use bash shell to solve syntax error

Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: do not use system site-packages

Don't use site-packages, since the host might have the same version, and
it won't have the ditaa and breathe.

Require Cython to make the venv sufficient.

Signed-off-by: Sage Weil <sage@redhat.com>

admin/doc-requirements.txt: bump Sphinx and breathe

* use the latest Sphinx
* use the latest breathe. also, since there is no need to be compatible
  with python2. we can move to 4.14.0 which is python3 only.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/workunit/rbd: fixed QoS throughput unit parsing

The 'rbd bench' command was recently modified to print IEC units
instead of bytes/sec. This broke the handling for QoS throughput
tests since it was incorrectly evaluating the available RBD
throughput. Additionally, the QoS tests should use a "<="
comparison operator since the QoS is the upper-bound limit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32293 into master

* refs/pull/32293/head:
script/build-integration-branch: Add usage

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32295 from tchaikov/wip-stop.sh

stop.sh: do not try to contact mon unless cluster is up

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

cephadm: clean up shell containers on exit

The run_cmd() was already passing --rm; do it for shell_cmd too so that
we don't leave behind a mess from shell commands, ceph-volume, etc.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do not force container removal on unit stop

The -f arg for 'podman rm' will remove the container without actually
stopping it, leaving the running containers and processes (conmon and
daemon) running and generally making a mess of things.

Doing 'podman rm' without the -f on a running container will complain
that the container is still running.

So, first stop, then remove.

Fixes: f1de7060762d31e567a277c02b0d8d533fbb4661
Signed-off-by: Sage Weil <sage@redhat.com>

stop.sh: use ${CEPH_BIN} and -c ${conf_fn}

to make sure ${CEPH_BIN} and ${CEPH_CONF_PATH} are respected

Signed-off-by: Kefu Chai <kchai@redhat.com>

stop.sh: do not try to contact mon unless cluster is up

otherwise "ceph mon metadata" will hang, when we try to stop an
already-stopped cluster. and `vstart_runner.py` always calls `stop.sh`
before setting up a cluster.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: pickup the recent future optimizations.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

script/build-integration-branch: Add usage

Also add a --no-date to make it possible to re-use an existing branch name

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32207 from bk201/wip-43136

mgr/dashboard: Fix OSD IDs are not displayed when using cephadm backend

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #32266 from xiexingguo/wip-balancer-efficiency-1

mgr/balancer: avoid pulling pg_dump twice

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32237 from tchaikov/wip-cmake-rdkafka

cmake: check version of librdkafka

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #31144 from badone/wip-use-dnf-on-rhel-or-centos-8

install-deps.sh: Use dnf for rhel/centos 8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31884 into master

* refs/pull/31884/head:
qa/tasks: Fix the volume ls in test_volume_rm
qa/tasks: tests for 'fs volume create' and 'fs volume ls'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29059 into master

* refs/pull/29059/head:
mds: complete all the replay op when mds is restarted ,no matter the session is closed or not.

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31662 into master

* refs/pull/31662/head:
mds: fix revoking caps after after stale->resume circle
client: fix incorrect debug message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32162 into master

* refs/pull/32162/head:
qa: note timeout in debug message

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31118 into master

* refs/pull/31118/head:
mds: Reorganize class members in MDSMap header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31123 into master

* refs/pull/31123/head:
mds : optimization functions,get_dirfrags_under, to speed up processing directories with tens of millions of files

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31595 into master

* refs/pull/31595/head:
mds: optimize function, fragset_t::simplify, to improve the efficiency of  merging fragment

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #32020 into master

* refs/pull/32020/head:
mds: fix assert(omap_num_objs <= MAX_OBJECTS) of OpenFileTable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: note timeout in debug message

Some tests set this to a dynamic value, it'd be helpful to know how long
a test is planning to wait.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32042 into master

* refs/pull/32042/head:
mds: update default MDS cache size memory limit
common/options: note runtime effect for some MDS configs

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

qa/suites/rbd/openstack: use 18.04, not 16.04

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32275 into master

* refs/pull/32275/head:
qa/suites/upgrade: all upgrades to octopus on ubuntu only

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32274 from dillaman/wip-rbd-mirror-test

rbd-mirror: ensure deterministic ordering of method calls

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge PR #32212 into master

* refs/pull/32212/head:
rpm/cephadm: start description and summary with capital letter
rpm/cephadm: move HOMEDIR to /var/lib and make scriptlets idempotent on SUSE

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

Merge PR #32277 into master

* refs/pull/32277/head:
lvm/deactivate: add unit tests, remove --all

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: combine all host into one 'device ls' table

Signed-off-by: Sage Weil <sage@redhat.com>

python-common: 'ssd/nvme' -> 'ssd'

NVMe's are generally 'solid state disks'.  More importantly, NVMe is an
interface, not a device type, and there is talk of HDDs with NVMe
interfaces.

But mostly this is just makes for a more compact table.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: make 'device ls' table formatting consistent

- all caps headers
- align left by default
- reorder columns

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade: all upgrades to octopus on ubuntu only

We cannot do a traditional upgrade (install old package, start cluster,
install new package, ...) because nautilus is el7-only and octopus is
el8-only.

So, do these tests on ubuntu.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32217 from trociny/wip-41557-2

librbd: explicitly specify mode on mirror image enable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

lvm/deactivate: add unit tests, remove --all

Remove the --all flag until its actually implemented.

Fixes: https://tracker.ceph.com/issues/43330
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #32198 into master

* refs/pull/32198/head:
cephadm: Fix Python2.7 isssue
cephadm: fix how uid/gid is determined
cephadm: add sample configuration for prometheus
cephadm: support deployment of prometheus

Reviewed-by: Sebastian Wagner <swagner@suse.com>

rpm/cephadm: start description and summary with capital letter

This commit fixes RPMLINT issue:

cephadm.x86_64: E: summary-not-capitalized (Badness: 20) C cephadm utility to bootstrap Ceph clusters
Summary doesn't begin with a capital letter.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rpm/cephadm: move HOMEDIR to /var/lib and make scriptlets idempotent on SUSE

Also introduce a cephadm group. Since the cephadm package does not
require ceph-common, the ceph group is not available to be used.

Drop the -r option to useradd, because cephadm does not qualify as a system
user.

Finally, fix the SUSE RPM build. SUSE RPM builds using "osc" implement a check
that compares the cephadm package's "before" and "after" file lists, to enforce
idempotency of the scriptlets. The "userdel cephadm" in the %postun breaks this
check, so omit it from the SUSE RPM builds.

Fixes: https://tracker.ceph.com/issues/43285
Signed-off-by: Nathan Cutler <ncutler@suse.com>

rbd-mirror: ensure deterministic ordering of method calls

The mock tests will require method calls in a known order. Previously
different environments could evaluate the parameters in different
orders resulting in out-of-order method calls.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32211 from rhcs-dashboard/43284-fix-frontend-e2e-script

mgr/dashboard: run-frontend-e2e-tests.sh: allow user defined BASE_URL…

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31819 from ricardoasmarques/wip-42687

mgr/dashboard: iSCSI targets not available if any gateway is down

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #32246 into master

* refs/pull/32246/head:
mgr/cephadm: remove redundant /dev when blinking device light

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32257 from dillaman/wip-41556

rbd-mirror: extract journal replaying logic from image replayer

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/rbd: remind Selinux/AppArmor effect on libvirtd

If libvird is under Selinux/Apparmor enforced mode, libvirtd
may not be enable to do some operatoins on the rbd image. It
need remind this in document.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

doc/rbd: correct software stack of VM with Ceph

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

cephadm: Fix Python2.7 isssue

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: CephFS quota management (#31811)

mgr/dashboard: CephFS quota management

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: add password expiration date (#30939)

mgr/dashboard: add password expiration date

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Display OSD device class in OSD list (#32092)

mgr/dashboard: Display OSD device class in OSD list

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31815 from tchaikov/wip-mgr-dashboard-gevent

mgr/dashboard: install teuthology using pip

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>

rbd: show mirror mode in image info

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: improve mirror snapshot list formatting for status command

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: allow to specify mode on mirror image enable

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: explicitly specify mode on mirror image enable

- don't use journaling feature to chose the mode;
- provide new API function mirror_image_enable2;
- return back the old behavior to automatically enable/disable
  journaling feature on enabling/disabling image mirroring.

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/balancer: avoid pulling pg_dump twice

For large cluster with thousands of osds/pgs, it may take
seconds even mitutes to refresh the relevant stats.
Since we already have pulling them down at initialization,
make a direct use of them whenever possible.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

stop.sh: use bash shell to solve syntax error

When running on ubuntu 18.04 for aarch64, there is syntax error in
line 52, caused by the /bin/sh is defalut to dash, which is not
support array type.

Signed-off-by: luo rixin <luorixin@huawei.com>

rbd-mirror: switch image replayer to new standalone journal replayer

Remove all the original journal replaying code embedded in the image
replayer and instead rely on the new journal replayer class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: extract journal replay logic to its own class

This will help to greatly reduce the journal-specific code in the
current image replayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved image asok/perf counter name helper to utils

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved event pre-processor to sub-namespace

The current implementation is tied to journal-based replaying so
move it down into the journal sub-namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved replay status formatter to sub-namespace

The current implementation is tied to journal-based replaying so
move it down into a new journal sub-namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: drop 'using namespace' in image replayer state machine

This is going to cause conflicts w/ the '::journal' namespace and
future 'rbd::mirror::image_replayer::journal' namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32142 from changchengx/doc_libvirt

doc/rbd: simplify libvirt usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32243 from trociny/wip-43289

rbd-mirror: fix race on namespace replayer initialization failure

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32184 from trociny/wip-43004

rbd-mirror: rename per-image replication perf counters

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32113 from lixiaoy1/master

librbd: finish write request early

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32251 into master

* refs/pull/32251/head:
mds/MDSDaemon: fix asok 'exit' and 'respawn' commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32260 into master

* refs/pull/32260/head:
.organizationmap: Wido 42on -> 42on

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: fix how uid/gid is determined

The uid/gid is determined by looking at file ownership. It
was looking at /etc/ceph, but on current builds this is owned by root so switched to using /var/lib/ceph which
is the homedir of the ceph account.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: add sample configuration for prometheus

Provides an example definition that sets up an
initial prometheus server with a scrape of itself.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: support deployment of prometheus

Initial support to enable cephadm to deploy
monitoring containers like prometheus. This
patch adds support for prometheus. It is the callers
responsibility to provide a valid prometheus.yml
file passed within the --config-json parameter.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

.organizationmap: Wido 42on -> 42on

Signed-off-by: Sage Weil <sage@redhat.com>

cmake: check version of librdkafka

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #32158 into master

* refs/pull/32158/head:
cephadm: fix exception when no time sync is running
cephadm: properly extract osd fsid during adoption
cephadm: do ceph-volume activate+deactivate as part of systemd unit
ceph-daemon: behave with no /etc/ceph/ceph.conf

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: fix exception when no time sync is running

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: properly extract osd fsid during adoption

This needs to be passed to the deploy_daemon_units method so that we can
set up the pre and post c-v calls properly.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do ceph-volume activate+deactivate as part of systemd unit

1- Do the activate as part of the unit on start, since it may include
setup that disappears on a reboot, like starting up the dmcrypted volume.

2- Do the matching deactivate step when the container stops, like shutting
down the dmcrypt volume.

Right now the OSD is the only thing that needs this special behavior.

Signed-off-by: Sage Weil <sage@redhat.com>