cmake/mgr: use Python 3 virtualenv if mgr subinterpreter is Python 3

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #21056 from hitoshikamei/rbdmap-notrim

rbd: add notrim option to rbd map

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #21490 from tchaikov/wip-tox

mgr/dashboard: create venv for running tox

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: create venv for running tox

the tox shipped with centos is 1.4.2, and is way too old. it does not
allow us to define the dependency using generative-envlist.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/setup-virtualenv.sh: bump up the version of tox to 2.9.1

2.8.1 brings an important fix, see
https://github.com/tox-dev/tox/issues/515.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21489 from thmour/thmour_parallel_create

ceph-volume: allow parallel creates

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #16608 into master

* refs/pull/16608/head:
qa: whitelist mds down wrn during cephfs testing
mds: add config to disable fragmentation
qa: add max_mds thrash test
qa: mds_thrash updates for new max_mds behavior
doc: update upgrade procedure and release notes
qa: add test for cluster resizing
qa: remove use of mds deactivate
cephfs: add new down/joinable fs flags
mds: evict all clients if last mds shutting down
cephfs: deprecate ceph mds deactivate
cephfs: kill allow_dirfrags
cephfs: Kill allow_multimds
cephfs: Change behavior of cluster_down flag
mon/FSCommands: Set extra MDS to standby
cephfs: Health check changes
mon/MDSMonitor: Remove command support for legacy syntax

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21456 into master

* refs/pull/21456/head:
MDSMonitor: initialize new Filesystem epoch from pending

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #21457 into master

* refs/pull/21457/head:
mds: unset deleted vars in shutdown_pass

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21458 into master

* refs/pull/21458/head:
MDSMonitor: cleanup and protect fsmap access

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #21473 into master

* refs/pull/21473/head:
client: fix error operator precedence

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: unset deleted vars in shutdown_pass

So future passes do not try to delete again.

Fixes: http://tracker.ceph.com/issues/23766
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21480 from smithfarm/wip-backport-create-issue

scripts: new backport-create-issue script

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

script/backport-create-issue: enable script to be imported

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #21436 from trociny/wip-test-out_of_order

test/librbd: ensure OutOfOrder test has enough concurrent management ops

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21493 from trociny/wip-deep-copy-missing-return

librbd: missing 'return' in deep_copy::ObjectCopyRequest::send_read_object

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21385 from ricardoasmarques/wip-list-all-rbds

mgr/dashboard: RBD management (frontend)

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21451 from ceph/wip-rm23581

ceph-volume: automatic VDO detection

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: RBD management

Signed-off-by: Ricardo Marques <rimarques@suse.com>

librbd: missing 'return' in deep_copy::ObjectCopyRequest::send_read_object

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21383 from b-ranto/wip-mgr-rgw-metadata

prometheus: Implement rgw_metadata metric

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: Add 'executing' cell template

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Configurable type for 'cd-submit-button'

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add 'delete-confirmation-modal' component

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Force focus on submit

Forcing focus on submit button will guarantee that all inputs will lose focus
before submission.

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add get method to summary service

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21462 from ricardoasmarques/wip-backgroud-tasks-style

mgr/dashboard: Improve background tasks style

Reviewed-by: Laura Paduano <lpaduano@suse.com>

prometheus: Implement rgw_metadata metric

This maps rgw id to its hostname and ceph version running on the node.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #21443 from LenzGr/mgr-dashboard-documentation

doc: Improved dashboard documentation

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume: allow parallel creates

Currently the create code decides the vg_name "ceph-$cluster_fsid" as
the primary vg_name and creates a new name if this already exists.

If this code is run N times in parallel, the script will try to
create N times the vg with the name "ceph-$cluster_fsid" and it
will fail to create the N osds successfully.

Creating vgs with names like "ceph-$uuid4" lets our scripts to run
without any problems.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

Merge pull request #21485 from tchaikov/wip-tox

test/dashboard: hardcode .coverage path to workaround tox bugs

Reviewed-by: Ricardo Dias <rdias@suse.com>

test/dashboard: hardcode .coverage path to workaround tox bugs

rhel/centos 7.3 comes with tox 1.4.2, which has a bunch of issues related
to environment variable substitution. for instance, see
- https://github.com/tox-dev/tox/issues/515
so, instead of having different coverage files for py3 and py2.7, it
would be simpler to just use the same path for coverage. this approach
works because pytest-cov plugin erases the coverage data file at the
beginning of test[1].

---
[1] https://pypi.org/project/pytest-cov/#coverage-data-file .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21474 from jecluis/mimic-mon-pending-releasenotes

PendingReleaseNotes: mention some monitor changes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21428 from ivancich/wip-fix-existing-mclock-cost

osd: remove cost from mclock op queues; cost not handled well in dmclock

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21464 from LenzGr/fix-cmakelist

cmake: fix typo in status message

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21418 from jecluis/wip-fix-mon-fs-caps

mon/AuthMonitor: don't validate `fs authorize` caps with `valid_caps()`

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21450 from dzafman/wip-jq-compat

test: Use jq in a compatible way and for easier diff analysis

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: whitelist mds down wrn during cephfs testing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

script/backport-create-issue: explain why release_id is hard-coded

Signed-off-by: Nathan Cutler <ncutler@suse.com>

scripts: backport-create-issue (initial commit)

This is a standalone implementation of a script written by Loic Dachary.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #21483 from yuriw/wip-yuriw-crontab

qa/tests - one more typo fixed :(

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests - one more typo fixed :(

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21482 from yuriw/wip-yuriw-crontab

qa/tests - Fixed typo in crontab entry

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests - Fixed typo in crontab entry

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21454 from yuriw/wip-yuriw-crontab

qa/tests - added for the suites with subset be able to use 'testing' …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume util.prepare preserve order on incoming mount flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests update (preserved) order on mount flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume add VDO tag information

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests ensure non-duplicate flags in mount calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.prepare allow and dedupe extra flags for mounting

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare detect vdo from device for mounting filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mds: add config to disable fragmentation

For disaster situations or testing.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add max_mds thrash test

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: mds_thrash updates for new max_mds behavior

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: update upgrade procedure and release notes

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add test for cluster resizing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove use of mds deactivate

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs: add new down/joinable fs flags

Fixes: http://tracker.ceph.com/issues/20609
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: evict all clients if last mds shutting down

This allows all ranks to stop. If the administrator shuts down the cluster then
we should immediately shut down all client's I/O.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs: deprecate ceph mds deactivate

MDS deactivation is now handled by the max_mds parameter. Deprecate
ceph mds deactivate and note it to be removed in a future release.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: kill allow_dirfrags

As dirfrags are now standard in CephFS, remove the machinery for
tracking and enabling this feature.

ceph fs set <fs> allow_dirfrags is now deprecated and prints a warning
message.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: Kill allow_multimds

With multi-mds now declared stable, allow_multimds now defaults to 1.
Given the max_mds parameter, it is now redundant. Remove it, leaving a
comment placeholder in the features bitmap.

ceph fs set <fs> allow_multimds is now deprecated and prints a warning
message.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: Change behavior of cluster_down flag

Setting the cluster_down flag will now set all active MDS
to standby and clearing it will restore the previous max_mds.
Changing max_mds when the cluster_down flag is set will clear the
flag automatically.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

mon/FSCommands: Set extra MDS to standby

When reducing max_mds, deactivate any MDS rank greater than the
new value of max_mds.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: Health check changes

Issue the HEALTH_WARN MDS_UP_LESS_THAN_MAX whenever an MDS cluster
has fewer than its max_mds online.

Issue the HEALTH_ERR MDS_ALL_DOWN whenever an MDS cluster has no
active MDS.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

mon/MDSMonitor: Remove command support for legacy syntax

No longer check for commands for deprecated legacy filesystems.
These commands are long since deprecated and this code is no longer
needed.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

MDSMonitor: cleanup and protect fsmap access

This commit is designed to permanently protect the current epoch FSMap from
accidental changes. Modifications to the pending_fsmap are protected by
checking if the monitor is the leader. Additionally, code must explicitly
request a writeable reference.

Fixes http://tracker.ceph.com/issues/23762

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21478 from idryomov/wip-krbd-os-recommend

doc/start/os-recommendations.rst: bump krbd kernels

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/start/os-recommendations.rst: bump krbd kernels

Drop 4.4 -- even though it is an "Extended LTS" release, it predates
the OSD client rewrite and should no longer be recommended.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #21409 from ErwanAliasr1/evelu-readonly

ceph-volume: Using --readonly for {vg|pv|lv}s commands

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

client: fix error operator precedence

    /home/pdonnell/ceph/src/client/Client.cc: In member function ‘int Client::mount(const string&, const UserPerm&, bool)’:
    /home/pdonnell/ceph/src/client/Client.cc:5681:23: warning: suggest parentheses around ‘+’ inside ‘<<’ [-Wparentheses]
        return CEPH_FUSE_NO_MDS_UP;
                    ~~^~

Found by gcc. I am ashamed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

PendingReleaseNotes: mention some monitor changes

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/AuthMonitor: don't validate fs caps on authorize

The monitor will have no idea how to validate fs caps during `fs
authorize`. Instead, validate solely the built osd and mds caps.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #21468 from tchaikov/wip=cmake-make-check-dashboard

cmake: let "tests" depend on "mgr-dashboard-frontend-build"

Reviewed-by: John Spray <john.spray@redhat.com>

test/librbd: ensure OutOfOrder test has enough concurrent management ops

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21470 from tylerqi/master

doc: Fix a grammar error in rbd-snapshot.rst

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: Fix a grammar error in rbd-snapshot.rst

Signed-off-by: Zeqing Tyler Qi <aqize@126.com>

Merge pull request #20945 from liewegas/wip-pg-ls

mon/PGMap: remove or narrow columns 'pg ls' output

Reviewed-by: John Spray <jspray@redhat.com>

Merge pull request #21113 from kungf/scrub_interval_overflow

utime： fix __32u sec time overflow

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21145 from ceph/wip-ifed-store-test-improve

test/ceph_test_objectstore: make settings update and restore less error prone

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21371 from jcsp/wip-mon-optracker-cleanup

mon: cleanups to optracker code

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #21377 from ifed01/wip-ifed-learn-alloc-fragm

os/bluestore: add perf counter for allocator fragmentation.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21141 from shun-s/fix-rbd-close-race-with-rewatch

librbd: fix rbd close race with rewatch

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21343 from Songweibin/wip-trash-ls

rbd: 'trash ls -l' will display column titles if existed non-USER trash image only

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: let "tests" depend on "mgr-dashboard-frontend-build"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: fix typo in status message

Replaced " we are x84_64" with  " we are x86_64" in `src/CMakeLists.txt`

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: Improve background tasks style

Signed-off-by: Ricardo Marques <rimarques@suse.com>

MDSMonitor: initialize new Filesystem epoch from pending

Fixes: https://tracker.ceph.com/issues/23764
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests - added for the suites with subset be able to use 'testing' distro

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

ceph-volume lvm.activate detect vdo from tags for mounting filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.prepare allow extra ad-hoc mount flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare capture vdo-ness on LVM tags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests for is_vdo detection

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm helpers to detect vdo from a dm path

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21449 from rjfd/wip-dashboard-fix-frontend-build

cmake: fix frontend cmake build

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21360 from rjfd/wip-dashboard-rbd

mgr/dashboard: RBD management (backend)

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21445 from rjfd/wip-dashboard-fix-run-tox

mgr/dashboard: run-tox: pass CEPH_BUILD_DIR value into tox script

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: remove cost from mclock op queues; cost not handled well in dmclock library

The current version of the dmclock library does not handle operation
cost well. Therefore cost should not be passed into the library when
enqueuing operations; instead 0 should be passed in.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

test: Use jq in a compatible way and for easier diff analysis

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr/dashboard: fix frontend cmake build

Signed-off-by: Ricardo Dias <rdias@suse.com>