doc: updated the dashboard feature list

Updated and improved the Mgr Dashboard feature list: added RBD management
features.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #21446 from smithfarm/wip-ctest-py3-fallback

cmake/mgr: use Python 3 virtualenv if mgr subinterpreter is Python 3

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21477 from liewegas/wip-kvstore-tool

ceph-kvstore-tool: make it a bit more friendly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21488 from tchaikov/wip-i-choose-to-die-instead-of-leak

msg/async/EventKqueue: assert on OOM

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #21496 from ceph/wip-rm23678

ceph-volume: allow skipping systemd interactions on activate/create

Reviewed-by: Andrew Schoen <aschoen@redhat.com

Merge pull request #21466 from jcsp/wip-dashboard-cmake-all

cmake: include frontend build in 'all' target

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21497 from votdev/forceidentifier_datatable

mgr/dashboard: Add 'forceIdentifier' attribute to datatable

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21469 from liewegas/wip-log-nomsgs

mon/LogMonitor: do not crash on log sub w/ no messages

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #21467 from tchaikov/wip-perf-stats-ns-incarnation

include/ceph_features: fix OS_PERF_STAT_NS's incarnation

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #21459 from runsisi/wip-mgrc-register

mgr/MgrClient: service registration filtered by service name instead of daemon name

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21344 from yangdongsheng/lock_timeout

rbd: support lock_timeout in rbd mapping

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

cmake/mgr: use Python 3 virtualenv if mgr subinterpreter is Python 3

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Force identifier for RBD datatable.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Add 'forceIdentified' attribute to datatable

Signed-off-by: Volker Theile <vtheile@suse.com>

rbd: support lock_timeout in rbd mapping

Signed-off-by: Dongsheng Yang <dongsheng.yang@easystack.cn>

Merge pull request #21056 from hitoshikamei/rbdmap-notrim

rbd: add notrim option to rbd map

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #21490 from tchaikov/wip-tox

mgr/dashboard: create venv for running tox

Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: create venv for running tox

the tox shipped with centos is 1.4.2, and is way too old. it does not
allow us to define the dependency using generative-envlist.

Signed-off-by: Kefu Chai <kchai@redhat.com>

tools/setup-virtualenv.sh: bump up the version of tox to 2.9.1

2.8.1 brings an important fix, see
https://github.com/tox-dev/tox/issues/515.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21489 from thmour/thmour_parallel_create

ceph-volume: allow parallel creates

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #16608 into master

* refs/pull/16608/head:
qa: whitelist mds down wrn during cephfs testing
mds: add config to disable fragmentation
qa: add max_mds thrash test
qa: mds_thrash updates for new max_mds behavior
doc: update upgrade procedure and release notes
qa: add test for cluster resizing
qa: remove use of mds deactivate
cephfs: add new down/joinable fs flags
mds: evict all clients if last mds shutting down
cephfs: deprecate ceph mds deactivate
cephfs: kill allow_dirfrags
cephfs: Kill allow_multimds
cephfs: Change behavior of cluster_down flag
mon/FSCommands: Set extra MDS to standby
cephfs: Health check changes
mon/MDSMonitor: Remove command support for legacy syntax

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21456 into master

* refs/pull/21456/head:
MDSMonitor: initialize new Filesystem epoch from pending

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #21457 into master

* refs/pull/21457/head:
mds: unset deleted vars in shutdown_pass

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #21458 into master

* refs/pull/21458/head:
MDSMonitor: cleanup and protect fsmap access

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #21473 into master

* refs/pull/21473/head:
client: fix error operator precedence

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

mds: unset deleted vars in shutdown_pass

So future passes do not try to delete again.

Fixes: http://tracker.ceph.com/issues/23766
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21480 from smithfarm/wip-backport-create-issue

scripts: new backport-create-issue script

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

script/backport-create-issue: enable script to be imported

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-volume tests functional python tests for --no-systemd in activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.activate add --no-systemd flag to skip starting/enabling services

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.common add --no-systemd flag to skip enabling/starting services

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #21436 from trociny/wip-test-out_of_order

test/librbd: ensure OutOfOrder test has enough concurrent management ops

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21493 from trociny/wip-deep-copy-missing-return

librbd: missing 'return' in deep_copy::ObjectCopyRequest::send_read_object

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21385 from ricardoasmarques/wip-list-all-rbds

mgr/dashboard: RBD management (frontend)

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21451 from ceph/wip-rm23581

ceph-volume: automatic VDO detection

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr/dashboard: RBD management

Signed-off-by: Ricardo Marques <rimarques@suse.com>

librbd: missing 'return' in deep_copy::ObjectCopyRequest::send_read_object

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21383 from b-ranto/wip-mgr-rgw-metadata

prometheus: Implement rgw_metadata metric

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: Add 'executing' cell template

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Configurable type for 'cd-submit-button'

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add 'delete-confirmation-modal' component

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Force focus on submit

Forcing focus on submit button will guarantee that all inputs will lose focus
before submission.

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Add get method to summary service

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #21462 from ricardoasmarques/wip-backgroud-tasks-style

mgr/dashboard: Improve background tasks style

Reviewed-by: Laura Paduano <lpaduano@suse.com>

prometheus: Implement rgw_metadata metric

This maps rgw id to its hostname and ceph version running on the node.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #21443 from LenzGr/mgr-dashboard-documentation

doc: Improved dashboard documentation

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume: allow parallel creates

Currently the create code decides the vg_name "ceph-$cluster_fsid" as
the primary vg_name and creates a new name if this already exists.

If this code is run N times in parallel, the script will try to
create N times the vg with the name "ceph-$cluster_fsid" and it
will fail to create the N osds successfully.

Creating vgs with names like "ceph-$uuid4" lets our scripts to run
without any problems.

Signed-off-by: Theofilos Mouratidis <t.mour@cern.ch>

Merge pull request #21485 from tchaikov/wip-tox

test/dashboard: hardcode .coverage path to workaround tox bugs

Reviewed-by: Ricardo Dias <rdias@suse.com>

msg/async/EventKqueue: assert on OOM

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/dashboard: hardcode .coverage path to workaround tox bugs

rhel/centos 7.3 comes with tox 1.4.2, which has a bunch of issues related
to environment variable substitution. for instance, see
- https://github.com/tox-dev/tox/issues/515
so, instead of having different coverage files for py3 and py2.7, it
would be simpler to just use the same path for coverage. this approach
works because pytest-cov plugin erases the coverage data file at the
beginning of test[1].

---
[1] https://pypi.org/project/pytest-cov/#coverage-data-file .

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21474 from jecluis/mimic-mon-pending-releasenotes

PendingReleaseNotes: mention some monitor changes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21428 from ivancich/wip-fix-existing-mclock-cost

osd: remove cost from mclock op queues; cost not handled well in dmclock

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21464 from LenzGr/fix-cmakelist

cmake: fix typo in status message

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #21418 from jecluis/wip-fix-mon-fs-caps

mon/AuthMonitor: don't validate `fs authorize` caps with `valid_caps()`

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21450 from dzafman/wip-jq-compat

test: Use jq in a compatible way and for easier diff analysis

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa: whitelist mds down wrn during cephfs testing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

script/backport-create-issue: explain why release_id is hard-coded

Signed-off-by: Nathan Cutler <ncutler@suse.com>

scripts: backport-create-issue (initial commit)

This is a standalone implementation of a script written by Loic Dachary.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #21483 from yuriw/wip-yuriw-crontab

qa/tests - one more typo fixed :(

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests - one more typo fixed :(

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21482 from yuriw/wip-yuriw-crontab

qa/tests - Fixed typo in crontab entry

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests - Fixed typo in crontab entry

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21454 from yuriw/wip-yuriw-crontab

qa/tests - added for the suites with subset be able to use 'testing' …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume util.prepare preserve order on incoming mount flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests update (preserved) order on mount flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/dev/ceph-volume add VDO tag information

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests ensure non-duplicate flags in mount calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.prepare allow and dedupe extra flags for mounting

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.prepare detect vdo from device for mounting filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mds: add config to disable fragmentation

For disaster situations or testing.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add max_mds thrash test

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: mds_thrash updates for new max_mds behavior

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: update upgrade procedure and release notes

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add test for cluster resizing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove use of mds deactivate

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs: add new down/joinable fs flags

Fixes: http://tracker.ceph.com/issues/20609
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: evict all clients if last mds shutting down

This allows all ranks to stop. If the administrator shuts down the cluster then
we should immediately shut down all client's I/O.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs: deprecate ceph mds deactivate

MDS deactivation is now handled by the max_mds parameter. Deprecate
ceph mds deactivate and note it to be removed in a future release.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: kill allow_dirfrags

As dirfrags are now standard in CephFS, remove the machinery for
tracking and enabling this feature.

ceph fs set <fs> allow_dirfrags is now deprecated and prints a warning
message.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: Kill allow_multimds

With multi-mds now declared stable, allow_multimds now defaults to 1.
Given the max_mds parameter, it is now redundant. Remove it, leaving a
comment placeholder in the features bitmap.

ceph fs set <fs> allow_multimds is now deprecated and prints a warning
message.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: Change behavior of cluster_down flag

Setting the cluster_down flag will now set all active MDS
to standby and clearing it will restore the previous max_mds.
Changing max_mds when the cluster_down flag is set will clear the
flag automatically.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

mon/FSCommands: Set extra MDS to standby

When reducing max_mds, deactivate any MDS rank greater than the
new value of max_mds.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

cephfs: Health check changes

Issue the HEALTH_WARN MDS_UP_LESS_THAN_MAX whenever an MDS cluster
has fewer than its max_mds online.

Issue the HEALTH_ERR MDS_ALL_DOWN whenever an MDS cluster has no
active MDS.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

mon/MDSMonitor: Remove command support for legacy syntax

No longer check for commands for deprecated legacy filesystems.
These commands are long since deprecated and this code is no longer
needed.

Signed-off-by: Douglas Fuller <dfuller@redhat.com>

MDSMonitor: cleanup and protect fsmap access

This commit is designed to permanently protect the current epoch FSMap from
accidental changes. Modifications to the pending_fsmap are protected by
checking if the monitor is the leader. Additionally, code must explicitly
request a writeable reference.

Fixes http://tracker.ceph.com/issues/23762

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21478 from idryomov/wip-krbd-os-recommend

doc/start/os-recommendations.rst: bump krbd kernels

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/start/os-recommendations.rst: bump krbd kernels

Drop 4.4 -- even though it is an "Extended LTS" release, it predates
the OSD client rewrite and should no longer be recommended.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-kvstore-tool: default ot level 2 for rocksdb

This gets rid of the wall of rocksdb logs every time.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-kvstore-tool: ignore default config file and mon config by default

This is a low-level tool and it is super annoying to have to pass
--no-mon-config every time I use it.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21409 from ErwanAliasr1/evelu-readonly

ceph-volume: Using --readonly for {vg|pv|lv}s commands

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

client: fix error operator precedence

    /home/pdonnell/ceph/src/client/Client.cc: In member function ‘int Client::mount(const string&, const UserPerm&, bool)’:
    /home/pdonnell/ceph/src/client/Client.cc:5681:23: warning: suggest parentheses around ‘+’ inside ‘<<’ [-Wparentheses]
        return CEPH_FUSE_NO_MDS_UP;
                    ~~^~

Found by gcc. I am ashamed.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

PendingReleaseNotes: mention some monitor changes

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

mon/AuthMonitor: don't validate fs caps on authorize

The monitor will have no idea how to validate fs caps during `fs
authorize`. Instead, validate solely the built osd and mds caps.

Signed-off-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #21468 from tchaikov/wip=cmake-make-check-dashboard

cmake: let "tests" depend on "mgr-dashboard-frontend-build"

Reviewed-by: John Spray <john.spray@redhat.com>

test/librbd: ensure OutOfOrder test has enough concurrent management ops

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21470 from tylerqi/master

doc: Fix a grammar error in rbd-snapshot.rst

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: Fix a grammar error in rbd-snapshot.rst

Signed-off-by: Zeqing Tyler Qi <aqize@126.com>

Merge pull request #20945 from liewegas/wip-pg-ls

mon/PGMap: remove or narrow columns 'pg ls' output

Reviewed-by: John Spray <jspray@redhat.com>

Merge pull request #21113 from kungf/scrub_interval_overflow

utime： fix __32u sec time overflow

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21145 from ceph/wip-ifed-store-test-improve

test/ceph_test_objectstore: make settings update and restore less error prone

Reviewed-by: Sage Weil <sage@redhat.com>