mgr/dashboard: prettify code

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Move unit-test-helper into the new testing folder

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22996 from batrick/mds-state-dot

doc: add mds state transition diagram

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22581 from fullerdj/wip-djf-24544

rgw: change default rgw_thread_pool_size to 512

mds: remove ascii art version of mds state diagram

Now refer to the user documentation. Any updates should go there!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: create mds state documentation

Fixes: http://tracker.ceph.com/issues/22989
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20330 from joke-lee/wip-rgw-postobjloc

rgw: return valid Location element, PostObj

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22937 from tianshan/fix_index_suggest_ut_fail

rgw: continue enoent index in dir_suggest

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #23095 from trociny/wip-24898

 librbd: ensure exclusive lock acquired when removing sync point snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23167 from tchaikov/wip-tools-rbd-cleanup

tools/rbd: assert(g_ceph_context) not g_conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22419 from votdev/hide_progress_bar

mgr/dashboard: Hide progress bar in case of an error

Merge pull request #23182 from alfredodeza/wip-ansible-26

ceph-volume tests/functional use Ansible 2.6

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23132 from dillaman/wip-25000

librbd: deep-copy should not write to objects that cannot exist

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22792 from jcsp/wip-mgrmodule-none-docs

mgr: improve docs for MgrModule methods

Merge pull request #22794 from wido/mgr-metadata-none

mgr: Ignore daemon if no metadata was returned

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22827 from theanalyst/wip-device-py3

mgr: devicehealth: dont error on dict iteritems

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22951 from jcsp/wip-mgr-intermodule

mgr: enable inter-module calls

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #23181 from tspmelo/wip-remove-karma

mgr/dashboard: Remove karma packages

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23115 from tspmelo/wip-angular-cmake

mgr/dashboard: Remove angular build progress logs during cmake

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #23159 from smithfarm/wip-example-no-boost

example/librados: remove dependency on Boost system library

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22978 from tchaikov/wip-docker-test-cleanup

test/docker-test-helper.sh: move "cp .git/HEAD" out of loop

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23194 from alfredodeza/wip-lvm-docs

ceph-volume: expand on why ceph-disk was replaced

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23029 from dillaman/wip-24909

osd: do not treat an IO hint as an IOP for PG stats

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23186 from smithfarm/wip-4640

doc/man: mention import and export commands in rados manpage

doc/mgr: add docs for missing functions

Some functions had docstrings but were
not linked into plugins.rst, some needed
more text added.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/ceph-volume expand on the ceph-disk replacement reasons

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume reference the new ceph-disk-replacement section

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional use testinfra latest for ansible 2.6

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man: mention import and export commands in rados manpage

Fixes: http://tracker.ceph.com/issues/4640
Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-volume tests/functional use Ansible 2.6

The ceph-ansible project now refuses to use 2.4 even though it works
with that version

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Remove karma packages

Since we are no longer using karma for the frontend unit tests, we will remove
all the related packages and configurations.

This includes the removal of the phantomjs package.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23114 from smithfarm/wip-24800

doc: use :command: for subcommands in ceph-bluestore-tool manpage

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Remove angular build progress logs during cmake

Currently Angular CLI is polluting the jenkins logs, by inserting a new line
each time the build process is updated.
With this change Angular CLI will only output the necessary information about
the start and end of the build.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Hide progress bar in case of an error

- Enhance datatable to handle loading errors.
- Add warning panel component
- Modify the icon of the error panel component
- Use cd-[info|warning|error]-panel in the cd-view-cache component

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23166 from tchaikov/wip-cmake-cleanups

cmake: cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22875 from smithfarm/wip-backport-script

script: new ceph-backport.sh script

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20401 from smithfarm/wip-22558

build/ops: rpm: provide files moved from ceph-test …

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/rbd: assert(g_ceph_context) not g_conf

see also 813cd5d5910c266c076d38277eddf32d908d5ae3

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract ceph-dencoder related stuff out

let's modularize the cmake file piecemeal.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Finddpdk: add dpdk::${component} libs

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23160 from smithfarm/wip-24017

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Reviewed-by: Sage Weil <sage@redhat.com>

example/librados: remove dependency on Boost system library

109e6022beb0920f2a4746bd8c541e975494f251 introduced "-lboost_system" into
example/librados/Makefile but the Boost system library is no longer required
to compile and link hello_world.cc.

Fixes: http://tracker.ceph.com/issues/25054
Signed-off-by: Nathan Cutler <ncutler@suse.com>

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Fixes: http://tracker.ceph.com/issues/25051
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: provide files moved from ceph-test

d7b493a7108a68302bc0f48337bf3c253a720266 moved three binaries from the
ceph-test package to other packages, causing trouble with upgrades in
test lab environments (users typically do not install the ceph-test
package).

While the original upgrade troubles have already been addressed, RPM
has a "Provides" mechanism for easing the pain, so it makes sense to
use it.

Fixes: http://tracker.ceph.com/issues/22558
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #23139 into master

* refs/pull/23139/head:
mds: avoid using g_conf().get_val<...>(...) in hot path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22977 into master

* refs/pull/22977/head:
qa/tasks/cephfs: add test for discontinuous mdsmap
mds: handle discontinuous mdsmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23022 into master

* refs/pull/23022/head:
mds: health warning for slow metadata IO

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23063 into master

* refs/pull/23063/head:
mds: increase debug level for dropped client cap msg

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23062 into master

* refs/pull/23062/head:
mds: dump recent events on respawn

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23061 into master

* refs/pull/23061/head:
mds: print mdsmap processed at low debug level

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #23135 into master

* refs/pull/23135/head:
mds: refactor ostream passing

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #22947 into master

* refs/pull/22947/head:
test: validate empty json output during recursive scrub
test: subclass test_scrub.Workload from CephFSTestCase
mds: dump minimal cli output during recursive scrub trigger

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23009 into master

* refs/pull/23009/head:
client: wire up ceph_ll_fallocate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23151 from neha-ojha/wip-25008

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Merge pull request #23123 from yuriw/wip-yuriw-24926-master

qa/tests: Set ansible-version: 2.5

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #22982 into master

* refs/pull/22982/head:
os/bluestore: Add new perfcounter which record latency of

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #23041 into master

* refs/pull/23041/head:
mon: manage paxos_services alloc with unique_ptr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23047 into master

* refs/pull/23047/head:
common: SubProcess needs SIGTERM, from the includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23053 into master

* refs/pull/23053/head:
mon: emplace mon_caps in Monitor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23060 into master

* refs/pull/23060/head:
test: Only inquire bluestore when it's build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23078 into master

* refs/pull/23078/head:
spdk: -march=core2 instead of -march=native

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr: generic self test command

Avoid need for each module to expose a self-test
command: they can just implement the method,
and then get it called via the selftest module.

As well as fewer LOC, this means that the self
test commands are not cluttering the interface
for end users, as they've invisible until
the selftest module is loaded.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: update mgr plugin page for remote calls

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: enable inter-module calls

This is being done by passing native CPython objects
back and forth.  It's safe because sub-interpreters in CPython
share memory allocation infrastructure and share the GIL.

With a view to PEP554, we limit inter-interpreter calls
to pickleable objects, so that this may be implemented
using byte-arrays in future.

This infrastructure should enable:
 - the dashboard to display the status of other modules, for
   example the set of progress indicators from `progress`
 - dashboard and restful to share an underlying long running
   job mechanism.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #22919 from liewegas/wip-rgw-cls-test

qa/suites/rgw/verify/tasks/cls_rgw: test cls_rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22975 from tone-zhang/vstart-spdk

vstart.sh: Support SPDK in Ceph development deployment

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22790 from joke-lee/WIP-admin-rest-api-user-info-with-access-key

rgw: support admin rest api get user info through user's access-key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23099 from adamemerson/wip-all-your-fault

rgw: All Your Fault

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #23058 from mkogan1/cont-19282-fix-rgw-multisite-datasync_v2

rgw: add curl_low_speed_limit and curl_low_speed_time config to avoid

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23119 from cbodley/wip-rgw-link-tenant

rgw: RGWBucket::link supports tenant

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23010 from noahdesu/mgr-expose-ec-profiles

mgr: expose ec profiles through manager

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #23016 from rzarzynski/wip-osd-cleanup-log_op_stats

osd: clean up and avoid extra ref-counting in PrimaryLogPG::log_op_stats.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22993 from jcsp/wip-progress

mgr/progress: introduce the `progress` module

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

rgw: return valid Location element, PostObj

PostResponse should contain ETag and urlencode bucket name
object name and tenant name and correct Location uri

Fixes: http://tracker.ceph.com/issues/22927
Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

mds: avoid using g_conf().get_val<...>(...) in hot path

g_conf->get_val<>()'s overhead is high

Fixes: http://tracker.ceph.com/issues/24820
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

vstart.sh: Support SPDK in Ceph development deployment

Support SPDK in vstart.sh, the developer can debug SPDK within
development deployment with the option
"--bluestore-spdk <vendor>:<device>".

Signed-off-by: tone.zhang <tone.zhang@arm.com>

Merge pull request #23121 from noahdesu/mgr-crash-history

mgr/crash: add hour granularity crash summary

mds: refactor ostream passing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: deep-copy should not write to objects that cannot exist

If an image has a snapshot and is subsequently shrunk, ensure that
discard ops are not sent to the deep-copy objects beyond the bounds
of the image at a given snapshot/HEAD. Only a remove op should be
expected in such cases.

Fixes: http://tracker.ceph.com/issues/25000
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/crash: json report of recent crashes

reports crashes per entity type with hour granularity. primarily for
consumption by the insights module.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

mgr/crash: add timestamp filter helper

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23129 from alfredodeza/wip-rm24952

ceph-volume lvm.listing only include devices if they exist

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23097 into master

* refs/pull/23097/head:
vstart_runner: pass ctx to FuseMount and load mount info

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #23077 from smithfarm/wip-etc-sudoers

build/ops: rpm: make sudo a build dependency

Reviewed-by: Tim Serong <tserong@suse.com>

build/ops: rpm: make sudo a build dependency

ede36df3253bd67bfe206abc0446f1f156feae64 added sudo as a runtime dependency,
but that commit also causes OBS builds to fail with:

[10666s] ... checking filelist
[10667s] ceph-osd-14.0.0-lp150.1332.1.x86_64.rpm: directories not owned by a package:
[10667s]  - /etc/sudoers.d

Since the "sudo" package is a runtime dependency of ceph-osd, it is guaranteed
to be installed on the target system and there is no reason for ceph-osd to own
this directory.

Having sudo as a build dependency makes OBS happy, and does no harm.

Follow-on fix for https://github.com/ceph/ceph/pull/22745

Signed-off-by: Nathan Cutler <ncutler@suse.com>

script: new ceph_backport.sh script

Once the local backport git branch has been prepared (i.e. after cherry-picking
stage), this script automates the process of opening the backport PR and
updating the backport tracker issue.

Parts of this script were written by Loic Dachary.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #23089 from smithfarm/wip-seastar-build-deps

build/ops: rpm: fix seastar build dependencies for SUSE

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: use :command: for subcommands in ceph-bluestore-tool manpage

Older versions of Sphinx, such as the one in CentOS 7, do not render "..
option::" lines correctly if the option contains a hyphen but does not start
with a hyphen. And ceph-bluestore-tool appears to be the only Ceph manpage
affected by this bug.

Fixes: http://tracker.ceph.com/issues/24800
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: fix seastar build dependencies for SUSE

cryptopp-devel is called libcryptopp-devel on SUSE.

/usr/bin/protoc is included in protobuf-devel on SUSE.

numactl-devel is called libnuma-devel on SUSE

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/tasks/cephfs: add test for discontinuous mdsmap

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: handle discontinuous mdsmap

There are two cases that a mds can get discontinuous mdsmap:
 - the mdsmap was sent by other mds
 - connection to monitor was reset

Monitor does not preserve old version mdsmap. There is no easy way to
ensure mds always gets continuous mdsmap. Instead, making mds handle
discontinuous mdsmap is not difficult. When failover happens in multimds
cluster, survivor mds may miss the mdsmap that indicates old mds failed
and/or the mdsmap that indicates new mds started to replay. But the
survivor mds always get the mdsmap that indicates the new mds entered
resolve state.

Fixes: http://tracker.ceph.com/issues/24856
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/tests: Set ansible-version: 2.5

Fixes http://tracker.ceph.com/issues/24926
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

ceph-volume lvm.listing only include devices if they exist

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rgw: RGWBucket::link supports tenant

use tenant-aware helper functions to generate metadata keys/oids

this allows 'radosgw-admin bucket link' to succeed for a tenanted user,
provided that the bucket belongs to the same tenant

Fixes: http://tracker.ceph.com/issues/22666
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23117 from ceph/wip-rm24796

ceph-volume: unmount lvs correctly before zapping

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #23098 from neha-ojha/wip-limit-pglog

osd: limit pg log length under all circumstances

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22347 from cbodley/wip-24367

rgw: fixes for sync of versioned objects

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23032 from cbodley/wip-24873

radosgw-admin: 'sync error trim' loops until complete

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23116 from alfredodeza/wip-rm24961

ceph-volume tests/functional run lvm list after OSD provisioning

Reviewed-by: Andrew Schoen <aschoen@redhat.com>