test: Add test for repair of bad object info data_digest on all copies

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix pg repair to handle object info data_digest mismatched, but replicas match each other.

Fixes: https://tracker.ceph.com/issues/25085
Signed-off-by: David Zafman <dzafman@redhat.com>

tools: Add testing feature "corrupt-info" to ceph-objectstore-tool

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Improve logic in be_compare_scrubmaps()

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #22695 from a2batic/scss

mgr/dashboard: Color variables for color codes

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23100 from adamemerson/wip-robustly-notify

rgw: Robustly notify

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #23183 from ricardoasmarques/wip-fix-url-prefix

mgr/dashboard: Fix cherrypy static content URL prefix config

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Color variables for color codes
Assign variables to colors present in css files
Changed navigation icons padding because the notification dropdown was going out of window.
Fixes: http://tracker.ceph.com/issues/24575
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #22857 from tspmelo/wip-move-test-file

mgr/dashboard: Move unit-test-helper into the new testing folder

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #22624 from ashutosh-narkar/opa-ceph

rgw: Initial work for OPA-Ceph integration

Merge PR #23220 into master

* refs/pull/23220/head:
osd/OSDMap: CRUSH_TUNABLES5 added in jewel, not kraken

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Initial work for OPA-Ceph integration

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>

osd/OSDMap: CRUSH_TUNABLES5 added in jewel, not kraken

In 0ceb5c0f3694725c7043ea99b077816da2ca44a7 I screwed up the version that
CRUSH_TUANBLES5 (chooseleaf_stable) was added: it comes from
043a7378ec4e828f2568ea3906d9076848161ccd, v10.0.3, which leads up to
jewel.

This, in turn, means that when jewel tunables are set we do not actually
encode the jewel tunables.  At least, luminous doesn't; jewel uses a
different code path to decode how to encode the OSDMap.

This, in turn, leads to crc errors:

- luminous mons running
- require_osd_release is jewel
- jewel tunables are set.  mon in-memory crushmap gets chooseleaf_stable=1
- osds are luminous, but don't see the new field in the crushmap
- upgrade finishes, and require_osd_Release becomes luminous
- the full osdmap encodes the chooseleaf_stable field, finally, and it
  is 1
- the osds try to encode the same thing, but their crushmap has 0.
-> crc mismatch

This is a regression from 0ceb5c0f3694725c7043ea99b077816da2ca44a7,
backported to luminous in 686b054f48c50859bd5e8bad93007bf9d7d7899c in
12.2.6.

Fixes: 686b054f48c50859bd5e8bad93007bf9d7d7899c
Fixes: http://tracker.ceph.com/issues/25057
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: prettify code

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Move unit-test-helper into the new testing folder

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22996 from batrick/mds-state-dot

doc: add mds state transition diagram

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22581 from fullerdj/wip-djf-24544

rgw: change default rgw_thread_pool_size to 512

mds: remove ascii art version of mds state diagram

Now refer to the user documentation. Any updates should go there!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: create mds state documentation

Fixes: http://tracker.ceph.com/issues/22989
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20330 from joke-lee/wip-rgw-postobjloc

rgw: return valid Location element, PostObj

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22937 from tianshan/fix_index_suggest_ut_fail

rgw: continue enoent index in dir_suggest

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #23095 from trociny/wip-24898

 librbd: ensure exclusive lock acquired when removing sync point snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23167 from tchaikov/wip-tools-rbd-cleanup

tools/rbd: assert(g_ceph_context) not g_conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22419 from votdev/hide_progress_bar

mgr/dashboard: Hide progress bar in case of an error

Merge pull request #23182 from alfredodeza/wip-ansible-26

ceph-volume tests/functional use Ansible 2.6

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23132 from dillaman/wip-25000

librbd: deep-copy should not write to objects that cannot exist

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22792 from jcsp/wip-mgrmodule-none-docs

mgr: improve docs for MgrModule methods

Merge pull request #22794 from wido/mgr-metadata-none

mgr: Ignore daemon if no metadata was returned

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22827 from theanalyst/wip-device-py3

mgr: devicehealth: dont error on dict iteritems

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22951 from jcsp/wip-mgr-intermodule

mgr: enable inter-module calls

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #23181 from tspmelo/wip-remove-karma

mgr/dashboard: Remove karma packages

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23115 from tspmelo/wip-angular-cmake

mgr/dashboard: Remove angular build progress logs during cmake

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #23159 from smithfarm/wip-example-no-boost

example/librados: remove dependency on Boost system library

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #22978 from tchaikov/wip-docker-test-cleanup

test/docker-test-helper.sh: move "cp .git/HEAD" out of loop

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23194 from alfredodeza/wip-lvm-docs

ceph-volume: expand on why ceph-disk was replaced

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23029 from dillaman/wip-24909

osd: do not treat an IO hint as an IOP for PG stats

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23186 from smithfarm/wip-4640

doc/man: mention import and export commands in rados manpage

doc/mgr: add docs for missing functions

Some functions had docstrings but were
not linked into plugins.rst, some needed
more text added.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/ceph-volume expand on the ceph-disk replacement reasons

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume reference the new ceph-disk-replacement section

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional use testinfra latest for ansible 2.6

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man: mention import and export commands in rados manpage

Fixes: http://tracker.ceph.com/issues/4640
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Fix cherrypy static content URL prefix config

Fixes: https://tracker.ceph.com/issues/25067
Signed-off-by: Ricardo Marques <rimarques@suse.com>

ceph-volume tests/functional use Ansible 2.6

The ceph-ansible project now refuses to use 2.4 even though it works
with that version

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Remove karma packages

Since we are no longer using karma for the frontend unit tests, we will remove
all the related packages and configurations.

This includes the removal of the phantomjs package.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23114 from smithfarm/wip-24800

doc: use :command: for subcommands in ceph-bluestore-tool manpage

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Remove angular build progress logs during cmake

Currently Angular CLI is polluting the jenkins logs, by inserting a new line
each time the build process is updated.
With this change Angular CLI will only output the necessary information about
the start and end of the build.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Hide progress bar in case of an error

- Enhance datatable to handle loading errors.
- Add warning panel component
- Modify the icon of the error panel component
- Use cd-[info|warning|error]-panel in the cd-view-cache component

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23166 from tchaikov/wip-cmake-cleanups

cmake: cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22875 from smithfarm/wip-backport-script

script: new ceph-backport.sh script

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20401 from smithfarm/wip-22558

build/ops: rpm: provide files moved from ceph-test …

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/rbd: assert(g_ceph_context) not g_conf

see also 813cd5d5910c266c076d38277eddf32d908d5ae3

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract ceph-dencoder related stuff out

let's modularize the cmake file piecemeal.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Finddpdk: add dpdk::${component} libs

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23160 from smithfarm/wip-24017

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Reviewed-by: Sage Weil <sage@redhat.com>

example/librados: remove dependency on Boost system library

109e6022beb0920f2a4746bd8c541e975494f251 introduced "-lboost_system" into
example/librados/Makefile but the Boost system library is no longer required
to compile and link hello_world.cc.

Fixes: http://tracker.ceph.com/issues/25054
Signed-off-by: Nathan Cutler <ncutler@suse.com>

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Fixes: http://tracker.ceph.com/issues/25051
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: provide files moved from ceph-test

d7b493a7108a68302bc0f48337bf3c253a720266 moved three binaries from the
ceph-test package to other packages, causing trouble with upgrades in
test lab environments (users typically do not install the ceph-test
package).

While the original upgrade troubles have already been addressed, RPM
has a "Provides" mechanism for easing the pain, so it makes sense to
use it.

Fixes: http://tracker.ceph.com/issues/22558
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #23139 into master

* refs/pull/23139/head:
mds: avoid using g_conf().get_val<...>(...) in hot path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22977 into master

* refs/pull/22977/head:
qa/tasks/cephfs: add test for discontinuous mdsmap
mds: handle discontinuous mdsmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23022 into master

* refs/pull/23022/head:
mds: health warning for slow metadata IO

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23063 into master

* refs/pull/23063/head:
mds: increase debug level for dropped client cap msg

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23062 into master

* refs/pull/23062/head:
mds: dump recent events on respawn

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23061 into master

* refs/pull/23061/head:
mds: print mdsmap processed at low debug level

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #23135 into master

* refs/pull/23135/head:
mds: refactor ostream passing

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #22947 into master

* refs/pull/22947/head:
test: validate empty json output during recursive scrub
test: subclass test_scrub.Workload from CephFSTestCase
mds: dump minimal cli output during recursive scrub trigger

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23009 into master

* refs/pull/23009/head:
client: wire up ceph_ll_fallocate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23151 from neha-ojha/wip-25008

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Merge pull request #23123 from yuriw/wip-yuriw-24926-master

qa/tests: Set ansible-version: 2.5

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #22982 into master

* refs/pull/22982/head:
os/bluestore: Add new perfcounter which record latency of

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #23041 into master

* refs/pull/23041/head:
mon: manage paxos_services alloc with unique_ptr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23047 into master

* refs/pull/23047/head:
common: SubProcess needs SIGTERM, from the includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23053 into master

* refs/pull/23053/head:
mon: emplace mon_caps in Monitor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23060 into master

* refs/pull/23060/head:
test: Only inquire bluestore when it's build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23078 into master

* refs/pull/23078/head:
spdk: -march=core2 instead of -march=native

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr: generic self test command

Avoid need for each module to expose a self-test
command: they can just implement the method,
and then get it called via the selftest module.

As well as fewer LOC, this means that the self
test commands are not cluttering the interface
for end users, as they've invisible until
the selftest module is loaded.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: update mgr plugin page for remote calls

Signed-off-by: John Spray <john.spray@redhat.com>

mgr: enable inter-module calls

This is being done by passing native CPython objects
back and forth.  It's safe because sub-interpreters in CPython
share memory allocation infrastructure and share the GIL.

With a view to PEP554, we limit inter-interpreter calls
to pickleable objects, so that this may be implemented
using byte-arrays in future.

This infrastructure should enable:
 - the dashboard to display the status of other modules, for
   example the set of progress indicators from `progress`
 - dashboard and restful to share an underlying long running
   job mechanism.

Signed-off-by: John Spray <john.spray@redhat.com>

rgw: Robustly notify

If at first you don't succeed, try, try again.

Keep track of what you succeed with and what you don't.

If you succeeded with something on a previous try, don't worry if you
didn't get it this time.

After a reasonable number of tries, give up.

http://tracker.ceph.com/issues/24963

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #22919 from liewegas/wip-rgw-cls-test

qa/suites/rgw/verify/tasks/cls_rgw: test cls_rgw

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22975 from tone-zhang/vstart-spdk

vstart.sh: Support SPDK in Ceph development deployment

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22790 from joke-lee/WIP-admin-rest-api-user-info-with-access-key

rgw: support admin rest api get user info through user's access-key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23099 from adamemerson/wip-all-your-fault

rgw: All Your Fault

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #23058 from mkogan1/cont-19282-fix-rgw-multisite-datasync_v2

rgw: add curl_low_speed_limit and curl_low_speed_time config to avoid

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #23119 from cbodley/wip-rgw-link-tenant

rgw: RGWBucket::link supports tenant

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #23010 from noahdesu/mgr-expose-ec-profiles

mgr: expose ec profiles through manager

Reveiwed-by: John Spray <john.spray@redhat.com>

Merge pull request #23016 from rzarzynski/wip-osd-cleanup-log_op_stats

osd: clean up and avoid extra ref-counting in PrimaryLogPG::log_op_stats.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22993 from jcsp/wip-progress

mgr/progress: introduce the `progress` module

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

rgw: return valid Location element, PostObj

PostResponse should contain ETag and urlencode bucket name
object name and tenant name and correct Location uri

Fixes: http://tracker.ceph.com/issues/22927
Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

mds: avoid using g_conf().get_val<...>(...) in hot path

g_conf->get_val<>()'s overhead is high

Fixes: http://tracker.ceph.com/issues/24820
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

vstart.sh: Support SPDK in Ceph development deployment

Support SPDK in vstart.sh, the developer can debug SPDK within
development deployment with the option
"--bluestore-spdk <vendor>:<device>".

Signed-off-by: tone.zhang <tone.zhang@arm.com>

Merge pull request #23121 from noahdesu/mgr-crash-history

mgr/crash: add hour granularity crash summary

mds: refactor ostream passing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: deep-copy should not write to objects that cannot exist

If an image has a snapshot and is subsequently shrunk, ensure that
discard ops are not sent to the deep-copy objects beyond the bounds
of the image at a given snapshot/HEAD. Only a remove op should be
expected in such cases.

Fixes: http://tracker.ceph.com/issues/25000
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/crash: json report of recent crashes

reports crashes per entity type with hour granularity. primarily for
consumption by the insights module.

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

mgr/crash: add timestamp filter helper

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #23129 from alfredodeza/wip-rm24952

ceph-volume lvm.listing only include devices if they exist

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #23097 into master

* refs/pull/23097/head:
vstart_runner: pass ctx to FuseMount and load mount info

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #23077 from smithfarm/wip-etc-sudoers

build/ops: rpm: make sudo a build dependency

Reviewed-by: Tim Serong <tserong@suse.com>