Merge pull request #23237 from neha-ojha/wip-whitelist-slow-requests

qa/tasks/thrashosds-health.yaml: whitelist slow requests

Merge pull request #23141 from alfredodeza/wip-rm24993

ceph-volume process.call with stdin in Python 3 fix

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #22533 from linuxbox2/wip-rgw-24507

rgw: add option for relaxed region enforcement

qa/tasks/thrashosds-health.yaml: whitelist slow requests

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #23074 from tchaikov/wip-seastar-config

common/config: add ConfigProxy for crimson

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #23232 into master

* refs/pull/23232/head:
mon: don't require CEPHX_V2 from mons until nautilus

ceph-volume tests run to log_descriptors, verifying utf-8

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process decode to utf-8 when logging descriptors

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.process ensure that stdin works on py3 and py2.7

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume process encode stdin with utf-8 to prevent bytes TypeError

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mon: don't require CEPHX_V2 from mons until nautilus

The mimic 13.2.0 didn't have it.

Fixes: 3dc80e5f9b6ebf1bc1cecbd95b288005216bdbec
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #23231 from tchaikov/wip-fio

tests/fio: fix build failures and ensure this is covered by run-make-check.sh

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

cmake: sync up with upstream FIO

to pick up the change to FTBFS with C++ engine.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22695 from a2batic/scss

mgr/dashboard: Color variables for color codes

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23100 from adamemerson/wip-robustly-notify

rgw: Robustly notify

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #23183 from ricardoasmarques/wip-fix-url-prefix

mgr/dashboard: Fix cherrypy static content URL prefix config

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Color variables for color codes
Assign variables to colors present in css files
Changed navigation icons padding because the notification dropdown was going out of window.
Fixes: http://tracker.ceph.com/issues/24575
Signed-off-by: Kanika Murarka <kmurarka@redhat.com>

Merge pull request #22857 from tspmelo/wip-move-test-file

mgr/dashboard: Move unit-test-helper into the new testing folder

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #22624 from ashutosh-narkar/opa-ceph

rgw: Initial work for OPA-Ceph integration

Merge PR #23220 into master

* refs/pull/23220/head:
osd/OSDMap: CRUSH_TUNABLES5 added in jewel, not kraken

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Initial work for OPA-Ceph integration

Signed-off-by: Ashutosh Narkar <anarkar4387@gmail.com>

common/config_proxy: document Config{Values,Proxy}

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: update test_alien_echo.cc with f80b848d

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build c-ares if it's not found or not new enough

Signed-off-by: Kefu Chai <kchai@redhat.com>

c-ares: add c-ares submodule

c-ares is a C library for asynchronous DNS requests, seastar requires
c-ares 1.13.0 or up. but xenial and EPEL offer 1.10.0, so we need to
build and ship it as part of Ceph.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/common: add ceph::common::ConfigProxy

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: change md_config_obs_impl's template param to ConfigProxy

we were using LockPolicy as this tempalte class's template parameter,
but turns out ConfigProxy is a better parameter for it. as ConfigProxy is
"closer" to md_config_obs_impl than LockPolicy is.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: change md_config_impl<> back to plain class

now that all of the variables that change depending on seastar/alien
have been extracted out of md_config_impl<>, there is no need to make it
a template anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm,install-deps,cmake: install {c-ares,fmt}-devel

* ceph.spec.in: c-ares-devel and fmt-devel are installed as
  dependencies fo seastar.
* install-deps.sh: libfmt-dev is missing in xenial, so we only
  install libc-ares-dev for ubuntu in install-deps.sh .
* cmake: build libfmt if it's not found.

Signed-off-by: Kefu Chai <kchai@redhat.com>

fmt: add libfmt submodule

libfmt is a library for fast printf() support in C++, it is used by
seastar. but it's missing in xenial. so we need to ship it in our dist
tarball and build if if it's not found in system.

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: use the modernized cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

seastar: sync with upstream

to pick up the commit of 67bb775e, which marks
foreign_ptr::get_owner_shard() const.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: mark do_argv_commands() const

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: extract ObserverMgr from md_config_impl<>

crimson will have sharded<PGShard> (name may vary in future), and
underlying ObjectStore instances. they will be consumers of crimson
ConfigProxy, and will register them as an observer of config changes.
but as always, in crimson osd, world is sharded, each ConfigProxy will
only take care of the sharded services living on the shard which it is
in charge of. in other words, unlike alien's ConfigProxy, the
"observers" data structure is not shared, they are sharded as well. so
we need to extract the observers out of md_config_impl<>, and move it
into ConfigProxy.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSDMap: CRUSH_TUNABLES5 added in jewel, not kraken

In 0ceb5c0f3694725c7043ea99b077816da2ca44a7 I screwed up the version that
CRUSH_TUANBLES5 (chooseleaf_stable) was added: it comes from
043a7378ec4e828f2568ea3906d9076848161ccd, v10.0.3, which leads up to
jewel.

This, in turn, means that when jewel tunables are set we do not actually
encode the jewel tunables.  At least, luminous doesn't; jewel uses a
different code path to decode how to encode the OSDMap.

This, in turn, leads to crc errors:

- luminous mons running
- require_osd_release is jewel
- jewel tunables are set.  mon in-memory crushmap gets chooseleaf_stable=1
- osds are luminous, but don't see the new field in the crushmap
- upgrade finishes, and require_osd_Release becomes luminous
- the full osdmap encodes the chooseleaf_stable field, finally, and it
  is 1
- the osds try to encode the same thing, but their crushmap has 0.
-> crc mismatch

This is a regression from 0ceb5c0f3694725c7043ea99b077816da2ca44a7,
backported to luminous in 686b054f48c50859bd5e8bad93007bf9d7d7899c in
12.2.6.

Fixes: 686b054f48c50859bd5e8bad93007bf9d7d7899c
Fixes: http://tracker.ceph.com/issues/25057
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: prettify code

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Move unit-test-helper into the new testing folder

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22996 from batrick/mds-state-dot

doc: add mds state transition diagram

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22581 from fullerdj/wip-djf-24544

rgw: change default rgw_thread_pool_size to 512

mds: remove ascii art version of mds state diagram

Now refer to the user documentation. Any updates should go there!

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: create mds state documentation

Fixes: http://tracker.ceph.com/issues/22989
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20330 from joke-lee/wip-rgw-postobjloc

rgw: return valid Location element, PostObj

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #22937 from tianshan/fix_index_suggest_ut_fail

rgw: continue enoent index in dir_suggest

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #23095 from trociny/wip-24898

 librbd: ensure exclusive lock acquired when removing sync point snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #23167 from tchaikov/wip-tools-rbd-cleanup

tools/rbd: assert(g_ceph_context) not g_conf

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22419 from votdev/hide_progress_bar

mgr/dashboard: Hide progress bar in case of an error

Merge pull request #23182 from alfredodeza/wip-ansible-26

ceph-volume tests/functional use Ansible 2.6

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23132 from dillaman/wip-25000

librbd: deep-copy should not write to objects that cannot exist

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22792 from jcsp/wip-mgrmodule-none-docs

mgr: improve docs for MgrModule methods

Merge pull request #22794 from wido/mgr-metadata-none

mgr: Ignore daemon if no metadata was returned

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22827 from theanalyst/wip-device-py3

mgr: devicehealth: dont error on dict iteritems

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #22951 from jcsp/wip-mgr-intermodule

mgr: enable inter-module calls

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #23181 from tspmelo/wip-remove-karma

mgr/dashboard: Remove karma packages

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #23115 from tspmelo/wip-angular-cmake

mgr/dashboard: Remove angular build progress logs during cmake

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #23159 from smithfarm/wip-example-no-boost

example/librados: remove dependency on Boost system library

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common/config: extract "changed" from md_config_impl<> to ConfigValues

for the same reason why we moved ObserverMgr out of md_config_impl<>,
"changed" cannot be shared among shards, we should move "changed" out of
md_config_impl. we can put "changed" into ConfigValues or
ConfigProxy, IMHO, it would be simpler if "changed" resides in
ConfigValues: less fragmented this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: move _apply_changes() calls up into ConfigProxy

see also the previous the commit, to reuse injectargs() and
set_mon_vals() in seastar's ConfigProxy, we need to move apply_changes()
call out of them, as this call is implemented differently in seastar's
ConfigProxy.

finalize_reexpand_meta() is not used by OSD at this moment, so we leave
it as it is for now.

* move _apply_changes() calls up into ConfigProxy
* move update_legacy_vals() out of _apply_changes() into the
callers of the latter. as it changes "values", in seastar's ConfigProxy,
it belongs to the first step of changing a config value

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: promote lock from md_config_t to ConfigProxy

seastar's ConfigProxy and alien's ConfigProxy follow different threading
models and expose different methods. the former updates a setting with 3
steps:
1. create a local copy of current setting, and apply the proposed change
   to the copy
2. populate the updated change with a foreign_ptr<> to all shards
   (including itself)
3. on each shards, call apply_changes() to get the interested observers
   updated, please note, apply_changes() should only update the local
   observers on current shard.

while the alien's ConfigProxy do all the job in a single synchronized
call,
but we can split it into a finer-grained steps:
1. apply the proposed change in-place
2. apply_changes() to get the interested observers updated.

so, to reuse the code across these two implementations, for instance,
set_mon_vals() will be implemented in ConfigProxy instead, so we can
have different behavior in different ConfigProxy classes. if we keep
using the existing single-piece md_config_t::set_mon_vals(), we have no
chance to differentiate the apply_changes() for seastar port. but the
alien implementation requires a grand lock protecting set_val() and
apply_changes(), so we have to move the lock from md_config_t up to
ConfigProxy. it's also simpler this way, as we don't need an extra layer
to have a dummy Mutex for seastar's ConfigProxy. as only the alien's
ConfigProxy requires the lock.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/config: do not use unique_ptr<> for config

it's not necessary to do so.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22978 from tchaikov/wip-docker-test-cleanup

test/docker-test-helper.sh: move "cp .git/HEAD" out of loop

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #23194 from alfredodeza/wip-lvm-docs

ceph-volume: expand on why ceph-disk was replaced

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #23029 from dillaman/wip-24909

osd: do not treat an IO hint as an IOP for PG stats

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #23186 from smithfarm/wip-4640

doc/man: mention import and export commands in rados manpage

doc/mgr: add docs for missing functions

Some functions had docstrings but were
not linked into plugins.rst, some needed
more text added.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/ceph-volume expand on the ceph-disk replacement reasons

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/ceph-volume reference the new ceph-disk-replacement section

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests/functional use testinfra latest for ansible 2.6

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc/man: mention import and export commands in rados manpage

Fixes: http://tracker.ceph.com/issues/4640
Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Fix cherrypy static content URL prefix config

Fixes: https://tracker.ceph.com/issues/25067
Signed-off-by: Ricardo Marques <rimarques@suse.com>

ceph-volume tests/functional use Ansible 2.6

The ceph-ansible project now refuses to use 2.4 even though it works
with that version

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: Remove karma packages

Since we are no longer using karma for the frontend unit tests, we will remove
all the related packages and configurations.

This includes the removal of the phantomjs package.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #23114 from smithfarm/wip-24800

doc: use :command: for subcommands in ceph-bluestore-tool manpage

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Remove angular build progress logs during cmake

Currently Angular CLI is polluting the jenkins logs, by inserting a new line
each time the build process is updated.
With this change Angular CLI will only output the necessary information about
the start and end of the build.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Hide progress bar in case of an error

- Enhance datatable to handle loading errors.
- Add warning panel component
- Modify the icon of the error panel component
- Use cd-[info|warning|error]-panel in the cd-view-cache component

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #23166 from tchaikov/wip-cmake-cleanups

cmake: cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #22875 from smithfarm/wip-backport-script

script: new ceph-backport.sh script

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20401 from smithfarm/wip-22558

build/ops: rpm: provide files moved from ceph-test …

Reviewed-by: Kefu Chai <kchai@redhat.com>

tools/rbd: assert(g_ceph_context) not g_conf

see also 813cd5d5910c266c076d38277eddf32d908d5ae3

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract ceph-dencoder related stuff out

let's modularize the cmake file piecemeal.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Finddpdk: add dpdk::${component} libs

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23160 from smithfarm/wip-24017

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Reviewed-by: Sage Weil <sage@redhat.com>

example/librados: remove dependency on Boost system library

109e6022beb0920f2a4746bd8c541e975494f251 introduced "-lboost_system" into
example/librados/Makefile but the Boost system library is no longer required
to compile and link hello_world.cc.

Fixes: http://tracker.ceph.com/issues/25054
Signed-off-by: Nathan Cutler <ncutler@suse.com>

tests: upgrade/luminous-x: whitelist REQUEST_SLOW for rados_mon_thrash

Fixes: http://tracker.ceph.com/issues/25051
Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: rpm: provide files moved from ceph-test

d7b493a7108a68302bc0f48337bf3c253a720266 moved three binaries from the
ceph-test package to other packages, causing trouble with upgrades in
test lab environments (users typically do not install the ceph-test
package).

While the original upgrade troubles have already been addressed, RPM
has a "Provides" mechanism for easing the pain, so it makes sense to
use it.

Fixes: http://tracker.ceph.com/issues/22558
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #23139 into master

* refs/pull/23139/head:
mds: avoid using g_conf().get_val<...>(...) in hot path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22977 into master

* refs/pull/22977/head:
qa/tasks/cephfs: add test for discontinuous mdsmap
mds: handle discontinuous mdsmap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23022 into master

* refs/pull/23022/head:
mds: health warning for slow metadata IO

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #23063 into master

* refs/pull/23063/head:
mds: increase debug level for dropped client cap msg

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23062 into master

* refs/pull/23062/head:
mds: dump recent events on respawn

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23061 into master

* refs/pull/23061/head:
mds: print mdsmap processed at low debug level

Reviewed-by: John Spray <john.spray@redhat.com>

Merge PR #23135 into master

* refs/pull/23135/head:
mds: refactor ostream passing

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #22947 into master

* refs/pull/22947/head:
test: validate empty json output during recursive scrub
test: subclass test_scrub.Workload from CephFSTestCase
mds: dump minimal cli output during recursive scrub trigger

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #23009 into master

* refs/pull/23009/head:
client: wire up ceph_ll_fallocate

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #23151 from neha-ojha/wip-25008

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Merge pull request #23123 from yuriw/wip-yuriw-24926-master

qa/tests: Set ansible-version: 2.5

qa/suites/powercycle: whitelist MDS_SLOW_REQUEST

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #22982 into master

* refs/pull/22982/head:
os/bluestore: Add new perfcounter which record latency of

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #23041 into master

* refs/pull/23041/head:
mon: manage paxos_services alloc with unique_ptr

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #23047 into master

* refs/pull/23047/head:
common: SubProcess needs SIGTERM, from the includes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #23053 into master

* refs/pull/23053/head:
mon: emplace mon_caps in Monitor

Reviewed-by: Kefu Chai <kchai@redhat.com>