test: Sort pool list because the order isn't guaranteed from "balancer pool ls"

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Change default upmap_max_deviation to 5

Fixes: https://tracker.ceph.com/issues/43312
Signed-off-by: David Zafman <dzafman@redhat.com>

osdmaptool: Add --upmap-active to simulate active upmap balancing

Signed-off-by: David Zafman <dzafman@redhat.com>

doc: Add upmap options to osdmaptool man page and give example

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: osdmaptool document non-upmap options that were missing

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Fix test case for pool based balancing instead of rule batched

Signed-off-by: David Zafman <dzafman@redhat.com>

Revert "tools: osdmaptool sync with balancer module behavior"

This was the rules batching version of osdmaptool

This reverts commit f165d4ca39b1edef4bf22a7597af79339c95026f.

Signed-off-by: David Zafman <dzafman@redhat.com>

Partially revert "mgr/balancer: balance pools with same crush_rule in batch"

This partially reverts commit 3a730d751deff892e7a0bddba87eba3dbb829c3e

Fixes: https://tracker.ceph.com/issues/43307
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #32358 from idryomov/wip-krbd-py3-fixup

qa: krbd_exclusive_option.sh: fixup for json.tool ordering change

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32498 into master

* refs/pull/32498/head:
doc: mounting CephFS subdirectory and Persistent Mounts cleanup

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32480 into master

* refs/pull/32480/head:
qa: use -D_GNU_SOURCE when compiling fsync-tester.c
qa: add comment about fsync-tester distribution

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29995 into master

* refs/pull/29995/head:
mds: add config to require forward to auth MDS

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31494 into master

* refs/pull/31494/head:
qa: force creation of fs with EC default data pool
qa: add tests for adding EC data pools
mon/MDSMonitor: warn when creating fs with default EC data pool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32015 into master

* refs/pull/32015/head:
doc/cephfs/standby: document mds_join_fs
mgr/cephadm: map mds daemons to a particular fs
mon/MDSMonitor: respect mfs fscid preference
mon/MDSMonitor: assign standbys to their preferred fscid
mds/FSMap: track preferred fscid for standby daemons
mds: add mds_join_fs option; pass via MMDSBeacon

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32268 into master

* refs/pull/32268/head:
mds: only enable batch getattr/lookup for non-snap request.
mds: pin dentry when attaching batch lookup to it

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32307 into master

* refs/pull/32307/head:
mon: print FSMap if standbys exist

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32308 into master

* refs/pull/32308/head:
mon/MDSMonitor: cleanup check_subs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32467 into master

* refs/pull/32467/head:
qa: fix various py3 cephfs qa bugs

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32164 into master

* refs/pull/32164/head:
mount.ceph: give a hint message when no mds is up or cluster is laggy

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #32363 into master

* refs/pull/32363/head:
qa: add .qa link

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32426 from tchaikov/wip-qa-test_envlibrados_for_rocksdb.sh

qa: enable CRB repo for RHEL8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa: enable CRB repo for RHEL8

instead of enabling PowerTools repo, we need to enable CodeReady Builder
repo for RHEL8

also, since we are moving to RHEL8, there is no need to install cmake3
specifically for CentOS. CentOS 8 comes with cmake3.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32242 from rishabh-d-dave/cv-methods-for-lvm-opt-S

ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

qa: krbd_exclusive_option.sh: fixup for json.tool ordering change

In Python 3.5 json.tool was changed to produce unsorted output and
--sort-keys option was added to compensate.  This wasn't caught by
4fe245cc2f2d ("qa: update krbd tests for python3") because it raced
with 50933b863a1d ("qa: krbd_exclusive_option.sh: update for recent
kernel changes").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

Filters can be passed to these commands by using option '-S'.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

doc: mounting CephFS subdirectory and Persistent Mounts cleanup

Fixes: https://tracker.ceph.com/issues/37746
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #32485 from tchaikov/wip-clang

common,crimson: fixes to compile with clang and libc++

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31728 from luis-henrix/copy-from-notrunc

osd: add a 'copy-from2' operation that includes truncate_{seq,size} parameters

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #32488 from ronen-fr/clang_3

crimson: CLANG-related fixes to errorator.h

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32490 from rzarzynski/wip-crimson-missing_err_include

crimson: add missing include in common/errorator.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: add missing include in common/errorator.h

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: fix non-standard C++ constructs in errorator.h

Fixing use patterns accepted by gcc, but frowned upon by Clang:

Put noexcept specifier after attributes specifiers (as
per the C++ standard $11.3.5)

Limit the use of a not-completely-defined class names to
within function bodies, per the standard ("A class is considered
a completely-defined object type (or complete type) at the
closing } of the class-specifier.
Within the class member-specification, the class is regarded as
complete within function bodies, default arguments, noexcept-specifiers,
and default member initializers (including such things in nested classes).
Otherwise it is regarded as incomplete within its own class member-specification.")

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson: fix FTBFS on Clang induced by errorator.

Reported-by Ronen Friedman <rfriedma@redhat.com>
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: add missing #include

clang++-9.0.1 and libc++ require them to build

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not use auto in param list of function prototype

src/crimson/net/Socket.cc:114:57: error: 'auto' not allowed in function
prototype
static inline seastar::future<> close_and_handle_errors(auto& out) {
                                                        ^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not capture unused variables

to silence warnings like

src/crimson/net/Socket.cc:93:41: warning: lambda capture 'this' is not
used [-Wunused-lambda-capture]
    return in.read_exactly(bytes).then([this](auto buf) {
                                        ^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32474 from kalebskeithley/fedora32

do_cmake.sh: fedora-32 (rawhide) build with python-3.8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32481 from tchaikov/wip=admin-build-doc-python3

admin/build-doc: always install python3-* for build deps

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

admin/build-doc: always install python3-* for build deps

since we've dropped the support of python36-*, i.e. python packages
provided by EPEL7 before RHEL7/CentOS7 included python3. as before the
inclusion of python3 as supported python3, python packages are named
python36-*. and they don't provide python3-*. so we had to install
python36-* explicitly. now that we are able to use the python3-*
packages, we can just install python3-*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: fix various py3 cephfs qa bugs

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use -D_GNU_SOURCE when compiling fsync-tester.c

Otherwise loff_t is unavailable for some distributions.

Fixes: https://tracker.ceph.com/issues/43460
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add comment about fsync-tester distribution

I was initially confused why this would not be included in the QA suite.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32451 from rs-fabrica/doc_rados_operations_crush-map-edits

doc/rados/operations/crush-map-edits: recompile and set instructions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32442 into master

* refs/pull/32442/head:
doc/developer: fix dev mailing list address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/developer: fix dev mailing list address

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

do_cmake.sh: fedora-32 (rawhide) build with python-3.8

Fedora-32 now has python-3.8.

Fedora-29 is EOL. All non-EOL versions of Fedora have python-3.7, except
Fedora-32

(Unclear to me why PYBUILD="3" isn't used, like Ubuntu and FreeBSD. That
would eliminate breakage when pyhon versions change in Rawhide.)

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge PR #32457 into master

* refs/pull/32457/head:
doc: fix Getting Started with CephFS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31309 from matthewoliver/rgw_swift_auth_note

doc: Add a RGW swift auth note

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31432 from zhangsw/rgw-cleanup-cls-reshard-const

rgw:cleanup the "magic string" usage in cls_rgw_client.cc

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #31429 from zhangsw/rgw-cleanup-cls-guard-bucket-resharding

rgw: correct some error log about reshard in cls_rgw.cc.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31404 from zhangsw/rgw-cleanup-deletelc

rgw:cleanup:remove un-used class member in RGWDeleteLC

Merge pull request #31345 from zhangsw/rgw-cleanup-rgw-admin-check-instance

rgw:cleanup:remove un-used create_new_bucket_instance in rgw_admin.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32351 from mykaul/osd_fsid

ceph-volume/lvm/activate.py: clarify error message: fsid refers to osd_fsid

Merge pull request #27792 from smithfarm/wip-make-dist

make-dist: drop Python 2/3 autoselect

Reviewed-by: Kefu Chai <kchai@redhat.com>

lvm/activate.py: clarify error message: fsid refers to osd_fsid

Users complained[1] the error message isn't clear, and they thought
it referred to the cluster fsid instead of the osd_fsid.
Made it clearer.

[1] rook/rook#4547

Fixes: https://tracker.ceph.com/issues/43442
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

doc: fix Getting Started with CephFS

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: Add a RGW swift auth note

Swift accounts are not like normal accounts, they are more akin to a
bank account that multile people could share. Or in the case of a cloud
it is usually mapped to the tenant.

Radosgw deals with this with a user and subuser, which is great, but a
little confusing. So this patch adds a note to those used to the Swift
API to make it more clear.

Signed-off-by: Matthew Oliver <moliver@suse.com>

doc/rados/operations/crush-map-edits: including recompile and set crushmapo instructions

Including missing recompile and set crushmap sections in crush-map-edits

Signed-off-by: Rodrigo Severo <rodrigo@fabricadeideias.com>

Merge pull request #32433 from tchaikov/wip-43414

rgw: svc.bucket: assign to optional<> using "="

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #32415 into master

* refs/pull/32415/head:
include/cpp-btree: use inline variable to define btree_is_key_compare_to
include/cpp-btree: use std::conditional_t<>
include/cpp-btree: use static_assert()
include/interval_set: use interval_set when appropriate
include/interval_set: rename some types
include/cpp-btree: add emplace_hint()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32399 from dillaman/wip-41556-2

rbd-mirror: continue to isolate journal replay logic

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32427 from tchaikov/qa-devices-schema-with-path

qa: add "path" to "device" output schema

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32430 from tchaikov/wip-rgw-rgw_log_entry

rgw: initialize member variables of rgw_log_entry

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32438 from tchaikov/wip-vstart-runner-py3

qa/tasks/vstart_runner: write string to StringIO

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/vstart_runner: write string to StringIO

we use `six.StringIO` for stdout when writing output messages from
`self.subproc.communicate()`, and `six.StringIO` accepts strings, so we
need to decode the output before sending them to `six.StringIO`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: drop Python 2/3 autoselect

virtualenv will prefer Python 3 over Python 2 if both are available.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: svc.bucket: assign to optional<> using "="

instead of using

*optional_val = value;

we should use:

optional_val = value;

otherwise we will have following runtime assertion failure like

/usr/include/c++/8/optional:945: constexpr _Tp& std::_Optional_base<_Tp, true, true>::_M_get() [with _Tp = RGWBucketInfo*]: Assertion 'this->_M_is_engaged()' failed.

when compiled with the `_GLIBCXX_ASSERTIONS` macro defined. and recent
`redhat-rpm-config` adds `-D_GLIBCXX_ASSERTIONS` to
`__global_compiler_flags`, which is in turn included by `optflags`. and
`optflags` is pratically used everywhere.

Fixes: https://tracker.ceph.com/issues/43414
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: svc.bucket: move _orig_info out of "if" block

so `*orig_info` is valid out of the `if` block if it's set to
`&_orig_info`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: add "path" to "device" output schema

"path" was added by 2c0fd7d86827aa76b8d923870018365cdae4a6ad, so update
the test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: set proper state after journal replay shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: initialize member variables of rgw_log_entry

`check-generated.sh` tests the dencoder of a class by:

ceph-dencoder type rgw_log_entry select_test 2 dump_json > /tmp/foo
ceph-dencoder type rgw_log_entry select_test 2 encode decode dump_json > /tmp/bar
cmp /tmp/foo /tmp/bar

but the second test instance created by
`rgw_log_entry::generate_test_instances` is created using the default
constructor of `rgw_log_entry`. if any of the member variable is created
using random bits in the heap, there is chance that the test fails like:

2 rgw_log_entry
/tmp/typ-l5omDhmDf /tmp/typ-4aoRVkoiV differ: char 315, line 15
**** rgw_log_entry test 2 dump_json check failed ****
   ceph-dencoder type rgw_log_entry select_test 2 dump_json > /tmp/typ-l5omDhmDf
   ceph-dencoder type rgw_log_entry select_test 2 encode decode dump_json > /tmp/typ-4aoRVkoiV
15c15
<     "obj_size": 809908640,
---
>     "obj_size": 1122974112,

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: do not log error upon bootstrap cancel

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32150 from ricardoasmarques/set-osd-device-class

mgr/dashboard: Explicitly set/change the device class of an OSD

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #32396 from toliu/restful-api-handle-execv

mgr: close restful socket after exec

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32406 from neha-ojha/wip-43364

mon/PGMap.h: disable network stats in dump_osd_stats

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31722 from tchaikov/wip-mon-deprecated-tell

mon: make "compact" tell command, and add deprecate/obsolete check for tell commands

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31829 from zjcmszh/xattrs

pybind: add WriteOp::set_xattr() & rm_xattr()

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #32420 from pkulijiawei/issues-43417

common: skip interfaces starting with "lo" in find_ipv{4,6}_in_subnet()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32421 from tchaikov/wip-43416

mgr/restful: jsonify lists instead of maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32424 into master

* refs/pull/32424/head:
mgr/balancer: sort pool names in 'balancer ls' output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32340 into master

* refs/pull/32340/head:
cephadm: support deployment of node-exporter

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32417 into master

* refs/pull/32417/head:
mgr/test_orchestrator: fix update_mgrs assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/test_orchestrator: fix update_mgrs assert

Signed-off-by: Sage Weil <sage@redhat.com>

common: skip interfaces starting with "lo" in find_ipv{4,6}_in_subnet()

This will solve the issue that the osd can't restart after seting a
virtual local loopback IP.
In find_ipv4_in_subnet() and find_ipv6_in_subnet(), I use
boost::starts_with(addrs->ifa_name, "lo") to ship the interfaces
starting with "lo".

Fixes: https://tracker.ceph.com/issues/43417
Signed-off-by: Jiawei Li <lijiawei1@chinatelecom.cn>

mgr/balancer: sort pool names in 'balancer ls' output

Otherwise this will vary based on the internal key sort order the dict it
was generated from.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: support deployment of node-exporter

Patch to support cephadm deploying node-exporter
containers. The node-exporter daemon has no
configuration and just needs access to the hosts
filesystem to pull the necessary system metrics.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #32405 into master

* refs/pull/32405/head:
qa/workunits/rados/test_large_omap_detection: py3-ify

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32403 from dillaman/wip-42248

librbd: diff iterate with fast-diff now correctly includes parent

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32402 from dillaman/wip-43401

librbd: bump minor version to match octopus

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32401 from dillaman/wip-43378

librbd: remove pool objects when removing a namespace

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/restful: jsonify lists instead of maps

in python3, `map()` returns a `map` instance which is iterable, while in
python2, `map()` returns a `list`, but `JSONEncoder.default()` expects a
list or whatever which inherits from `json.JSONEncoder`.

in this change, we just return a materized list instead of a map for
json encoder.

Fixes: https://tracker.ceph.com/issues/43416
Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: use inline variable to define btree_is_key_compare_to

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: use std::conditional_t<>

instead of using a homebrew replacement

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: use static_assert()

instead of using a homebrew replacement

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: use interval_set when appropriate

it's an alias of `interval_set<T,Map>` under some circumstances

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: rename some types

to be consistent with STL conventions

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: add emplace_hint()

so we can construct the value to be inserted in-place, and move it into
the tree, instead of copying it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32409 from dillaman/wip-qa-rbd-python3

qa/workunits/rbd: fixed python interpreter for EL8

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: fix race between replay shutdown and notification

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32232 into master

* refs/pull/32232/head:
qa: no need to exclude ceph-mgr-diskprediction-cloud from package list to be installed
qa/packages: do not install ceph-mgr-diskprediction-cloud by default
ceph.spec.in: add runtime deps for mgr-diskprediction-cloud

Reviewed-by: Sage Weil <sage@redhat.com>