Merge pull request #32433 from tchaikov/wip-43414

rgw: svc.bucket: assign to optional<> using "="

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #32415 into master

* refs/pull/32415/head:
include/cpp-btree: use inline variable to define btree_is_key_compare_to
include/cpp-btree: use std::conditional_t<>
include/cpp-btree: use static_assert()
include/interval_set: use interval_set when appropriate
include/interval_set: rename some types
include/cpp-btree: add emplace_hint()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32399 from dillaman/wip-41556-2

rbd-mirror: continue to isolate journal replay logic

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32427 from tchaikov/qa-devices-schema-with-path

qa: add "path" to "device" output schema

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32430 from tchaikov/wip-rgw-rgw_log_entry

rgw: initialize member variables of rgw_log_entry

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32438 from tchaikov/wip-vstart-runner-py3

qa/tasks/vstart_runner: write string to StringIO

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/vstart_runner: write string to StringIO

we use `six.StringIO` for stdout when writing output messages from
`self.subproc.communicate()`, and `six.StringIO` accepts strings, so we
need to decode the output before sending them to `six.StringIO`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: svc.bucket: assign to optional<> using "="

instead of using

*optional_val = value;

we should use:

optional_val = value;

otherwise we will have following runtime assertion failure like

/usr/include/c++/8/optional:945: constexpr _Tp& std::_Optional_base<_Tp, true, true>::_M_get() [with _Tp = RGWBucketInfo*]: Assertion 'this->_M_is_engaged()' failed.

when compiled with the `_GLIBCXX_ASSERTIONS` macro defined. and recent
`redhat-rpm-config` adds `-D_GLIBCXX_ASSERTIONS` to
`__global_compiler_flags`, which is in turn included by `optflags`. and
`optflags` is pratically used everywhere.

Fixes: https://tracker.ceph.com/issues/43414
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: svc.bucket: move _orig_info out of "if" block

so `*orig_info` is valid out of the `if` block if it's set to
`&_orig_info`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: add "path" to "device" output schema

"path" was added by 2c0fd7d86827aa76b8d923870018365cdae4a6ad, so update
the test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: set proper state after journal replay shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: initialize member variables of rgw_log_entry

`check-generated.sh` tests the dencoder of a class by:

ceph-dencoder type rgw_log_entry select_test 2 dump_json > /tmp/foo
ceph-dencoder type rgw_log_entry select_test 2 encode decode dump_json > /tmp/bar
cmp /tmp/foo /tmp/bar

but the second test instance created by
`rgw_log_entry::generate_test_instances` is created using the default
constructor of `rgw_log_entry`. if any of the member variable is created
using random bits in the heap, there is chance that the test fails like:

2 rgw_log_entry
/tmp/typ-l5omDhmDf /tmp/typ-4aoRVkoiV differ: char 315, line 15
**** rgw_log_entry test 2 dump_json check failed ****
   ceph-dencoder type rgw_log_entry select_test 2 dump_json > /tmp/typ-l5omDhmDf
   ceph-dencoder type rgw_log_entry select_test 2 encode decode dump_json > /tmp/typ-4aoRVkoiV
15c15
<     "obj_size": 809908640,
---
>     "obj_size": 1122974112,

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: do not log error upon bootstrap cancel

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32150 from ricardoasmarques/set-osd-device-class

mgr/dashboard: Explicitly set/change the device class of an OSD

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #32396 from toliu/restful-api-handle-execv

mgr: close restful socket after exec

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32406 from neha-ojha/wip-43364

mon/PGMap.h: disable network stats in dump_osd_stats

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31722 from tchaikov/wip-mon-deprecated-tell

mon: make "compact" tell command, and add deprecate/obsolete check for tell commands

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31829 from zjcmszh/xattrs

pybind: add WriteOp::set_xattr() & rm_xattr()

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #32420 from pkulijiawei/issues-43417

common: skip interfaces starting with "lo" in find_ipv{4,6}_in_subnet()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32421 from tchaikov/wip-43416

mgr/restful: jsonify lists instead of maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32424 into master

* refs/pull/32424/head:
mgr/balancer: sort pool names in 'balancer ls' output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32340 into master

* refs/pull/32340/head:
cephadm: support deployment of node-exporter

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32417 into master

* refs/pull/32417/head:
mgr/test_orchestrator: fix update_mgrs assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/test_orchestrator: fix update_mgrs assert

Signed-off-by: Sage Weil <sage@redhat.com>

common: skip interfaces starting with "lo" in find_ipv{4,6}_in_subnet()

This will solve the issue that the osd can't restart after seting a
virtual local loopback IP.
In find_ipv4_in_subnet() and find_ipv6_in_subnet(), I use
boost::starts_with(addrs->ifa_name, "lo") to ship the interfaces
starting with "lo".

Fixes: https://tracker.ceph.com/issues/43417
Signed-off-by: Jiawei Li <lijiawei1@chinatelecom.cn>

mgr/balancer: sort pool names in 'balancer ls' output

Otherwise this will vary based on the internal key sort order the dict it
was generated from.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: support deployment of node-exporter

Patch to support cephadm deploying node-exporter
containers. The node-exporter daemon has no
configuration and just needs access to the hosts
filesystem to pull the necessary system metrics.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #32405 into master

* refs/pull/32405/head:
qa/workunits/rados/test_large_omap_detection: py3-ify

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32403 from dillaman/wip-42248

librbd: diff iterate with fast-diff now correctly includes parent

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32402 from dillaman/wip-43401

librbd: bump minor version to match octopus

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32401 from dillaman/wip-43378

librbd: remove pool objects when removing a namespace

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/restful: jsonify lists instead of maps

in python3, `map()` returns a `map` instance which is iterable, while in
python2, `map()` returns a `list`, but `JSONEncoder.default()` expects a
list or whatever which inherits from `json.JSONEncoder`.

in this change, we just return a materized list instead of a map for
json encoder.

Fixes: https://tracker.ceph.com/issues/43416
Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: use inline variable to define btree_is_key_compare_to

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: use std::conditional_t<>

instead of using a homebrew replacement

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: use static_assert()

instead of using a homebrew replacement

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: use interval_set when appropriate

it's an alias of `interval_set<T,Map>` under some circumstances

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: rename some types

to be consistent with STL conventions

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: add emplace_hint()

so we can construct the value to be inserted in-place, and move it into
the tree, instead of copying it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32409 from dillaman/wip-qa-rbd-python3

qa/workunits/rbd: fixed python interpreter for EL8

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: fix race between replay shutdown and notification

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32232 into master

* refs/pull/32232/head:
qa: no need to exclude ceph-mgr-diskprediction-cloud from package list to be installed
qa/packages: do not install ceph-mgr-diskprediction-cloud by default
ceph.spec.in: add runtime deps for mgr-diskprediction-cloud

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32341 from dillaman/wip-43379

rbd-mirror: do not auto-create peers in non-default namespaces

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32202 from xiexingguo/wip-pg-stuck-undersized

osd/PeeringState: transit async_recovery_targets back into acting before backfilling

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32408 from dillaman/wip-devstack

qa/suites/rbd: fixed download path for Ubuntu Bionic

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32305 from neha-ojha/wip-43308

osd/PeeringState.cc: don't let num_objects become negative

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32354 from rzarzynski/wip-log-id2name

log: print pthread ID / name mapping in recent events dump.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32364 from rzarzynski/wip-bug-42780

common: fix deadlocky inflight op visiting in OpTracker.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32316 from bk201/wip-43363

mgr/cephadm: fix type of timeout options

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32270 from sebastian-philipp/orchestrator-completion-already-finished

mgr/orchestrator: Raise more expressive Error, if completion already …

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32185 from batrick/interval-set-c++17

include: fix interval_set const_iterator call operator type

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30344 from shenhang/jsoninfo

osd/OSDMap.cc: add more info in json output of osd stat

Reviewed-by: Коренберг Марк <socketpair@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32183 from aclamk/fix-getxattr-leak

librados: fix leak in getxattr and getxattrs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32265 from simon-rock/simon_work_adv

tool: ceph_monstore_tool: --readable=0 => --readable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32222 from toabctl/qa-flake8-py3

qa: Run flake8 on python2 and python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32312 from badone/wip-nature-theme-changes

doc: Modify nature theme

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32366 from liu-chunmei/vstart

vstart.sh: move extra_seastar_args up in vstart.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32391 from tchaikov/wip-boost-1.72

install-deps.sh, cmake: use boost 1.72 on bionic

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32407 into master

* refs/pull/32407/head:
qa/suites/rados: move cephadm_orchestrator to el8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32392 into master

* refs/pull/32392/head:
cephadm: fix v1/v2 ip/addrv handling; explicitly check bind to ip:port

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32377 into master

* refs/pull/32377/head:
qa/suites/rados/thrash-old-clients: configure mons in terms of addrvecs
qa/suites/rados/thrash-old-clients: hammer: fix package list
qa/tasks/cephadm: set .conf to cluster config object
qa/tasks/cephadm: archive /var/log/ceph logs too (not just cluster dir)
qa/tasks/cephadm: client keyring
qa/tasks/cephadm: setup thrashers ctx item
qa/tasks/ceph_manager: asok commands via cephadm shell
qa/suites/rados/thrash-old-clients: stick to el7
qa/tasks/cephadm: check cluster log; support log-whitelist
qa/suites/rados/thrash-old-clienets: python-foo to python3-foo
qa/suites/rados/thrash-old-clients: add new exclude_packages
qa/suites/rados/thrash-old-clients: use cephadm
mon/ConfigMonitor: make legacy mon addr/port parseable by legacy code

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/thrash-old-clients: configure mons in terms of addrvecs

This is more explicit.  More importantly, the 'mon update' command
can't handle an "ip:port"; it wants either a CIDR, bare IP, or addrvec.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: hammer: fix package list

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: set .conf to cluster config object

ceph.py does this and ceph_manager.py kill_osd relies on it.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: archive /var/log/ceph logs too (not just cluster dir)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: client keyring

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32390 into master

* refs/pull/32390/head:
cephadm: create /var/run/ceph/$fsid as needed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32281 into master

* refs/pull/32281/head:
mgr/cephadm: fix =[] default arg to _create_daemon
qa/suites/rados/cephadm: enable mgr debugging
cephadm: restart non-ceph daemons on reconfig
mgr/cephadm/test_cephadm: fix expected output
mgr/cephadm: fix remapping of hosts and services
mgr/cephadm: add 'reconfig' service action
cephadm: move mon creation config bits into mgr/cephadm
cephadm: fix bootstrap message 'ssh' -> 'cephadm'
cephadm: add --reconfig option for 'deploy'
mgr/orchestrator: drop 'reload' service[-instance] action

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32338 from trociny/wip-peer-list-default-ns

librbd: look for mirror peers in default namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rados: move cephadm_orchestrator to el8

The python3-remoto dependency does not exist on 18.04 (or any ubuntu or
debian AFAICS).

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd: fixed python interpreter for EL8

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: fixed download path for Ubuntu Bionic

The OS image was changed in a9ee4bcf241 from Xenial to Bionic,
but the Bionic image path is incorrect.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/tasks/cephadm: setup thrashers ctx item

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph_manager: asok commands via cephadm shell

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_large_omap_detection: py3-ify

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap.h: disable network stats in dump_osd_stats

Signed-off-by: Neha Ojha <nojha@redhat.com>

make-dist: package boost v1.72 instead of v1.67

to be consistent with the version used by "make check"

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: build boost v1.72 instead of v1.67

so we can drop two patches which are already included by v1.72

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not use small_vector

if `std::is_nothrow_move_constructible<small_vector<peer_shard_t, 2>>`
also depends on if the allocator throws when moving from a small_vector
to a new one, as we need to allocate memory for the destination
small_vector.

so, in this change, the optimization for small_vector is dropped,
because the default allocator used by small_vector throws.

Signed-off-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: use "apt" for checking install status

as "dpkg -s <package-name>" always returns 0 even if the <package> is
removed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,rgw: workaround for boost 1.72

see also https://github.com/boostorg/coroutine/issues/46

will need to remove this change once we require boost >= 1.73

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: diff iterate with fast-diff now correctly includes parent

When whole-object and include-parent options are enabled, the
diff will now include the parent image diffs. Previously, the
parent image diffs were not included when fast-diff was enabled
but was included when fast-diff was disabled.

Fixes: https://tracker.ceph.com/issues/42248
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: fix =[] default arg to _create_daemon

This is a no-no (pass by reference, not value)!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/cephadm: enable mgr debugging

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: bump minor version to match octopus

This will assist layered projects like QEMU and libvirt
determine which features are supported.

Fixed: https://tracker.ceph.com/issues/43401
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove pool objects when removing a namespace

Avoid leaving RBD pool objects within a pool namespace that is
being deleted.

Fixes: https://tracker.ceph.com/issues/43378
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr: close restful socket after exec

Signed-off-by: liushi <liu.shi@navercorp.com>

Merge pull request #32368 from xiexingguo/wip-balancer-efficiency-4

osd/OSDMap: consider overfull osds only when trying to do upmap

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #32289 from xiexingguo/wip-balancer-efficiency-2

mgr/balancer: eliminate usage of MS infrastructure for upmap mode

Reviewed-by: Yan Jun <yan.jun8@zte.com.cn>

install-deps.sh: install boost-1.72 for bionic

and uninstall boost-1.67 before installing boost-1.72

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: move extra_seastar_agrs up in vstart.sh

add extra_seastar_args in --mkfs

Signed-off-by: Chunmei Liu <chunmei.liu@intel.com>

rbd-mirror: clear init/shutdown context before completion

This fixes a potential issue where the init fails and shut down
is called with the original init context still set.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cephadm: create /var/run/ceph/$fsid as needed

The get_container_mounts method only passed this through if it already
exists so that shell etc commands will work even when this dir isn't
present yet.  Normally the systemd unit creates it.  This may not be the
case on initial bootstrap, of even later if a deploy happens but a unit
hasn't started yet.

Precreate it during bootstrap and other times before we do a deplot so
that the unit.run command has this passed through.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: fix v1/v2 ip/addrv handling; explicitly check bind to ip:port

- Handle the various v1 and v2 cases
- Handle --mon-ip with a port
- Explicitly test bind to the IP + port combination

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32389 into master

* refs/pull/32389/head:
test_cephadm.sh: pass --fsid to shell command

Reviewed-by: Michael Fritch <mfritch@suse.com>

rbd-mirror: consolidate prepare local/remote image steps to bootstrap

Now that the bootstrap state machine is simplier, it makes it cleaner to
move the prepare steps into the bootstrap state machine. Once snapshot
mirror support is added, these prepare state can be swapped out depending
on the mirroring type.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cephadm: restart non-ceph daemons on reconfig

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm/test_cephadm: fix expected output

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix remapping of hosts and services

This was nondeterministically remapping services to different hosts
depending on which hosts had stale caches because the 'hosts' list order
didn't match 'results' + 'in_cache'.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32286 into master

* refs/pull/32286/head:
cephadm: correct ipv6 support in port open detection

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/cephadm: add 'reconfig' service action

Signed-off-by: Sage Weil <sage@redhat.com>