osd: Again remove deprecated full/nearfull from osdmap

Fixes: https://tracker.ceph.com/issues/43485
Caused by: dff411f1905cc69bfb2cfa8b62a00b4702e6aa46

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #32242 from rishabh-d-dave/cv-methods-for-lvm-opt-S

ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

ceph-volume: add methods to pass filters to pvs, vgs and lvs commands

Filters can be passed to these commands by using option '-S'.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #32485 from tchaikov/wip-clang

common,crimson: fixes to compile with clang and libc++

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #31728 from luis-henrix/copy-from-notrunc

osd: add a 'copy-from2' operation that includes truncate_{seq,size} parameters

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #32488 from ronen-fr/clang_3

crimson: CLANG-related fixes to errorator.h

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32490 from rzarzynski/wip-crimson-missing_err_include

crimson: add missing include in common/errorator.h

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: add missing include in common/errorator.h

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: fix non-standard C++ constructs in errorator.h

Fixing use patterns accepted by gcc, but frowned upon by Clang:

Put noexcept specifier after attributes specifiers (as
per the C++ standard $11.3.5)

Limit the use of a not-completely-defined class names to
within function bodies, per the standard ("A class is considered
a completely-defined object type (or complete type) at the
closing } of the class-specifier.
Within the class member-specification, the class is regarded as
complete within function bodies, default arguments, noexcept-specifiers,
and default member initializers (including such things in nested classes).
Otherwise it is regarded as incomplete within its own class member-specification.")

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

crimson: fix FTBFS on Clang induced by errorator.

Reported-by Ronen Friedman <rfriedma@redhat.com>
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: add missing #include

clang++-9.0.1 and libc++ require them to build

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not use auto in param list of function prototype

src/crimson/net/Socket.cc:114:57: error: 'auto' not allowed in function
prototype
static inline seastar::future<> close_and_handle_errors(auto& out) {
                                                        ^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: do not capture unused variables

to silence warnings like

src/crimson/net/Socket.cc:93:41: warning: lambda capture 'this' is not
used [-Wunused-lambda-capture]
    return in.read_exactly(bytes).then([this](auto buf) {
                                        ^~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32474 from kalebskeithley/fedora32

do_cmake.sh: fedora-32 (rawhide) build with python-3.8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32481 from tchaikov/wip=admin-build-doc-python3

admin/build-doc: always install python3-* for build deps

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

admin/build-doc: always install python3-* for build deps

since we've dropped the support of python36-*, i.e. python packages
provided by EPEL7 before RHEL7/CentOS7 included python3. as before the
inclusion of python3 as supported python3, python packages are named
python36-*. and they don't provide python3-*. so we had to install
python36-* explicitly. now that we are able to use the python3-*
packages, we can just install python3-*.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32451 from rs-fabrica/doc_rados_operations_crush-map-edits

doc/rados/operations/crush-map-edits: recompile and set instructions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32442 into master

* refs/pull/32442/head:
doc/developer: fix dev mailing list address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/developer: fix dev mailing list address

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

do_cmake.sh: fedora-32 (rawhide) build with python-3.8

Fedora-32 now has python-3.8.

Fedora-29 is EOL. All non-EOL versions of Fedora have python-3.7, except
Fedora-32

(Unclear to me why PYBUILD="3" isn't used, like Ubuntu and FreeBSD. That
would eliminate breakage when pyhon versions change in Rawhide.)

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge PR #32457 into master

* refs/pull/32457/head:
doc: fix Getting Started with CephFS

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31309 from matthewoliver/rgw_swift_auth_note

doc: Add a RGW swift auth note

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31432 from zhangsw/rgw-cleanup-cls-reshard-const

rgw:cleanup the "magic string" usage in cls_rgw_client.cc

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #31429 from zhangsw/rgw-cleanup-cls-guard-bucket-resharding

rgw: correct some error log about reshard in cls_rgw.cc.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31404 from zhangsw/rgw-cleanup-deletelc

rgw:cleanup:remove un-used class member in RGWDeleteLC

Merge pull request #31345 from zhangsw/rgw-cleanup-rgw-admin-check-instance

rgw:cleanup:remove un-used create_new_bucket_instance in rgw_admin.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32351 from mykaul/osd_fsid

ceph-volume/lvm/activate.py: clarify error message: fsid refers to osd_fsid

Merge pull request #27792 from smithfarm/wip-make-dist

make-dist: drop Python 2/3 autoselect

Reviewed-by: Kefu Chai <kchai@redhat.com>

lvm/activate.py: clarify error message: fsid refers to osd_fsid

Users complained[1] the error message isn't clear, and they thought
it referred to the cluster fsid instead of the osd_fsid.
Made it clearer.

[1] rook/rook#4547

Fixes: https://tracker.ceph.com/issues/43442
Signed-off-by: Yaniv Kaul <ykaul@redhat.com>

doc: fix Getting Started with CephFS

Signed-off-by: Jos Collin <jcollin@redhat.com>

doc: Add a RGW swift auth note

Swift accounts are not like normal accounts, they are more akin to a
bank account that multile people could share. Or in the case of a cloud
it is usually mapped to the tenant.

Radosgw deals with this with a user and subuser, which is great, but a
little confusing. So this patch adds a note to those used to the Swift
API to make it more clear.

Signed-off-by: Matthew Oliver <moliver@suse.com>

doc/rados/operations/crush-map-edits: including recompile and set crushmapo instructions

Including missing recompile and set crushmap sections in crush-map-edits

Signed-off-by: Rodrigo Severo <rodrigo@fabricadeideias.com>

Merge pull request #32433 from tchaikov/wip-43414

rgw: svc.bucket: assign to optional<> using "="

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #32415 into master

* refs/pull/32415/head:
include/cpp-btree: use inline variable to define btree_is_key_compare_to
include/cpp-btree: use std::conditional_t<>
include/cpp-btree: use static_assert()
include/interval_set: use interval_set when appropriate
include/interval_set: rename some types
include/cpp-btree: add emplace_hint()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32399 from dillaman/wip-41556-2

rbd-mirror: continue to isolate journal replay logic

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32427 from tchaikov/qa-devices-schema-with-path

qa: add "path" to "device" output schema

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32430 from tchaikov/wip-rgw-rgw_log_entry

rgw: initialize member variables of rgw_log_entry

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32438 from tchaikov/wip-vstart-runner-py3

qa/tasks/vstart_runner: write string to StringIO

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/vstart_runner: write string to StringIO

we use `six.StringIO` for stdout when writing output messages from
`self.subproc.communicate()`, and `six.StringIO` accepts strings, so we
need to decode the output before sending them to `six.StringIO`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

make-dist: drop Python 2/3 autoselect

virtualenv will prefer Python 3 over Python 2 if both are available.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: svc.bucket: assign to optional<> using "="

instead of using

*optional_val = value;

we should use:

optional_val = value;

otherwise we will have following runtime assertion failure like

/usr/include/c++/8/optional:945: constexpr _Tp& std::_Optional_base<_Tp, true, true>::_M_get() [with _Tp = RGWBucketInfo*]: Assertion 'this->_M_is_engaged()' failed.

when compiled with the `_GLIBCXX_ASSERTIONS` macro defined. and recent
`redhat-rpm-config` adds `-D_GLIBCXX_ASSERTIONS` to
`__global_compiler_flags`, which is in turn included by `optflags`. and
`optflags` is pratically used everywhere.

Fixes: https://tracker.ceph.com/issues/43414
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: svc.bucket: move _orig_info out of "if" block

so `*orig_info` is valid out of the `if` block if it's set to
`&_orig_info`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: add "path" to "device" output schema

"path" was added by 2c0fd7d86827aa76b8d923870018365cdae4a6ad, so update
the test accordingly

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: set proper state after journal replay shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: initialize member variables of rgw_log_entry

`check-generated.sh` tests the dencoder of a class by:

ceph-dencoder type rgw_log_entry select_test 2 dump_json > /tmp/foo
ceph-dencoder type rgw_log_entry select_test 2 encode decode dump_json > /tmp/bar
cmp /tmp/foo /tmp/bar

but the second test instance created by
`rgw_log_entry::generate_test_instances` is created using the default
constructor of `rgw_log_entry`. if any of the member variable is created
using random bits in the heap, there is chance that the test fails like:

2 rgw_log_entry
/tmp/typ-l5omDhmDf /tmp/typ-4aoRVkoiV differ: char 315, line 15
**** rgw_log_entry test 2 dump_json check failed ****
   ceph-dencoder type rgw_log_entry select_test 2 dump_json > /tmp/typ-l5omDhmDf
   ceph-dencoder type rgw_log_entry select_test 2 encode decode dump_json > /tmp/typ-4aoRVkoiV
15c15
<     "obj_size": 809908640,
---
>     "obj_size": 1122974112,

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: do not log error upon bootstrap cancel

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32150 from ricardoasmarques/set-osd-device-class

mgr/dashboard: Explicitly set/change the device class of an OSD

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #32396 from toliu/restful-api-handle-execv

mgr: close restful socket after exec

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32406 from neha-ojha/wip-43364

mon/PGMap.h: disable network stats in dump_osd_stats

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31722 from tchaikov/wip-mon-deprecated-tell

mon: make "compact" tell command, and add deprecate/obsolete check for tell commands

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31829 from zjcmszh/xattrs

pybind: add WriteOp::set_xattr() & rm_xattr()

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #32420 from pkulijiawei/issues-43417

common: skip interfaces starting with "lo" in find_ipv{4,6}_in_subnet()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32421 from tchaikov/wip-43416

mgr/restful: jsonify lists instead of maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32424 into master

* refs/pull/32424/head:
mgr/balancer: sort pool names in 'balancer ls' output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32340 into master

* refs/pull/32340/head:
cephadm: support deployment of node-exporter

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32417 into master

* refs/pull/32417/head:
mgr/test_orchestrator: fix update_mgrs assert

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/test_orchestrator: fix update_mgrs assert

Signed-off-by: Sage Weil <sage@redhat.com>

common: skip interfaces starting with "lo" in find_ipv{4,6}_in_subnet()

This will solve the issue that the osd can't restart after seting a
virtual local loopback IP.
In find_ipv4_in_subnet() and find_ipv6_in_subnet(), I use
boost::starts_with(addrs->ifa_name, "lo") to ship the interfaces
starting with "lo".

Fixes: https://tracker.ceph.com/issues/43417
Signed-off-by: Jiawei Li <lijiawei1@chinatelecom.cn>

mgr/balancer: sort pool names in 'balancer ls' output

Otherwise this will vary based on the internal key sort order the dict it
was generated from.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: support deployment of node-exporter

Patch to support cephadm deploying node-exporter
containers. The node-exporter daemon has no
configuration and just needs access to the hosts
filesystem to pull the necessary system metrics.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

Merge PR #32405 into master

* refs/pull/32405/head:
qa/workunits/rados/test_large_omap_detection: py3-ify

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32403 from dillaman/wip-42248

librbd: diff iterate with fast-diff now correctly includes parent

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32402 from dillaman/wip-43401

librbd: bump minor version to match octopus

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32401 from dillaman/wip-43378

librbd: remove pool objects when removing a namespace

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/restful: jsonify lists instead of maps

in python3, `map()` returns a `map` instance which is iterable, while in
python2, `map()` returns a `list`, but `JSONEncoder.default()` expects a
list or whatever which inherits from `json.JSONEncoder`.

in this change, we just return a materized list instead of a map for
json encoder.

Fixes: https://tracker.ceph.com/issues/43416
Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: use inline variable to define btree_is_key_compare_to

simpler this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: use std::conditional_t<>

instead of using a homebrew replacement

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: use static_assert()

instead of using a homebrew replacement

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: use interval_set when appropriate

it's an alias of `interval_set<T,Map>` under some circumstances

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/interval_set: rename some types

to be consistent with STL conventions

Signed-off-by: Kefu Chai <kchai@redhat.com>

include/cpp-btree: add emplace_hint()

so we can construct the value to be inserted in-place, and move it into
the tree, instead of copying it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32409 from dillaman/wip-qa-rbd-python3

qa/workunits/rbd: fixed python interpreter for EL8

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: fix race between replay shutdown and notification

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32232 into master

* refs/pull/32232/head:
qa: no need to exclude ceph-mgr-diskprediction-cloud from package list to be installed
qa/packages: do not install ceph-mgr-diskprediction-cloud by default
ceph.spec.in: add runtime deps for mgr-diskprediction-cloud

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32341 from dillaman/wip-43379

rbd-mirror: do not auto-create peers in non-default namespaces

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32202 from xiexingguo/wip-pg-stuck-undersized

osd/PeeringState: transit async_recovery_targets back into acting before backfilling

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32408 from dillaman/wip-devstack

qa/suites/rbd: fixed download path for Ubuntu Bionic

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32305 from neha-ojha/wip-43308

osd/PeeringState.cc: don't let num_objects become negative

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #32354 from rzarzynski/wip-log-id2name

log: print pthread ID / name mapping in recent events dump.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32364 from rzarzynski/wip-bug-42780

common: fix deadlocky inflight op visiting in OpTracker.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32316 from bk201/wip-43363

mgr/cephadm: fix type of timeout options

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32270 from sebastian-philipp/orchestrator-completion-already-finished

mgr/orchestrator: Raise more expressive Error, if completion already …

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32185 from batrick/interval-set-c++17

include: fix interval_set const_iterator call operator type

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30344 from shenhang/jsoninfo

osd/OSDMap.cc: add more info in json output of osd stat

Reviewed-by: Коренберг Марк <socketpair@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32183 from aclamk/fix-getxattr-leak

librados: fix leak in getxattr and getxattrs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32265 from simon-rock/simon_work_adv

tool: ceph_monstore_tool: --readable=0 => --readable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32222 from toabctl/qa-flake8-py3

qa: Run flake8 on python2 and python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32312 from badone/wip-nature-theme-changes

doc: Modify nature theme

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32366 from liu-chunmei/vstart

vstart.sh: move extra_seastar_args up in vstart.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32391 from tchaikov/wip-boost-1.72

install-deps.sh, cmake: use boost 1.72 on bionic

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32407 into master

* refs/pull/32407/head:
qa/suites/rados: move cephadm_orchestrator to el8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32392 into master

* refs/pull/32392/head:
cephadm: fix v1/v2 ip/addrv handling; explicitly check bind to ip:port

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32377 into master

* refs/pull/32377/head:
qa/suites/rados/thrash-old-clients: configure mons in terms of addrvecs
qa/suites/rados/thrash-old-clients: hammer: fix package list
qa/tasks/cephadm: set .conf to cluster config object
qa/tasks/cephadm: archive /var/log/ceph logs too (not just cluster dir)
qa/tasks/cephadm: client keyring
qa/tasks/cephadm: setup thrashers ctx item
qa/tasks/ceph_manager: asok commands via cephadm shell
qa/suites/rados/thrash-old-clients: stick to el7
qa/tasks/cephadm: check cluster log; support log-whitelist
qa/suites/rados/thrash-old-clienets: python-foo to python3-foo
qa/suites/rados/thrash-old-clients: add new exclude_packages
qa/suites/rados/thrash-old-clients: use cephadm
mon/ConfigMonitor: make legacy mon addr/port parseable by legacy code

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/thrash-old-clients: configure mons in terms of addrvecs

This is more explicit.  More importantly, the 'mon update' command
can't handle an "ip:port"; it wants either a CIDR, bare IP, or addrvec.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash-old-clients: hammer: fix package list

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: set .conf to cluster config object

ceph.py does this and ceph_manager.py kill_osd relies on it.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: archive /var/log/ceph logs too (not just cluster dir)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: client keyring

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32390 into master

* refs/pull/32390/head:
cephadm: create /var/run/ceph/$fsid as needed

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32281 into master

* refs/pull/32281/head:
mgr/cephadm: fix =[] default arg to _create_daemon
qa/suites/rados/cephadm: enable mgr debugging
cephadm: restart non-ceph daemons on reconfig
mgr/cephadm/test_cephadm: fix expected output
mgr/cephadm: fix remapping of hosts and services
mgr/cephadm: add 'reconfig' service action
cephadm: move mon creation config bits into mgr/cephadm
cephadm: fix bootstrap message 'ssh' -> 'cephadm'
cephadm: add --reconfig option for 'deploy'
mgr/orchestrator: drop 'reload' service[-instance] action

Reviewed-by: Sebastian Wagner <swagner@suse.com>