mon/OSDMonitor: do not no_reply on ready_to_merge message

The C_ReplyMap will reply to this op.  Do not no_reply, or else an
ill-timed mon election will mean we do nothing for this request.

Fixes: http://tracker.ceph.com/issues/37512
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25122 from noahdesu/insights-osd-space

pybind/mgr: add osd space utilization to insights report

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #25308 into master

* refs/pull/25308/head:
osd/OSD: OSD::mkfs asserts when reusing disk with existing superblock.
os/bluestore: add main device expand capability.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25328 into master

* refs/pull/25328/head:
spec: fix cython package version to less than 0.29

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25330 from alfredodeza/wip-rm37449

ceph-volume  zap: improve zapping to remove all partitions and all LVs, encrypted or not

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests patch udevadm so it doesn't actually call udevadm in a test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk use device.path for error reporting

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add back sdc1 partitions with parted to redeploy after zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add back partitions with parted to redeploy after zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional recreate partitions previously removed with --destroy

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure that both disk and device report is_device

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure is_encrypted works for lv, partition, and mapper devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure partitions are listed as top-level keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add mapper test for dm paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests update is_mapper unit test when zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device check both dm and mapper paths for is_mapper property

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm skip empty vg_name removals

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.arg_validators optionally skip gpt header check

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device if no vgs, set it to an empty list

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device set vgs to Device object if available

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util update is_device when reported as device or disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util add method to detect encryption on devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk add a helper to remove partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk include partitions as top level devices in get_devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk fix issue when capturing partition information

Before, the `if` condition meant that it would only work when the output
was '1', which is incorrect as that would only happen if a partition was
the first one, ignoring any other partition. The contents of that file
is the partition number, not a boolean to tell if it is a partition or
not.

It now includes the `holders` file contents which is needed for
dm-mapper work

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap split zapping in methods per device type

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25201 from jan--f/c-v-inventory-add-device_id

ceph-volume add device_id to inventory listing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastien Han <shan@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25203 from jan--f/c-v-batch-fix-py3

ceph-volume: fix Batch object in py3 environments

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25178 from joke-lee/wip-fix-rgw-multisite-info-get-set-caps-check

rgw: get or set realm zonegroup zone need check user's caps

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume: add Batch smoketest

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: fix Batch object in py3 environments

Fixes: https://tracker.ceph.com/issues/37356
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr: handle inconsistent api state

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #25240 from cbodley/wip-rgw-list-buckets-cleanup

rgw: remove duplicated RGWRados::list_buckets_ helpers

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #25196 into master

* refs/pull/25196/head:
mon: auto clean up stale upmap items
osd/OSDMap: add pg-existence sanity check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25303 into master

* refs/pull/25303/head:
mgr/DaemonState: fix get_config_defaults locking

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: removed unused RGWRados::list_buckets_*

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: use metadata mgr to list buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25331 from jtlayton/wip-volumes

volumes mgr module fixes

Merge pull request #25161 from wjwithagen/wjw-fix-stringlist

rgw: need to give a type in list constructor

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25145 from adamemerson/wip-23817

Don't treat colons specially when matching resource field of ARNs in S3 Policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #25123 from yehudasa/wip-37281

rgw: change the way sysobj filters raw attributes, fix bucket sync state xattrs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25078 from mkoderer/rgw-ux

[rgw]: Fix help of radosgw-admin user info in case no uid

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25068 from cbodley/wip-rgw-rados-yield

rgw: RADOS::Obj::operate takes optional_yield

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25310 from cbodley/wip-37448

rgw: data sync accepts ERR_PRECONDITION_FAILED on remove_object()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25081 from cbodley/wip-37091

rgw: catch exceptions from librados::NObjectIterator

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25329 from jtlayton/wip-orchestrator

orchestrator_cli: fix HandleCommandResult invocations in _status()

Merge pull request #25314 from tchaikov/wip-37432

qa/suites/rados/upgrade: set require-osd-release to nautilus

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/volumes: fix "perm" field on ceph fs volume commands

They're all currently set as "r" but most of them should be "rw".

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/volumes: fix description text for ceph fs volume create

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/volumes: call correct method to wait for the orchestrator

We need to use _orchestrator_wait here.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

orchestrator_cli: fix HandleCommandResult invocations in _status()

These strings were being passed in as positional parameters, but we
need to pass them in as odata=.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #25128 from dragonylffly/rb

tools/rados: allow reuse object for write test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25317 from tchaikov/wip-37444

mgr: don't write to output if EOPNOTSUPP

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25318 from javacruft/collections.abc

qa,pybind,tools: Correct usage of collections.abc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24822 from Rjerk/wip-fix-devicehealth-typo-error

mgr/devicehealth: fix is_valid_daemon_name typo error

Reviewed-by: Kefu Chai <kchai@redhat.com>

spec: fix cython package version to less than 0.29

Cython version 0.29 removed the support for python subinterpreters,
which completely breaks ceph-mgr funcionality.

See cython repo commit:
https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50

Fixes: http://tracker.ceph.com/issues/37472
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #25323 from sebastian-philipp/orchestrator-utf8

mgr/orchestrator: remove unicode whitespaces

Reviewed-by: Jeff Layton <jlayton@redhat.com>

mgr/orchestrator: remove unicode whitespaces

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25311 from wjwithagen/wjw-fix-cores-EXPECT_DEATH

test: suppress core dumping in there tests as well

Reviewed-by: Kefu Chai <kchai@redhat.com>

Correct usage of collections.abc

Some classes should still be imported directly from collections;
only OrderedDict, Iterable and Callable (in the context of the
ceph codebase) are found in collections.abc.

The current code works due to the fallback support for Python 2.

Signed-off-by: James Page <james.page@ubuntu.com>

osd/OSD: OSD::mkfs asserts when reusing disk with existing superblock.

Fixes: http://tracker.ceph.com/issues/37404
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: add main device expand capability.

One can do that via ceph-bluestore-tool's bluefs-bdev-expand command

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test: suppress core dumping in there tests as well

On FreeBSD coredumps are enabled by default.
And thus tests in teardown() in ceph-helpers.sh can fail because of
unwaranted cores.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

tools/rados: allow reuse object for write test

Currently, for rados bench write test, it always
creates new objects for testing. Create operation
refers to non-neglectable metadata overhead, especially
for small write performance. This patch allows to
reuse objects for write testing

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

Merge pull request #20952 from shangfufei/wip-async-external-event

msg/async: Skip the duplicated processing of the same link

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25302 from wjwithagen/wjw-wip-build-kv_store_2

include: define errnos if not defined for better portablity

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25264 from tchaikov/wip-mds-mem-frag

denc: fix internal fragmentation when decoding ptr in bl

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24878 from trociny/wip-36656

mgr: race between daemon state and service map in 'service status'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24733 from tchaikov/wip-19348

pybind: simplify timeout handling in run_in_thread()

Reviewed-by: Rishabh Dave <ridave@redhat.com>

mgr: don't write to output if EOPNOTSUPP

if process_pg_map_command() fails to fulfill the request, we should keep
the odata intact. and let the module take care of it.

before this change, we always write to odata even if
process_pg_map_command() returns -EOPNOTSUPP, this leaves unnecessary
leftover in the output, like pg_info,pg_ready.

after this change, we won't touch odata if process_pg_map_command()
returns -EOPNOTSUPP. and odata will be filled with whatever the python
module returns.

Fixes: http://tracker.ceph.com/issues/37444
Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: Skip the duplicated processing of the same link.

Update Event.cc

Signed-off-by: shangfufei <shangfufei@inspur.com>

qa/suites/rados/upgrade: set require-osd-release to nautilus

* add qa/releases/nautilus.yaml so it can be reused.
* use releases/nautilus.yaml in luminous-x upgrade test, so
  test_librbd_python.sh is able to use the feature introduced in
  nautilus.

Fixes: http://tracker.ceph.com/issues/37432
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #24580 into master

* refs/pull/24580/head:
mds: bump mds protocol version
mds: handle fragment notify race

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/rgw: add credentials to zone 'remove' in test_zonegroup_remove()

zone credentials are required to 'period update --commit' from
--rgw-zone remove

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Conflicts:
src/test/rgw/rgw_multi/tests.py

rgw: get or set realm zonegroup zone need check user's caps

fix: https://tracker.ceph.com/issues/37352

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

rgw: data sync accepts ERR_PRECONDITION_FAILED on remove_object()

sync of deletes uses an If-UnModified-Since precondition, but does not
handle the corresponding ERR_PRECONDITION_FAILED error. treating this as
a failure means that we'll keep retrying the delete which will never
succeed. break this loop by treating ERR_PRECONDITION_FAILED as a
success

Fixes: http://tracker.ceph.com/issues/37448
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #25272 into master

* refs/pull/25272/head:
qa: add simple test-volumes.sh workunit and run it from fs/basic_functional
vstart.sh: create default fs via 'fs volume create'
mgr/volumes: fix oremote
mon/MgrMonitor: enable volumes module by default
mgr: create `volumes` module
mgr: cleaner constructor for CommandResult
mgr: block for latest osdmap after command execution
mgr: add MgrModule.mon_command helper
ceph_volume_client: enable using existing rados inst
mon: give ceph-mgr access to 'fs' commands

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #25160 from theanalyst/rgw-empty-zg-fix

rgw: handle cases around zone deletion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24662 from theanalyst/reshard-stale-cleanup

rgw: reshard stale instance cleanup

qa: add simple test-volumes.sh workunit and run it from fs/basic_functional

Signed-off-by: Sage Weil <sage@redhat.com>

build: FreeBSD allow kv_store tools to be build

This requires defining
`EUCLEAN`
`EKEYREJECTED`
which are defined in Linux asm-generic/error.h

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/DaemonState: fix get_config_defaults locking

A fixup for 1c2bc53df73d9891330cf0b18f5193167d64df61.

Fixes: http://tracker.ceph.com/issues/36590
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25290 from noahdesu/dashboard-tox-workdir

mgr/dashboard: use dedicated tox working dir

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #25179 from tspmelo/wip-check-i18n

mgr/dashboard: Add i18n validation script

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #24900 from zmc/wip-minimal-health

mgr/dashboard: Replace dashboard service

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

ceph-volume: add test for disk.udevadm_property

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: add device_id field to inventory output

This intends to mimic the C++ implementation in src/common/blkdev.cc.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

mon: auto clean up stale upmap items

See https://www.spinics.net/lists/ceph-users/msg49518.html

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25280 from tchaikov/wip-ceph-py3

ceph.in: write bytes to stdout in raw_write()

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25261 from sebastian-philipp/mgr-handle-command-result

mgr: Add `HandleCommandResult` namedtuple

Merge pull request #25255 from tchaikov/wip-ceph-dencoder-without-cephfs

tools/ceph-dencoder: conditionally link against mds

Merge pull request #25244 from trociny/wip-crash-procname

mgr/crash: add process name to crash metadata

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24990 from dragonylffly/wip-fix-bench

tools/rados: always call rados.shutdown() before exit()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24939 from marcosps/simplify_examples

examples/librados: Remove not needed else clauses

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #24502 into master

* refs/pull/24502/head:
crushtool: implement --rebuild-class-roots command
crushtool: make --reweight re-sum choose_args weight-sets too
crushtool: --reweight should only reweight nonshadow roots
crush/CrushWrapper: reclassify: use default parent for created buckets
crush/CrushWrapper: reclassify: handle to-be-created buckets that we need twice
test/cli/crushtool/reclassify: add second gabe test case
crushtool: add --set-subtree-class; do not set class via --reclassify-root
test/cli/crushtool/reclassify: add reclassify test cases
doc/rados/operations/crush*: document reclassify
doc/rados/operations/crush: remove instructions for separate crush trees for ssd
crushtool: add --compare command
crushtool: implement --reclassify
crush/CrushCompiler: fix id scan to include class ids

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crushtool: implement --rebuild-class-roots command

This simply rebuilds the class roots.  Normally this should create no
change in the map since whatever was making changes to the map before
should have rebuild the shadow roots at that point.

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: make --reweight re-sum choose_args weight-sets too

This ensures that the weights add us for each weight-set (and each
position).  Note that since we don't have anything that actually
creates positional weight-sets, the behavior here might not be what we
want in the end, but for the compat weight-sets (no position), we *do*
keep the weights as a properly summing tree.

Signed-off-by: Sage Weil <sage@redhat.com>

crushtool: --reweight should only reweight nonshadow roots

The shadow roots will be then be rebuilt accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: reclassify: use default parent for created buckets

Note that these last 2 fixes also fix the 'h' test case.  \o/

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushWrapper: reclassify: handle to-be-created buckets that we need twice

Signed-off-by: Sage Weil <sage@redhat.com>