common: drop the doubly-linked-list compatibility of bl::buffers_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: switch bl::buffers_t to a singly-linked list. Drop bi::list.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: bufferlist::buffers_t wraps bi::list now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

include: buffer_raw.h uses C++11-compatible std::aligned_storage.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: factories of buffer::ptr_node produce std::unique_ptr now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

client: create buffer::ptr_node directly.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os/bluestore: KernelDevice creates buffer::ptr_node directly.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tests: bench_bufferlist_alloc() gives a try to node_ptr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: hypercombined bufferlist.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: proliferate ptr_node in the bufferlist implementation.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: switch bufferlist to use boost::intrusive for _buffers.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

os: Transaction::_update_op_bl doesn't copy bl::_buffers anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: modernize and refactor occurrences of bufferlist::buffers_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: abstract the container type of bufferlist::_buffers.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #25126 from xiexingguo/wip-heartbeat-bs

osd: two heartbeat fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #25122 from noahdesu/insights-osd-space

pybind/mgr: add osd space utilization to insights report

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #25308 into master

* refs/pull/25308/head:
osd/OSD: OSD::mkfs asserts when reusing disk with existing superblock.
os/bluestore: add main device expand capability.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25328 into master

* refs/pull/25328/head:
spec: fix cython package version to less than 0.29

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25330 from alfredodeza/wip-rm37449

ceph-volume  zap: improve zapping to remove all partitions and all LVs, encrypted or not

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests patch udevadm so it doesn't actually call udevadm in a test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk use device.path for error reporting

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add back sdc1 partitions with parted to redeploy after zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add back partitions with parted to redeploy after zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional recreate partitions previously removed with --destroy

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure that both disk and device report is_device

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure is_encrypted works for lv, partition, and mapper devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure partitions are listed as top-level keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add mapper test for dm paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests update is_mapper unit test when zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device check both dm and mapper paths for is_mapper property

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm skip empty vg_name removals

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.arg_validators optionally skip gpt header check

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device if no vgs, set it to an empty list

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device set vgs to Device object if available

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util update is_device when reported as device or disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util add method to detect encryption on devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk add a helper to remove partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk include partitions as top level devices in get_devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk fix issue when capturing partition information

Before, the `if` condition meant that it would only work when the output
was '1', which is incorrect as that would only happen if a partition was
the first one, ignoring any other partition. The contents of that file
is the partition number, not a boolean to tell if it is a partition or
not.

It now includes the `holders` file contents which is needed for
dm-mapper work

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap split zapping in methods per device type

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25201 from jan--f/c-v-inventory-add-device_id

ceph-volume add device_id to inventory listing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastien Han <shan@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25203 from jan--f/c-v-batch-fix-py3

ceph-volume: fix Batch object in py3 environments

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25178 from joke-lee/wip-fix-rgw-multisite-info-get-set-caps-check

rgw: get or set realm zonegroup zone need check user's caps

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-volume: add Batch smoketest

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: fix Batch object in py3 environments

Fixes: https://tracker.ceph.com/issues/37356
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr: handle inconsistent api state

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #25240 from cbodley/wip-rgw-list-buckets-cleanup

rgw: remove duplicated RGWRados::list_buckets_ helpers

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #25196 into master

* refs/pull/25196/head:
mon: auto clean up stale upmap items
osd/OSDMap: add pg-existence sanity check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25303 into master

* refs/pull/25303/head:
mgr/DaemonState: fix get_config_defaults locking

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: removed unused RGWRados::list_buckets_*

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: use metadata mgr to list buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25331 from jtlayton/wip-volumes

volumes mgr module fixes

Merge pull request #25161 from wjwithagen/wjw-fix-stringlist

rgw: need to give a type in list constructor

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25145 from adamemerson/wip-23817

Don't treat colons specially when matching resource field of ARNs in S3 Policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #25123 from yehudasa/wip-37281

rgw: change the way sysobj filters raw attributes, fix bucket sync state xattrs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25078 from mkoderer/rgw-ux

[rgw]: Fix help of radosgw-admin user info in case no uid

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25068 from cbodley/wip-rgw-rados-yield

rgw: RADOS::Obj::operate takes optional_yield

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25310 from cbodley/wip-37448

rgw: data sync accepts ERR_PRECONDITION_FAILED on remove_object()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25081 from cbodley/wip-37091

rgw: catch exceptions from librados::NObjectIterator

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25329 from jtlayton/wip-orchestrator

orchestrator_cli: fix HandleCommandResult invocations in _status()

Merge pull request #25314 from tchaikov/wip-37432

qa/suites/rados/upgrade: set require-osd-release to nautilus

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/volumes: fix "perm" field on ceph fs volume commands

They're all currently set as "r" but most of them should be "rw".

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/volumes: fix description text for ceph fs volume create

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/volumes: call correct method to wait for the orchestrator

We need to use _orchestrator_wait here.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

orchestrator_cli: fix HandleCommandResult invocations in _status()

These strings were being passed in as positional parameters, but we
need to pass them in as odata=.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #25128 from dragonylffly/rb

tools/rados: allow reuse object for write test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25317 from tchaikov/wip-37444

mgr: don't write to output if EOPNOTSUPP

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25318 from javacruft/collections.abc

qa,pybind,tools: Correct usage of collections.abc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24822 from Rjerk/wip-fix-devicehealth-typo-error

mgr/devicehealth: fix is_valid_daemon_name typo error

Reviewed-by: Kefu Chai <kchai@redhat.com>

spec: fix cython package version to less than 0.29

Cython version 0.29 removed the support for python subinterpreters,
which completely breaks ceph-mgr funcionality.

See cython repo commit:
https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50

Fixes: http://tracker.ceph.com/issues/37472
Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #25323 from sebastian-philipp/orchestrator-utf8

mgr/orchestrator: remove unicode whitespaces

Reviewed-by: Jeff Layton <jlayton@redhat.com>

mgr/orchestrator: remove unicode whitespaces

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25311 from wjwithagen/wjw-fix-cores-EXPECT_DEATH

test: suppress core dumping in there tests as well

Reviewed-by: Kefu Chai <kchai@redhat.com>

Correct usage of collections.abc

Some classes should still be imported directly from collections;
only OrderedDict, Iterable and Callable (in the context of the
ceph codebase) are found in collections.abc.

The current code works due to the fallback support for Python 2.

Signed-off-by: James Page <james.page@ubuntu.com>

osd/OSD: OSD::mkfs asserts when reusing disk with existing superblock.

Fixes: http://tracker.ceph.com/issues/37404
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: add main device expand capability.

One can do that via ceph-bluestore-tool's bluefs-bdev-expand command

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test: suppress core dumping in there tests as well

On FreeBSD coredumps are enabled by default.
And thus tests in teardown() in ceph-helpers.sh can fail because of
unwaranted cores.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

tools/rados: allow reuse object for write test

Currently, for rados bench write test, it always
creates new objects for testing. Create operation
refers to non-neglectable metadata overhead, especially
for small write performance. This patch allows to
reuse objects for write testing

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

Merge pull request #20952 from shangfufei/wip-async-external-event

msg/async: Skip the duplicated processing of the same link

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25302 from wjwithagen/wjw-wip-build-kv_store_2

include: define errnos if not defined for better portablity

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25264 from tchaikov/wip-mds-mem-frag

denc: fix internal fragmentation when decoding ptr in bl

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24878 from trociny/wip-36656

mgr: race between daemon state and service map in 'service status'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24733 from tchaikov/wip-19348

pybind: simplify timeout handling in run_in_thread()

Reviewed-by: Rishabh Dave <ridave@redhat.com>

mgr: don't write to output if EOPNOTSUPP

if process_pg_map_command() fails to fulfill the request, we should keep
the odata intact. and let the module take care of it.

before this change, we always write to odata even if
process_pg_map_command() returns -EOPNOTSUPP, this leaves unnecessary
leftover in the output, like pg_info,pg_ready.

after this change, we won't touch odata if process_pg_map_command()
returns -EOPNOTSUPP. and odata will be filled with whatever the python
module returns.

Fixes: http://tracker.ceph.com/issues/37444
Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/async: Skip the duplicated processing of the same link.

Update Event.cc

Signed-off-by: shangfufei <shangfufei@inspur.com>

qa/suites/rados/upgrade: set require-osd-release to nautilus

* add qa/releases/nautilus.yaml so it can be reused.
* use releases/nautilus.yaml in luminous-x upgrade test, so
  test_librbd_python.sh is able to use the feature introduced in
  nautilus.

Fixes: http://tracker.ceph.com/issues/37432
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #24580 into master

* refs/pull/24580/head:
mds: bump mds protocol version
mds: handle fragment notify race

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/rgw: add credentials to zone 'remove' in test_zonegroup_remove()

zone credentials are required to 'period update --commit' from
--rgw-zone remove

Signed-off-by: Casey Bodley <cbodley@redhat.com>
Conflicts:
src/test/rgw/rgw_multi/tests.py

rgw: get or set realm zonegroup zone need check user's caps

fix: https://tracker.ceph.com/issues/37352

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

rgw: data sync accepts ERR_PRECONDITION_FAILED on remove_object()

sync of deletes uses an If-UnModified-Since precondition, but does not
handle the corresponding ERR_PRECONDITION_FAILED error. treating this as
a failure means that we'll keep retrying the delete which will never
succeed. break this loop by treating ERR_PRECONDITION_FAILED as a
success

Fixes: http://tracker.ceph.com/issues/37448
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #25272 into master

* refs/pull/25272/head:
qa: add simple test-volumes.sh workunit and run it from fs/basic_functional
vstart.sh: create default fs via 'fs volume create'
mgr/volumes: fix oremote
mon/MgrMonitor: enable volumes module by default
mgr: create `volumes` module
mgr: cleaner constructor for CommandResult
mgr: block for latest osdmap after command execution
mgr: add MgrModule.mon_command helper
ceph_volume_client: enable using existing rados inst
mon: give ceph-mgr access to 'fs' commands

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #25160 from theanalyst/rgw-empty-zg-fix

rgw: handle cases around zone deletion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24662 from theanalyst/reshard-stale-cleanup

rgw: reshard stale instance cleanup

qa: add simple test-volumes.sh workunit and run it from fs/basic_functional

Signed-off-by: Sage Weil <sage@redhat.com>

build: FreeBSD allow kv_store tools to be build

This requires defining
`EUCLEAN`
`EKEYREJECTED`
which are defined in Linux asm-generic/error.h

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/DaemonState: fix get_config_defaults locking

A fixup for 1c2bc53df73d9891330cf0b18f5193167d64df61.

Fixes: http://tracker.ceph.com/issues/36590
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #25290 from noahdesu/dashboard-tox-workdir

mgr/dashboard: use dedicated tox working dir

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #25179 from tspmelo/wip-check-i18n

mgr/dashboard: Add i18n validation script

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #24900 from zmc/wip-minimal-health

mgr/dashboard: Replace dashboard service

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

ceph-volume: add test for disk.udevadm_property

Signed-off-by: Jan Fajerski <jfajerski@suse.com>