Merge pull request #25365 from xiexingguo/wip-37493

osd/OSDMap: fix upmap mis-killing for erasure-coded PGs

Merge pull request #25262 from aclamk/wip-kvstore-tool-dump

ceph-kvstore-tool: dump fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25383 from dmick/wip-container-name

common/util.cc: add CONTAINER_NAME processing for metadata

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25400 from dillaman/wip-ceph-cli

ceph: do not raise StopIteration within generator

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25392 from majianpeng/async-rdma

msg/async/rdma: Delete duplicate header file

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph: do not raise StopIteration within generator

This is implicit when a generator exits and under Python 3
it results in a "StopIteration" exception being raised when
"ceph --help" is run.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #24129 into master

* refs/pull/24129/head:
common: drop ::length() from other implementations of the OpQueue.
common, osd: kill OpQueue::length() to minimize inter-core traffic.

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge PR #25388 into master

* refs/pull/25388/head:
mon/OSDMonitor: do not no_reply on ready_to_merge message

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #25192 from ProgVal/doc-rbd-features

rbd: Document new RBD feature flags and version support

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25359 from zmc/wip-dashboard-fix-teuth-runner

mgr/dashboard: Fix PYTHONPATH for test runner

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #25231 from rhcs-dashboard/37371-logs-links-permissions

mgr/dashboard: 'Logs' links permission in Landing Page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #25322 from Devp00l/wip-issue-37468

mgr/dashboard: Document custom RESTController endpoints

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

msg/async/rdma: Delete duplicate header file

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

bluestore/tools: Add option 'dump' to ceph-kvstore-tool to print both key and values.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #25335 from ifed01/wip-ifed-fix-enospc

os/bluestore: fix and unify log output on allocation failure

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25343 from liewegas/wip-36612

msg/async: do not trigger RESETSESSION from connect fault during connection phase

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

osd/OSDMap: fix upmap mis-killing for erasure-coded PGs

The up-set of erasure-coded PGs may include CRUSH_ITEM_NONE,
which as a result causes mis-killing of valid upmap items.

Fixes: https://tracker.ceph.com/issues/37493
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: ningtao <ningtao@sangfor.com.cn>

Merge pull request #25336 from dragonylffly/wip-silence-warning

crush/CrushWrapper: silence compiler warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #25373 into master

* refs/pull/25373/head:
ceph_argparse: fix CephBool to CephChoices conversion

Reviewed-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: do not no_reply on ready_to_merge message

The C_ReplyMap will reply to this op.  Do not no_reply, or else an
ill-timed mon election will mean we do nothing for this request.

Fixes: http://tracker.ceph.com/issues/37512
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #25361 from badone/wip-add-luminous-12.2.9-and-10

doc/releases: Add luminous releases 12.2.9 and 10

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>

common/util.cc: add CONTAINER_NAME processing for metadata

Kubernetes clusters use POD_NAME; simple containers should be
distinguishable and don't have 'pods'.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

doc/rbd: Document new RBD feature flags and librbd/krbd version support

Sources:

layering:

* https://github.com/ceph/ceph/commit/f87c441f6891e5e72d5e712d9a19e45a1b9a97eb
* krbd: https://github.com/torvalds/linux/commit/770eba6e295fd36e43881176ee0644b9cc2803f1

striping:

* https://github.com/ceph/ceph/commit/2e403ef2c98bb64220060c06b435fb1ffe08294e
* krbd: https://github.com/torvalds/linux/commit/770eba6e295fd36e43881176ee0644b9cc2803f1

exclusive-lock:

* https://github.com/ceph/ceph/commit/fa96bb4db47c500947efcab51892f45a04b03d30
* krbd: https://github.com/torvalds/linux/commit/ed95b21a4b0a71ef89306cdeb427d53cc9cb343f

object-map:

* https://github.com/ceph/ceph/commit/7e751ce9125cbeb04b9551afff4b91c6255c03f9

fast-diff:

* https://github.com/ceph/ceph/commit/16ad44cebcc5d2464be38b95f16e0d63d2b29ed1

deep-flatten:

* https://github.com/ceph/ceph/commit/b556d3173ae4483e681bb21658a7d484a2627386

journaling:

* https://github.com/ceph/ceph/commit/750771c6ff07f6fcf0235172ec68ff4d907c621d
* not a default: https://github.com/ceph/ceph/blob/1a14963/src/test/cli/rbd/help.t#L229

data-pool:

* https://github.com/ceph/ceph/commit/c5f22900474e0eed88910341de57e17e1cc154f0
* description: https://tracker.ceph.com/issues/17409
* krbd: https://github.com/torvalds/linux/commit/7e97332ea9caad3b7c6d86bc3b982e17eda2f736

operations:

* https://github.com/ceph/ceph/commit/209f6fc5a44e2d068ce0dd8b3e9a25ee700ad0e0
* description: https://github.com/torvalds/linux/commit/e573427a440fd67d3f522357d7ac901d59281948
* krbd: https://github.com/torvalds/linux/commit/e573427a440fd67d3f522357d7ac901d59281948

migrating:

* https://github.com/ceph/ceph/commit/2f7c4ae3fc7ec7a2bfe99e116bf5e4f71f24c6a1

Signed-off-by: Valentin Lorentz <progval+git@progval.net>

mgr/dashboard: Fix 'resource' typo

Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Document custom RESTController endpoints

Extended HACKING.rst by the fact how to extend a RESTController with
a custom endpoint which has all permissions preserved.

Fixes: https://tracker.ceph.com/issues/37468
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #25305 from cbodley/wip-37440

qa/rgw: fix invalid syntax error in radosgw_admin_rest.py

Reviewed-by: Volker Theile <vtheile@suse.com>

os/bluestore: fix and unify log output on allocation failure

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: 'Logs' links permission in Landing Page

- 'Logs' links in Landing Page cards' popovers
for cards 'Cluster Status' (when not in HEALTH_OK) and
'PG Status' appear only when the user has
the appropriate permissions.
- Some refactor.
- Added test: ensure that clickable text (popover) is shown
when there are health checks' messages.

Fixes: https://tracker.ceph.com/issues/37371
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #25366 from jtlayton/wip-orchestrator

orchestrator: add the ability to remove services

Merge pull request #25344 from sebastian-philipp/volumes-py3

mgr/volumes: Fix Python 3 import error

Reviewed-by: Ricardo Dias <rdias@suse.com>

ceph_argparse: fix CephBool to CephChoices conversion

Commit 525623b has introduced a regression that makes Ceph clients of
versions prior to Nautilus to receive invalid unicode characters in the
commands list JSON sent by the monitors.

The unicode characters are caused by memory corruption in a dangling
pointer stored by a std::string_view. The dangling pointer belonged to
an std::string object that was already destructed when the
std::string_view is used.
The fix in this PR makes the std::string object to live enough time for
the std::string_view to be used safely.

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #24445 from jmolmo/wip_ansible_orchestrator

mgr/ansible: Ansible orchestrator module

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge pull request #25339 from jtlayton/wip-volumes

mgr/volumes: fix orchestrator remove operation

common: drop ::length() from other implementations of the OpQueue.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common, osd: kill OpQueue::length() to minimize inter-core traffic.

Before
------
```
-    1,50%     1,34%  msgr-worker-0  ceph-osd  [.] WeightedPriorityQueue<OpQueueItem, unsigned long>::Queue::insert
```

Where self if 1.34%
```
       │       mov    %r12d,0x2c(%r14)
       │               }
       │               ++size;
 17,33 │1ba:   addl   $0x1,0x2ec48(%r14)
       │             }
```

After
-----
```
-    1,05%     0,93%  msgr-worker-0  ceph-osd  [.] WeightedPriorityQueue<OpQueueItem, unsigned long>::Queue::insert
```

Let's annotate hottest place (self is 0,93%):
```
       │           //Find the upper bound, cache the previous value and if we should
       │           //store it in the left or right node
       │           bool left_child = true;
       │           while(x){
       │ a0:   mov    %rax,%rbx
       │              ++depth;
       │              y = x;
       │              x = (left_child = comp(key, x)) ?
  8,86 │ a3:   cmp    0x20(%rbx),%r15
       │     _ZN5boost9intrusive31default_rbtree_node_traits_implIPvE9get_rightERKPNS0_11r
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

orchestrator_cli: add a service rm command

Signed-off-by: Jeff Layton <jlayton@redhat.com>

orchestrator/rook: add a rm_service backend command

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/volumes: fix orchestrator remove operation

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #25126 from xiexingguo/wip-heartbeat-bs

osd: two heartbeat fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

doc/releases: Add luminous releases 12.2.9 and 10

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

mgr/dashboard: Fix PYTHONPATH for test runner

A new mgr module ('volumes') pulls in src/pybind/ceph_volume_client.py; the
wrapper script that we use to run backend API tests needs to include that in
PYTHONPATH, or else the cluster will be stuck in HEALTH_WARN with the message:
"Module 'volumes' has failed dependency: No module named ceph_volume_client".

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #25122 from noahdesu/insights-osd-space

pybind/mgr: add osd space utilization to insights report

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #25308 into master

* refs/pull/25308/head:
osd/OSD: OSD::mkfs asserts when reusing disk with existing superblock.
os/bluestore: add main device expand capability.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25328 into master

* refs/pull/25328/head:
spec: fix cython package version to less than 0.29

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25330 from alfredodeza/wip-rm37449

ceph-volume  zap: improve zapping to remove all partitions and all LVs, encrypted or not

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests patch udevadm so it doesn't actually call udevadm in a test

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk use device.path for error reporting

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add back sdc1 partitions with parted to redeploy after zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional add back partitions with parted to redeploy after zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional recreate partitions previously removed with --destroy

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure that both disk and device report is_device

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure is_encrypted works for lv, partition, and mapper devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.util ensure partitions are listed as top-level keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests add mapper test for dm paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests update is_mapper unit test when zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device check both dm and mapper paths for is_mapper property

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm skip empty vg_name removals

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.arg_validators optionally skip gpt header check

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device if no vgs, set it to an empty list

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.device set vgs to Device object if available

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util update is_device when reported as device or disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util add method to detect encryption on devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk add a helper to remove partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/rgw: radosgw_admin_rest.py uses /admin/user?list for user listing

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph-volume util.disk include partitions as top level devices in get_devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume util.disk fix issue when capturing partition information

Before, the `if` condition meant that it would only work when the output
was '1', which is incorrect as that would only happen if a partition was
the first one, ignoring any other partition. The contents of that file
is the partition number, not a boolean to tell if it is a partition or
not.

It now includes the `holders` file contents which is needed for
dm-mapper work

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume lvm.zap split zapping in methods per device type

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25201 from jan--f/c-v-inventory-add-device_id

ceph-volume add device_id to inventory listing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastien Han <shan@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25203 from jan--f/c-v-batch-fix-py3

ceph-volume: fix Batch object in py3 environments

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25178 from joke-lee/wip-fix-rgw-multisite-info-get-set-caps-check

rgw: get or set realm zonegroup zone need check user's caps

Reviewed-by: Casey Bodley <cbodley@redhat.com>

msg/async: do not trigger RESETSESSION from connect fault during connection phase

Previously, if we got a connection fault during the connect/connect_reply
phase, we would increment connect_seq on the client side and trigger a
RESETSESSION on the server side (because there was not yet an existing
connection to replace).  This led to dropped messages, usually in the
form of stuck peering in the rados/thrash suite.

The problem is that the condition for 'reconnect' vs 'backoff' inherited
the test from SimpleMessenger, which had only a STATE_CONNECTING.  In
contract, AsyncMessenger also as CONNECTING_WAIT_BANNER_AND_IDENTIFY and
CONNECTING_SEND_CONNECT_MSG, and if we were in these states, we would
increment connect_seq instead of backing off and retrying (without an
increment).

Fix by adjusting the condition to match the range of CONNECTING states
in asyncmessenger.

Fixes: http://tracker.ceph.com/issues/36612
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/volumes: Fix Python 3 import error

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

crush/CrushWrapper: silence compiler warning

silence compiler warning for comparison
between signed and unsigned integer expressions

Signed-off-by: Li Wang <laurence.liwang@gmail.com>

ceph-volume: add Batch smoketest

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: fix Batch object in py3 environments

Fixes: https://tracker.ceph.com/issues/37356
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

pybind/mgr: handle inconsistent api state

Signed-off-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #25240 from cbodley/wip-rgw-list-buckets-cleanup

rgw: remove duplicated RGWRados::list_buckets_ helpers

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #25196 into master

* refs/pull/25196/head:
mon: auto clean up stale upmap items
osd/OSDMap: add pg-existence sanity check

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #25303 into master

* refs/pull/25303/head:
mgr/DaemonState: fix get_config_defaults locking

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: removed unused RGWRados::list_buckets_*

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: use metadata mgr to list buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25331 from jtlayton/wip-volumes

volumes mgr module fixes

Merge pull request #25161 from wjwithagen/wjw-fix-stringlist

rgw: need to give a type in list constructor

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25145 from adamemerson/wip-23817

Don't treat colons specially when matching resource field of ARNs in S3 Policy

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #25123 from yehudasa/wip-37281

rgw: change the way sysobj filters raw attributes, fix bucket sync state xattrs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25078 from mkoderer/rgw-ux

[rgw]: Fix help of radosgw-admin user info in case no uid

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #25068 from cbodley/wip-rgw-rados-yield

rgw: RADOS::Obj::operate takes optional_yield

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25310 from cbodley/wip-37448

rgw: data sync accepts ERR_PRECONDITION_FAILED on remove_object()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #25081 from cbodley/wip-37091

rgw: catch exceptions from librados::NObjectIterator

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25329 from jtlayton/wip-orchestrator

orchestrator_cli: fix HandleCommandResult invocations in _status()

Merge pull request #25314 from tchaikov/wip-37432

qa/suites/rados/upgrade: set require-osd-release to nautilus

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/volumes: fix "perm" field on ceph fs volume commands

They're all currently set as "r" but most of them should be "rw".

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/volumes: fix description text for ceph fs volume create

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mgr/volumes: call correct method to wait for the orchestrator

We need to use _orchestrator_wait here.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

orchestrator_cli: fix HandleCommandResult invocations in _status()

These strings were being passed in as positional parameters, but we
need to pass them in as odata=.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #25128 from dragonylffly/rb

tools/rados: allow reuse object for write test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25317 from tchaikov/wip-37444

mgr: don't write to output if EOPNOTSUPP

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25318 from javacruft/collections.abc

qa,pybind,tools: Correct usage of collections.abc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24822 from Rjerk/wip-fix-devicehealth-typo-error

mgr/devicehealth: fix is_valid_daemon_name typo error

Reviewed-by: Kefu Chai <kchai@redhat.com>

spec: fix cython package version to less than 0.29

Cython version 0.29 removed the support for python subinterpreters,
which completely breaks ceph-mgr funcionality.

See cython repo commit:
https://github.com/cython/cython/commit/7e27c7cd51a2f048cd6d3c246740cd977f8d2e50

Fixes: http://tracker.ceph.com/issues/37472
Signed-off-by: Ricardo Dias <rdias@suse.com>