rgw: Don't crash on copy when metadata directive not supplied

Fixes: http://tracker.ceph.com/issues/40416
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit d5a85e681f05f2b680bfe771ac16704b374218c7)

Merge pull request #28583 from ukernel/nautilus-40326

nautilus:  mds: change how mds revoke stale caps

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #28609 from batrick/i40324

nautilus: cephfs: ceph_volume_client: d_name needs to be converted to string before using

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29156 from smithfarm/wip-40839-nautilus

nautilus: cephfs-shell: TypeError in poutput

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29157 from smithfarm/wip-40842-nautilus

nautilus: cephfs: ceph-fuse: mount does not support the fallocate()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29158 from smithfarm/wip-40843-nautilus

nautilus: cephfs-shell: name 'files' is not defined error in do_rm()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29186 from xiaoxichen/wip-40874-nautilus

nautilus: cephfs: client: bump ll_ref from int32 to uint64_t

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29209 from smithfarm/wip-40316-nautilus

nautilus: tests: test_volume_client: print python version correctly

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29231 from smithfarm/wip-40438-nautilus

nautilus: cephfs: getattr on snap inode stuck

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #29233 from smithfarm/wip-40440-nautilus

nautilus: cephfs: mds: cannot switch mds state from standby-replay to active

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29275 from liewegas/wip-cephfs-meta-pool-priority-nautilus

nautilus: mon: set recovery priority etc on cephfs metadata pool

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29343 from pdvian/wip-40443-nautilus

nautilus: cephfs: client: set snapdir's link count to 1

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29344 from pdvian/wip-40445-nautilus

nautilus: cephfs: mds: cleanup unneeded client_snap_caps when splitting snap inode

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29060 from sebastian-philipp/nautilus-orchestrator_cache

nautilus: mgr/orchestrator: Cache and DeepSea iSCSI + NFS

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #28187 from pdvian/wip-39516-nautilus

nautilus: tests: osd-backfill-space.sh test failed in TEST_backfill_multi_partial()

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #28573 from smithfarm/wip-40281-nautilus

nautilus: bluestore: 50-100% iops lost due to bluefs_preextend_wal_files = false

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29007 from smithfarm/wip-40750-nautilus

nautilus: build/ops: rpm: drop SuSEfirewall2

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #29102 from tspmelo/wip-40733-nautilus

nautilus: mgr/dashboard: Fix npm vulnerabilities

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29162 from smithfarm/wip-40837-nautilus

nautilus: bluestore: Set concurrent max_background_compactions in rocksdb to 2

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29204 from smithfarm/wip-40319-nautilus

nautilus: tests: make: *** [hello_world_cpp] Error 127 in rados

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29391 from pdvian/wip-40625-nautilus

nautilus: osd/OSD: auto mark heartbeat sessions as stale and tear them down

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #29464 from jan--f/wip-41058-nautilus

nautilus: ceph-volume does not recognize wal/db partitions created by ceph-disk

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: fall back to PARTTYPE if PARTLABEL is empty

In some cases ceph-disk does not populate PARTLABEL for wal and db
partitions. This commit adds the assumption that the empty string is a
valid label and falls back to identifying those disks by PARTTYPE.

Fixes: https://tracker.ceph.com/issues/40917
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 82d2ae7ffdf62ca93c7848fb62e506db4cda893c)

ceph-volume: adjust tests for empty PARTLABEL fields

It seems like ceph-disk sometimes creates OSDs where the wal and db
partitions have no PARTLABEL set. This commit alters tests so that the
empty string is considered a valid label.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit bb29110c3c6e081568231131c70522dee6d12bb5)

ceph-volume: refactor ceph-disk_member unittests

This adds a parameterized fixture as input to the tests. Should make
writing new tests easier since only fixture parameters need to be added.
In the future we should be able to reuse the fixtures to build other
fixtures and reuse the parametrization.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit f42287c3416ad09c112e64756f5051c5c13aa344)

Merge pull request #29453 from neha-ojha/wip-41052-nautilus

nautilus: qa/tasks/cbt.py: change port to work with client_endpoints

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/tasks/cbt.py: change port to work with client_endpoints

hack to work with https://github.com/ceph/cbt/pull/160/

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit ee02416c654581d77d5f976da89b4a33dd1172e3)

Merge pull request #28740 from smithfarm/wip-40546-nautilus

nautilus: common: Keyrings created by ceph auth get are not suitable for ceph auth import

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28810 from ifed01/wip-ifed-db-stats-nau

nautilus: tools: ceph-kvstore-tool: print db stats

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29252 from smithfarm/wip-40180-nautilus

nautilus: tests: qa/standalone/scrub/osd-scrub-snaps.sh sometimes fails

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29309 from smithfarm/wip-40216-nautilus

nautilus: rgw_file:  fix invalidation of top-level directories

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #29254 from smithfarm/wip-40465-nautilus

nautilus: core: osd beacon sometimes has empty pg list

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28528 from pdvian/wip-39743-nautilus

nautilus: mon: paxos: introduce new reset_pending_committing_finishers for safety

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28575 from smithfarm/wip-40381-nautilus

nautilus: rbd: tests: add "rbd diff" coverage to suite

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28576 from smithfarm/wip-40382-nautilus

nautilus: tests: expected MON_CLOCK_SKEW but got none

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28870 from pdvian/wip-40235-nautilus

nautilus: rbd/action: fix error getting positional argument

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29193 from pdvian/wip-40272-nautilus

nautilus: rbd-mirror: link against the specified alloc library

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29244 from pdvian/wip-40274-nautilus

nautilus: core: librados: move buffer free functions to inline namespace

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29246 from pdvian/wip-40276-nautilus

nautilus: rbd: librbd: do not unblock IO prior to growing object map during resize

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #29315 from pdvian/wip-40293-nautilus

nautilus: rbd-nbd: sscanf return 0 mean not-match.

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/orchestrator: add --refresh to `ceph orchestrator service ls`

`ceph orchestrator device ls` already supports --refresh to force
a reload, we should support this for `ceph orchestrator service ls`
as well.

Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit d5e320ae541cabb2bfdfe20e68fa88233999b63c)

mgr/orchestrator: allow listing iscsi services

Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit e2bebee712a0716c6d009e8f5559fbe247123161)

mgr/deepsea: return ganesha and iscsi endpoint URLs

This updates describe_service() to include nfs and iscsi services
(deepsea internally refers to these as "ganesha" and "igw" roles).
Additionally, if deepsea sets any of container_id, service, version,
rados_config_location, service_url, status or status_desc, these will
come through now too.

This relies on https://github.com/SUSE/DeepSea/pull/1606 for the new
functionality, but if run against an older version of deepsea, it will
continue to operate as it did before, it just won't include
rados_config_location or service_url data.

I've also updated the caching in describe_service() so that if the
cache needs filling, it just requests all services from DeepSea, caches
everything, then does filtering on the results.  Previously, if the
cache was empty and only one service type was requested, the cache
would then *only* include that service and no other items, so subsequent
requests for different service types would return nothing until the
cache expired.

Fixes: https://tracker.ceph.com/issues/39095
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit 1af7c08a96ed82b8824e644b2884d963793299b2)

Merge pull request #28962 from ifed01/wip-ifed-kv-prefetch-nau

nautilus: os/bluestore: support RocksDB prefetch in buffered read mode

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28639 from xiexingguo/wip-pr-28404-for-n

nautilus: osd/OSD: keep synchronizing with mon if stuck at booting

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28891 from ifed01/wip-ifed-fix-alloc-dump-nau

nautilus: os/bluestore: more smart allocator dump when lacking space for bluefs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28892 from ifed01/wip-ifed-fix-no-compress-nau

nautilus: os/bluestore: load OSD all compression settings unconditionally.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28893 from ifed01/wip-ifed-remove-assert-bs-tool-nau

nautilus: os/bluestore/bluestore-tool: minor fixes around migrate

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28963 from ifed01/wip-ifed-add-omap-tail-nau

nautilus: os/bluestore: create the tail when first set FLAG_OMAP

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28966 from liewegas/wip-set-aio-write-max-nautilus

nautilus: os/bluestore: fix >2GB bluefs writes

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29023 from ifed01/wip-ifed-fix-stuplid-alloc-len0-nau

nautilus: os/bluestore: avoid length overflow in extents returned by Stupid Allocator

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #29147 from pdvian/wip-40267-nautilus

nautilus: common: OutputDataSocket retakes mutex on error path

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29168 from b-ranto/wip-n-mgr-metadata

nautilus: mgr: Add mgr metdata to prometheus exporter module

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #29173 from neha-ojha/wip-40583-nautilus

nautilus:common/options.cc: Lower the default value of osd_deep_scrub_large_omap_object_key_threshold

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29194 from pdvian/wip-40273-nautilus

nautilus: mgr: use ipv4 default when ipv6 was disabled

Merge pull request #29207 from smithfarm/wip-40904-nautilus

nautilus: mgr/influx: module fails due to missing close() method

Merge pull request #29227 from ifed01/wip-ifed-slow-ops2-nau

nautilus: bluestore: add slow op detection for collection_listing

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28756 from smithfarm/wip-40231-nautilus

nautilus: core: maybe_remove_pg_upmap can be super inefficient for large clusters

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28869 from dzafman/wip-backport-40073

nautilus: osd: Fix the way that auto repair triggers after regular scrub

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28993 from smithfarm/wip-40730-nautilus

nautilus: mon: auth mon isn't loading full KeyServerData after restart

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29115 from liewegas/wip-39693-nautilus

nautilus: osd/PG: fix cleanup of pgmeta-like objects on PG deletion

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29140 from liewegas/wip-40441-nautilus

nautilus: msg/async, v2: make the reset_recv_state() unconditional

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29141 from liewegas/wip-msg-async-1-nautilus

nautilus: msg/async: avoid unnecessary costly wakeups for outbound messages

Merge pull request #29142 from liewegas/wip-msg-noneed-set-connection-nautilus

nautilus: msg/async: no-need set connection for Message.

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #29143 from liewegas/wip-secure-nautilus

nautilus: msg/async: enable secure mode by default, no longer experimental

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29159 from smithfarm/wip-40845-nautilus

nautilus: core: MDSMonitor: use stringstream instead of dout for mds repaired

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29188 from badone/wip-nautilus-lazy-omap-stats-backport-tracker-40744

nautilus: core: lazy omap stat collection

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28550 from smithfarm/wip-40007-nautilus

nautilus: rgw: fix prefix handling in LCFilter

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28648 from pdvian/wip-40107-nautilus

nautilus: rgw: Librgw doesn't GC deleted object correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28712 from pdvian/wip-40134-nautilus

nautilus: rgw: TempURL should not allow PUTs with the X-Object-Manifest.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28713 from smithfarm/wip-40505-nautilus

nautilus: rgw: fix miss get ret in STSService::storeARN

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28714 from smithfarm/wip-40515-nautilus

nautilus: rgw: multisite: DELETE Bucket CORS is not forwarded to master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28715 from smithfarm/wip-40508-nautilus

nautilus: rgw: conditionally allow builtin users with non-unique email addresses

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28728 from joke-lee/nautilus

nautilus: rgw, Policy should be url_decode when assume_role

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28729 from joke-lee/nautilus-backport-28172

nautilus: rgw: fix rgw crash and set correct error code

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28735 from smithfarm/wip-40125-nautilus

nautilus: rgw: hadoop-s3a suite failing with more ansible errors

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28736 from smithfarm/wip-40129-nautilus

nautilus: rgw: Swift interface: server side copy fails if object name contains "?"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28737 from smithfarm/wip-40137-nautilus

nautilus: rgw: the Multi-Object Delete operation of S3 API wrongly handles the Code response element

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28739 from smithfarm/wip-40142-nautilus

nautilus: rgw: multisite: 'radosgw-admin bucket sync status' should call syncs_from(source.name) instead of id

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28751 from pdvian/wip-40150-nautilus

nautilus: rgw: add 'GET /admin/realm?list' api to list realms

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28854 from pdvian/wip-40226-nautilus

nautilus: rgw_file: include tenant when hashing bucket names

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #28886 from pdvian/wip-40263-nautilus

nautilus: rgw_file: all directories are virtual with respect to contents

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29154 from smithfarm/wip-40591-nautilus

nautilus: rgw: deleting bucket can fail when it contains unfinished multipart uploads

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29163 from smithfarm/wip-40851-nautilus

nautilus: rgw: multisite: radosgw-admin commands should not modify metadata on a non-master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29205 from smithfarm/wip-40760-nautilus

nautilus: rgw: Save an unnecessary copy of RGWEnv

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29265 from smithfarm/wip-40124-nautilus

nautilus: rgw: hadoop-s3a suite failing with more ansible errors

Merge pull request #29286 from smithfarm/wip-40355-nautilus

nautilus: rgw: provide admin-friendly reshard status output

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29287 from smithfarm/wip-40358-nautilus

nautilus: rgw: set null version object issues

Merge pull request #29310 from smithfarm/wip-40349-nautilus

nautilus: rgw/OutputDataSocket: append_output(buffer::list&) says it will (but does not) discard output at data_max_backlog

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29311 from smithfarm/wip-40352-nautilus

nautilus: rgw: multisite: RGWListBucketIndexesCR for data full sync needs pagination

Merge pull request #29313 from smithfarm/wip-40512-nautilus

nautilus: rgw: Put LC doesn't clear existing lifecycle

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29325 from smithfarm/wip-40450-nautilus

nautilus: core: s3tests-test-readwrite failed in rados run (Connection refused)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28549 from smithfarm/wip-39749-nautilus

nautilus: rgw: multisite: fix --bypass-gc flag for 'radosgw-admin bucket rm'

Reviewed-by: Casey Bodley <cbodley@redhat.com>

osd/OSD: auto mark heartbeat sessions as stale and tear them down

The primary benefit is that the OSD doesn't need to keep a flood of
blocked heartbeat messages around in memory.
This prevents OSDs from accumulating heartbeat messages due to a
broken switch and then exhausting the whole node's memory:

Jun 11 04:19:26 host-192-168-9-12 kernel: [409881.137077] Out of memory:
Kill process 1471476 (ceph-osd) score 47 or sacrifice child
Jun 11 04:19:26 host-192-168-9-12 kernel: [409881.146054] Killed process
1471476 (ceph-osd) total-vm:4822548kB, anon-rss:3097860kB,
file-rss:2556kB, shmem-rss:0kB

Fixes: http://tracker.ceph.com/issues/40586
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 6cc90f363b8096d2d5fad30e57426d0cea9e3478)

Merge pull request #29040 from dcasier/wip-40100-nautilus

nautilus: ceph-volume: lvm.activate: Return an error if WAL/DB devices absent

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

mds: cleanup unneeded client_snap_caps when splitting snap inode

Fixes: http://tracker.ceph.com/issues/39987
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit df79944a773ee0d77babe343b0b7107d58e5a1cd)

client: set snapdir's link count to 1

Otherwise nfs-ganesha treats snapdir as stale.

Fixes: https://tracker.ceph.com/issues/40101
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 7ad002307d7d34578478b1fee02be071d6b65250)

qa/rgw: update default port in perl workunits

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1643879218638915073d5310b859a94d10ffeac6)

qa/rgw: extra s3tests tasks use rgw endpoint configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 09e992ff01b4ce286540e1230a30df67103f5968)

rgw_file: pretty-print fh_key

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 84a2c22108a3d15b836680894ba750b637e49cd2)