Merge pull request #20710 from pdvian/wip-23174-luminous

luminous: common/dns_resolv.cc: Query for AAAA-record if ms_bind_ipv6 is True

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #20766 from t19903/wip-cache-tier-forward-hang

luminous: Objecter: add ignore overlay flag if got redirect reply

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21079 from tchaikov/wip-23471-luminous

luminous: add --add-bucket and --move options to crushtool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/singleton/diverget_priors*: broaden whitelist

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 39e5efbad2c2834acf8e6463258b148321ea85a4)

Merge pull request #20762 from pdvian/wip-23180-luminous

luminous: radosgw-admin data sync run crashes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20953 from pdvian/wip-23347-luminous

luminous: rgw: fix memory fragmentation problem reading data from client.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21054 from pdvian/wip-23225-luminous

luminous: cls/rgw: fix bi_log_iterate_entries return wrong truncated

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20722 from theanalyst/wip-luminous-23221

luminous: fixes for openssl & libcurl

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21214 from smithfarm/wip-23025-luminous

luminous: rgw: data sync of versioned objects, note updating bi marker

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20896 from smithfarm/wip-23349-luminous

luminous: Couldn't init storage provider (RADOS)

Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #20250 from smithfarm/wip-22856-luminous

luminous: build Debian installation packages failure

Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #20737 from pdvian/wip-23178-luminous

luminous: build/ops: run-make-check.sh: fix SUSE support

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20748 from smithfarm/wip-jinja2-luminous

build/ops: deb: move python-jinja2 dependency to mgr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21016 from pdvian/wip-23077-luminous

luminous: mon: ops get stuck in "resend forwarded message to leader"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21111 from wido/luminous-21813

luminous: common/ipaddr: Do not select link-local IPv6 addresses

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21118 from smithfarm/wip-23500-luminous

luminous: snapmapper inconsistency, crash on luminous

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21134 from smithfarm/wip-23522-luminous

luminous: tests: unittest_pglog timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: log unlink_instance mtime as object's mtime

This is needed so that when squashing entries during data
sync we get a consistent mtime for that object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit bef22609de95843008b5d02eb075252bb97fbe90)

common: FreeBSD wants the correct struct selection for ipv6

Lets see if this also works for Linux

Fixes: http://tracker.ceph.com/issues/21813
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit 3806d8ec7c145d0239c94fc5b601d460b36757a5)

Merge pull request #21177 from tchaikov/wip-luminous-pr-21159

luminous: src: s/--use-wheel//

Reviewed-by: Nathan Cutler <ncutler@suse.com>

src: s/--use-wheel//

--use-wheel was deprecated in favor of --only-binary in pip v7.0.0. and
--use-wheel was removed in a recent release of pip. but some packages
are source packages, so we cannot simply replace use-wheel with
only-binary. so a simpler approach is to drop --use-wheel option, as pip
respects --find-links, and will find the required package from the
wheelhouse.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a80cce4ba6afee921a7c6adce802c91e30f8c0a9)

Conflicts:
src/ceph-detect-init/CMakeLists.txt
src/ceph-disk/CMakeLists.txt: trivial resolution
src/pybind/mgr/dashboard/CMakeLists.txt: dashboard2 is not
in luminous, so drop this change.

tests: remove TestPGLog ASSERT_DEATH test

Fixes: http://tracker.ceph.com/issues/23504
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit adcba97a41795c25de8b6dfe263a1014564a305f)

mon: mark mgr reports and osd_failure as no_reply

see also: #20517

Fixes: http://tracker.ceph.com/issues/22114
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0daccfb4951471ab3f70ae87870f23e14357d9c8)

Merge pull request #20254 from k0ste/luminous_backports

luminous: mgr: prometheus: set metadata metrics value to '1' (#22717).

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20334 from tchaikov/wip-pr-20255-luminous

luminous: config: Change bluestore_cache_kv_max to type INT64

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20358 from pdvian/wip-22940-luminous

luminous: Double free in rados_getxattrs_next

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20359 from xiexingguo/wip-luminous-balancer-fixes

luminous: mgr/balancer: pool-specific optimization support and bug fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20399 from smithfarm/wip-22942-luminous

luminous: ceph osd force-create-pg cause all ceph-mon to crash and unable to come up again

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20433 from pdvian/wip-22971-luminous

mon: allow removal of tier of ec overwritable pool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20495 from pdvian/wip-23024-luminous

luminous: osd: Fix assert when checking missing version

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20554 from pdvian/wip-23074-luminous

luminous: bluestore: statfs available can go negative

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #20585 from pdvian/wip-23114-luminous

luminous: librados: make OPERATION_FULL_FORCE the default for rados_remove()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20609 from pdvian/wip-23075-luminous

luminous: osd: objecter sends out of sync with pg epochs for proxied ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20988 from pdvian/wip-23410-luminous

luminous: legal: remove doc license ambiguity

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

osd/PG: simplify replica purged_snaps update

This dependency on the ondisk version dates back before argonaut, and no
longer makes sense.  Once the snap is trimmed by the primary, and
purged_snaps is updated, the replica can (must!) blindly follow suit.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 231ec67b7a6d0ca228266cccd0ef53a77b5428e6)

osd/PG: print debug if we fail to decode snapbl

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2f5a24d8f8fe7dd69dbbe3b3883d24d7ea0fdd41)

osd/PG: wait for scan_snaps() mapper repair

Wait for our repair to apply before continuing to avoid reading the
broken state again.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ea531df216c65a3773a55b5f6c42af20e5004263)

Conflicts:
src/osd/PG.cc (luminous uses osd->store->apply_transaction() where
                      master has osd->store->queue_transaction())

osd/SnapMapper: catch empty snaps on add_oid()

This shouldn't ever happen.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6668c16484be3f98a4a8a3806a338b2185a58a47)

osd/SnapMapper: clean up broken snapmapper entry if present

Better to clean up than to crash the OSD.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit fc378a0dc2aafbeee44871a304717b916a3f9f88)

osd: osd_debug_verify_snaps_on_info -> osd_debug_verify_snaps

Old option is unused

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit dce6f9d5328c50444fe3e2421f9ab1dc56bd597b)

Merge pull request #20825 from rzarzynski/wip-22534-luminous

luminous: rocksdb: incorporate the fix in RocksDB for no fast CRC32 path.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20604 from pdvian/wip-23101-luminous

luminous: systemd: Wait 10 seconds before restarting ceph-mgr

Reviewed-by: Wido den Hollander <wido@widodh.nl>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20651 from gregsfortytwo/luminous-23141-rotational

luminous: bluestore: correctly check all block devices to decide if journal is_…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20664 from tchaikov/wip-luminous-22344

luminous: ceph.in: print all matched commands if arg missing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20771 from tchaikov/wip-23212-luminous

luminous: os/bluestore: recalc_allocated() when decoding bluefs_fnode_t

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #20781 from wido/luminous-zabbix

mgr/zabbix: Backports to Luminous

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20785 from badone/wip-luminous-osd-removing

luminous: osd: add numpg_removing metric

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20957 from pdvian/wip-23351-luminous

luminous: os/filestore: fix do_copy_range replay bug

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20964 from ifed01/wip-ifed-reap-col-luminous

luminous: os/bluestore: backport fixes around _reap_collection

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20725 from smithfarm/wip-23230-luminous

luminous: Update mgr/restful documentation

Reviewed-By: Boris Ranto <branto@redhat.com>

test/cli/crushtool: add test for --add-bucket and --move

Fixes: http://tracker.ceph.com/issues/23471
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 8ea03a78d5b689fa1ee5f9729d310f417a9c6275)

crush/CrushWrapper: lower log level of check_item_loc()

it's not an error if the item is not at loc.

Fixes: http://tracker.ceph.com/issues/23471
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c1f43343450a84cbc1bde016e58b3b51479c3ab5)

crushtool: add --move

Fixes: http://tracker.ceph.com/issues/23471
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 22c943511e5d5187235d2455af8ad4158bf0053f)

Conflicts: src/tools/crushtool.cc: argparse_withargs() was introduced in
  3fc7b2fd, but the backport of that commit removed it. so we should use
  ceph_argparse_witharg() instead.

crushtool: add --add-bucket

Fixes: http://tracker.ceph.com/issues/23471
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4525003908f47dc8162c4778b00d87c87ea81dba)

Conflicts:
src/tools/crushtool.cc: the original commit uses C++17 features
heavily. in the backport, they are re-implemented using C++11.

Merge PR #20527 into luminous

* refs/pull/20527/head:
mds: various fixes for backport
mds: convert to boost::string_view
test/encoding: refactor to avoid escaping shell magic
mds: minor refactor of SimpleLock
mds: track Capability in mempool
mds: move CInode container members to mempool
mds: move CDentry container members to mempool
mds: move CDir container members to mempool
mds: put MDSCacheObject compact_map in mempool
common: use size_t for object size
mds: convert to allocator agnostic string_view
mds: simplify initialization
compact_*: support mempool allocated containers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20251 from pdvian/wip-22862-luminous

luminous: cephfs-journal-tool: Fix Dumper destroyed before shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20302 from pdvian/wip-22891-luminous

luminous: qa: synchronize kcephfs suites with fs/multimds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20342 from batrick/i22688

luminous: client: release revoking Fc after invalidate cache

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #20354 from pdvian/wip-22935-luminous

luminous: Client: setattr should drop "Fs" rather than "As" for mtime and size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20356 from pdvian/wip-22936-luminous

luminous : Client:Fix readdir bug

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20417 from pdvian/wip-22966-luminous

luminous : qa: adjust cephfs full test for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20522 into luminous

* refs/pull/20522/head:
qa: ignore full cluster warnings for mds-full

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20457 into luminous

* refs/pull/20457/head:
qa: ignore bad backtrace cluster wrn
qa/cephfs: Add tests to validate scrub functionality
cephfs: Add option to load invalid metadata from disk
cephfs: Reset scrub data when inodes move

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20432 from pdvian/wip-22969-luminous

luminous: mds: fix session reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20455 from pdvian/wip-23013-luminous

luminous: mds: optimize getattr file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20523 from pdvian/wip-23061-luminous

luminous:qa: ignore MON_DOWN while thrashing mons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20525 from pdvian/wip-23062-luminous

luminous: qa: ignore io pause warnings in mds-full test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20626 from pdvian/wip-23150-luminous

luminous: mds: add uptime to MDS status

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20654 from pdvian/wip-23156-luminous

luminous: common: omit short option for id in help for clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20904 from pdvian/wip-23314-luminous

luminous: client: allow client to use caps that are revoked but not yet returned

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #20960 from pdvian/wip-23355-luminous

luminous: client: use either dentry_invalidate_cb or remount_cb to invalidate k…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21072 from ceph/backport-cv-vagrant-reload

luminous ceph-volume improve robustness when reloading vms in tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests create a retry script for vagrant reload

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3be723d4d9816e6b487681d686f87cf41f67abd0)

ceph-volume tests use a reload script in simple functional tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b1b9c7a4162dcaa919c58c106915e77bab5eb40a)

ceph-volume tests use a reload script in lvm functional tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a54f5037861063d985b8ac1d158b919f119637b2)

Merge pull request #20253 from pdvian/wip-22857-luminous

luminous: librbd: potential crash if object map check encounters error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20416 from pdvian/wip-22964-luminous

luminous: rbd-mirror: fix potential infinite loop when formatting status message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20454 from pdvian/wip-23011-luminous

luminous: [journal] allocating a new tag after acquiring the lock should use on-disk committed position

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20550 from pdvian/wip-23064-luminous

luminous: test/librbd: utilize unique pool for cache tier testing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20617 from smithfarm/wip-perms-ocf-rbd-luminous

luminous: cmake: rbd resource agent needs to be executable.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20628 from pdvian/wip-23152-luminous

luminous: TestLibRBD.RenameViaLockOwner may still fail with -ENOENT

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20715 from pdvian/wip-23177-luminous

luminous: qa/workunits/rbd: switch devstack tempest to 17.2.0 tag

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20860 from pdvian/wip-23304-luminous

luminous: parent blocks are still seen after a whole-object discard

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #20967 from pdvian/wip-23407-luminous

luminous: cls/rbd: group_image_list incorrectly flagged as RW

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20991 from trociny/luminous-23423

luminous: librados/snap_set_diff: don't assert on empty snapset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21001 from pdvian/wip-23414-luminous

luminous: PendingReleaseNotes: add note about upgrading MDS

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

PendingReleaseNotes: add note about upgrading MDS

Fixes: https://tracker.ceph.com/issues/23172
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 9c809d3c58126694e577ade6d017e61a6c283411)

Conflicts:
PendingReleaseNotes: added section 12.2.5 with cephfs release notes

cls/rgw: fix bi_log_iterate_entries return wrong truncated

if there are over 1000 entries of instance keys, cls_cxx_map_get_vals
will get truncated=true, but bilogs already reach the end.

fixes: http://tracker.ceph.com/issues/22737

Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit de791830097d9eec454ea7450f76c67508fa80fa)

Merge pull request #21031 from ceph/wip-yuriw-fix-j-x

qa/tests - Change machine type from 'vps' to 'ovh' as 'vps' does not …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests - Change machine type from 'vps' to 'ovh' as 'vps' does not exist anymore

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21015 from dzafman/wip-remove-bc-luminous

luminous: test: Replace bc command with printf command

Reviewed-by: Nathan Cutler <ncutler@suse.com>

test: Replace bc command with printf command

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 293ac9895f86b1258434d0dc539a7d4b7046cb6d)

mon: mark OSD beacons and pg_create messages as no_reply

Fixes: http://tracker.ceph.com/issues/22114
Reported-by: Hongpeng Lu <ludehp@163.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit be1f999f5e1e902f60c50433251c90926538f406)

mon: mark manager beacons as no_reply

We never reply to manager beacons, and we have to mark them
that way or else forwarded messages pile up pending replies
and things eventually block.

Fixes: http://tracker.ceph.com/issues/22114
Reported-by: Hongpeng Lu <ludehp@163.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 57ecc1ef1c1d4df973517c71ba02e5d1a16fb9c1)

Merge pull request #20736 from pdvian/wip-23176-luminous

luminous: qa/rgw: remove some civetweb overrides for beast testing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20859 from pdvian/wip-23302-luminous

luminous: rgw: add radosgw-admin sync error trim to trim sync error log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20889 from pdvian/wip-23310-luminous

luminous: rgw: s3website error handler uses original object name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20932 from pdvian/wip-23318-luminous

luminous: rgw: fix crash with rgw_run_sync_thread false

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librados/snap_set_diff: don't assert on empty snapset

Instead treat the diff as a full-object delta.

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 2be4840afd4fc54c2ad06ccc2aac903e45bd37f5)

Conflicts:
src/librbd/deep_copy/ObjectCopyRequest.cc
src/librbd/deep_copy/ObjectCopyRequest.h
            deep_copy::ObjectCopyRequest does not exist in luminous,
            fixed rbd_mirror::image_sync::ObjectCopyRequest instead

Merge pull request #20937 from pdvian/wip-23346-luminous

luminous: rgw: reject encrypted object COPY before supported

Reviewed-by: Casey Bodley <cbodley@redhat.com>