Merge pull request #20722 from theanalyst/wip-luminous-23221

luminous: fixes for openssl & libcurl

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21214 from smithfarm/wip-23025-luminous

luminous: rgw: data sync of versioned objects, note updating bi marker

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20896 from smithfarm/wip-23349-luminous

luminous: Couldn't init storage provider (RADOS)

Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #20250 from smithfarm/wip-22856-luminous

luminous: build Debian installation packages failure

Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #20737 from pdvian/wip-23178-luminous

luminous: build/ops: run-make-check.sh: fix SUSE support

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20748 from smithfarm/wip-jinja2-luminous

build/ops: deb: move python-jinja2 dependency to mgr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21016 from pdvian/wip-23077-luminous

luminous: mon: ops get stuck in "resend forwarded message to leader"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21111 from wido/luminous-21813

luminous: common/ipaddr: Do not select link-local IPv6 addresses

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21118 from smithfarm/wip-23500-luminous

luminous: snapmapper inconsistency, crash on luminous

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21134 from smithfarm/wip-23522-luminous

luminous: tests: unittest_pglog timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: log unlink_instance mtime as object's mtime

This is needed so that when squashing entries during data
sync we get a consistent mtime for that object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit bef22609de95843008b5d02eb075252bb97fbe90)

common: FreeBSD wants the correct struct selection for ipv6

Lets see if this also works for Linux

Fixes: http://tracker.ceph.com/issues/21813
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit 3806d8ec7c145d0239c94fc5b601d460b36757a5)

Merge pull request #21177 from tchaikov/wip-luminous-pr-21159

luminous: src: s/--use-wheel//

Reviewed-by: Nathan Cutler <ncutler@suse.com>

src: s/--use-wheel//

--use-wheel was deprecated in favor of --only-binary in pip v7.0.0. and
--use-wheel was removed in a recent release of pip. but some packages
are source packages, so we cannot simply replace use-wheel with
only-binary. so a simpler approach is to drop --use-wheel option, as pip
respects --find-links, and will find the required package from the
wheelhouse.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a80cce4ba6afee921a7c6adce802c91e30f8c0a9)

Conflicts:
src/ceph-detect-init/CMakeLists.txt
src/ceph-disk/CMakeLists.txt: trivial resolution
src/pybind/mgr/dashboard/CMakeLists.txt: dashboard2 is not
in luminous, so drop this change.

tests: remove TestPGLog ASSERT_DEATH test

Fixes: http://tracker.ceph.com/issues/23504
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit adcba97a41795c25de8b6dfe263a1014564a305f)

mon: mark mgr reports and osd_failure as no_reply

see also: #20517

Fixes: http://tracker.ceph.com/issues/22114
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0daccfb4951471ab3f70ae87870f23e14357d9c8)

Merge pull request #20254 from k0ste/luminous_backports

luminous: mgr: prometheus: set metadata metrics value to '1' (#22717).

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20334 from tchaikov/wip-pr-20255-luminous

luminous: config: Change bluestore_cache_kv_max to type INT64

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20358 from pdvian/wip-22940-luminous

luminous: Double free in rados_getxattrs_next

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20359 from xiexingguo/wip-luminous-balancer-fixes

luminous: mgr/balancer: pool-specific optimization support and bug fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20399 from smithfarm/wip-22942-luminous

luminous: ceph osd force-create-pg cause all ceph-mon to crash and unable to come up again

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20433 from pdvian/wip-22971-luminous

mon: allow removal of tier of ec overwritable pool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20495 from pdvian/wip-23024-luminous

luminous: osd: Fix assert when checking missing version

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20554 from pdvian/wip-23074-luminous

luminous: bluestore: statfs available can go negative

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #20585 from pdvian/wip-23114-luminous

luminous: librados: make OPERATION_FULL_FORCE the default for rados_remove()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20609 from pdvian/wip-23075-luminous

luminous: osd: objecter sends out of sync with pg epochs for proxied ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20988 from pdvian/wip-23410-luminous

luminous: legal: remove doc license ambiguity

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

osd/PG: simplify replica purged_snaps update

This dependency on the ondisk version dates back before argonaut, and no
longer makes sense.  Once the snap is trimmed by the primary, and
purged_snaps is updated, the replica can (must!) blindly follow suit.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 231ec67b7a6d0ca228266cccd0ef53a77b5428e6)

osd/PG: print debug if we fail to decode snapbl

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2f5a24d8f8fe7dd69dbbe3b3883d24d7ea0fdd41)

osd/PG: wait for scan_snaps() mapper repair

Wait for our repair to apply before continuing to avoid reading the
broken state again.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ea531df216c65a3773a55b5f6c42af20e5004263)

Conflicts:
src/osd/PG.cc (luminous uses osd->store->apply_transaction() where
                      master has osd->store->queue_transaction())

osd/SnapMapper: catch empty snaps on add_oid()

This shouldn't ever happen.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6668c16484be3f98a4a8a3806a338b2185a58a47)

osd/SnapMapper: clean up broken snapmapper entry if present

Better to clean up than to crash the OSD.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit fc378a0dc2aafbeee44871a304717b916a3f9f88)

osd: osd_debug_verify_snaps_on_info -> osd_debug_verify_snaps

Old option is unused

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit dce6f9d5328c50444fe3e2421f9ab1dc56bd597b)

Merge pull request #20825 from rzarzynski/wip-22534-luminous

luminous: rocksdb: incorporate the fix in RocksDB for no fast CRC32 path.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20604 from pdvian/wip-23101-luminous

luminous: systemd: Wait 10 seconds before restarting ceph-mgr

Reviewed-by: Wido den Hollander <wido@widodh.nl>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20651 from gregsfortytwo/luminous-23141-rotational

luminous: bluestore: correctly check all block devices to decide if journal is_…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20664 from tchaikov/wip-luminous-22344

luminous: ceph.in: print all matched commands if arg missing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20771 from tchaikov/wip-23212-luminous

luminous: os/bluestore: recalc_allocated() when decoding bluefs_fnode_t

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #20781 from wido/luminous-zabbix

mgr/zabbix: Backports to Luminous

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20785 from badone/wip-luminous-osd-removing

luminous: osd: add numpg_removing metric

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20957 from pdvian/wip-23351-luminous

luminous: os/filestore: fix do_copy_range replay bug

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20964 from ifed01/wip-ifed-reap-col-luminous

luminous: os/bluestore: backport fixes around _reap_collection

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20725 from smithfarm/wip-23230-luminous

luminous: Update mgr/restful documentation

Reviewed-By: Boris Ranto <branto@redhat.com>

Merge PR #20527 into luminous

* refs/pull/20527/head:
mds: various fixes for backport
mds: convert to boost::string_view
test/encoding: refactor to avoid escaping shell magic
mds: minor refactor of SimpleLock
mds: track Capability in mempool
mds: move CInode container members to mempool
mds: move CDentry container members to mempool
mds: move CDir container members to mempool
mds: put MDSCacheObject compact_map in mempool
common: use size_t for object size
mds: convert to allocator agnostic string_view
mds: simplify initialization
compact_*: support mempool allocated containers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20251 from pdvian/wip-22862-luminous

luminous: cephfs-journal-tool: Fix Dumper destroyed before shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20302 from pdvian/wip-22891-luminous

luminous: qa: synchronize kcephfs suites with fs/multimds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20342 from batrick/i22688

luminous: client: release revoking Fc after invalidate cache

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #20354 from pdvian/wip-22935-luminous

luminous: Client: setattr should drop "Fs" rather than "As" for mtime and size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20356 from pdvian/wip-22936-luminous

luminous : Client:Fix readdir bug

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20417 from pdvian/wip-22966-luminous

luminous : qa: adjust cephfs full test for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20522 into luminous

* refs/pull/20522/head:
qa: ignore full cluster warnings for mds-full

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20457 into luminous

* refs/pull/20457/head:
qa: ignore bad backtrace cluster wrn
qa/cephfs: Add tests to validate scrub functionality
cephfs: Add option to load invalid metadata from disk
cephfs: Reset scrub data when inodes move

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20432 from pdvian/wip-22969-luminous

luminous: mds: fix session reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20455 from pdvian/wip-23013-luminous

luminous: mds: optimize getattr file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20523 from pdvian/wip-23061-luminous

luminous:qa: ignore MON_DOWN while thrashing mons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20525 from pdvian/wip-23062-luminous

luminous: qa: ignore io pause warnings in mds-full test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20626 from pdvian/wip-23150-luminous

luminous: mds: add uptime to MDS status

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20654 from pdvian/wip-23156-luminous

luminous: common: omit short option for id in help for clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20904 from pdvian/wip-23314-luminous

luminous: client: allow client to use caps that are revoked but not yet returned

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #20960 from pdvian/wip-23355-luminous

luminous: client: use either dentry_invalidate_cb or remount_cb to invalidate k…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21072 from ceph/backport-cv-vagrant-reload

luminous ceph-volume improve robustness when reloading vms in tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests create a retry script for vagrant reload

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3be723d4d9816e6b487681d686f87cf41f67abd0)

ceph-volume tests use a reload script in simple functional tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b1b9c7a4162dcaa919c58c106915e77bab5eb40a)

ceph-volume tests use a reload script in lvm functional tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a54f5037861063d985b8ac1d158b919f119637b2)

Merge pull request #20253 from pdvian/wip-22857-luminous

luminous: librbd: potential crash if object map check encounters error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20416 from pdvian/wip-22964-luminous

luminous: rbd-mirror: fix potential infinite loop when formatting status message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20454 from pdvian/wip-23011-luminous

luminous: [journal] allocating a new tag after acquiring the lock should use on-disk committed position

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20550 from pdvian/wip-23064-luminous

luminous: test/librbd: utilize unique pool for cache tier testing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20617 from smithfarm/wip-perms-ocf-rbd-luminous

luminous: cmake: rbd resource agent needs to be executable.

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20628 from pdvian/wip-23152-luminous

luminous: TestLibRBD.RenameViaLockOwner may still fail with -ENOENT

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20715 from pdvian/wip-23177-luminous

luminous: qa/workunits/rbd: switch devstack tempest to 17.2.0 tag

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20860 from pdvian/wip-23304-luminous

luminous: parent blocks are still seen after a whole-object discard

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #20967 from pdvian/wip-23407-luminous

luminous: cls/rbd: group_image_list incorrectly flagged as RW

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20991 from trociny/luminous-23423

luminous: librados/snap_set_diff: don't assert on empty snapset

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21001 from pdvian/wip-23414-luminous

luminous: PendingReleaseNotes: add note about upgrading MDS

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

PendingReleaseNotes: add note about upgrading MDS

Fixes: https://tracker.ceph.com/issues/23172
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 9c809d3c58126694e577ade6d017e61a6c283411)

Conflicts:
PendingReleaseNotes: added section 12.2.5 with cephfs release notes

Merge pull request #21031 from ceph/wip-yuriw-fix-j-x

qa/tests - Change machine type from 'vps' to 'ovh' as 'vps' does not …

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests - Change machine type from 'vps' to 'ovh' as 'vps' does not exist anymore

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #21015 from dzafman/wip-remove-bc-luminous

luminous: test: Replace bc command with printf command

Reviewed-by: Nathan Cutler <ncutler@suse.com>

test: Replace bc command with printf command

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 293ac9895f86b1258434d0dc539a7d4b7046cb6d)

mon: mark OSD beacons and pg_create messages as no_reply

Fixes: http://tracker.ceph.com/issues/22114
Reported-by: Hongpeng Lu <ludehp@163.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit be1f999f5e1e902f60c50433251c90926538f406)

mon: mark manager beacons as no_reply

We never reply to manager beacons, and we have to mark them
that way or else forwarded messages pile up pending replies
and things eventually block.

Fixes: http://tracker.ceph.com/issues/22114
Reported-by: Hongpeng Lu <ludehp@163.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>
(cherry picked from commit 57ecc1ef1c1d4df973517c71ba02e5d1a16fb9c1)

Merge pull request #20736 from pdvian/wip-23176-luminous

luminous: qa/rgw: remove some civetweb overrides for beast testing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20859 from pdvian/wip-23302-luminous

luminous: rgw: add radosgw-admin sync error trim to trim sync error log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20889 from pdvian/wip-23310-luminous

luminous: rgw: s3website error handler uses original object name

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20932 from pdvian/wip-23318-luminous

luminous: rgw: fix crash with rgw_run_sync_thread false

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librados/snap_set_diff: don't assert on empty snapset

Instead treat the diff as a full-object delta.

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 2be4840afd4fc54c2ad06ccc2aac903e45bd37f5)

Conflicts:
src/librbd/deep_copy/ObjectCopyRequest.cc
src/librbd/deep_copy/ObjectCopyRequest.h
            deep_copy::ObjectCopyRequest does not exist in luminous,
            fixed rbd_mirror::image_sync::ObjectCopyRequest instead

Merge pull request #20937 from pdvian/wip-23346-luminous

luminous: rgw: reject encrypted object COPY before supported

Reviewed-by: Casey Bodley <cbodley@redhat.com>

legal: remove doc license ambiguity

The license applicable to the files under doc/ was originally declared (in the
top-level file COPYING) to be "Creative Commons Attribution-ShareAlike (CC
BY-SA)" by ed0653b493a3f919a3abc37a0aa9b5aa29ae0b0e

This license declaration omitted a version number.

Some time later ef7418421b3748c712019c8aedd02b8005c1e1ea was merged, mentioning
CC-BY-SA-1.0 as one of the Ceph source code licenses. Although the purpose of that
commit was only to summarize the canonical license information from COPYING, it
unintentionally became the only place in the source code where the doc license
version was specified.

In March 2018, the Debian project warned that CC-BY-SA-1.0 does not meet its
criteria for inclusion in the "free" section of the Debian distribution. (For
that, at least version 3.0 must be used.)

This new commit removes the doc license ambiguity by setting it to CC-BY-SA-3.0
in all places where the license is mentioned.

The exact spelling of the license name is taken from https://spdx.org/licenses/

Fixes: http://tracker.ceph.com/issues/23336
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit e294dd59100d962753f4dee8f7156be5a7fba875)

Merge pull request #20672 from linuxbox2/luminous-21942

luminous: rgw_log, rgw_file: account for new required envvars

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #20741 from pdvian/wip-23179-luminous

luminous: rgw: can't download object with range when compression enabled

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20157 from pdvian/wip-22812-luminous

luminous: rgw: log the right http status code in civetweb frontend's access log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20252 from pdvian/wip-22858-luminous

luminous: rgw: allow beast frontend to listen on specific IP address

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20345 from pdvian/wip-22889-luminous

luminous: rgw_file: avoid evaluating nullptr for readdir offset

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #20476 from pdvian/wip-23020-luminous

luminous: rgw: fix the max-uploads parameter not work

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20477 from pdvian/wip-23022-luminous

luminous: rgw: upldate the max-buckets when the quota is uploaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20586 from pdvian/wip-23102-luminous

luminous: rgw: parse old rgw_obj with namespace correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20072 from smithfarm/wip-22766-luminous

luminous: RGW doesn't check time skew in auth v4 http header request

Reviewed-by: Robin H. Johnson <robbat2@gentoo.org>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/bluestore: fix the wrong usage for map_any.

If Onode::flushing_count > 0, it should return true rahter than false.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
(cherry picked from commit 039ee8328e92d6834cd0cd4b77af75dc55c65221)

os/bluestore: remove lock for removed_collections.

Current, queue removed-collections and handle removed-collections
in the same thread. So no need a lock to protect.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
(cherry picked from commit 7646b7407b7cb6fe6f1b4bf3eb9e20d7cdfc5bce)