Merge pull request #20890 from pdvian/wip-23312-luminous

luminous: invalid JSON returned when querying pool parameters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20998 from pdvian/wip-23412-luminous

luminous: log: Fix AddressSanitizer: new-delete-type-mismatch

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #21053 from pdvian/wip-23224-luminous

luminous: mgr: quieten logging on missing OSD stats

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21059 from pdvian/wip-23226-luminous

luminous: os/bluestore: trim cache every 50ms (instead of 200ms)

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21090 from tchaikov/wip-23477-luminous

luminous: common: should not check for VERSION_ID

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #20675 from rzarzynski/wip-bs-no-blobrefing-in-do_read-luminous

luminous: os/bluestore: avoid unneeded BlobRefing in _do_read().

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20676 from rzarzynski/wip-bs-dump-onode-by-ref-luminous

luminous: os/bluestore: _dump_onode() don't prolongate Onode anymore.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20712 from pdvian/wip-23175-luminous

luminous: mgr: die on bind() failure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20867 from pdvian/wip-23306-luminous

luminous: msg/async: size of EventCenter::file_events should be greater than fd.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #21198 into luminous

* refs/pull/21198/head:
ceph_authtool: add mode option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20642 from b-ranto/wip-mgr-prom-luminous

[luminous] Backport recent prometheus exporter changes

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20668 from pdvian/wip-23160-luminous

luminous: osd: Remove partially created pg known as DNE

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #20710 from pdvian/wip-23174-luminous

luminous: common/dns_resolv.cc: Query for AAAA-record if ms_bind_ipv6 is True

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #20766 from t19903/wip-cache-tier-forward-hang

luminous: Objecter: add ignore overlay flag if got redirect reply

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21079 from tchaikov/wip-23471-luminous

luminous: add --add-bucket and --move options to crushtool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/rados/singleton/diverget_priors*: broaden whitelist

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 39e5efbad2c2834acf8e6463258b148321ea85a4)

Merge pull request #20762 from pdvian/wip-23180-luminous

luminous: radosgw-admin data sync run crashes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20953 from pdvian/wip-23347-luminous

luminous: rgw: fix memory fragmentation problem reading data from client.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21054 from pdvian/wip-23225-luminous

luminous: cls/rgw: fix bi_log_iterate_entries return wrong truncated

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20722 from theanalyst/wip-luminous-23221

luminous: fixes for openssl & libcurl

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21214 from smithfarm/wip-23025-luminous

luminous: rgw: data sync of versioned objects, note updating bi marker

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20896 from smithfarm/wip-23349-luminous

luminous: Couldn't init storage provider (RADOS)

Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #20250 from smithfarm/wip-22856-luminous

luminous: build Debian installation packages failure

Reviewed-By: Sage Weil <sage@redhat.com>

Merge pull request #20737 from pdvian/wip-23178-luminous

luminous: build/ops: run-make-check.sh: fix SUSE support

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20748 from smithfarm/wip-jinja2-luminous

build/ops: deb: move python-jinja2 dependency to mgr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21016 from pdvian/wip-23077-luminous

luminous: mon: ops get stuck in "resend forwarded message to leader"

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #21111 from wido/luminous-21813

luminous: common/ipaddr: Do not select link-local IPv6 addresses

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21118 from smithfarm/wip-23500-luminous

luminous: snapmapper inconsistency, crash on luminous

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #21134 from smithfarm/wip-23522-luminous

luminous: tests: unittest_pglog timeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

pybind/mgr/prometheus: extend daemon metadata with hostname and ceph version

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit d03c79676b1761b016ceaed82f7e3f03f99cda73)

pybind/mgr/prometheus: add fs and mds metadata metrics

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 0095e6a1ba0fb9473cfb60b6977498f2c1159079)

pybind/mgr/prometheus: new mon metrics

Remove mon_quorum_count and replace it with per-MON quorum status
(mon_quorum_status). Also add mon_metadata metrics.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 71a6408b1adfedab64208fe1a23231df43960924)

pybind/mgr/prometheus: add Metrics class to manage Metric instances

The central change of this commit is that per-daemon metrics are now
managed by first appending the metric (using Metrics.append) to a
staging area. Then the metrics for specific paths (metric names) are
overwritten by the staged metrics (by calling Metrics.reset). This gets
rid of metrics from daemon that are no longer in the cluster. I.e. when
ceph no longer reports metrics for one OSD daemon (because it was
removed from the cluster) the prometheus module will no longer export
metrics for that daemon.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit f88877ab0bd895d8b67222225c22a12d479b7548)

pybind/mgr/prometheus: don't crash on OSDs without metadata

Fix issue where the ceph_exporter crashes after a Ceph upgrade with a broken OSD - that OSD was never online with Luminous and thus we have no metadata for it

Signed-off-by: Christopher Blum zeichenanonym@web.de
(cherry picked from commit 3beee663f6fd6a16cef5a65ea5f9937cbea7a537)

pybing/mgr/prometheus: tidy up cherrypy engine start and stop

Calling cherrypy.engine.block() in the stanby module results in a failing
mgr failover.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ff471d485cd2068b6425b74a17875d8d6a823595)

pybind/mgr/prometheus: return 503 if MON cluster is down

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit eda9f15dd939de3e7205ceb0dfcc9fcfce7d89bb)

pybind/mgr/prometheus: add StandbyModule; return empty answer

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 18ffd7529b0975897121d8a5c37d0d20560a1fc7)

mgr/prometheus: Expose OSD Flags

Signed-off-by: Boris Ranto <branto@redhat.com>

mgr/prometheus: Fix pg_* counts

Currently, the pg_* counts are not computed properly. We split the
current state by '+' sign but do not add the pg count to the already
found pg count. Instead, we overwrite any existing pg count with the new
count. This patch fixes it by adding all the pg counts together for all
the states.

It also introduces a new pg_total metric for pg_total that shows the
total count of PGs.

Signed-off-by: Boris Ranto <branto@redhat.com>

mgr/prometheus: Skip bogus entries

The osd data can contain bogus '-' entries, skip these when populating
osd metadata and disk occupation.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 965aaad5273eac311e4535559a0e0a9df8a39bee)

rgw: log unlink_instance mtime as object's mtime

This is needed so that when squashing entries during data
sync we get a consistent mtime for that object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit bef22609de95843008b5d02eb075252bb97fbe90)

ceph_authtool: add mode option

We now have the ability to set the keyring file mode with the help of
'--mode MODE'. The mode needs to be specified in octal using the format:
0600.

Closes: http://tracker.ceph.com/issues/23513
Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit b179cd275526613a3838cb52e6f1c35950b6f3a8)

common: FreeBSD wants the correct struct selection for ipv6

Lets see if this also works for Linux

Fixes: http://tracker.ceph.com/issues/21813
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit 3806d8ec7c145d0239c94fc5b601d460b36757a5)

Merge pull request #21177 from tchaikov/wip-luminous-pr-21159

luminous: src: s/--use-wheel//

Reviewed-by: Nathan Cutler <ncutler@suse.com>

src: s/--use-wheel//

--use-wheel was deprecated in favor of --only-binary in pip v7.0.0. and
--use-wheel was removed in a recent release of pip. but some packages
are source packages, so we cannot simply replace use-wheel with
only-binary. so a simpler approach is to drop --use-wheel option, as pip
respects --find-links, and will find the required package from the
wheelhouse.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a80cce4ba6afee921a7c6adce802c91e30f8c0a9)

Conflicts:
src/ceph-detect-init/CMakeLists.txt
src/ceph-disk/CMakeLists.txt: trivial resolution
src/pybind/mgr/dashboard/CMakeLists.txt: dashboard2 is not
in luminous, so drop this change.

tests: remove TestPGLog ASSERT_DEATH test

Fixes: http://tracker.ceph.com/issues/23504
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit adcba97a41795c25de8b6dfe263a1014564a305f)

mon: mark mgr reports and osd_failure as no_reply

see also: #20517

Fixes: http://tracker.ceph.com/issues/22114
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0daccfb4951471ab3f70ae87870f23e14357d9c8)

Merge pull request #20254 from k0ste/luminous_backports

luminous: mgr: prometheus: set metadata metrics value to '1' (#22717).

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20334 from tchaikov/wip-pr-20255-luminous

luminous: config: Change bluestore_cache_kv_max to type INT64

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20358 from pdvian/wip-22940-luminous

luminous: Double free in rados_getxattrs_next

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20359 from xiexingguo/wip-luminous-balancer-fixes

luminous: mgr/balancer: pool-specific optimization support and bug fixes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20399 from smithfarm/wip-22942-luminous

luminous: ceph osd force-create-pg cause all ceph-mon to crash and unable to come up again

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20433 from pdvian/wip-22971-luminous

mon: allow removal of tier of ec overwritable pool

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20495 from pdvian/wip-23024-luminous

luminous: osd: Fix assert when checking missing version

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20554 from pdvian/wip-23074-luminous

luminous: bluestore: statfs available can go negative

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #20585 from pdvian/wip-23114-luminous

luminous: librados: make OPERATION_FULL_FORCE the default for rados_remove()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20609 from pdvian/wip-23075-luminous

luminous: osd: objecter sends out of sync with pg epochs for proxied ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20988 from pdvian/wip-23410-luminous

luminous: legal: remove doc license ambiguity

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

osd/PG: simplify replica purged_snaps update

This dependency on the ondisk version dates back before argonaut, and no
longer makes sense.  Once the snap is trimmed by the primary, and
purged_snaps is updated, the replica can (must!) blindly follow suit.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 231ec67b7a6d0ca228266cccd0ef53a77b5428e6)

osd/PG: print debug if we fail to decode snapbl

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2f5a24d8f8fe7dd69dbbe3b3883d24d7ea0fdd41)

osd/PG: wait for scan_snaps() mapper repair

Wait for our repair to apply before continuing to avoid reading the
broken state again.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ea531df216c65a3773a55b5f6c42af20e5004263)

Conflicts:
src/osd/PG.cc (luminous uses osd->store->apply_transaction() where
                      master has osd->store->queue_transaction())

osd/SnapMapper: catch empty snaps on add_oid()

This shouldn't ever happen.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6668c16484be3f98a4a8a3806a338b2185a58a47)

osd/SnapMapper: clean up broken snapmapper entry if present

Better to clean up than to crash the OSD.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit fc378a0dc2aafbeee44871a304717b916a3f9f88)

osd: osd_debug_verify_snaps_on_info -> osd_debug_verify_snaps

Old option is unused

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit dce6f9d5328c50444fe3e2421f9ab1dc56bd597b)

Merge pull request #20825 from rzarzynski/wip-22534-luminous

luminous: rocksdb: incorporate the fix in RocksDB for no fast CRC32 path.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20604 from pdvian/wip-23101-luminous

luminous: systemd: Wait 10 seconds before restarting ceph-mgr

Reviewed-by: Wido den Hollander <wido@widodh.nl>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20651 from gregsfortytwo/luminous-23141-rotational

luminous: bluestore: correctly check all block devices to decide if journal is_…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20664 from tchaikov/wip-luminous-22344

luminous: ceph.in: print all matched commands if arg missing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20771 from tchaikov/wip-23212-luminous

luminous: os/bluestore: recalc_allocated() when decoding bluefs_fnode_t

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

Merge pull request #20781 from wido/luminous-zabbix

mgr/zabbix: Backports to Luminous

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #20785 from badone/wip-luminous-osd-removing

luminous: osd: add numpg_removing metric

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20957 from pdvian/wip-23351-luminous

luminous: os/filestore: fix do_copy_range replay bug

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20964 from ifed01/wip-ifed-reap-col-luminous

luminous: os/bluestore: backport fixes around _reap_collection

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20725 from smithfarm/wip-23230-luminous

luminous: Update mgr/restful documentation

Reviewed-By: Boris Ranto <branto@redhat.com>

test/cli/crushtool: add test for --add-bucket and --move

Fixes: http://tracker.ceph.com/issues/23471
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 8ea03a78d5b689fa1ee5f9729d310f417a9c6275)

crush/CrushWrapper: lower log level of check_item_loc()

it's not an error if the item is not at loc.

Fixes: http://tracker.ceph.com/issues/23471
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c1f43343450a84cbc1bde016e58b3b51479c3ab5)

crushtool: add --move

Fixes: http://tracker.ceph.com/issues/23471
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 22c943511e5d5187235d2455af8ad4158bf0053f)

Conflicts: src/tools/crushtool.cc: argparse_withargs() was introduced in
  3fc7b2fd, but the backport of that commit removed it. so we should use
  ceph_argparse_witharg() instead.

crushtool: add --add-bucket

Fixes: http://tracker.ceph.com/issues/23471
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4525003908f47dc8162c4778b00d87c87ea81dba)

Conflicts:
src/tools/crushtool.cc: the original commit uses C++17 features
heavily. in the backport, they are re-implemented using C++11.

test: remove distro_version assert in distro detect test

VERSION_ID is optional in /etc/os-release, we removed the error
log in #17787, and the test also needs to be fixed.

Fixes: http://tracker.ceph.com/issues/23477
Signed-off-by: Shengjing Zhu <zhsj@umcloud.com>
(cherry picked from commit d446915ea4c02c3874d60067f88eb6492c1bf5df)

common/util: do not print error if VERSION_ID is missing

per os-release(5), VERSION_ID is optional.

Fixes: http://tracker.ceph.com/issues/23477
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit eff4961f4608384519aca0918e8ab9c7ac650f32)

Merge PR #20527 into luminous

* refs/pull/20527/head:
mds: various fixes for backport
mds: convert to boost::string_view
test/encoding: refactor to avoid escaping shell magic
mds: minor refactor of SimpleLock
mds: track Capability in mempool
mds: move CInode container members to mempool
mds: move CDentry container members to mempool
mds: move CDir container members to mempool
mds: put MDSCacheObject compact_map in mempool
common: use size_t for object size
mds: convert to allocator agnostic string_view
mds: simplify initialization
compact_*: support mempool allocated containers

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20251 from pdvian/wip-22862-luminous

luminous: cephfs-journal-tool: Fix Dumper destroyed before shutdown

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20302 from pdvian/wip-22891-luminous

luminous: qa: synchronize kcephfs suites with fs/multimds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20342 from batrick/i22688

luminous: client: release revoking Fc after invalidate cache

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #20354 from pdvian/wip-22935-luminous

luminous: Client: setattr should drop "Fs" rather than "As" for mtime and size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20356 from pdvian/wip-22936-luminous

luminous : Client:Fix readdir bug

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20417 from pdvian/wip-22966-luminous

luminous : qa: adjust cephfs full test for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20522 into luminous

* refs/pull/20522/head:
qa: ignore full cluster warnings for mds-full

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20457 into luminous

* refs/pull/20457/head:
qa: ignore bad backtrace cluster wrn
qa/cephfs: Add tests to validate scrub functionality
cephfs: Add option to load invalid metadata from disk
cephfs: Reset scrub data when inodes move

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20432 from pdvian/wip-22969-luminous

luminous: mds: fix session reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20455 from pdvian/wip-23013-luminous

luminous: mds: optimize getattr file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20523 from pdvian/wip-23061-luminous

luminous:qa: ignore MON_DOWN while thrashing mons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20525 from pdvian/wip-23062-luminous

luminous: qa: ignore io pause warnings in mds-full test

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20626 from pdvian/wip-23150-luminous

luminous: mds: add uptime to MDS status

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20654 from pdvian/wip-23156-luminous

luminous: common: omit short option for id in help for clients

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20904 from pdvian/wip-23314-luminous

luminous: client: allow client to use caps that are revoked but not yet returned

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #20960 from pdvian/wip-23355-luminous

luminous: client: use either dentry_invalidate_cb or remount_cb to invalidate k…

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21072 from ceph/backport-cv-vagrant-reload

luminous ceph-volume improve robustness when reloading vms in tests

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests create a retry script for vagrant reload

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3be723d4d9816e6b487681d686f87cf41f67abd0)

ceph-volume tests use a reload script in simple functional tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b1b9c7a4162dcaa919c58c106915e77bab5eb40a)