osd: only exit if *latest* map(s) say we are destroyed

It's possible our current map is older, we were destroyed then, but in
newer maps our osd was recreated.  This happens when the oldest map after
a recreated osd happens to land on an epoch where the osd was marked
destroyed.

Fix by only exiting if one of the newest maps says we are (still)
destroyed.

Fixes: http://tracker.ceph.com/issues/22673
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17520 from joscollin/wip-cleanup-drop-return

rgw: Drop unnecessary return

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #19803 into master

* refs/pull/19803/head:
mds: get rid of unnecessary if condition

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19837 into master

* refs/pull/19837/head:
client: avoid recursive lock in ll_get_vino

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge PR #19860 into master

* refs/pull/19860/head:
cephfs-journal-tool: add usage help info for header set <field>
mds: add error log info
cephfs-journal-tool: add "set pool_id" option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19891 into master

* refs/pull/19891/head:
mds: handle 'inode gets queued for recovery multiple times'
mds: use elist to track recover queue items

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19911 into master

* refs/pull/19911/head:
cephfs: Switch MIN/MAX for std::min/max and use intarith templates

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19876 from theanalyst/build/do-cmake-args

build: do_cmake: allow ARGS to be overridden

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19433 from liewegas/wip-pg-removal

osd: put pg removal in op_wq

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

osd/PrimaryLogPG: do on_shutdown on removal

This cleans up our reservations and misc other state in OSDService that
needs to be cleaned up.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: track deleted pools' pg_nums for calculating split

This is needed to determine (quickly) whether PGs have split.  Calling
get_pg_num() on the latest map does not work when the pool has been
deleted from that map.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19850 from dzafman/wip-calc-stats

osd/PG: re-write of _update_calc_stats and improve pg degraded state

Fixes: http://tracker.ceph.com/issues/20059
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

tests: recovery-unfound-found test needs to account for correct misplaced calculations

The test expected HEALTH_OK when in a state with misplaced objects therefore HEALTH_WARN

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #19613 from qrGitHub/wip-rgw-optimize-bucketLCPrepare

rgw: optimize function bucket_lc_prepare

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19878 from cbodley/wip-rgw-get-omap-keys-cr

rgw: fix handling of ENOENT in RGWRadosGetOmapKeysCR

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #19897 from cbodley/wip-rgw-put-system-obj

rgw: rgw_put_system_obj takes bufferlist

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #19945 from tchaikov/wip-deprecated-in-cxx17

librbd,librados: do not include stdbool.h in C++ headers

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19261 from dillaman/wip-22286

mgr: disconnect unregistered service daemon when report received

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #19942 from shinobu-x/wip-sk-minmax-ms_async_op_threads

common: Add min/max of ms_async_op_threads

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19397 from xxhdx1985126/master

mon: update PaxosService::cached_first_committed in PaxosService::maybe_trim()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19822 from trociny/wip-ggate-ceph-globals

rbd-ggate: fix parsing ceph global options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #19673 from zouaiguo/wip_erase_code

mon/OSDMonitor.cc : set erasure-code-profile to "" when create replicated pools.

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19896 from tchaikov/wip-vive-cxx-17

msg/simple: s/ceph::size/std::size/

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19870 from liewegas/wip-strip-ceph-base

debian/rules: strip ceph-base libraries

Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #18755 from aclamk/wip-reverted-approx-size

osd/PG: revert approx size

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18191 from hjwsm1989/osd-mark-down

qa/standalone/osd/osd-mark-down: create pool to get updated osdmap faster

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19877 from adamemerson/wip-dencish

denc cleanups and other fixes

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Verify stat calculations during backfill

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Verify stat calculations during recovery

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-helpers.sh: Add flush_pg_stats() to wait_for_clean() to make it reliable

osd-scrub-repair.sh: Fixes for omap keys landing on different OSDs due to flush

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Don't start recovery for missing until active pg state set

I was seeing recovery hang when it is started before _activate_committed()
The state machine passes into "Active" but this transitions to activating
pg state and after commmitted into "active" pg state.

Signed-off-by: David Zafman <dzafman@redhat.com>

qa: Ignore degraded PGs when injecting random eio errors

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Improve the way insufficient targets is handled to be compatible with EC

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Improve pg degraded state setting based on _update_calc_stats() degraded count

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #19946 from liewegas/wip-22677

qa/workunits/rados/test_rados_tool: fix stray |, race

osd: Handling when recovery sources have missing

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Base pg degraded state on num_degraded_objects

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Rewrite _update_calc_stats() to make it cleaner and more accurate

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: cleanup: Remove unused const vars

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: cleanup: Fix log message

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/workunits/rados/test_rados_tool: remove stray |, fix race

Fixes: http://tracker.ceph.com/issues/22676
Signed-off-by: Sage Weil <sage@redhat.com>

librbd,librados: do not include stdbool.h in C++ headers

stdbool.h is offered to provide C++ keywords for C source file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #12431 from VictorDenisov/snapshot_rename

rbd: group snapshot rename

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cephfs: Switch MIN/MAX for std::min/max and use intarith templates

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Add group snapshot rename commands

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Add group snap rename method

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

qa/workunits/rados/test_rados_tool.sh: -x

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19913 from adamemerson/wip-minimax-rados

rados: Prefer templates to macros

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19914 from tchaikov/wip-cmake-check-boost-in-src

cmake: check version of boost in src/boost

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19926 from batrick/i22527

mon: revert mds metadata argument name change

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19903 from dillaman/wip-librbd-op-features

librbd: add preliminary support for new operation feature bit

Reviewed-by: Mykola Golub <mgolub@suse.com>

common: Add min/max of ms_async_op_threads

Signed-off-by: Shinobu Kinjo <shinobu@redhat.com>

Merge pull request #19793 from badone/wip-blocking-deep-scrub

ceph.in: Add blocking mode for scrub and deep-scrub

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19910 from jdurgin/wip-21897

config: lower default omap entries recovered at once

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19894 into master

* refs/pull/19894/head:
cmake: link against ceph-common instead of common

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19920 into master

* refs/pull/19920/head:
client: fix cap revoke race
client: release revoking Fc after invalidate cache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19885 into master

* refs/pull/19885/head:
qa: increase osd count for ec testing
qa: add missing openstack configs

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: RGWRados::put_system_obj takes const bufferlist

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rados: Switch MIN/MAX for std::min/max and use intarith templates

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19900 from dillaman/wip-rbd-lock-list

rbd: adjusted "lock list" JSON and XML formatted output

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: rgw_put_system_obj takes bufferlist

all callers were converting from bufferlist to const char*, then
RGWRados::put_system_obj() copied that back into a new bufferlist

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19933 from qrGitHub/wip-rgw-useless-line4

rgw: drop redundant RGW_OP_STAT_OBJ check

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd: adjusted "lock list" JSON and XML formatted output

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: operations should return -EROFS if disabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: retrieve the op features on image refresh

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track new operation features within image

This will initially be utilized to restrict older clients from
performing operations against an image if (1) it doesn't
support the new feature bit, or (2) doesn't support the specific
enabled op feature.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

include: Add templates along side macros in intarith

The macros can be removed later.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: drop redundant RGW_OP_STAT_OBJ check

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

denc: Clean up tuples even more

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

types: Output of arbitrary std::tuples

Again, using ceph::for_each

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

encoding: Encode/decode of arbitrary tuples

Using ceph::for_each to avoid fiddling with indices everywhere.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Add ceph::for_each for tuples

Expand to pairs, optionals, and variants later.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

common: Make maybe functions support std::optional

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

denc: Simplify tuple decode/encode

Use index_sequence instead of rolling our own support infrastructure.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

denc: Fix std::array bound_encode

std::arrays are not variable length structures and thus are not
encoded with a length.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

denc: Remove unneeded enable_ifs

The idiom
```
template<typename U = T>
static enable_if_t<!!sizeof(U) && ... >
```

Exists to force SFINAE dependent on some aspect of T on functions that
are themselves members of a template parameterized on T. Without some
other expression conjoined to it, this construct is just noise.

Also replace the reserve switch with constexpr if.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

denc: Simplify is_any_of

We can't simplify underlying type in the obvious way since template
evaluation is strict.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

denc: Support std::optional

Since we'll probably want to use it at some point.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

{common,mds,rbd}: Fix unused lambda capture warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

denc: Unbreak Clang support

We were using an enable_if<sizeof(U)> idiom that Clang rejects as an
illicit narrowing conversion.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

doc/man: Add '--block' option for ceph command

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19886 from liu-chunmei/fix_dpdk_worker_references_issue

osd: fix dpdk worker references issue

Reviewed-by: Haomai Wang <haomai@xsky.com>

rbd: hide implicit / non-mutable features from the CLI help

The layering and data-pool features are enabled automatically
when striping or a data pool is specified, respectively. For
the 'rbd feature XYZ' command, hide all non-mutable features.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: record final pg_pool_t when a pool is deleted

Also, prevent OSD start if we have a PG whose pool is deleted and no
stored pool info.  (User should downgrade, let PG deletion complete, then
upgrade.)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: simplify snapmapper init

Use PGPool info; don't look at map (pool may have been deleted).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop PGPool auid member

It's at info.auid.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: tolerate startup for pg with no pool in osdmap

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: clear requeued peering events

Broken by fffcc8a50e684c9b67643ec002869a8b7024b01c

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3f07137f5f6b0448271be273db7b375ab1844d38)

osd/PG: use local_reserver to schedule delete

Use the reserver so that delete competes for the same slot(s) as recovery
and such.

Priority below recovery normally, unless the OSD is getting fullish, in
which case we set a very high priority.  We have to be careful here because
backfill will back off when the OSD gets full(ish) but log recovery does
not.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: identify splits even if pool is eventually deleted

Previously we wouldn't bother splitting if the pool was going away; now
we walk the pg forward and will process the split even if the pool is
later deleted.  Adjust the loop to terminate gracefully if that happens.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clean up pending splits when removing a pg

Say we get an osdmap indicating a pg will split, but the pg is deleting and
finishes its delete before the pg consumes that map.  We need to clean up
the pending split state.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: always call complete_split()

We need to complete the split regardless of whether the pg will get
removed after.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do not clean up split state on deleted pools

We want the PGs to get created and then process the newer osdmap(s) to
delete themselves.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: start delete after initial delete start info is applied

This allows us to eliminate the flush in _delete_some().

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: wait for commit *and* readable before deleting more of PG

For filestore, waiting for onreadable ensures that (1) the backend has done
(all) of the deletion work (we are throttled) and (2) that the flush() will
not block.  So, all good.

For bluestore, onreadable happens at queue time, so the flush() was needed
to throttle progress.  However, we don't want to block the op thread on
flush.  And waiting for commit isn't sufficient because that would not
capture the filestore apply work.

Fix by waiting for both commit and readable before doing more deletion
work.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: document split tracking structures a bit

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop advance_pg() split_pgs arg

We can do the same work inside the function and simplify the code.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: do not publish_stats_to_osd in state exit() methods

exit() can happen due to AdvMap and a peering interval change, but it
runs before we have updated any of our internal state about whether we
are the primary, whether our pool is deleted and the pg no longer exists,
and so on.  The publish depends on (1) being primary, and (2) will crash
if the pool is gone from the OSDMap.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop dup call to publish_stats_to_osd in Active AdvMap handler

Signed-off-by: Sage Weil <sage@redhat.com>

osd: queue peering events directly, without pg lookup

We do not need to look up the PG in order to queue a peering event now
that the queue is based on spg_t and handles the validity checks (based on
the epoch) for us.

Signed-off-by: Sage Weil <sage@redhat.com>