osd/PrimaryLogPG: do on_shutdown on removal

This cleans up our reservations and misc other state in OSDService that
needs to be cleaned up.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: track deleted pools' pg_nums for calculating split

This is needed to determine (quickly) whether PGs have split.  Calling
get_pg_num() on the latest map does not work when the pool has been
deleted from that map.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: record final pg_pool_t when a pool is deleted

Also, prevent OSD start if we have a PG whose pool is deleted and no
stored pool info.  (User should downgrade, let PG deletion complete, then
upgrade.)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: simplify snapmapper init

Use PGPool info; don't look at map (pool may have been deleted).

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop PGPool auid member

It's at info.auid.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: tolerate startup for pg with no pool in osdmap

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: clear requeued peering events

Broken by fffcc8a50e684c9b67643ec002869a8b7024b01c

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3f07137f5f6b0448271be273db7b375ab1844d38)

osd/PG: use local_reserver to schedule delete

Use the reserver so that delete competes for the same slot(s) as recovery
and such.

Priority below recovery normally, unless the OSD is getting fullish, in
which case we set a very high priority.  We have to be careful here because
backfill will back off when the OSD gets full(ish) but log recovery does
not.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: identify splits even if pool is eventually deleted

Previously we wouldn't bother splitting if the pool was going away; now
we walk the pg forward and will process the split even if the pool is
later deleted.  Adjust the loop to terminate gracefully if that happens.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: clean up pending splits when removing a pg

Say we get an osdmap indicating a pg will split, but the pg is deleting and
finishes its delete before the pg consumes that map.  We need to clean up
the pending split state.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: always call complete_split()

We need to complete the split regardless of whether the pg will get
removed after.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: do not clean up split state on deleted pools

We want the PGs to get created and then process the newer osdmap(s) to
delete themselves.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: start delete after initial delete start info is applied

This allows us to eliminate the flush in _delete_some().

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: wait for commit *and* readable before deleting more of PG

For filestore, waiting for onreadable ensures that (1) the backend has done
(all) of the deletion work (we are throttled) and (2) that the flush() will
not block.  So, all good.

For bluestore, onreadable happens at queue time, so the flush() was needed
to throttle progress.  However, we don't want to block the op thread on
flush.  And waiting for commit isn't sufficient because that would not
capture the filestore apply work.

Fix by waiting for both commit and readable before doing more deletion
work.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: document split tracking structures a bit

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop advance_pg() split_pgs arg

We can do the same work inside the function and simplify the code.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: do not publish_stats_to_osd in state exit() methods

exit() can happen due to AdvMap and a peering interval change, but it
runs before we have updated any of our internal state about whether we
are the primary, whether our pool is deleted and the pg no longer exists,
and so on.  The publish depends on (1) being primary, and (2) will crash
if the pool is gone from the OSDMap.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop dup call to publish_stats_to_osd in Active AdvMap handler

Signed-off-by: Sage Weil <sage@redhat.com>

osd: queue peering events directly, without pg lookup

We do not need to look up the PG in order to queue a peering event now
that the queue is based on spg_t and handles the validity checks (based on
the epoch) for us.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: carry pg ref for final pg delete txn

FileStore breaks of the Sequencer is destroyed while the txn is in flight.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: enqueue peering evt with pgid, not PG*

Signed-off-by: Sage Weil <sage@redhat.com>

osd: remove old pg removal infrastructure

Another queue bites the dust! \o/

Signed-off-by: Sage Weil <sage@redhat.com>

osd: leave PG registered (and stray) during delete; reimplement pg deletion

A lot of awkward complexity is implemented in OSD to handle PGs that aren't in
pg_map and are in the process of being deleted.  This is hard because if the
PG is recreated (or split, or whatever) then we need to stop the deletion and
create a fresh PG referencing the same data.

Instead, leave deleting PGs registered and Stray, with a new peering state
Stray/Deleting.  Let them continue to process OSDMaps, splits, peering intervals,
and so on.  If they are not fully deleted, they'll go back to Reset -> Stray and
so on and the new primary will get the notify and decide what to do with them
(usually instruct them to delete again).

This (1) streamlines and cleans up the code structure, and also (2) gets rid of
the special purpose RemoveWQ and moves the delete work into the main op_wq
where it can get throttled and so on.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: tolerate missing pool

These assertions will soon no longer be valid.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: tolerate pool deletion

- the epoch the pool is deleted is an interval change
- no changes are possible after that

Also, use a pg_pool_t pointer to avoid the repeat lookups.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-names!-in!-space!' of git://github.com/adamemerson/ceph

Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
# Conflicts:
# src/cls/rbd/cls_rbd.cc
# src/cls/rbd/cls_rbd_types.cc

Merge pull request #19099 from ifed01/wip-ifed-faster-pglog

osd: eliminate ineffective container operations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19180 from theanalyst/tools/rados-omap-clear

tools: rados add a cli option to clear omap keys

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19872 from tchaikov/wip-random_shuffle

osd: s/random_shuffle()/shuffle()/

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #19881 from liewegas/wip-kill-crush-location

ceph-crush-location: remove

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #11544 from VictorDenisov/consistency_groups_snapshots

librbd: group snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: fix valgrind false positive from new boost release

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19924 from rzarzynski/wip-rgw-drop-dump_uri_from_state

rgw: drop dump_uri_from_state() which isn't used anymore.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: ensure enums are prefixed w/ RBD_ to avoid collisions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix false-positive compiler warning

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/pybind: corrected failures due to test execution ordering

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: group snapshots should be linked to image pool (not data pool)

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: restore snap list by image id functionality

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: combined group_snap_add and group_snap_update

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix ImageCtx::get_snap_id method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fixed memory leak and use-after-free in group snap API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: utilize '<name>@<snap>' format when creating/removing group snaps

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd: added snap_get_group_namespace API method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added missing C API version for rbd_snap_get_group_namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: moved new snapshot API methods to separate class

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: wrap object section around group snap list formatter

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: translate pool id into name

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: translate group pool to name

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: fix cli tests

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: drop consistency from Info

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: produce state string instead of integer

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Add snap rm alias

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Change message format

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Inline options function

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Drop consistency from tools

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Fix section name

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

test/pybind: drop consistency from test name

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Drop consistency from help

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Drop consistency from pybind

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Fix indentation

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: use RBD_GROUP_INVALID_POOL

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Drop word consistency in librbd

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: remove get_snap_id_from_namespace

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: fixup Move function declaration

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: rename group state to 'incomplete'

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: initialize group state

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: Remove h_group_create

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: Address comments

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

qa/workunits/rbd: Add tests for consistency groups

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Add group snapshot command

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Add group snapshot operations

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: Add group snapshot operations to cls_rbd

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

Merge pull request #19919 from wjwithagen/wip-wjw-rbd-ggate-fix

rbd-ggate: fix syntax error.

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #19917 from yaozongyou/rgw-remove-unused-cls-user-add-bucket

rgw: remove unused cls_user_add_bucket

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19699 from qrGitHub/wip-rgw-useless-line

rgw: remove useless lines in RGWDeleteBucket::execute

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #19632 from Rubab-Syed/outreachy_mgr_dashboard

mgr/dashboard: Add monitor list

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19898 from tchaikov/wip-cmake-build-with-clang

cmake: BuildBoost.cmake: use specified compiler for building boost

Reviewed-By: Casey Bodley <cbodley@redhat.com>

rgw: drop dump_uri_from_state() which isn't used anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rbd-ggate: fix syntax error.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: remove unused cls_user_add_bucket

the function
`int cls_user_add_bucket(rgw_raw_obj& obj, list<cls_user_bucket_entry>& entries);`
is unsed and not implemented, so remove it.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #19879 from jcsp/wip-doc-dashboard-standby

doc: update mgr/dashboard doc about standbys

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19642 from xiexingguo/wip-pending-new-state

osd/OSDMap.h: toss osd out if it has no more pending states

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19901 from vico-research-and-consulting/master

mgr/zabbix: monitoring template improvements

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #18833 from yuyuyu101/wip-async-perf

msg/async/AsyncConnection: combine multi alloc into one

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18906 from ovh/bp-buffer-crc-improvements

common/buffer: switch crc cache to single pair instead of map

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19547 from tchaikov/wip-22344

ceph.in: print all matched commands if arg missing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19832 from shinobu-x/min_max_filestore

os: FileStore, Using stl min | max, MIN | MAX macros instead

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19874 from myoungwon/wip-fix-22641

osd: fix a valgrind issue (conditional jump depends on uninitialized value)

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/zabbix: monitoring template improvements

The changes need some review because i am currently not able to test this in a environment.
- added a host screen which displays all graphs in a convenient way
- replaced AVG by MAX Function in Graphs where maximum values are of
  high interest from operational view
- changed graph type to stacked where combined behavior is of interest
  (i.e. bandwidth and storage consumption)

Signed-off-by: Marc Schoechlin <ms@256bit.org>

include: Remove fallback usings from encoding.h

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test: Use unqualified encode/decode

This is a portion of Part 1 of the namespace project: using ADL
properly in encode and decode so we can use namespaces easily in Ceph.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

tools: Use unqualified encode/decode

This is a portion of Part 1 of the namespace project: using ADL
properly in encode and decode so we can use namespaces easily in Ceph.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Use unqualified encode/decode

This is a portion of Part 1 of the namespace project: using ADL
properly in encode and decode so we can use namespaces easily in Ceph.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd_replay: Use unqualified encode/decode

This is a portion of Part 1 of the namespace project: using ADL
properly in encode and decode so we can use namespaces easily in Ceph.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osdc: Use unqualified encode/decode

This is a portion of Part 1 of the namespace project: using ADL
properly in encode and decode so we can use namespaces easily in Ceph.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd: Use unqualified encode/decode

This is a portion of Part 1 of the namespace project: using ADL
properly in encode and decode so we can use namespaces easily in Ceph.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

os: Use unqualified encode/decode

This is a portion of Part 1 of the namespace project: using ADL
properly in encode and decode so we can use namespaces easily in Ceph.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

objclass: Use unqualified encode/decode

This is a portion of Part 1 of the namespace project: using ADL
properly in encode and decode so we can use namespaces easily in Ceph.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

msg: Use unqualified encode/decode

This is a portion of Part 1 of the namespace project: using ADL
properly in encode and decode so we can use namespaces easily in Ceph.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mon: Use unqualified encode/decode

This is a portion of Part 1 of the namespace project: using ADL
properly in encode and decode so we can use namespaces easily in Ceph.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>