osd/ECBackend: debug ec scrub error paths

Signed-off-by: Sage Weil <sage@redhat.com>

osd: document scrub options

Signed-off-by: Sage Weil <sage@redhat.com>

osd: allow limited scrub preemption

If we receive a write within the scrub range, abort the scrub chunk and
shrink the chunk size.  If we do this too many times do not preempt and
allow the scrub to complete (to avoid scrub starvation due to client io).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: piecewise scrub

Perform scrub in stages, with each unit of work requeuing an item in the
work queue.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: flush osr before listing scrub objects

We need to make sure previous transactions are readable before we list
objects.  This avoids races like, say,

- ecbackend rolls forward and removes a rollback object
- bluestore delays the txn apply due to some prior op
- scrub lists objects, sees rollback object it doesn't expect

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ECBackend: turn be_deep_scrub options into legacy

We don't have a lightweight mechanism for doing trivial config options
that is better than legacy_config_opts.h yet.  Until then,

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: disable osd_debug_deep_scrub_sleep in case it is set

Otherwise the final scrub may take too long.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/*Backend: debug: inject sleep during deep scrub

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PG: drop waiting_on, use waiting_on_whom

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #12431 from VictorDenisov/snapshot_rename

rbd: group snapshot rename

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: Add group snapshot rename commands

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Add group snap rename method

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

Merge pull request #19913 from adamemerson/wip-minimax-rados

rados: Prefer templates to macros

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19914 from tchaikov/wip-cmake-check-boost-in-src

cmake: check version of boost in src/boost

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19926 from batrick/i22527

mon: revert mds metadata argument name change

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19903 from dillaman/wip-librbd-op-features

librbd: add preliminary support for new operation feature bit

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #19793 from badone/wip-blocking-deep-scrub

ceph.in: Add blocking mode for scrub and deep-scrub

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19910 from jdurgin/wip-21897

config: lower default omap entries recovered at once

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #19894 into master

* refs/pull/19894/head:
cmake: link against ceph-common instead of common

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19920 into master

* refs/pull/19920/head:
client: fix cap revoke race
client: release revoking Fc after invalidate cache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #19885 into master

* refs/pull/19885/head:
qa: increase osd count for ec testing
qa: add missing openstack configs

Reviewed-by: Sage Weil <sage@redhat.com>

rados: Switch MIN/MAX for std::min/max and use intarith templates

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #19900 from dillaman/wip-rbd-lock-list

rbd: adjusted "lock list" JSON and XML formatted output

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #19933 from qrGitHub/wip-rgw-useless-line4

rgw: drop redundant RGW_OP_STAT_OBJ check

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rbd: adjusted "lock list" JSON and XML formatted output

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: operations should return -EROFS if disabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: retrieve the op features on image refresh

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: track new operation features within image

This will initially be utilized to restrict older clients from
performing operations against an image if (1) it doesn't
support the new feature bit, or (2) doesn't support the specific
enabled op feature.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

include: Add templates along side macros in intarith

The macros can be removed later.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: drop redundant RGW_OP_STAT_OBJ check

Signed-off-by: Bingyin Zhang <zhangbingyin@cloudin.cn>

doc/man: Add '--block' option for ceph command

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #19886 from liu-chunmei/fix_dpdk_worker_references_issue

osd: fix dpdk worker references issue

Reviewed-by: Haomai Wang <haomai@xsky.com>

rbd: hide implicit / non-mutable features from the CLI help

The layering and data-pool features are enabled automatically
when striping or a data pool is specified, respectively. For
the 'rbd feature XYZ' command, hide all non-mutable features.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: revert mds metadata argument name change

Partial revert of PR #18512: d1877b68d90163897245a9801cff0ea419ff28e4

Fixes: http://tracker.ceph.com/issues/22527
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge branch 'wip-names!-in!-space!' of git://github.com/adamemerson/ceph

Reviewed-by: Jesse Williamson <jwilliamson@suse.de>
# Conflicts:
# src/cls/rbd/cls_rbd.cc
# src/cls/rbd/cls_rbd_types.cc

Merge pull request #19099 from ifed01/wip-ifed-faster-pglog

osd: eliminate ineffective container operations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19180 from theanalyst/tools/rados-omap-clear

tools: rados add a cli option to clear omap keys

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19872 from tchaikov/wip-random_shuffle

osd: s/random_shuffle()/shuffle()/

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #19881 from liewegas/wip-kill-crush-location

ceph-crush-location: remove

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #11544 from VictorDenisov/consistency_groups_snapshots

librbd: group snapshots

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: fix valgrind false positive from new boost release

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #19924 from rzarzynski/wip-rgw-drop-dump_uri_from_state

rgw: drop dump_uri_from_state() which isn't used anymore.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: ensure enums are prefixed w/ RBD_ to avoid collisions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix false-positive compiler warning

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/pybind: corrected failures due to test execution ordering

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: group snapshots should be linked to image pool (not data pool)

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: restore snap list by image id functionality

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: combined group_snap_add and group_snap_update

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fix ImageCtx::get_snap_id method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: fixed memory leak and use-after-free in group snap API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: utilize '<name>@<snap>' format when creating/removing group snaps

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

pybind/rbd: added snap_get_group_namespace API method

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added missing C API version for rbd_snap_get_group_namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: moved new snapshot API methods to separate class

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: wrap object section around group snap list formatter

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: translate pool id into name

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: translate group pool to name

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: fix cli tests

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: drop consistency from Info

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: produce state string instead of integer

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Add snap rm alias

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Change message format

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Inline options function

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Drop consistency from tools

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Fix section name

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

test/pybind: drop consistency from test name

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Drop consistency from help

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Drop consistency from pybind

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Fix indentation

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: use RBD_GROUP_INVALID_POOL

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Drop word consistency in librbd

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: remove get_snap_id_from_namespace

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: fixup Move function declaration

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: rename group state to 'incomplete'

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: initialize group state

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: Remove h_group_create

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: Address comments

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

qa/workunits/rbd: Add tests for consistency groups

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

rbd: Add group snapshot command

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

librbd: Add group snapshot operations

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

cls_rbd: Add group snapshot operations to cls_rbd

Signed-off-by: Victor Denisov <denisovenator@gmail.com>

Merge pull request #19919 from wjwithagen/wip-wjw-rbd-ggate-fix

rbd-ggate: fix syntax error.

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #19917 from yaozongyou/rgw-remove-unused-cls-user-add-bucket

rgw: remove unused cls_user_add_bucket

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19699 from qrGitHub/wip-rgw-useless-line

rgw: remove useless lines in RGWDeleteBucket::execute

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #19632 from Rubab-Syed/outreachy_mgr_dashboard

mgr/dashboard: Add monitor list

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19898 from tchaikov/wip-cmake-build-with-clang

cmake: BuildBoost.cmake: use specified compiler for building boost

Reviewed-By: Casey Bodley <cbodley@redhat.com>

rgw: drop dump_uri_from_state() which isn't used anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

client: fix cap revoke race

If caps are been revoking by the auth MDS, don't consider them as
issued even they are still issued by non-auth MDS. The non-auth
MDS should also be revoking/exporting these caps, the client just
hasn't received the cap revoke/export message.

The race I encountered is: When caps are exporting to new MDS, the
client receives cap import message and cap revoke message from the
new MDS, then receives cap export message from the old MDS. When
the client receives cap revoke message from the new MDS, the revoking
caps are still issued by the old MDS, so the client does nothing.
Later when the cap export message is received, the client removes
the caps issued by the old MDS. (Another way to fix the race is
calling ceph_check_caps() in handle_cap_export())

copid from kernel commit b1530f57

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

client: release revoking Fc after invalidate cache

this issue is likely revealed by commit 7bc1fa04
"client: avoid adding checked inode to the delayed_caps"

Fixes: http://tracker.ceph.com/issues/22652
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

rbd-ggate: fix syntax error.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: remove unused cls_user_add_bucket

the function
`int cls_user_add_bucket(rgw_raw_obj& obj, list<cls_user_bucket_entry>& entries);`
is unsed and not implemented, so remove it.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #19879 from jcsp/wip-doc-dashboard-standby

doc: update mgr/dashboard doc about standbys

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: check version of boost in src/boost

after bumping up the required boost version, old copy of boost in
src/boost will not work. so we need to error out early.

Signed-off-by: Kefu Chai <kchai@redhat.com>

config: lower default omap entries recovered at once

For large omap DBs, reading 64k leads to heartbeat timeouts.  There
are numerous callchains leading to this recovery step, many of which
do not have heartbeat handles, so for an easily backported version
just change the default number of entries read. DBs approaching 100GB
may require an even lower setting, but this should be good enough for
most clusters, without sacrificing recovery speed.

Fixes: http://tracker.ceph.com/issues/21897
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19642 from xiexingguo/wip-pending-new-state

osd/OSDMap.h: toss osd out if it has no more pending states

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19901 from vico-research-and-consulting/master

mgr/zabbix: monitoring template improvements

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #18833 from yuyuyu101/wip-async-perf

msg/async/AsyncConnection: combine multi alloc into one

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18906 from ovh/bp-buffer-crc-improvements

common/buffer: switch crc cache to single pair instead of map

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #19547 from tchaikov/wip-22344

ceph.in: print all matched commands if arg missing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #19832 from shinobu-x/min_max_filestore

os: FileStore, Using stl min | max, MIN | MAX macros instead

Reviewed-by: Sage Weil <sage@redhat.com>