rgw-admin: bucket sync info command

Dumps bucket sources

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pass through bucket sync pairs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define bucket sync pipes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: read local bucket sources

And also add functionality to get bucket sync policy handler (in cr).
Next we'll need to compare local bucket sources with what policy
handler defines, and act if it's different.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: defined bucket sync sources manager

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: split RGWDataSyncEnv

Keep only the environment related fields in there, and have the
sync info related stuff in the containing RGWDataSyncCtx.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reduce use of sync_env->store

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: rename current bucket sync crs

Add the 'pipe' term as bucket can be synced on multiple pipes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ctl.bucket: add bucket sync related methods

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.bucket_sync: add new svc for dealing with bucket sync policy

svc.bucket_sync_sobj keeps a cache of bucket sync policies (per bucket
instance), and chains to the metadata cache.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework sync policy

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define sync policy structures

Defines sync policy for both zones and buckets.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused variable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: separate source and dest bucket

This is still incomplete. The idea is that source bucket and destination
bucket do not need to reflect the same entity.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: integrate bucket sync decisions

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: store data log entry if bucket sync policy not empty

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket-granularity sync: define policy structure

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: test-multi: craft admin command line options differently

The problem was specifically with the "--rgw-cache-enabled false" param. This
doesn't work like that anymore, need to either pass --rgw-cache-enabled=false,
or --no-rgw-cache-enabled.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: dump command if not found

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: split remove and delete aliases

Since "zonegroup remove" and "zonegroup delete" are different commands,
don't have them share the same aliases.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add "zonegroup remove zone" command

Does the same as "zonegroup remove", but should be used instead of it
as we also have a different "zonegroup delete" command.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: cmdline, dump expected commands on error

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: adjust commands

Add missing definitions needed for backward compatibility.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: use strongly typed enum for cmdline options

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: cmdline, use generic std::any to hold value

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: put cmd parser in its own class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: command line aliases

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: remove old manual command parsing code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: new command parsing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #32929 from jan--f/wip-43844-master

ceph-volume: batch bluestore fix create_lvs call

Merge pull request #32836 from s0nea/wip-dashboard-boolean-settings

mgr/dashboard: show checkboxes for booleans

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: extend doc by Grafana setting (#32460)

mgr/dashboard: extend doc by Grafana setting

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: disable 'Add Capability' button in rgw user edi… (#32610)

mgr/dashboard: disable 'Add Capability' button in rgw user edit form.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume: batch bluestore fix create_lvs call

Fixes: https://tracker.ceph.com/issues/43844
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #32925 into master

* refs/pull/32925/head:
os/bluestore: fix duplicate onode instance while doing omap repair.
test/store_test: introduce test case for per-pool omap repair.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32850 into master

* refs/pull/32850/head:
osdc/Objecter: debug pause/unpause transition

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32752 into master

* refs/pull/32752/head:
cephadm: remove `logs` command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32920 into master

* refs/pull/32920/head:
qa/suites/rados/rest: run restful test on el8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

os/bluestore: fix duplicate onode instance while doing omap repair.

Fixes: https://tracker.ceph.com/issues/43824
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/store_test: introduce test case for per-pool omap repair.

This also reproduces
https://tracker.ceph.com/issues/43824

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #32812 into master

* refs/pull/32812/head:
mgr/cephadm: get --osd-uuid deploy arg from osdmap
mgr/cephadm: fetch keyring in _create_daemon()

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32885 into master

* refs/pull/32885/head:
qa/suites/rados/thrash: debug monc

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32886 into master

* refs/pull/32886/head:
cephadm: no --no-systemd arg to ceph-volume deactivate

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32851 into master

* refs/pull/32851/head:
qa/standalone/osd/osd-backfill-recovery-log.sh: fix TEST_backfill_log_2
qa/standalone/osd/osd-backfill-recovery-log.sh: fix TEST_backfill_log_1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

qa/suites/rados/rest: run restful test on el8

For some reason the requests library has trouble connecting from
ubuntu 18.04.  I reproduced this locally on my 18.04 desktop, although
there it fails on the first API request instead of the last (as in QA).

In any case, this appears to be a client library problem.

Fixes: https://tracker.ceph.com/issues/43720
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32898 into master

* refs/pull/32898/head:
qa/suites/rados/singleton/all/recovery-preemption: fix pg log length

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32617 from taodd/bug-43563

rgw: update the hash source for multipart entries during resharding

Reviewed-by: Eric J. Ivancich <ivancich@redhat.com>

Merge pull request #32487 from ofriedma/wip-ofriedma-copy-part-hdr

rgw: Fix upload part copy range able to get almost any string

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #31887 from ofriedma/wip-ofriedma-lc-hdr-mid

rgw: LC expiration header should present midnight expiration date

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #30778 from joke-lee/rgw-ipv6

rgw: fix SignatureDoesNotMatch when use ipv6 address in s3 client

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #31977 from ofriedma/wip-ofriedma-dynamic-disable

rgw: Fix dynamic resharding not working for empty zonegroup in period

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #31185 from joke-lee/rgw-user-modify-placement

rgw: support specify user default placement and placement_tags when create or modify user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32235 from jan--f/c-v-prepare-size-args

ceph-volume: add sizing arguments to prepare

mgr/dashboard: unify indentation in dashboard.rst

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

cephadm: remove `logs` command

Fixes: https://tracker.ceph.com/issues/43618
Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/singleton/all/recovery-preemption: fix pg log length

This was broken by the variable PG log lengths in
9c69c2f7cc585b5e13e4d1b0432016d38135a3de.

Disable the new option to get (roughly) the old behavior, or at least the
short logs that we want to trigger some backfill.

Fixes: https://tracker.ceph.com/issues/43810
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32853 into master

* refs/pull/32853/head:
qa/suites/fs/upgrade: finish at octopus
qa/suites/fs/upgrade: set min-compat-client to octopus
qa/suites/fs/upgrade: set pg_autoscale_mode=off after upgrade

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32883 into master

* refs/pull/32883/head:
qa/workunits/rados/test_crash.sh: do not fail if coredump not found
rgw: set bucket quota only when necessary

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: show checkboxes for booleans

The frontend showed textboxes for the dashboard settings because
the actual type information was missing here. The REST API then
returned the default type 'str'.

Edit the e2e test case in order to update a different setting as
the 'editMgrModule' method can't handle checkboxes.

Fixes: https://tracker.ceph.com/issues/43769
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

monitoring: add details to Prometheus' alerts (#32803)

monitoring: add details to Prometheus' alerts

Reviewed-by: Andreas Jaeger <aj@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #32434 from trociny/wip-41559

mgr/rbd_support: implement mirror snapshot scheduler

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: optimize mirror_image_info_list (don't open image context)

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: make mirror get info request not require image context

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rados/test_crash.sh: do not fail if coredump not found

Fixes: https://tracker.ceph.com/issues/43653
Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: no --no-systemd arg to ceph-volume deactivate

ceph-volume lvm deactivate: error: unrecognized arguments: --no-systemd

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash: debug monc

Hunting https://tracker.ceph.com/issues/43825

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: set bucket quota only when necessary

this change fixes the regression introduced by 9a7f76a7

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rbd_support: optimize scanning images in mirror snapshot mode

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add API to list mirror image info

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/rbd_support: implement mirror snapshot scheduler

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/rbd_support: split module.py into several files

(it is becoming too large)

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32734 from trociny/wip-43589

librbd: don't call refresh from mirror::GetInfoRequest state machine

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32766 from rzarzynski/wip-common-ceph_atomic

crimson, common: introduce ceph::atomic and apply it on bufferlist.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32760 from trociny/wip-43666

librbd: fix potential snapshot remove failure due to duplicate RPC messages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32730 from tchaikov/wip-mgr-maybe-unused

mgr: better error handling when reading option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32758 from ifed01/wip-ifed-fix-legacy-omap

os/bluestore: upgrade legacy omap to per-pool format automatically.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32764 from liewegas/fix-43552

mon/Session: only index osd ids >= 0

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32753 from mgfritch/cephadm-tox-escape-seq

cephadm: fix tox DeprecationWarning

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32352 from rzarzynski/wip-bug-43070

msg/async: reset the V1's session_security in proper EventCenter.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32785 from rzarzynski/wip-encoding-no-bl-copy_in-2

common: encode for std::list<T> doesn't use bl::copy_in() anymore.

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/fs/upgrade: finish at octopus

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: get --osd-uuid deploy arg from osdmap

During the initial OSD creation, we had the osd_uuid handy.  However, on
reconfig/redeploy, we don't.

Refactor a bit so that this is extracted from the osdmap by the
_create_daemon() method.  While we're at it, make the deployment case
verify that the deployed OSD uuid matches the one in the osdmap.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/fs/upgrade: set min-compat-client to octopus

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30272 from ivancich/wip-move-some-bucket-listing-logic-to-cls

rgw: move delimiter-based bucket listing/filtering logic to cls

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/fs/upgrade: set pg_autoscale_mode=off after upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32750 into master

* refs/pull/32750/head:
qa/suites/upgrade/*-x/stress-split: run latest python tests at end
qa/tasks/ceph_manager: kludge around /var/log/ceph permissions
mgr/pg_autoscaler: don't check anything until cluster is nautilus
qa/suites/upgrade: install python3-* as part of final upgrade step
qa/tasks/radosbench: only set object size if != block size
qa/tasks/ceph: simplify mon_health_to_clog suppression during restart
cls_hello: alias write_return_data -> writes_dont_return_data
ceph_test_cls_hello: only do returndata test on octopus+
qa: remove unnecessary package excludes in upgrades/nautilus*
qa: exclude cephadm from upgrade/nautilus-x
qa/suites/upgrade/mimic-x/parallel: fix msgr2 vs octopus ordering
qa/suites/upgrade/nautilus-x/stress-split: fix msgr2 vs octopus ordering
qa/suites/upgrade/mimic-x: fix msgr2 vs octopus ordering
qa/suites/upgrade/nautilus-x: end on octopus
qa/suites/upgrade/mimic-x: finish at octopus
qa/suites/upgrade/nautilus-x: disable TOO_FEW_PGS warning
qa/tasks/ceph: set mon_health_to_clog=false via mon config
qa/suites/upgrade/mimic-x: disable TOO_FEW_PGS warning

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/standalone/osd/osd-backfill-recovery-log.sh: fix TEST_backfill_log_2

Fixes: https://tracker.ceph.com/issues/43807
Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/standalone/osd/osd-backfill-recovery-log.sh: fix TEST_backfill_log_1

Fixes: https://tracker.ceph.com/issues/43807
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #32804 into master

* refs/pull/32804/head:
pybind: Add test for listxattr
pybind: Add listxattr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32786 into master

* refs/pull/32786/head:
mon/ConfigMonitor: put global/ keys in global section
mon/ConfigMonitor: transition old keys to have global/ prefix
mon/ConfigMonitor: always prefix global config keys with global/
mon/ConfigMonitor: switch to use pending_cleanup boost::optional

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32665 into master

* refs/pull/32665/head:
os/bluestore: avoid race between split_cache and get/put pin/unpin
os/bluestore: remove no-op line from split_cache

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge PR #32755 into master

* refs/pull/32755/head:
pybind/mgr/*: fix config_notify handling of default values

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #32809 into master

* refs/pull/32809/head:
common/options: bluestore 64k min_alloc_size for HDD

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: let newer RGWs work with older OSDs re: cls-side filtering

Previous commits moved bucket list filtering when a delimiter was
specified to the osd/cls layer. However, since rgw's are often
upgraded before osd's are, until we reach verison ceph version 16, an
rgw cannot assume that the osd/cls did the filtering. This is
addressed in the following ways....

First rgw_cls_list_ret now indicates whether filtering was done on the
osd/cls side.

And second, the old filtering code in the rgw is maintained in
RGWRados::Bucket::List::list_objects_ordered, so it can still be
triggered when all osd's are not doing the filtering.

Once we reach ceph version 16, and there is no chance that the rgw is
working with a osd running "young" version 14 code, we can remove the
backward compatibility code in
RGWRados::Bucket::List::list_objects_ordered.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: add unit testing for delimiter filtering in cls

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: move some bucket index delimiter filtering logic to cls

Currently the cls code that returns bucket index entries does no
filtering when a delimiter is specified. This means that the cls code
will return many entries that the rgw code does not need. This means
unnecessary bucket index entries will be encoded, decoded, and consume
network bandwidth.

A change is made to send a delimiter, if it's specified, to the cls
code, so the filtering can be done there.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #32735 from trociny/wip-doc-mirror-snapshot

doc: document "mode" param for "rbd mirror image enable" command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: reformat some code to make it easier to read

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: rename search variable to better indicate semantics

cls_cxx_map_get_vals takes a parameter of where in the sequence of
keys to start returning values. In code it's often referred to as the
"start_key", but in fact it only returns values who have keys *after*
it. To clarify the semantics, rename the variable to "start_after_key"
in the bucket listing functions in both rgw and cls.

Also, declare a type RGWRados::check_filter_t, so the type does not
have to be fully literalized multiple times.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: change scope of bucket index entry flags

Redefine global constants that define flags in bucket index entries to
struct-scoped.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>