rgw: move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy: cleanups and fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_bucket operator fixes

Fix smaller-than operator, ostream operator.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: flow manager generates explicit pipes

Have multiple pipes instead of having a single pipe that can describe
 multiple different zones have multiple pipes, each point at a single
zone (or all zones).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use single entity in expanded sync determination

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rename some datastructures to signify plurality

pipe -> pipes, entity -> entities

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket flow manager initialization changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add 'sync info' command

Also move code into header so that flow manager could be used in radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: manage sync policy at the bucket level

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: code cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group pipe create and remove

also rename sync group flow add to sync group flow create.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group get command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group flow remove command

and also some code cleanup.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group flow add command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync, change "target" to "dest"

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: json dump sync groups map as vector

looks nicer

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group create/remove commands

Still incomplete

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy logic

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: redo policy definitions

Still incomplete

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: compilation fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync policy related commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add sync policy to zonegroup

and move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: bucket sync info command

Dumps bucket sources

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pass through bucket sync pairs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define bucket sync pipes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: read local bucket sources

And also add functionality to get bucket sync policy handler (in cr).
Next we'll need to compare local bucket sources with what policy
handler defines, and act if it's different.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: defined bucket sync sources manager

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: split RGWDataSyncEnv

Keep only the environment related fields in there, and have the
sync info related stuff in the containing RGWDataSyncCtx.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reduce use of sync_env->store

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: rename current bucket sync crs

Add the 'pipe' term as bucket can be synced on multiple pipes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ctl.bucket: add bucket sync related methods

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.bucket_sync: add new svc for dealing with bucket sync policy

svc.bucket_sync_sobj keeps a cache of bucket sync policies (per bucket
instance), and chains to the metadata cache.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework sync policy

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define sync policy structures

Defines sync policy for both zones and buckets.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused variable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: separate source and dest bucket

This is still incomplete. The idea is that source bucket and destination
bucket do not need to reflect the same entity.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: integrate bucket sync decisions

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: store data log entry if bucket sync policy not empty

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket-granularity sync: define policy structure

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: test-multi: craft admin command line options differently

The problem was specifically with the "--rgw-cache-enabled false" param. This
doesn't work like that anymore, need to either pass --rgw-cache-enabled=false,
or --no-rgw-cache-enabled.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: dump command if not found

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: split remove and delete aliases

Since "zonegroup remove" and "zonegroup delete" are different commands,
don't have them share the same aliases.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add "zonegroup remove zone" command

Does the same as "zonegroup remove", but should be used instead of it
as we also have a different "zonegroup delete" command.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: cmdline, dump expected commands on error

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: adjust commands

Add missing definitions needed for backward compatibility.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: use strongly typed enum for cmdline options

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: cmdline, use generic std::any to hold value

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: put cmd parser in its own class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: command line aliases

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: remove old manual command parsing code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: new command parsing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #32929 from jan--f/wip-43844-master

ceph-volume: batch bluestore fix create_lvs call

Merge pull request #32836 from s0nea/wip-dashboard-boolean-settings

mgr/dashboard: show checkboxes for booleans

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: extend doc by Grafana setting (#32460)

mgr/dashboard: extend doc by Grafana setting

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: disable 'Add Capability' button in rgw user edi… (#32610)

mgr/dashboard: disable 'Add Capability' button in rgw user edit form.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume: batch bluestore fix create_lvs call

Fixes: https://tracker.ceph.com/issues/43844
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #32925 into master

* refs/pull/32925/head:
os/bluestore: fix duplicate onode instance while doing omap repair.
test/store_test: introduce test case for per-pool omap repair.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32850 into master

* refs/pull/32850/head:
osdc/Objecter: debug pause/unpause transition

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32752 into master

* refs/pull/32752/head:
cephadm: remove `logs` command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32920 into master

* refs/pull/32920/head:
qa/suites/rados/rest: run restful test on el8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

os/bluestore: fix duplicate onode instance while doing omap repair.

Fixes: https://tracker.ceph.com/issues/43824
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/store_test: introduce test case for per-pool omap repair.

This also reproduces
https://tracker.ceph.com/issues/43824

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #32812 into master

* refs/pull/32812/head:
mgr/cephadm: get --osd-uuid deploy arg from osdmap
mgr/cephadm: fetch keyring in _create_daemon()

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32885 into master

* refs/pull/32885/head:
qa/suites/rados/thrash: debug monc

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32886 into master

* refs/pull/32886/head:
cephadm: no --no-systemd arg to ceph-volume deactivate

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32851 into master

* refs/pull/32851/head:
qa/standalone/osd/osd-backfill-recovery-log.sh: fix TEST_backfill_log_2
qa/standalone/osd/osd-backfill-recovery-log.sh: fix TEST_backfill_log_1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

qa/suites/rados/rest: run restful test on el8

For some reason the requests library has trouble connecting from
ubuntu 18.04.  I reproduced this locally on my 18.04 desktop, although
there it fails on the first API request instead of the last (as in QA).

In any case, this appears to be a client library problem.

Fixes: https://tracker.ceph.com/issues/43720
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32898 into master

* refs/pull/32898/head:
qa/suites/rados/singleton/all/recovery-preemption: fix pg log length

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32617 from taodd/bug-43563

rgw: update the hash source for multipart entries during resharding

Reviewed-by: Eric J. Ivancich <ivancich@redhat.com>

Merge pull request #32487 from ofriedma/wip-ofriedma-copy-part-hdr

rgw: Fix upload part copy range able to get almost any string

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #31887 from ofriedma/wip-ofriedma-lc-hdr-mid

rgw: LC expiration header should present midnight expiration date

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #30778 from joke-lee/rgw-ipv6

rgw: fix SignatureDoesNotMatch when use ipv6 address in s3 client

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #31977 from ofriedma/wip-ofriedma-dynamic-disable

rgw: Fix dynamic resharding not working for empty zonegroup in period

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #31185 from joke-lee/rgw-user-modify-placement

rgw: support specify user default placement and placement_tags when create or modify user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32235 from jan--f/c-v-prepare-size-args

ceph-volume: add sizing arguments to prepare

mgr/dashboard: unify indentation in dashboard.rst

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

cephadm: remove `logs` command

Fixes: https://tracker.ceph.com/issues/43618
Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/singleton/all/recovery-preemption: fix pg log length

This was broken by the variable PG log lengths in
9c69c2f7cc585b5e13e4d1b0432016d38135a3de.

Disable the new option to get (roughly) the old behavior, or at least the
short logs that we want to trigger some backfill.

Fixes: https://tracker.ceph.com/issues/43810
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32853 into master

* refs/pull/32853/head:
qa/suites/fs/upgrade: finish at octopus
qa/suites/fs/upgrade: set min-compat-client to octopus
qa/suites/fs/upgrade: set pg_autoscale_mode=off after upgrade

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32883 into master

* refs/pull/32883/head:
qa/workunits/rados/test_crash.sh: do not fail if coredump not found
rgw: set bucket quota only when necessary

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: show checkboxes for booleans

The frontend showed textboxes for the dashboard settings because
the actual type information was missing here. The REST API then
returned the default type 'str'.

Edit the e2e test case in order to update a different setting as
the 'editMgrModule' method can't handle checkboxes.

Fixes: https://tracker.ceph.com/issues/43769
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

monitoring: add details to Prometheus' alerts (#32803)

monitoring: add details to Prometheus' alerts

Reviewed-by: Andreas Jaeger <aj@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #32434 from trociny/wip-41559

mgr/rbd_support: implement mirror snapshot scheduler

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: optimize mirror_image_info_list (don't open image context)

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: make mirror get info request not require image context

Signed-off-by: Mykola Golub <mgolub@suse.com>

qa/workunits/rados/test_crash.sh: do not fail if coredump not found

Fixes: https://tracker.ceph.com/issues/43653
Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: no --no-systemd arg to ceph-volume deactivate

ceph-volume lvm deactivate: error: unrecognized arguments: --no-systemd

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/thrash: debug monc

Hunting https://tracker.ceph.com/issues/43825

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: set bucket quota only when necessary

this change fixes the regression introduced by 9a7f76a7

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rbd_support: optimize scanning images in mirror snapshot mode

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: add API to list mirror image info

Signed-off-by: Mykola Golub <mgolub@suse.com>

mgr/rbd_support: implement mirror snapshot scheduler

Signed-off-by: Mykola Golub <mgolub@suse.com>

pybind/mgr/rbd_support: split module.py into several files

(it is becoming too large)

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #32734 from trociny/wip-43589

librbd: don't call refresh from mirror::GetInfoRequest state machine

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32766 from rzarzynski/wip-common-ceph_atomic

crimson, common: introduce ceph::atomic and apply it on bufferlist.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #32760 from trociny/wip-43666

librbd: fix potential snapshot remove failure due to duplicate RPC messages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32730 from tchaikov/wip-mgr-maybe-unused

mgr: better error handling when reading option

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32758 from ifed01/wip-ifed-fix-legacy-omap

os/bluestore: upgrade legacy omap to per-pool format automatically.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #32764 from liewegas/fix-43552

mon/Session: only index osd ids >= 0

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>