rgw: sync: pipe rules handler

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate sync pipe params down the stack

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add option to configure sync pipe prefix, tags

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: extend data structures to extend bucket sync functionality

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: use hints when determining sync pipes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: resolve pipes by bucket sync hints

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync info shows and resolve bucket sync hints

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: manage bucket sync deps index

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: identify potential related (for sync) buckets on bucket update

Generate a diff off old and new bucket infos.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.zone: keep policy handler for all zones

So that we can use the correct policy handler when checking source bucket.
Also build list of source zones for data sync out of all potential zones,
and not just the ones that are enabled, because it can be enabled at the
bucket level.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: trigger per shard sync by source and target

and tie to data log sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: rework RGWGetBucketPeersCR

so that it returns pipes with bucket info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: more work towards new policy integration

RGWGetBucketPeersCR will be able to return correct buckets
that need to sync, either by a target bucket, and/or a source
zone and a source bucket (partially implemented).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: bucket sync manager adjustments for new system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: encoding fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: modify sync policy handler api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: bucket sync status changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix, cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync flow handler: integrate with flow manager

Convert zone names to zone ids, adjust interfaces.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync flow: init zone level sync flow manager

Hold the zone level sync flow manager in the zone svc. Convert old
zonegroup sync config into new sync policy structure.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy mgr: more fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy: cleanups and fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_bucket operator fixes

Fix smaller-than operator, ostream operator.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: flow manager generates explicit pipes

Have multiple pipes instead of having a single pipe that can describe
 multiple different zones have multiple pipes, each point at a single
zone (or all zones).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use single entity in expanded sync determination

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rename some datastructures to signify plurality

pipe -> pipes, entity -> entities

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket flow manager initialization changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add 'sync info' command

Also move code into header so that flow manager could be used in radosgw-admin

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: manage sync policy at the bucket level

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: code cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group pipe create and remove

also rename sync group flow add to sync group flow create.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group get command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group flow remove command

and also some code cleanup.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group flow add command

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync, change "target" to "dest"

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: json dump sync groups map as vector

looks nicer

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync group create/remove commands

Still incomplete

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy logic

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: redo policy definitions

Still incomplete

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: compilation fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: sync policy related commands

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add sync policy to zonegroup

and move code around

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: bucket sync info command

Dumps bucket sources

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pass through bucket sync pairs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define bucket sync pipes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: read local bucket sources

And also add functionality to get bucket sync policy handler (in cr).
Next we'll need to compare local bucket sources with what policy
handler defines, and act if it's different.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: defined bucket sync sources manager

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: split RGWDataSyncEnv

Keep only the environment related fields in there, and have the
sync info related stuff in the containing RGWDataSyncCtx.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: reduce use of sync_env->store

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: rename current bucket sync crs

Add the 'pipe' term as bucket can be synced on multiple pipes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ctl.bucket: add bucket sync related methods

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.bucket_sync: add new svc for dealing with bucket sync policy

svc.bucket_sync_sobj keeps a cache of bucket sync policies (per bucket
instance), and chains to the metadata cache.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework sync policy

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: define sync policy structures

Defines sync policy for both zones and buckets.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused variable

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: separate source and dest bucket

This is still incomplete. The idea is that source bucket and destination
bucket do not need to reflect the same entity.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: integrate bucket sync decisions

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: store data log entry if bucket sync policy not empty

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket-granularity sync: define policy structure

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: test-multi: craft admin command line options differently

The problem was specifically with the "--rgw-cache-enabled false" param. This
doesn't work like that anymore, need to either pass --rgw-cache-enabled=false,
or --no-rgw-cache-enabled.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: dump command if not found

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: split remove and delete aliases

Since "zonegroup remove" and "zonegroup delete" are different commands,
don't have them share the same aliases.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add "zonegroup remove zone" command

Does the same as "zonegroup remove", but should be used instead of it
as we also have a different "zonegroup delete" command.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: cmdline, dump expected commands on error

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: adjust commands

Add missing definitions needed for backward compatibility.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: use strongly typed enum for cmdline options

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: cmdline, use generic std::any to hold value

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: put cmd parser in its own class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: command line aliases

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: remove old manual command parsing code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: new command parsing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #32929 from jan--f/wip-43844-master

ceph-volume: batch bluestore fix create_lvs call

Merge pull request #32836 from s0nea/wip-dashboard-boolean-settings

mgr/dashboard: show checkboxes for booleans

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: extend doc by Grafana setting (#32460)

mgr/dashboard: extend doc by Grafana setting

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

mgr/dashboard: disable 'Add Capability' button in rgw user edi… (#32610)

mgr/dashboard: disable 'Add Capability' button in rgw user edit form.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume: batch bluestore fix create_lvs call

Fixes: https://tracker.ceph.com/issues/43844
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #32925 into master

* refs/pull/32925/head:
os/bluestore: fix duplicate onode instance while doing omap repair.
test/store_test: introduce test case for per-pool omap repair.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32850 into master

* refs/pull/32850/head:
osdc/Objecter: debug pause/unpause transition

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32752 into master

* refs/pull/32752/head:
cephadm: remove `logs` command

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32920 into master

* refs/pull/32920/head:
qa/suites/rados/rest: run restful test on el8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

os/bluestore: fix duplicate onode instance while doing omap repair.

Fixes: https://tracker.ceph.com/issues/43824
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/store_test: introduce test case for per-pool omap repair.

This also reproduces
https://tracker.ceph.com/issues/43824

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #32812 into master

* refs/pull/32812/head:
mgr/cephadm: get --osd-uuid deploy arg from osdmap
mgr/cephadm: fetch keyring in _create_daemon()

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32885 into master

* refs/pull/32885/head:
qa/suites/rados/thrash: debug monc

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #32886 into master

* refs/pull/32886/head:
cephadm: no --no-systemd arg to ceph-volume deactivate

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32851 into master

* refs/pull/32851/head:
qa/standalone/osd/osd-backfill-recovery-log.sh: fix TEST_backfill_log_2
qa/standalone/osd/osd-backfill-recovery-log.sh: fix TEST_backfill_log_1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

qa/suites/rados/rest: run restful test on el8

For some reason the requests library has trouble connecting from
ubuntu 18.04.  I reproduced this locally on my 18.04 desktop, although
there it fails on the first API request instead of the last (as in QA).

In any case, this appears to be a client library problem.

Fixes: https://tracker.ceph.com/issues/43720
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32898 into master

* refs/pull/32898/head:
qa/suites/rados/singleton/all/recovery-preemption: fix pg log length

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32617 from taodd/bug-43563

rgw: update the hash source for multipart entries during resharding

Reviewed-by: Eric J. Ivancich <ivancich@redhat.com>

Merge pull request #32487 from ofriedma/wip-ofriedma-copy-part-hdr

rgw: Fix upload part copy range able to get almost any string

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #31887 from ofriedma/wip-ofriedma-lc-hdr-mid

rgw: LC expiration header should present midnight expiration date

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #30778 from joke-lee/rgw-ipv6

rgw: fix SignatureDoesNotMatch when use ipv6 address in s3 client

Reviewed-By: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #31977 from ofriedma/wip-ofriedma-dynamic-disable

rgw: Fix dynamic resharding not working for empty zonegroup in period

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #31185 from joke-lee/rgw-user-modify-placement

rgw: support specify user default placement and placement_tags when create or modify user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32235 from jan--f/c-v-prepare-size-args

ceph-volume: add sizing arguments to prepare

mgr/dashboard: unify indentation in dashboard.rst

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

cephadm: remove `logs` command

Fixes: https://tracker.ceph.com/issues/43618
Signed-off-by: Michael Fritch <mfritch@suse.com>