cls_hello: fix typo

This was mostly harmless but this makes the warning go away.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32623 into master

* refs/pull/32623/head:
ceph_release: octopus rc 15.1.0

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32828 from leseb/raw-db-wal

ceph-volume: add db and wal support to raw mode

Merge pull request #32941 from cbodley/wip-qa-rgw-multisite-pubsub-checkpoint

qa/rgw: multisite checkpoints consider pubsub zone

Merge PR #32943 into master

* refs/pull/32943/head:
qa/tasks/ceph_manager: fix chmod on log dir during pg export copy

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32932 from smanjara/add-bucket-chown-nonmaster

rgw/multisite: warn if bucket chown command is run on non-master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #32957 from tchaikov/wip-mgr-cephadm

mgr/cephadm: init attrs created by settattr()

Merge pull request #32893 from sebastian-philipp/orchestrator-document-client-usage

mgr/orchestrator: Add doc about how to use OrchestratorClientMixin

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #32939 into master

* refs/pull/32939/head:
mon/MgrMonitor.cc: add always_on_modules to the output of "ceph mgr module ls"

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: drop "type: ignore" annotations

since these options are already initialized in the constructor, there is
no need to ignore their types when checking those lines using mypy

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: init attrs created by settattr()

to address the test falure caused by mypy:
```
mypy run-test: commands[0] | mypy --config-file=../../mypy.ini ansible/module.py cephadm/module.py mgr_module.py mgr_util.py orchestrator.py orchestrator_cli/module.py rook/module.py
test_orchestrator/module.py
cephadm/module.py: note: In member "_check_for_strays" of class "CephadmOrchestrator":
cephadm/module.py:596: error: "CephadmOrchestrator" has no attribute "warn_on_stray_hosts"
cephadm/module.py:596: error: "CephadmOrchestrator" has no attribute "warn_on_stray_services"
cephadm/module.py:599: error: "CephadmOrchestrator" has no attribute "warn_on_stray_services"
Found 3 errors in 1 file (checked 8 source files)
```
see also https://github.com/python/mypy/issues/5719

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add bucket_chown to list of commands that are recommended to be run on master zone.

Fixes: https://tracker.ceph.com/issues/43884
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge PR #32823 into master

* refs/pull/32823/head:
common/bl: fix memory corruption in bufferlist::claim_append()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32940 into master

* refs/pull/32940/head:
qa: remove rados/basic/tasks/rgw_snaps.yml

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #32942 into master

* refs/pull/32942/head:
osd: dispatch_context and queue split finish on early bail-out

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31879 from joke-lee/rgw-sts-post-upload-auth

rgw: use STSEngine::authenticate when post upload with x_amz_security_token

Merge pull request #31214 from IlsooByun/lc_stale_head

rgw: prevent LC from reading stale head when transitioning object

ceph-volume: add db and wal support to raw mode

Using the raw mode, the OSD preparation can now be called with extra
arguments to specify a block device for either rocksdb db or wal.

The code expects the device to be available and no validation will be
performed. Also, the entire block device will be consumed.

The goal is run this on Kubernetes with Rook and OSD are running on PVC.
Users will request PVC from a certain size, the request will be
acknowledged and a block will be created, then attached to the machine.
Later on consumed by Rook's preparation job.

Signed-off-by: Sébastien Han <seb@redhat.com>

Merge pull request #31700 from rishabh-d-dave/cv-list-using-dash-S

ceph-volume: refactor listing.py

Merge pull request #32808 from theanalyst/rgw/test-iam-bits

test: modify iam tests to use a function to set bits

Reviewed-By: Adam Emersen <aemerson@redhat.com>
Reviewed-By: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: smart: add support for NVMe (#32708)

mgr/dashboard: smart: add support for NVMe

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #32938 from trociny/wip-fix-freebsd-build

librbd: fix build on freebsd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/rgw: multisite checkpoints consider pubsub zone

pubsub zones are configured to only sync_from the master zone, so bucket
checkpoints should ignore its sync status with respect to other zones

Fixes: https://tracker.ceph.com/issues/43768
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32502 into master

* refs/pull/32502/head:
pybind/mgr: Add ceph_module.pyi to improve type checking

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge PR #32903 into master

* refs/pull/32903/head:
mgr/devicehealth: fix telemetry stops sending device reports after 48 hours

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32365 into master

* refs/pull/32365/head:
mon: Don't put session during feature change

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32897 into master

* refs/pull/32897/head:
osd/PG: do not leak cluster message when there's no con

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31686 from yehudasa/wip-rgw-sync-bucket-6

rgw: bucket granularity sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/tasks/ceph_manager: fix chmod on log dir during pg export copy

With cephadm, we should chmod both /var/log/ceph and /var/log/ceph/$fsid.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: bucket sync: invalidate hints data if raced with another writer

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

osd: dispatch_context and queue split finish on early bail-out

If we bail out of advance_pg early because there is an upcoming merge, we
still need to dispatch_context() on rctx before we drop the PG lock.  And
the rctx that we submit needs to include the on_applied finisher comit
to call _finish_splits.

This is noticeable (at least) when there is a split and merge that are
both known.  When we process the split, the new child is added to new_pgs.
When we get to the merge epoch, we stop early and take the bail-out
path.

Fix by adding a dispatch_context call for this path.  And further make sure
that both dispatch_context callers in this function queue up the
new_pgs event.

Fixes: https://tracker.ceph.com/issues/43825
Signed-off-by: Sage Weil <sage@redhat.com>

qa: remove rados/basic/tasks/rgw_snaps.yml

rgw_snaps tasks should not be running in the rados suite.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: fix compilation after rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mon/MgrMonitor.cc: add always_on_modules to the output of "ceph mgr module ls"

Makes it easier to know what the always_on_modules for each release are, without
having to look at the code.

Signed-off-by: Neha Ojha <nojha@redhat.com>

rgw: don't include empty buckets in potential related buckets

Being used by the sync hint system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep sync_policy as optional member inside RGWBucketInfo

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move a few basic types into rgw_basic_types.{h,cc}

To help with the fight against circular dependencies

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: initiate bucket meta sync if can't find sync handler

If bucket sync handler returns -ENOENT, try to get bucket info. This can
trigger bucket meta sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trim bilog: send bucket param instead of source-bucket param

For backward compatibility, so that older rgws could handle it appropriately.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: don't allow replication api on legacy config

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

PendingReleaseNotes: rgw, bucket granularity

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clarify api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: get_hint_entities(), call read_bucket_info once

once per bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: no create param in find directional/symmetrical

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: don't use optional for some members

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix a potential leak

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove return std::move()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fetch_remote_obj() create default acls if not found

Create default acls if the original object did not have acl header.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove superfluous alias

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: doc fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: find first key with longest matching prefix

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: fix bucket sync markers for non-trivial case

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket autotrim: fetch aggregated bilog info of all bucket targets

Add a new option to the bilog status request that allows returning the
min status for all buckets in a zone that sync from a specific bucket.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket trim: only fetch status from relevant zones

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket sync: use list instead of vector

Keeping pointers into entries inside the container, vector might be
reallocated.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy doc

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_argparse: binary flag value can be separate arg

Resurrect old behavior where a binary flag could be detached from the
argument, e.g., "--rgw-cache-enabled false".

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: fix use of opt_storage_class

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa: s3a-hadoop: fix user creation

This breaks with the new more strict radosgw-admin command line parsing

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: source zone, dest zone converted to zone id

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pipe map in pipe_set should be multimap

Can hold multiple entries for the same pipe id (if flow defines
more than a single entry).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: flatten binary encoding for rgw_sync_pipe_params

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement s3 get bucket replication api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement s3 delete bucket replication api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: put bucket replication extension

Now able to set specific source and destination zones

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implement s3 put-bucket-replication api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix pipes zone wildcard match logic

and improve logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: bucket sync status related fixes and improvements

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use negative shard id when creating sync pair

negative shard id means the bucket has a single shard. This
is to be backward compatible.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unneeded #warnings

and other cleanups

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: only update data log if bucket exports data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pubsub: user source bucket owner for event, not dest

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove a #warning with a comment

It's a nice to have, not crucial

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: object sync info oid depends on pipe

The object sync info object (used in cloud sync module) depends on pipe. Different
oid is crafted if source and destination buckets are not the same.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: guard against racing source object change

Source object might have changed since we tried to sync it,
and the decisions we made originally might be irrelevant.
After reading object header, check if the original dest params
are the same dest params we find now for this object, otherwise
need to retry.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: override dest object owner if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: check user permission in user level sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: auth: api to create auth identity without req_state

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api to check bucket and object permissions without need for req_state

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: user mode sync pipes fetch objects using effective uid

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: prepare for system and user mode changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: handle preconfition failed error

Handle precondition failed error. This would happen in case we cannot
sync the object due to either newer local changes, or if the policy
doesn't allow syncing the specific object. Handle this as success, but
log a message.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: http client: propagate errors in client callbacks

If one of the callbacks return error, stop the processing and set that
as the request error.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: filter fetch remote obj after reading remote metadata

In the sync case: find the appropriate pipe params that match this
remote object (if matches tags / prefix), and adjust fetch if needed --
abort if doesn't match tags. Later will adjust acls as needed (if user
sync and not system).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: find pipe rules param for specific obj changes

 - reuse RGWObjTags::tag_map_t instead of vector of strings
 - fix the prefix search

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: add mode (system, user) as a pipe param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add delimiter to tags parsing

Otherwise it will use '=' as a delimiter, and need to preserve it as
part of the tag initially.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: minor message fix

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync policy: reorganize a bit

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: json encoding filter

Can create and register a filter (that will be pointed at by the formatter
as an external feature), that could be then used to replace default
encoders for specific data types. This can be useful in cases where
we want to mutate the output (for example: translate zone ids to zone
names on the fly).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, radosgw-admin: explicit zone id type

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: resolve hints when initializing policy handler

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: modify zone_trace for bucket granularity info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data sync: incremental prefixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: full sync checks for rules prefixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use pipe handlers instead of pipes off pipe_set

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: pipe rules handler

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate sync pipe params down the stack

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: add option to configure sync pipe prefix, tags

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sync: extend data structures to extend bucket sync functionality

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>