rgw: fix leak of metadata backend context

Op had no virtual destructor, so the derived Op_ManagedCtx destructor
wasn't being called to free its context. just use Op_ManagedCtx directly

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: user service removes old access keys in remove_old_indexes

Signed-off-by: Casey Bodley <cbodley@redhat.com>

common: remove ceph::optional_ref_default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: simplify RGWSI_RADOS params with default arguments

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: simplify RGWOTPCtl params with default arguments

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: simplify RGWBucketCtl params with default arguments

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: simplify RGWUserCtl params with default arguments

Params types needed to define a non-default constructor as a workaround
for https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88165

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make optional_yield explicit in RGWUserCtl interfaces

the purpose of null_yield is to highlight synchronous function calls so
they can be found with grep. having optional_yield parameters or member
variables that default to null_yield obscures these synchronous calls,
so optional_yield arguments should be an explicit part of interfaces

moved optional_yield out of RGWUserCtl's various Param types, and
updated all callers to pass in null_yield (except for a few cases where
the caller had access to a valid one)

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/static_ptr.h: add include guard

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_meta_be: add missing header

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common/ptr_wrapper: a pointer wrapper with id

Can be used to enforce strong typing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

common: add optional_ref_default<>

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.mdlog: init oldest log period only if we're syncing

Need to diffrerentiate between need to sync and actually syncing. Zone might need
to sync, but we're running in radosgw-admin and we don't actually sync.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: list_prefixed_objs() callback: don't take substring

We already get the substring as input, don't do it again.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_bucket_parse_bucket_instance returns bucket instance and not key

rgw_bucket_parse_bucket_instance() was returning a meta key, e.g., bucket:instance
and not just 'instance'. This caused issue when using read_bucket_instance_info().
Adjusted other users of this function, now that api changed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.user: call meta_be->put() and not ->put_entry()

put_entry() doesn't write to mdlog.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use period object for period history init, explicit constructor

Was passing the period id (string) which triggered implicit construction
of the period object. Instead pass the period object itself and make
constructor explicit to avoid future similar issues.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: relax bucket meta instance parsing

allow both [tenant/]bucket:instance and [tenant:]bucket:instance

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a few initialization fixes

Was passing wrong variable in constructor initialization, renamed
handler member name to avoid future confusion.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: silence log message

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove ep_objv field from RGWBucketInfo

This field was only initialized if we also read the bucket entrypoint,
which is not always the case. Added ep objv_tracker field on req_state
instead, and changed logic in delete_bucket() to make sure we do
the right thing.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: binfo_cache: keep entries by both bucket name and bucket instance

So that requests to read_bucket_instance_info() could be satisfied through
cache.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unneeded warning

No need to pass mtime there

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add missing cache_info param

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add pool to uinfo cache keys

To differentiate between the different uinfo cache indexes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused field from struct rgw_bucket

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: code cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: propagate sysobj_ctx to required bucket meta calls

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.meta_be: propagate backend ctx params

This is needed so that we can provide a backend-specific params from
the top level. For example in the current case we need in some cases
to be able to provide a sysobj_ctx that will be used instead of a
newly generated one. The layers in between don't need to know about
the backend specifics.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta: remove #warning

No need for fetching attrs, these could be read via the get() if meta
object has pattrs set (happens in the RGWBucketEntryMetadataObject case).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix user_ctl->read_info_by_access_key()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more metadata related initialization fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: metadata handlers init fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialization fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: unitests and other tools fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: api fixes

rgw-admin now compiles

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: resurrect metadata mutate

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: resurrect metadata mutate

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more api fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api fixes and adjustments

radosgw compiles

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta handler provides shard id info of meta key

In the future we should maybe look into providing this info differently,
and without exposing data representation in this layer, but for now
it will do.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWBucket::link() update

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

svc: add ctl.user->read_stats_async() and use it instead of old api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix pool initialization

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use stl.user->read_stats() instead of cls_user_get_header()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.datalog_rados: add new svc, move data_log out of RGWRados

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_rados.cc mostly compiles

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more api adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.bilog: move more functionality in

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add svc.bilog_rados

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.bucket: adding missing files

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.bucket: split into svc.bucket, svc.bucket_sobj

separate abstraction and implementation for clearer intention

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following api changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ctl.bucket: add read_bucket_info()

if bucket_id is not specified then try to read entrypoint first.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move bucket overwrite code to common path

In order to do that needed to make sure we have the old_info
and identify overwrites. Moved the bilog and data log related overwrite
handlers into svc.bucket, and also the bucket meta handler now
implements put_checked() and not using the generic sobj call.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update otp meta handler, add a new otp svc

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.be: don't alloc get params, use lambda

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWRados::lock_exclusive, ::unlock now being handled at svc.cls

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fixes following api changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: timelog goes into svc.cls.timelog

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move async rados processor insnace into svc.rados

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta_be, bucket: meta be context can be strongly typed

When passing around the ctx for higher level non-generic functionality
it is better to have a strongly typed context. This is especially important
in the bucket entrypoint and bucket instance case where we can accidentally
confuse the two. There were cases where we created one and used it where
the other one was needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more compilation fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more compilation fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api adjustments and other issues fixed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix meta handlers initialization

Split alloc into alloc + init, so that we don't need to get all the params
through the sync module.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move mdlog related functionality to the be_sobj impl

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket: metadata handler, fix put

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: acl: api adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move a lot of cls_user utility methods into svc_user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc: new bucket index service

Currently only specialized and incomplete rados version.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_rados: keep ioctx on the pool

and rgw_rados_ref now holds Pool and not IoCtx directly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adding missing files

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more refactoring work

a lot went into this, geting closer to have both user and bucket stuff
adapted to the new system. Will need to create a new bucket index service
to deal with its management (init, stat, etc.).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.user_rados: split svc.user

Need to have rados specific implementation: use of cls_user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta listing: adapt to new system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_bucket: move more code into ctl

Also some work on the handlers list operations (which can now be done
through the meta backend).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.sysobj: implement paged object listing

This was missing. Also modify svc_sys_obj.h to avoid direct reference
to svc_sys_obj_core.h, other than what's needed for using ceph::static_ptr --
which we use to avoid extra allocation.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ctl.bucket: move [un]link bucket into ctl

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more work towards updated bucket meta interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use optional method params

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ctl.bucket: implement bucket ctl utilities

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.bucket: reimplement

Have svc.bucket do only the low level storage functionality. Next
will need to have helpers that use these (similar to what was done
for users).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta_be: provide a backend handler

RGW_MetaBackend_Handler can create a backend context. It is created
for each backend user (meta handler).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_user adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_user api adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial cleanups

in rgw_user.cc

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: consolidate some higher level apis into ctl structure

Similar to svc, just for higher level apis.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more refactoring work

notable changes are around user metadata. Create an api that uses
the service interface (that requires backend context) and use it
for higher level functions. Still a lot to do.

Following rebase, started modifying the internal meta apis to deal
with optional_yield param. Passing it from top level metadata handlers
and from ctls, all the way down to the services. However, since following
commit will introduce many more changes, this was only done for the
user apis.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more meta backend work

and a lot of compilation fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta put: revise

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket and user meta: implement higher level service utils

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta_be_sobj interface changes and adapt otp, bucket meta handlers

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start adapting otp metadata handler

and other changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move service_module init into svc.sync_modules

So that it could be used now in svc.bucket without needing store.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more progress with meta restructure

Getting closer, but not there yet

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial meta manager refactoring

I didn't intend for it to be such a big commit, and it's not
even compiling yet. This changes the structure of how
the metadata manager and handlers work.
The idea is to be able to relatively easily hook in different
meta backends (or same backends with different handling -- such
as the otp).
Added new services for meta, meta backend, and meta backend sysobj
implementation.
The meta backend service is responsible for the final data storage,
and updating the meta log (log might be split later on, but at the
moment it keeping it together for simplicity).
The handlers themselves are the ones responsible for reading or
modifying the metadata. This means that they need to call the
meta backend service instead of calling the utility functions.
The utility functions need to call the handlers, and not the other
way around. Handlers can have utility methods to assist.

Left to do: get everything actually compiling and implemented. The
structure is there, now need to fill in the gaps.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc bucket: set bucket instance info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>