rgw: move a lot of cls_user utility methods into svc_user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc: new bucket index service

Currently only specialized and incomplete rados version.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_rados: keep ioctx on the pool

and rgw_rados_ref now holds Pool and not IoCtx directly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: adding missing files

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more refactoring work

a lot went into this, geting closer to have both user and bucket stuff
adapted to the new system. Will need to create a new bucket index service
to deal with its management (init, stat, etc.).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.user_rados: split svc.user

Need to have rados specific implementation: use of cls_user

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta listing: adapt to new system

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_bucket: move more code into ctl

Also some work on the handlers list operations (which can now be done
through the meta backend).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.sysobj: implement paged object listing

This was missing. Also modify svc_sys_obj.h to avoid direct reference
to svc_sys_obj_core.h, other than what's needed for using ceph::static_ptr --
which we use to avoid extra allocation.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ctl.bucket: move [un]link bucket into ctl

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more work towards updated bucket meta interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use optional method params

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: ctl.bucket: implement bucket ctl utilities

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc.bucket: reimplement

Have svc.bucket do only the low level storage functionality. Next
will need to have helpers that use these (similar to what was done
for users).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta_be: provide a backend handler

RGW_MetaBackend_Handler can create a backend context. It is created
for each backend user (meta handler).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_user adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_user api adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: trivial cleanups

in rgw_user.cc

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: consolidate some higher level apis into ctl structure

Similar to svc, just for higher level apis.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more refactoring work

notable changes are around user metadata. Create an api that uses
the service interface (that requires backend context) and use it
for higher level functions. Still a lot to do.

Following rebase, started modifying the internal meta apis to deal
with optional_yield param. Passing it from top level metadata handlers
and from ctls, all the way down to the services. However, since following
commit will introduce many more changes, this was only done for the
user apis.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more meta backend work

and a lot of compilation fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta put: revise

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket and user meta: implement higher level service utils

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: meta_be_sobj interface changes and adapt otp, bucket meta handlers

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: start adapting otp metadata handler

and other changes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move service_module init into svc.sync_modules

So that it could be used now in svc.bucket without needing store.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more progress with meta restructure

Getting closer, but not there yet

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial meta manager refactoring

I didn't intend for it to be such a big commit, and it's not
even compiling yet. This changes the structure of how
the metadata manager and handlers work.
The idea is to be able to relatively easily hook in different
meta backends (or same backends with different handling -- such
as the otp).
Added new services for meta, meta backend, and meta backend sysobj
implementation.
The meta backend service is responsible for the final data storage,
and updating the meta log (log might be split later on, but at the
moment it keeping it together for simplicity).
The handlers themselves are the ones responsible for reading or
modifying the metadata. This means that they need to call the
meta backend service instead of calling the utility functions.
The utility functions need to call the handlers, and not the other
way around. Handlers can have utility methods to assist.

Left to do: get everything actually compiling and implemented. The
structure is there, now need to fill in the gaps.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc bucket: set bucket instance info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move helper code out of rgw_rados.cc

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: header cleanup

move implementation out of header.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use bucket service for get_bucket_info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_bucket: initial work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: no need to pass in RGWRados to rgw_get_system_obj()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move admin socket cache code into svc_sys_obj_cache

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move rgw_compression_info_from_attrset() to a better home

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_cls: replace svc_mfa with a more generic svc_cls

Consolidate objclass util services.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: switch to using svc.mfa

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc/mfa: new service for mfa functionality

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move manifest code around, initial tier separation

cascading changes, and minor improvements.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more headers cleanup

Dependency reduction

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: object expirer: move code around

move code out of RGWRados, refactor a bit to use rados svc.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more headers cleanups

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: header cleanup: remove code from rgw_rados.h

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #29374 into master

* refs/pull/29374/head:
cephfs-shell: Fix flake8 line too long error
cephfs-shell: Fix flake8 line break after binary operator error
cephfs-shell: Fix flake8 continuation line over-indented for hanging indent error
cephfs-shell: Fix flake8 missing whitespace around arithmetic operator error
cephfs-shell: Fix multiple spaces after ':' flake8 error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27780 from theanalyst/cls-timeindex-denc

cls: add timeindex types to ceph-dencoder

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29182 from dcasier/mgr-dashboard-ganesha-fix-cephfs-name

mgr/dashboard: ganesha: Specify the name of the filesystem (create_path)

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #29138 from Reperator/dashboard-pw-hash

mgr/dashboard: support setting password hashes

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

cephfs-shell: Fix flake8 line too long error

https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 line break after binary operator error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 continuation line over-indented for hanging indent error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 missing whitespace around arithmetic operator error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix multiple spaces after ':' flake8 error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #29272 from votdev/issue_40925

mgr/dashboard: Various UI issues related to CephFS

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29216 from rafaelquint/configtest

mgr/dashboard: E2E test to verify Configuration editing functionality

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29324 from adk3798/typepages

mgr/dashboard: Explicitly type page variables

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29371 from bk201/wip-40971

mgr/dashboard: fix MDS counter chart is not displayed

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #29341 from zhangbingyi1/master

test/librados: avoid residual crush rule after test case execution

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix MDS counter chart is not displayed

Unit of timetamp for performance counters is changed to nanoseconds [1].
Adapt this for CephFS MDS performance counters in backend.

[1]: https://github.com/ceph/ceph/pull/28882

Fixes: https://tracker.ceph.com/issues/40971
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

test/librados: avoid residual crush rule after test case execution
Fixes: http://tracker.ceph.com/issues/40970
Signed-off-by: Bingyi Zhang <zhangbingyi@umcloud.com>

Merge pull request #29348 from myoungwon/fix-wrong-class

osd: fix wrong arguments when dropping refcount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29345 from majianpeng/bluestore-flush-waiter

os/bluestore: avoid unnecessary notify.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29126 from xiexingguo/wip-clamp-ratio-to-0

mgr/progress: clamp pg recovery ratio to 0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28012 from rouming/bootstrap-after-setuid

global/global_init: do first transport connection after setuid()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27985 from ifed01/wip-ifed-slow-omaps

tools/ceph-objectstore-tool: introduce list-slow-omap command.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29238 into master

* refs/pull/29238/head:
os/bluestore: it's better to erase spanning blob only once

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29239 into master

* refs/pull/29239/head:
qa/tasks/ceph_manager.py: always use self.logger

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #29305 into master

* refs/pull/29305/head:
mgr: Reduce logging noise when handling commands

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29307 into master

* refs/pull/29307/head:
monmaptool: added --addv option to usage description

Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29316 into master

* refs/pull/29316/head:
common,tools: make sure the destination buffer can handle the size of the string
src/tools: initialize variables before the goto statement
src/mount: check before dereference buf
src/crush: check before dereference out2
src/test: s/strcpy/strncpy

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29358 into master

* refs/pull/29358/head:
mgr/telemetry: salt osd ids too

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #29054 from dillaman/wip-40621

mgr/rbd_support: support scheduling long-running background operations

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #29361 from tchaikov/wip-qa-osd-pause

qa: fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28685 into master

* refs/pull/28685/head:
client: more precise CEPH_CLIENT_CAPS_PENDING_CAPSNAP
client: unify kicking cap flushes and kicking snapcap flushes
client: define helper function that sends flushsnap message
client: cleanup tracking of early kicked flushing caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28793 into master

* refs/pull/28793/head:
qa/cephfs: make run_shell() accept args as str too
qa/vstart_runner.py: ignores when source and destination are same
test_cephfs_shell: add a test for cd with arguments
test_cephfs_shell: add a test for cd with no arguments
test_cephfs_shell: group test methods
test_cephfs_shell: add a new method to return script output
test_cephfs_shell: allow running CephFS shell script
cephfs-shell: cd with no args shouldn't print an error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29064 into master

* refs/pull/29064/head:
mds: execute PurgeQueue on_error handler in finisher

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29302 into master

* refs/pull/29302/head:
cephfs: fix a memory leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29066 into master

* refs/pull/29066/head:
mds: Move inline functions with loops or switch statements to CInode.cc
mds: Reorganize class members in CInode header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29336 into master

* refs/pull/29336/head:
qa: wait for MDS to come back after removing it

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #29359 from jmunhoz/wip-msite-az-tests

rgw: az: add archive zone tests

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard/qa: add more fields to report

to be in sync with #28847

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/rest/mgr-restful: whitelist OSDMAP_FLAGS

`rest/test-restful.sh` calls `test_mgr_rest_api.py`, which in turn
calls

```
    ('patch',  '/config/osd', {'pause': True}),
```

and rest module translates it to `ceph osd set key=pause`

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: az: add archive zone tests

Signed-off-by: Javier M. Mellid <jmunhoz@igalia.com>

mgr/telemetry: salt osd ids too

Better to fully obfuscate here.

This has a nice side-effect of assigning entity names that are *globally*
unique across the full telemetry data set, since the salts are unique and
sha1 is (sufficiently) collision-free.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29330 into master

* refs/pull/29330/head:
mgr/telemetry: obscure entity_name with a salt

Reviewed-by: Dan Mick <dmick@redhat.com>

mgr/dashboard: Explicitly type page variables

Explicitly gives a type to page variables.
Helps make better use of Typescript's static type checking.
Undoes mistake from #29070.

Fixes: https://tracker.ceph.com/issues/40961
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

mgr: Reduce logging noise when handling commands

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29237 from adk3798/usertest

mgr/dashboard: RGW user E2E Tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #29192 from ceph/dependabot/npm_and_yarn/src/pybind/mgr/dashboard/frontend/lodash-4.17.13

build(deps): bump lodash from 4.17.11 to 4.17.13 in /src/pybind/mgr/dashboard/frontend

Reviewed-by: Tiago Melo <tmelo@suse.com>

pybind/mgr: don't log exception when cannot find RBD task by id

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: fix wrong arguments when dropping refcount

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

Merge pull request #29198 from rhcs-dashboard/e2e-ci

mgr/dashboard: run e2e tests against prod build (jenkins job)

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29280 from rhcs-dashboard/bucket-tests

mgr/dashboard: RGW bucket creation when no placement target received

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: Various UI issues related to CephFS

* Check if data is accessible, otherwise set grafanaId to undefined.
* Catch errors in 'Clients' tab and stop loading spinner. Display ViewCacheStatus Exception.
* Render the 'Enabled' column using the checkIcon cell template as other lists do.

Fixes: https://tracker.ceph.com/issues/40925
Signed-off-by: Volker Theile <vtheile@suse.com>

os/bluestore: avoid unnecessary notify.

Add parameter count waiting list for Onode::flush to avoid unnecessary
notify.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common,tools: make sure the destination buffer can handle the size of the string

fix klocwork issues:
Width is not specified for 's' conversion specifier.
This can result in an overflow of the buffer
provided in argument 3 of a call to 'sscanf'

Signed-off-by: songweibin <song.weibin@zte.com.cn>

src/tools: initialize variables before the goto statement

CID 172143 (#15 of 15): Branch past initialization (PW.BRANCH_PAST_INITIALIZATION)
1. branch_past_initialization:
2. name_at_decl_position: variable "oid" (declared at line 220)
3. name_at_decl_position: variable "exkeys" (declared at line 241)
4. name_at_decl_position: variable "exvals" (declared at line 242)
5. name_at_decl_position: variable "safe" (declared at line 278)

Signed-off-by: songweibin <song.weibin@zte.com.cn>

src/mount: check before dereference buf

CID 180671 (#1 of 1): Dereference null return value (NULL_RETURNS)

Signed-off-by: songweibin <song.weibin@zte.com.cn>

src/crush: check before dereference out2

CID 174874 (#2 of 2): Dereference after null check (FORWARD_NULL)
30. var_deref_op: Dereference null pointer out2.

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #29264 from rjfd/wip-dashboard-fix-qa-rbd

mgr/dashboard: qa: fix RBD test when matching error strings

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>