Merge PR #24828 into master

* refs/pull/24828/head:
qa/osd-bluefs-volume-ops: use ceph-bluestore-tool for fsck
qa/osd-bluefs-volume-ops: reduce space usage for the test case

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #24948 into master

* refs/pull/24948/head:
os/bluestore: fix wal dir presence verification

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24014 from yehudasa/wip-rgw-svc-2

rgw: initial RGWRados refactoring work

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

rgw: shutdown services only if initialized

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librados_test_stub: implement move operations

Need to match whatever librados implements.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge PR #24995 into master

* refs/pull/24995/head:
doc: use 'proxy' cache mode in "Removing a Writeback Cache" procedure

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24896 from tchaikov/wip-librados-cleanup

librados,rpm,deb: various fixes to address librados3 transition and cleanups in librados

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: use 'proxy' cache mode in "Removing a Writeback Cache" procedure

The 'forward' mode is flawed as it allows the reordering of requests.

Signed-off-by: Mykola Golub <mgolub@suse.com>

test/rbd_mirror: fix linking

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_notify: call set_enabled() when registering callback

svc_notify starts before svc_sysobj_cache, this way we make sure
svc_sysobj_cache gets the status callback triggered.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sysobj_core: filter sysobj attrs

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librados: add move operations to IoCtx

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: chained cache shouldn't access service after it was unchained

This fixes a cleanup ordering issue. We can now have the chained cache
removed before and after the service is destructed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: AioThrottle uses RGWSI_RADOS::Obj

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RADOS::Pool::create() for single pool

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RADOS::Pool::lookup() uses Pool's pool name

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add Obj::aio_operate() for read ops

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rgw_rados_ref uses default copy/move

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix svc_zone compile with is_started()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix list_realms() and list_zones()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix chained cache put if cache isn't initialized

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc: support raw storage init

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move services shutdown around

fixes ordering

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: services shutdown

Explicitly make sure services are being shut down in appropriate order.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: unregister chained cache on shutdown

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc: hold services in unique_ptr

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't try to canonicalize raw obj

and other minor fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unneeded code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't include rgw_zone.h in svc_zone.h

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc: due to popular demand, get rid of service registry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: api fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more code removal and api adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: chained cache doesn't init if not caching

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc: remove SystemObject impl

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc: fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: services init fixes

First load all depending services, then initialize all.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: minor adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: api adjustments

now compiles

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc: more changes, add svc_sync_modules

radosgw actually compiles now

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rgw_zone api adjustments

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add missing files

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_finisher: create service

and other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_cache: more work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_notify: initial work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sysobj service: initial cache conversion

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc_sysobj: split core out

Split core sysobj interface from sysobj service so that cache could be
implemented.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: zone, sysobj services: add missing system obj calls

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: sys_obj service, move zone init to svc_zone

still wip

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc: more zone work, add zone_utils, quota services

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more refactoring work, initial work on svc_rados

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: svc: initial work on zone_svc

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: basic rgw services registry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #24719 from Devp00l/wip-issue-26861

mgr/dashboard: Remove _filterValue from CdFormGroup

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #24333 into master

* refs/pull/24333/head:
test: update librados tests for CLI arg syntax
librados: update for CLI arg format change
pybind: update python callers of force flags
mon: convert remaining confirmation flags to CephBool
ceph_argparse: introduce CephBool arguments
test: remove quirky argparse case
mgr,mon: use CephChoices for confirmation flags
test: add cases for CLI's --key=val style
mon: remove dead "cluster_snap" command
pybind: enable --keyword=arguments in ceph_argparse
ceph.in: some cleanups
ceph.in: misc cleanups
common: remove unused 'avail' field from commands
mon: fix help string for `osd crush rule create-replicated`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24930 from tspmelo/wip-multiple-actions

mgr/dashboard: Fix duplicate tasks

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #24974 from rhcs-dashboard/36712-cluser-hosts-fix

mgr/dashboard: fix for 'Cluster >> Hosts' page

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

rpm,deb: libradospp1-dev should depend on librados-dev

because librados.hpp `#include`s librados.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

examples/librados: link c++ samples against libradospp

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24530 from alimaredia/wip-rgw-dpp-gc-lc

rgw: refactor logging in gc and lc

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24510 from pritha-srivastava/wip-sts-buffer-overflow

rgw: Fix for buffer overflow in STS op_post().

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #24594 from theanalyst/wip-reshard-enoent

rgw: reshard add: fail correctly on a non existant bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #24638 from linuxbox2/wip-reshard-st

radosgw-admin: translate reshard status codes (trivial)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24816 from linuxbox2/wip-rgw-keystone-secret

rgw: move keystone secrets from ceph.conf to files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24869 from joke-lee/wip-op-mask-radosgw-admin-rest

rgw: admin rest api support op-mask

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #24887 from cbodley/wip-36662

rgw: beast frontend parses ipv6 addrs

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #23906 from dongbula/nf-add-latency-after-finish-processing

rgw: add latency info in the log of req done

Merge pull request #24947 from votdev/bug_36708

mgr/dashboard: tasks.mgr.dashboard.test_osd.OsdTest failures

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #24966 from daixiang0/correct-rbytes-description

doc: correct rbytes description

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: fix for 'Cluster >> Hosts' page
- Added permission mapping for 'tcmu-runner' service type.
- Avoid error caused by unmapped services' types:
  The hosts list should be visible even with not mapped services' types,
  as this mapping only decides whether the service being displayed has a link or not.

Fixes: https://tracker.ceph.com/issues/36712
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

doc: correct rbytes description

The default was changed from on to off in 4.6 because the way rbytes behaves can be surprising and sometimes confuses tools like rsync.

Signed-off-by: Xiang Dai <764524258@qq.com>

Merge pull request #24817 from tone-zhang/wip-64-assert

bluestore/NVMEDevice.cc: fix ceph_assert() when enable SPDK with 64KB kernel page size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24964 from Songweibin/wip-doc-rados

doc/rados: correct osd path in troubleshooting-mon.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #23940 from robbat2/wip-12713-chunked-encoding

rgw: use chunked encoding to get partial results out faster

rgw: add latency info in the log of req done

Signed-off-by: lvshuhua <lvshuhua@cmss.chinamobile.com>

qa: install libradospp-dev for librados_hello_world.yaml

libradospp-{dev,devel} is necessary for compiling sources in
examples/librados/hello_world.cc

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24749 from dzafman/wip-36474

Add support for osd_delete_sleep configuration value

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

debian: libradospp-dev should depend on libradospp1 not libradospp

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: remove rados_tmap_*

we have switched from tmap to omap long ago.

but keep the server side implementation around, in case ancient
client is still using these tmap APIs.

also, tmap_update() is kept, because librbd is using it for v1 image
backward compatibility.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: add new setxattr overload back

this change reverts b8ff781ddcf737882163cf56d7c9b11e815fb699

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: remove redundant IoCtx::nobjects_begin() variants

this change partially reverts 6eca7d08893abbe41f7fa6b2834fb05e84be572b

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados: deprecate ObjectWriteOperation::copy_from()

this change partially reverts 6eca7d08893abbe41f7fa6b2834fb05e84be572b

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: remove redundant implementations

this change reverts cac1d6f936c40003c5231d1051feb51198252715

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian: loosen the Depends of python bindings packages

- python-* packages are using the C APIs which are not changed across
librados2 and librados3.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24977 from alfredodeza/wip-bz1644265

ceph-volume util.encryption robust blkid+lsblk detection of lockbox

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume util.encryption robust blkid+lsblk detection of lockbox

Signed-off-by: Alfredo Deza <adeza@redhat.com>

mgr/dashboard: backend api tests: tasks.mgr.dashboard.test_osd.OsdTest failures

- Fix bug in Dashboard QA unit test framework. Don't set the application type header manually, this is done by the requests library if required.
- Enhance QA unit test helper: Print the response of the API request if it fails. This should help to identify the problem more easily.
- Fix bug in the OSD controller. A parameter needs to be converted to integer.
- Take care that the params of the request object are not modified.

The issue was introduced by PR https://github.com/ceph/ceph/pull/24475. The CherryPy json_in plugin disclosed the errorneous unit test helper implementation.

Fixes: https://tracker.ceph.com/issues/36708
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #24847 from pritha-srivastava/wip-rgw-assume-role-doc

rgw: Improvements to STS Lite documentation.

Merge pull request #24841 from b-ranto/wip-deep-scrub

mgr/restful: Fix deep-scrub typo

Reviewed-by: John Spray <john.spray@redhat.com>

test: update librados tests for CLI arg syntax

Signed-off-by: John Spray <john.spray@redhat.com>

librados: update for CLI arg format change

Signed-off-by: John Spray <john.spray@redhat.com>

mgr/restful: Fix deep-scrub typo

There is a typo in implemented commands for OSD. It reads deep_scrub
instead of deep-scrub which is the true valid command.

Fixes: http://tracker.ceph.com/issues/36720
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #24681 from tspmelo/wip-npm-update

mgr/dashboard: Update npm packages

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

doc/rados: correct osd path in troubleshooting-mon.rst

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #24937 from alfredodeza/wip-rm36704

ceph-volume tests.systemd update imports for systemd module

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #15047 into master

* refs/pull/15047/head:
tool/ceph_objectstore_tool: add new op that reset last_complete to last_update

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #24717 from trociny/wip-rbd-mirror-status-instance

rbd: show info about mirror daemon instance in image mirror status output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24815 from iridescent-rsy/remove-ceph_abort

rbd-nbd: do not ceph_abort() after print the usages

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24919 from trociny/wip-tests-migration-namespace

test/librbd: migration supporting namespace tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-volume tests.systemd update imports for systemd module

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume systemd import main so console_scripts work for executable

Signed-off-by: Alfredo Deza <adeza@redhat.com>