Project Zipper Part 1 - Framework and RGWRadosStore

This is the first part of Project Zipper, the Store Abstraction Layer.
It introduces the basic framework, and wraps RGWRados in RGWRadosStore.
The goal over the next few weeks is to do the same for user, bucket, and
object.  This will make most of the remaining users of RGWRados wrapped
in SAL classes, allowing it to be completely absorbed into the private
RGWRadosStore.  This will also expose all the APIs that need to be
pusheg up to higher layers in the SAL.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

mgr/dashboard: Fix e2e issue in HACKING.rst (#29640)

mgr/dashboard: Fix e2e issue in HACKING.rst

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29570 from rhcs-dashboard/new-bucket-utilities-adaptation

mgr/dashboard: adapt bucket tenant API tests to new behaviour

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29634 from rhcs-dashboard/mgr-module-fixes

mgr/dashboard: fix mgr module API tests

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29612 from tchaikov/wip-crimson-perf-test

crimson/test: add CBT based perf tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29644 from anthonyeleven/patch-3

doc: operations: correct 'comma-delimited'

Reviewed-by: Kefu Chai <kchai@redhat.com>

src/script: add run-cbt.sh

this script will be used by jenkins to drive the CBT based test

also, this test can be used by developer for testing crimson or classic
OSD.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/test: add perf tests for crimson

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/test: add script to convert teuthology task config to cbt config

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: operations: correct 'comma-delimited'

CIDR blocks are comma-separated, not comma-delimited.

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #29118 from cbodley/wip-rgw-metadata-servicification

rgw: metadata refactoring

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge branch 'wip-rgw-metadata-servicification'

 Conflicts:
src/rgw/rgw_auth.cc
src/rgw/rgw_auth_registry.h
src/rgw/rgw_auth_s3.h
src/rgw/rgw_bucket.cc
src/rgw/rgw_bucket.h
src/rgw/rgw_data_sync.h
src/rgw/rgw_frontend.h
src/rgw/rgw_log.h
src/rgw/rgw_main.cc
src/rgw/rgw_rados.cc
src/rgw/rgw_rados.h
src/rgw/rgw_rest_s3.h
src/rgw/rgw_rest_sts.h
src/rgw/rgw_swift_auth.h
src/rgw/rgw_user.cc
src/rgw/rgw_user.h
src/rgw/services/svc_sys_obj_core.h

mgr/dashboard: Fix e2e issue in HACKING.rst

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #29633 from hanfengzhe-hi/Fix-decompression-logprint

rgw:Fix rgw decompression log-print

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: fix mgr module API tests

Signed-off-by: alfonsomthd <almartin@redhat.com>

rgw:Fix rgw decompression log-print

The zlib compression takes effect in RGW。
When getting objects failed because of decompress-failed。“ceph-client.rgw” log printed “Compression failed with exit code......”，it should be “deCompression failed with exit code......”。

Signed-off-by: Han Fengzhe <hanfengzhe@hisilicon.com>

Merge pull request #29587 from yuvalif/wip-yuvali-fix-issue-41169

rgw: don't throw when accept errors are happening on frontend

Merge pull request #29615 from tchaikov/wip-qa/tasks/mgr/dashboard/test_health

qa/tasks/mgr/dashboard/test_health: add missing field for test_full_health

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

qa/tasks/mgr/dashboard/test_health: add missing field for test_full_health

fix regressions introduced by a076260e and d6ff61ed

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/dashboard/test_mgr_module: remove enable/disable test from MgrModuleTelemetryTest

telemetry is always enabled since 2d62d71cd46d791294b3803e034c0995fef08b33

Fixes: https://tracker.ceph.com/issues/41186
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29342 from Jeegn-Chen/wip-scrub-extended-sleep

osd: support osd_scrub_extended_sleep

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29522 from majianpeng/bluestore-optimization

os/bluestore: deferred IO notify and locking optimization

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29418 from junxfl/test1

OSD: modify n.cookie to op.notify.cookie

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29571 from tchaikov/wip-41177

ceph-objectstore-tool: update-mon-db: do not fail if incmap is missing

Reviewed-by: huangjun <huangjun@xsky.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: RGWUserCtl::PutParams::attrs defaults to nullptr

Signed-off-by: Casey Bodley <cbodley@redhat.com>

vstart: pass "--smp 1 --cpuset $osd" to crimson-osd

for optimal performance when testing multiple crimson-osd in a vstart cluster

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29605 from yaozongyou/remove-unused-marco

common: remove unused _STR and STRINGIFY macro

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29553 from cbodley/wip-rgw-http-unlock-suspend

rgw: http client drops lock before suspending coroutine

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

common: remove unused _STR and STRINGIFY macro

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #29547 from jan--f/c-v-always-log-to-stdout

ceph-volume: never log to stdout, use stderr instead

Merge pull request #29076 from tchaikov/wip-crimson-rep-op

crimson/osd: implement replicated write

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29378 from cyx1231st/rfc-seastar-msgr-lossless-master

crimson/net: lossless policy for v2 protocol

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: log if the dest of send_to_osd() is not valid

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not return "void"

to avoid potential confusions.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: write pg info after done with peering

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDRepOp

* add a `RepRequest` operation which is blocked by `ConnectionPipeline`
  and `PGPipeline`. these two pipelines are modeled after their
  counterparts of `ClientRequest`.
* add these two blockers to `PG` and `OSDConnectionPriv` accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: replicate transaction to peers

* handle `MOSDRepOpReply` message in osd, and pass it all the way down
  to `PGBackend`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement PeeringListener::on_activate()

see also f7b55ec14415811d750e79838f746a91106ba81e

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: minor logging cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: throw bad_peer_address when reconnect address doesn't match

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: REPLACING state to resolve racing and retain session

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: handle fault for READY, CONNECTING and ACCEPTING

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: WAIT state and backoff for client

Client goes to WAIT state when it is delayed to reconnect, or wants to
be replaced by a newly established socket.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: SERVER_WAIT state for accepting server

Server wait for peer client close the socket at SERVER_WAIT.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: STANDBY state for lossless server or peer

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow REPLACING state wait for protocol exit

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: send AckFrame for lossless policy

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: maintain the sent queue for lossless policy

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: reset write state with reset_write()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow connecting state reentrant

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: reset handshake status when connecting/accepting

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: pending_q to store the pending(sending) messages

We cannot left the pending messages in the out_q, because with lossless
policy, they can be partially sent and even acknowledged.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: wait_write_exit() to wait for writer stopped

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: is_queued() to check if there's any pending writes

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: fix variables for stateful connection

server_cookie, client_cookie, connect_seq and global_seq are identifiers
of a stateful connection.

We already have some related implementations, but they are stub code
when implement lossy policy and cannot work properly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

osd: support osd_scrub_extended_sleep

1. always take osd_scrub_sleep for manually initiated
   scrubs
2. when scrub_time_permit() return true for scheduled
   ones, the existing osd_scrub_sleep is used
3. when scrub_time_permit() return false for scheduled
   ones, there may be 2 scenarios
   3.1 if osd_scrub_extended_sleep <= osd_scrub_sleep,
       let's take osd_scrub_sleep
   3.2 otherwise, let's take osd_scrub_extended_sleep

Fixes: http://tracker.ceph.com/issues/40955
Signed-off-by: Jeegn Chen <jeegnchen@tencent.com>

crimson/net: clean up, exsiting_conn and existing_proto

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: next_step_t for explicit decision of next state

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: introduce 3 ways to abort the active protocol state

* abort_in_fault(): a fault is happening and needs to be handled.
* abort_protocol(): abort the current protocol state due to preemptive
                    state change.
* abort_in_close(): close this connection and abort the current protocol
                    state due to some fatal error.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

ceph-objectstore-tool: update-mon-db: do not fail if incmap is missing

there is chance that we could use an OSD which does not have incmap of a
certain epoch for rebuilding the monstore. and since OSD does not read
and store the incmap if the MOSDMap message already has the fullmap of
that fullmap, and if an OSD does not have previous fullmap, monitor
will just send it the fullmao. so it's not unusual that an OSD has
a fullmap of some epoch without corresponding incmap.

Fixes: https://tracker.ceph.com/issues/41177
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28157 from Kriechi/docs-rgw-ldap

docs: improve rgw ldap auth options

rgw: don't throw when accept errors are happening on frontend

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #28227 from sseshasa/monCachePriority

mon/OSDMonitor: Use generic priority cache tuner for mon caches

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29540 from cbodley/wip-rgw-user-rename

rgw: followup for 'user rename'

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29558 from theanalyst/rgw-cache-lock

rgw: fix unlock of shared lock in RGWCache

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: fix unlock of shared lock in RGWCache

similar to https://github.com/ceph/ceph/pull/29538/ we unlock a shared_lock with
unlock causing a crash. Also scope the single line if statements to make the
code more concise

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #29292 into master

* refs/pull/29292/head:
os/bluestore: warn on no per-pool omap
os/bluestore: fsck: warning (not error) by default on no per-pool omap
os/bluestore: fsck: int64_t for error count
os/bluestore: default size of 1 TB for testing
os/bluestore: behave if we *do* set PGMETA and PERPOOL flags
os/bluestore: do not set both PGMETA_OMAP and PERPOOL_OMAP
os/bluestore: fsck: only generate 1 error per omap_head
os/bluestore: make fsck repair convert to per-pool omap
os/bluestore: teach fsck to tolerate per-pool omap
os/bluestore: ondisk format change to 3 for per-pool omap
mon/PGMap: add data/omap breakouts for 'df detail' view
osd/osd_types: separate get_{user,allocated}_bytes() into data and omap variants
mon/PGMap: fix stored_raw calculation
mon/PGMap: add in actual omap usage into per-pool stats
osd: report per-pool omap support via store_statfs_t
os/bluestore: set per_pool_omap key on mkfs
osd/osd_types: count per-pool omap capable OSDs
os/bluestore: report omap_allocated per-pool
os/bluestore: add pool prefix to omap keys
kv/KeyValueDB: take key_prefix for estimate_prefix_size()
os/bluestore: fix manual omap key manipulation to use Onode::get_omap_key()
os/bluestore: make omap key helpers Onode methods
os/bluestore: add Onode::get_omap_prefix() helper
os/bluestore: change _do_omap_clear() args

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29528 from tchaikov/wip-build-doc-with-python3

admin/build-doc: use python3

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29560 from linuxbox2/wip-rgwf-advance

rgw_file: dont deadlock in advance_mtime()

os/bluestore: warn on no per-pool omap

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: never log to stdout, use stderr instead

We should never print log messages to stdout, as this should be reserved
for output of ceph-volume.

Fixes: https://tracker.ceph.com/issues/41158
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

mgr/dashboard: adapt bucket tenant tests to new behaviour

Fixes: https://tracker.ceph.com/issues/41175
Signed-off-by: alfonsomthd <almartin@redhat.com>

Merge pull request #29400 from wjwithagen/wjw-fix-do_freeBSD.sh

do_freebsd.sh: update build scripts to resemble Jenkins scripts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29495 from ifed01/wip-ifed-finisher-improve

common/Finisher: remove some lock acquisitions.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29477 from tchaikov/wip-osd-pg-as-mutex

osd: pg as a mutex

Reviewed-by: Greg Farnum gfarnum@redhat.com
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #29486 from runsisi/wip-fix-verbose

ceph.in: fix verbose print

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29465 from penglaiyxy/wip_bluestore_caculated_revert

os/bluestore: no need to add tail length (revert PR#29185)

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29488 from majianpeng/bluestore-remove-lock

os/bluestore: no need protected by OpSequencer::qlock.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29385 from kamoltat/wip-qa-tasks-mgr-test-progress-bug-fix

qa/tasks/mgr/test_progress.py: fix bug in 9b4dbf0

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29365 from liewegas/wip-telemetry-post

telemetry/server: misc fixes

rgw_file: dont deadlock in advance_mtime()

Fixes: https://tracker.ceph.com/issues/41173
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

os/bluestore: fsck: warning (not error) by default on no per-pool omap

Unconditionally error if the global per_pool_omap was set, though!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: int64_t for error count

Just to be on the safe side.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: default size of 1 TB for testing

Less ENOSPC from vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: behave if we *do* set PGMETA and PERPOOL flags

pgmeta trumps perpool.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not set both PGMETA_OMAP and PERPOOL_OMAP

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: only generate 1 error per omap_head

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make fsck repair convert to per-pool omap

- error if objects not per-pool
- error if per_pool_omap not set
- convert as we go, and set the flag at the end

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29541 from yuriw/wip-yuriw-upgrade-rbd-master

qa/tests: added rbd task on ec

Merge PR #27542 into master

* refs/pull/27542/head:
qa/tasks: Fix ambiguous store_thrash, thrash_store

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #29493 into master

* refs/pull/29493/head:
qa/tasks/mgr/mgr_test_case: get mgrmap from 'mgr dump', not status
qa/tasks/ceph_manager: no newlines in 'ceph -s' output
mon: make mon summary more concise in 'ceph -s'
mon/MgrStatMonitor: set initial service_map 'modified' to cluster mkfs
mon: remove double-nesting of "osdmap" for ceph status
mon/MgrMap: make print_summary (used by 'ceph -s') more concise

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29511 into master

* refs/pull/29511/head:
common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars
common/config: let diff show non-build defaults
common/config: do no include multiple 'default' values

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29517 into master

* refs/pull/29517/head:
osd: include PastInterals in pg_notify_t

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #29559 from adamemerson/wip-putobj-crash

rgw: Move upload_info declaration out of conditional

Reviewed-By: Casey Bodley <cbodley@redhat.com>

rgw: Move upload_info declaration out of conditional

That way we don't deconstruct it right after assigning a reference to
part of it.

Fixes: https://tracker.ceph.com/issues/41172
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #29449 from cbodley/wip-vstart-client-rgw

vstart: move common rgw config to [client.rgw]

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

qa/tests: added rbd task on ec

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

osd: include PastInterals in pg_notify_t

We use a pair<pg_notify_t,PastIntervals> everywhere a pg_notify_t is used.
This is silly; just make it a member instead.

Include some minor compat cruft so we can speak to pre-octopus OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29538 from cbodley/wip-rgw-data-changes-shared-unlock

rgw: fix unlock of shared lock in RGWDataChangesLog

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: http client drops mutex before suspending coroutine

the lock can't be held over async_wait(), because the call to finish()
won't be able to reacquire the lock in order to fire the completion.
coroutines may also resume on a different thread, and you can't unlock a
mutex that was locked on another thread

Reported-by: Yuval Lifshitz <yuvalif@yahoo.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>