rgw:Fix rgw decompression log-print

The zlib compression takes effect in RGW。
When getting objects failed because of decompress-failed。“ceph-client.rgw” log printed “Compression failed with exit code......”，it should be “deCompression failed with exit code......”。

Signed-off-by: Han Fengzhe <hanfengzhe@hisilicon.com>

Merge pull request #29587 from yuvalif/wip-yuvali-fix-issue-41169

rgw: don't throw when accept errors are happening on frontend

Merge pull request #29615 from tchaikov/wip-qa/tasks/mgr/dashboard/test_health

qa/tasks/mgr/dashboard/test_health: add missing field for test_full_health

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

qa/tasks/mgr/dashboard/test_health: add missing field for test_full_health

fix regressions introduced by a076260e and d6ff61ed

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29342 from Jeegn-Chen/wip-scrub-extended-sleep

osd: support osd_scrub_extended_sleep

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29522 from majianpeng/bluestore-optimization

os/bluestore: deferred IO notify and locking optimization

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29418 from junxfl/test1

OSD: modify n.cookie to op.notify.cookie

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29571 from tchaikov/wip-41177

ceph-objectstore-tool: update-mon-db: do not fail if incmap is missing

Reviewed-by: huangjun <huangjun@xsky.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29605 from yaozongyou/remove-unused-marco

common: remove unused _STR and STRINGIFY macro

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29553 from cbodley/wip-rgw-http-unlock-suspend

rgw: http client drops lock before suspending coroutine

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

common: remove unused _STR and STRINGIFY macro

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>

Merge pull request #29547 from jan--f/c-v-always-log-to-stdout

ceph-volume: never log to stdout, use stderr instead

Merge pull request #29076 from tchaikov/wip-crimson-rep-op

crimson/osd: implement replicated write

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29378 from cyx1231st/rfc-seastar-msgr-lossless-master

crimson/net: lossless policy for v2 protocol

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: log if the dest of send_to_osd() is not valid

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not return "void"

to avoid potential confusions.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: write pg info after done with peering

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDRepOp

* add a `RepRequest` operation which is blocked by `ConnectionPipeline`
  and `PGPipeline`. these two pipelines are modeled after their
  counterparts of `ClientRequest`.
* add these two blockers to `PG` and `OSDConnectionPriv` accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: replicate transaction to peers

* handle `MOSDRepOpReply` message in osd, and pass it all the way down
  to `PGBackend`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement PeeringListener::on_activate()

see also f7b55ec14415811d750e79838f746a91106ba81e

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: minor logging cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: throw bad_peer_address when reconnect address doesn't match

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: REPLACING state to resolve racing and retain session

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: handle fault for READY, CONNECTING and ACCEPTING

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: WAIT state and backoff for client

Client goes to WAIT state when it is delayed to reconnect, or wants to
be replaced by a newly established socket.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: SERVER_WAIT state for accepting server

Server wait for peer client close the socket at SERVER_WAIT.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: STANDBY state for lossless server or peer

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow REPLACING state wait for protocol exit

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: send AckFrame for lossless policy

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: maintain the sent queue for lossless policy

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: reset write state with reset_write()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow connecting state reentrant

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: reset handshake status when connecting/accepting

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: pending_q to store the pending(sending) messages

We cannot left the pending messages in the out_q, because with lossless
policy, they can be partially sent and even acknowledged.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: wait_write_exit() to wait for writer stopped

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: is_queued() to check if there's any pending writes

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: fix variables for stateful connection

server_cookie, client_cookie, connect_seq and global_seq are identifiers
of a stateful connection.

We already have some related implementations, but they are stub code
when implement lossy policy and cannot work properly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

osd: support osd_scrub_extended_sleep

1. always take osd_scrub_sleep for manually initiated
   scrubs
2. when scrub_time_permit() return true for scheduled
   ones, the existing osd_scrub_sleep is used
3. when scrub_time_permit() return false for scheduled
   ones, there may be 2 scenarios
   3.1 if osd_scrub_extended_sleep <= osd_scrub_sleep,
       let's take osd_scrub_sleep
   3.2 otherwise, let's take osd_scrub_extended_sleep

Fixes: http://tracker.ceph.com/issues/40955
Signed-off-by: Jeegn Chen <jeegnchen@tencent.com>

crimson/net: clean up, exsiting_conn and existing_proto

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: next_step_t for explicit decision of next state

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: introduce 3 ways to abort the active protocol state

* abort_in_fault(): a fault is happening and needs to be handled.
* abort_protocol(): abort the current protocol state due to preemptive
                    state change.
* abort_in_close(): close this connection and abort the current protocol
                    state due to some fatal error.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

ceph-objectstore-tool: update-mon-db: do not fail if incmap is missing

there is chance that we could use an OSD which does not have incmap of a
certain epoch for rebuilding the monstore. and since OSD does not read
and store the incmap if the MOSDMap message already has the fullmap of
that fullmap, and if an OSD does not have previous fullmap, monitor
will just send it the fullmao. so it's not unusual that an OSD has
a fullmap of some epoch without corresponding incmap.

Fixes: https://tracker.ceph.com/issues/41177
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28157 from Kriechi/docs-rgw-ldap

docs: improve rgw ldap auth options

rgw: don't throw when accept errors are happening on frontend

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #28227 from sseshasa/monCachePriority

mon/OSDMonitor: Use generic priority cache tuner for mon caches

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29540 from cbodley/wip-rgw-user-rename

rgw: followup for 'user rename'

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29558 from theanalyst/rgw-cache-lock

rgw: fix unlock of shared lock in RGWCache

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: fix unlock of shared lock in RGWCache

similar to https://github.com/ceph/ceph/pull/29538/ we unlock a shared_lock with
unlock causing a crash. Also scope the single line if statements to make the
code more concise

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #29292 into master

* refs/pull/29292/head:
os/bluestore: warn on no per-pool omap
os/bluestore: fsck: warning (not error) by default on no per-pool omap
os/bluestore: fsck: int64_t for error count
os/bluestore: default size of 1 TB for testing
os/bluestore: behave if we *do* set PGMETA and PERPOOL flags
os/bluestore: do not set both PGMETA_OMAP and PERPOOL_OMAP
os/bluestore: fsck: only generate 1 error per omap_head
os/bluestore: make fsck repair convert to per-pool omap
os/bluestore: teach fsck to tolerate per-pool omap
os/bluestore: ondisk format change to 3 for per-pool omap
mon/PGMap: add data/omap breakouts for 'df detail' view
osd/osd_types: separate get_{user,allocated}_bytes() into data and omap variants
mon/PGMap: fix stored_raw calculation
mon/PGMap: add in actual omap usage into per-pool stats
osd: report per-pool omap support via store_statfs_t
os/bluestore: set per_pool_omap key on mkfs
osd/osd_types: count per-pool omap capable OSDs
os/bluestore: report omap_allocated per-pool
os/bluestore: add pool prefix to omap keys
kv/KeyValueDB: take key_prefix for estimate_prefix_size()
os/bluestore: fix manual omap key manipulation to use Onode::get_omap_key()
os/bluestore: make omap key helpers Onode methods
os/bluestore: add Onode::get_omap_prefix() helper
os/bluestore: change _do_omap_clear() args

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29528 from tchaikov/wip-build-doc-with-python3

admin/build-doc: use python3

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29560 from linuxbox2/wip-rgwf-advance

rgw_file: dont deadlock in advance_mtime()

os/bluestore: warn on no per-pool omap

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: never log to stdout, use stderr instead

We should never print log messages to stdout, as this should be reserved
for output of ceph-volume.

Fixes: https://tracker.ceph.com/issues/41158
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #29400 from wjwithagen/wjw-fix-do_freeBSD.sh

do_freebsd.sh: update build scripts to resemble Jenkins scripts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29495 from ifed01/wip-ifed-finisher-improve

common/Finisher: remove some lock acquisitions.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29477 from tchaikov/wip-osd-pg-as-mutex

osd: pg as a mutex

Reviewed-by: Greg Farnum gfarnum@redhat.com
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #29486 from runsisi/wip-fix-verbose

ceph.in: fix verbose print

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29465 from penglaiyxy/wip_bluestore_caculated_revert

os/bluestore: no need to add tail length (revert PR#29185)

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29488 from majianpeng/bluestore-remove-lock

os/bluestore: no need protected by OpSequencer::qlock.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29385 from kamoltat/wip-qa-tasks-mgr-test-progress-bug-fix

qa/tasks/mgr/test_progress.py: fix bug in 9b4dbf0

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29365 from liewegas/wip-telemetry-post

telemetry/server: misc fixes

rgw_file: dont deadlock in advance_mtime()

Fixes: https://tracker.ceph.com/issues/41173
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

os/bluestore: fsck: warning (not error) by default on no per-pool omap

Unconditionally error if the global per_pool_omap was set, though!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: int64_t for error count

Just to be on the safe side.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: default size of 1 TB for testing

Less ENOSPC from vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: behave if we *do* set PGMETA and PERPOOL flags

pgmeta trumps perpool.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not set both PGMETA_OMAP and PERPOOL_OMAP

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: only generate 1 error per omap_head

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make fsck repair convert to per-pool omap

- error if objects not per-pool
- error if per_pool_omap not set
- convert as we go, and set the flag at the end

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29541 from yuriw/wip-yuriw-upgrade-rbd-master

qa/tests: added rbd task on ec

Merge PR #27542 into master

* refs/pull/27542/head:
qa/tasks: Fix ambiguous store_thrash, thrash_store

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #29493 into master

* refs/pull/29493/head:
qa/tasks/mgr/mgr_test_case: get mgrmap from 'mgr dump', not status
qa/tasks/ceph_manager: no newlines in 'ceph -s' output
mon: make mon summary more concise in 'ceph -s'
mon/MgrStatMonitor: set initial service_map 'modified' to cluster mkfs
mon: remove double-nesting of "osdmap" for ceph status
mon/MgrMap: make print_summary (used by 'ceph -s') more concise

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29511 into master

* refs/pull/29511/head:
common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars
common/config: let diff show non-build defaults
common/config: do no include multiple 'default' values

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29517 into master

* refs/pull/29517/head:
osd: include PastInterals in pg_notify_t

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #29559 from adamemerson/wip-putobj-crash

rgw: Move upload_info declaration out of conditional

Reviewed-By: Casey Bodley <cbodley@redhat.com>

rgw: Move upload_info declaration out of conditional

That way we don't deconstruct it right after assigning a reference to
part of it.

Fixes: https://tracker.ceph.com/issues/41172
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #29449 from cbodley/wip-vstart-client-rgw

vstart: move common rgw config to [client.rgw]

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

qa/tests: added rbd task on ec

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

osd: include PastInterals in pg_notify_t

We use a pair<pg_notify_t,PastIntervals> everywhere a pg_notify_t is used.
This is silly; just make it a member instead.

Include some minor compat cruft so we can speak to pre-octopus OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29538 from cbodley/wip-rgw-data-changes-shared-unlock

rgw: fix unlock of shared lock in RGWDataChangesLog

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: http client drops mutex before suspending coroutine

the lock can't be held over async_wait(), because the call to finish()
won't be able to reacquire the lock in order to fire the completion.
coroutines may also resume on a different thread, and you can't unlock a
mutex that was locked on another thread

Reported-by: Yuval Lifshitz <yuvalif@yahoo.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: http client doesn't need lock to access atomic 'done'

Reported-by: Yuval Lifshitz <yuvalif@yahoo.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29468 from dillaman/wip-41032

pybind/mgr/rbd_support: use image ids to detect duplicate tasks

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #29514 into master

* refs/pull/29514/head:
mgr/hello: typos
mgr/hello: a better command example
mgr/hello: boilerplate for handling config options
mgr/hello: fully document our options

Reviewed-by: Tim Serong <tserong@suse.com>

qa/tasks/mgr/mgr_test_case: get mgrmap from 'mgr dump', not status

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29546 into master

* refs/pull/29546/head:
mgr/telemetry: clear the event after being awaken by it

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Prevent clone when layering not enabled on pare… (#29317)

mgr/dashboard: Prevent clone when layering not enabled on parent image

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: fix RBD snapshot protected label (#29534)

mgr/dashboard: fix RBD snapshot protected label

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify fields on Crush map page (#29452)

mgr/dashboard: Verify fields on Crush map page

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify fields on Monitors page (#29450)

mgr/dashboard: Verify fields on Monitors page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: E2E Dashboard Page Test Suite (#29432)

mgr/dashboard: E2E Dashboard Page Test Suite

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify manager modules have editing functionali… (#29407)

mgr/dashboard: Verify manager modules have editing functionality

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/telemetry: clear the event after being awaken by it

otherwise telemetry will have a busy-loop once it's signaled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29494 from dzafman/wip-scrub-test

test: Bump sleep time for slower machines

Reviewed-by: Neha Ojha <nojha@redhat.com>

os/bluestore: add kv_drain_preceding_waiters indicate drain_preceding.

Condition: (txc->state == TransContext::STATE_PREPARE &&  deferred_aggressive) isn't
correctly for drain_preceding. So represent this event by adding this
parameter.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore: reduce unecessary notify.

In fact, only drain()/drain_preceding() need wakeup.
Other case it depend on kv_submitted_waiters.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rgw: add rename_swift_keys for user rename

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: consolidate RGWUser::execute_rename

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: user rename calls update() to write final user

this calls remove_old_indexes() to clean up the uid and buckets index
objects, and also sets 'old_info = user_info' so that radosgw-admin
dumps the json output of the renamed user instead of the old user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: user rename overwrites new user with --yes-i-really-mean-it

if a previous rename attempt fails to complete, it can be restarted with
--yes-i-really-mean-it to overwrite the 'stub' user created previously

Signed-off-by: Casey Bodley <cbodley@redhat.com>