Merge pull request #29553 from cbodley/wip-rgw-http-unlock-suspend

rgw: http client drops lock before suspending coroutine

Reviewed-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #29547 from jan--f/c-v-always-log-to-stdout

ceph-volume: never log to stdout, use stderr instead

Merge pull request #29076 from tchaikov/wip-crimson-rep-op

crimson/osd: implement replicated write

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29378 from cyx1231st/rfc-seastar-msgr-lossless-master

crimson/net: lossless policy for v2 protocol

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: log if the dest of send_to_osd() is not valid

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not return "void"

to avoid potential confusions.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: write pg info after done with peering

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: handle MOSDRepOp

* add a `RepRequest` operation which is blocked by `ConnectionPipeline`
  and `PGPipeline`. these two pipelines are modeled after their
  counterparts of `ClientRequest`.
* add these two blockers to `PG` and `OSDConnectionPriv` accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: replicate transaction to peers

* handle `MOSDRepOpReply` message in osd, and pass it all the way down
  to `PGBackend`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: implement PeeringListener::on_activate()

see also f7b55ec14415811d750e79838f746a91106ba81e

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: minor logging cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: throw bad_peer_address when reconnect address doesn't match

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: REPLACING state to resolve racing and retain session

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: handle fault for READY, CONNECTING and ACCEPTING

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: WAIT state and backoff for client

Client goes to WAIT state when it is delayed to reconnect, or wants to
be replaced by a newly established socket.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: SERVER_WAIT state for accepting server

Server wait for peer client close the socket at SERVER_WAIT.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: STANDBY state for lossless server or peer

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow REPLACING state wait for protocol exit

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: send AckFrame for lossless policy

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: maintain the sent queue for lossless policy

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: reset write state with reset_write()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: allow connecting state reentrant

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: reset handshake status when connecting/accepting

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: pending_q to store the pending(sending) messages

We cannot left the pending messages in the out_q, because with lossless
policy, they can be partially sent and even acknowledged.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: wait_write_exit() to wait for writer stopped

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: is_queued() to check if there's any pending writes

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: fix variables for stateful connection

server_cookie, client_cookie, connect_seq and global_seq are identifiers
of a stateful connection.

We already have some related implementations, but they are stub code
when implement lossy policy and cannot work properly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: clean up, exsiting_conn and existing_proto

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: next_step_t for explicit decision of next state

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: introduce 3 ways to abort the active protocol state

* abort_in_fault(): a fault is happening and needs to be handled.
* abort_protocol(): abort the current protocol state due to preemptive
                    state change.
* abort_in_close(): close this connection and abort the current protocol
                    state due to some fatal error.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #28157 from Kriechi/docs-rgw-ldap

docs: improve rgw ldap auth options

Merge pull request #28227 from sseshasa/monCachePriority

mon/OSDMonitor: Use generic priority cache tuner for mon caches

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29540 from cbodley/wip-rgw-user-rename

rgw: followup for 'user rename'

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29558 from theanalyst/rgw-cache-lock

rgw: fix unlock of shared lock in RGWCache

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: fix unlock of shared lock in RGWCache

similar to https://github.com/ceph/ceph/pull/29538/ we unlock a shared_lock with
unlock causing a crash. Also scope the single line if statements to make the
code more concise

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #29292 into master

* refs/pull/29292/head:
os/bluestore: warn on no per-pool omap
os/bluestore: fsck: warning (not error) by default on no per-pool omap
os/bluestore: fsck: int64_t for error count
os/bluestore: default size of 1 TB for testing
os/bluestore: behave if we *do* set PGMETA and PERPOOL flags
os/bluestore: do not set both PGMETA_OMAP and PERPOOL_OMAP
os/bluestore: fsck: only generate 1 error per omap_head
os/bluestore: make fsck repair convert to per-pool omap
os/bluestore: teach fsck to tolerate per-pool omap
os/bluestore: ondisk format change to 3 for per-pool omap
mon/PGMap: add data/omap breakouts for 'df detail' view
osd/osd_types: separate get_{user,allocated}_bytes() into data and omap variants
mon/PGMap: fix stored_raw calculation
mon/PGMap: add in actual omap usage into per-pool stats
osd: report per-pool omap support via store_statfs_t
os/bluestore: set per_pool_omap key on mkfs
osd/osd_types: count per-pool omap capable OSDs
os/bluestore: report omap_allocated per-pool
os/bluestore: add pool prefix to omap keys
kv/KeyValueDB: take key_prefix for estimate_prefix_size()
os/bluestore: fix manual omap key manipulation to use Onode::get_omap_key()
os/bluestore: make omap key helpers Onode methods
os/bluestore: add Onode::get_omap_prefix() helper
os/bluestore: change _do_omap_clear() args

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29528 from tchaikov/wip-build-doc-with-python3

admin/build-doc: use python3

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29560 from linuxbox2/wip-rgwf-advance

rgw_file: dont deadlock in advance_mtime()

os/bluestore: warn on no per-pool omap

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: never log to stdout, use stderr instead

We should never print log messages to stdout, as this should be reserved
for output of ceph-volume.

Fixes: https://tracker.ceph.com/issues/41158
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #29400 from wjwithagen/wjw-fix-do_freeBSD.sh

do_freebsd.sh: update build scripts to resemble Jenkins scripts

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29495 from ifed01/wip-ifed-finisher-improve

common/Finisher: remove some lock acquisitions.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29477 from tchaikov/wip-osd-pg-as-mutex

osd: pg as a mutex

Reviewed-by: Greg Farnum gfarnum@redhat.com
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #29486 from runsisi/wip-fix-verbose

ceph.in: fix verbose print

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29465 from penglaiyxy/wip_bluestore_caculated_revert

os/bluestore: no need to add tail length (revert PR#29185)

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29488 from majianpeng/bluestore-remove-lock

os/bluestore: no need protected by OpSequencer::qlock.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29385 from kamoltat/wip-qa-tasks-mgr-test-progress-bug-fix

qa/tasks/mgr/test_progress.py: fix bug in 9b4dbf0

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29365 from liewegas/wip-telemetry-post

telemetry/server: misc fixes

rgw_file: dont deadlock in advance_mtime()

Fixes: https://tracker.ceph.com/issues/41173
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

os/bluestore: fsck: warning (not error) by default on no per-pool omap

Unconditionally error if the global per_pool_omap was set, though!

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: int64_t for error count

Just to be on the safe side.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: default size of 1 TB for testing

Less ENOSPC from vstart.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: behave if we *do* set PGMETA and PERPOOL flags

pgmeta trumps perpool.

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: do not set both PGMETA_OMAP and PERPOOL_OMAP

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: fsck: only generate 1 error per omap_head

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore: make fsck repair convert to per-pool omap

- error if objects not per-pool
- error if per_pool_omap not set
- convert as we go, and set the flag at the end

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29541 from yuriw/wip-yuriw-upgrade-rbd-master

qa/tests: added rbd task on ec

Merge PR #27542 into master

* refs/pull/27542/head:
qa/tasks: Fix ambiguous store_thrash, thrash_store

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #29493 into master

* refs/pull/29493/head:
qa/tasks/mgr/mgr_test_case: get mgrmap from 'mgr dump', not status
qa/tasks/ceph_manager: no newlines in 'ceph -s' output
mon: make mon summary more concise in 'ceph -s'
mon/MgrStatMonitor: set initial service_map 'modified' to cluster mkfs
mon: remove double-nesting of "osdmap" for ceph status
mon/MgrMap: make print_summary (used by 'ceph -s') more concise

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29511 into master

* refs/pull/29511/head:
common/config: respect POD_MEMORY_REQUEST *and* POD_MEMORY_LIMIT env vars
common/config: let diff show non-build defaults
common/config: do no include multiple 'default' values

Reviewed-by: Mark Nelson <mnelson@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29517 into master

* refs/pull/29517/head:
osd: include PastInterals in pg_notify_t

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #29559 from adamemerson/wip-putobj-crash

rgw: Move upload_info declaration out of conditional

Reviewed-By: Casey Bodley <cbodley@redhat.com>

rgw: Move upload_info declaration out of conditional

That way we don't deconstruct it right after assigning a reference to
part of it.

Fixes: https://tracker.ceph.com/issues/41172
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #29449 from cbodley/wip-vstart-client-rgw

vstart: move common rgw config to [client.rgw]

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

qa/tests: added rbd task on ec

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

osd: include PastInterals in pg_notify_t

We use a pair<pg_notify_t,PastIntervals> everywhere a pg_notify_t is used.
This is silly; just make it a member instead.

Include some minor compat cruft so we can speak to pre-octopus OSDs.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #29538 from cbodley/wip-rgw-data-changes-shared-unlock

rgw: fix unlock of shared lock in RGWDataChangesLog

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: http client drops mutex before suspending coroutine

the lock can't be held over async_wait(), because the call to finish()
won't be able to reacquire the lock in order to fire the completion.
coroutines may also resume on a different thread, and you can't unlock a
mutex that was locked on another thread

Reported-by: Yuval Lifshitz <yuvalif@yahoo.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: http client doesn't need lock to access atomic 'done'

Reported-by: Yuval Lifshitz <yuvalif@yahoo.com>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29468 from dillaman/wip-41032

pybind/mgr/rbd_support: use image ids to detect duplicate tasks

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #29514 into master

* refs/pull/29514/head:
mgr/hello: typos
mgr/hello: a better command example
mgr/hello: boilerplate for handling config options
mgr/hello: fully document our options

Reviewed-by: Tim Serong <tserong@suse.com>

qa/tasks/mgr/mgr_test_case: get mgrmap from 'mgr dump', not status

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29546 into master

* refs/pull/29546/head:
mgr/telemetry: clear the event after being awaken by it

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Prevent clone when layering not enabled on pare… (#29317)

mgr/dashboard: Prevent clone when layering not enabled on parent image

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: fix RBD snapshot protected label (#29534)

mgr/dashboard: fix RBD snapshot protected label

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify fields on Crush map page (#29452)

mgr/dashboard: Verify fields on Crush map page

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify fields on Monitors page (#29450)

mgr/dashboard: Verify fields on Monitors page

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: E2E Dashboard Page Test Suite (#29432)

mgr/dashboard: E2E Dashboard Page Test Suite

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Verify manager modules have editing functionali… (#29407)

mgr/dashboard: Verify manager modules have editing functionality

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/telemetry: clear the event after being awaken by it

otherwise telemetry will have a busy-loop once it's signaled.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29494 from dzafman/wip-scrub-test

test: Bump sleep time for slower machines

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: add rename_swift_keys for user rename

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: consolidate RGWUser::execute_rename

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: user rename calls update() to write final user

this calls remove_old_indexes() to clean up the uid and buckets index
objects, and also sets 'old_info = user_info' so that radosgw-admin
dumps the json output of the renamed user instead of the old user

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: user rename overwrites new user with --yes-i-really-mean-it

if a previous rename attempt fails to complete, it can be restarted with
--yes-i-really-mean-it to overwrite the 'stub' user created previously

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: user rename operates on 'stub' user

user rename creates a 'stub' user and links all buckets to that before
modifying the existing user. once everything succeeds, the stub user is
updated to match the existing user and all index objects are updated to
point at it

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_remove_user_buckets_index

helper function to remove the user:buckets object.
rgw_remove_uid_index() now omits the object version tracker argument to
avoid reading the user info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: simplify bucket chown

removes unused 'attrs' argument, and takes uid + display name instead
of RGWUserInfo

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28813 from smanjara/wip-user-rename-working

rgw : Bucket mv, bucket chown and user rename utilities

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29425 from aclamk/wip-bluestore-monitor-allocations

[bluestore][tools] Inspect allocations in bluestore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: fix unlock of shared lock in RGWDataChangesLog

std::shared_mutex expects a call to unlock_shared() after lock_shared().
use the std::shared_lock guard to make it more obviously correct

Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc/rados/operations/health-checks: document BlueStore fragmentation and BlueFS space available features

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

mgr/dashboard: fix RBD snapshot protected label

Fixes: https://tracker.ceph.com/issues/41153
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #29438 from wjwithagen/wjw-fix-ceph-backport.sh-https-tracker

script/ceph-backport.sh: Use secure access for tracker.ceph.com

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

mgr/hello: typos

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/hello: a better command example

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/hello: boilerplate for handling config options

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: E2E Dashboard Page Test Suite

Fixes: https://tracker.ceph.com/issues/40549
Fixes: https://tracker.ceph.com/issues/40550
Fixes: https://tracker.ceph.com/issues/40561
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

crimson/net: improve get_global_seq()

Implement single global_seq and non-racing get_global_seq() interface.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

admin/build-doc: use python3

to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
https://github.com/sphinx-doc/sphinx/commit/e049f86b2de1cfdf8a74c88dc9593d047c85d5cb
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.

in this change:

* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
  venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
  stable.

Signed-off-by: Kefu Chai <kchai@redhat.com>