Merge PR #27823 into master

* refs/pull/27823/head:
msg/async/ProtocolV1: generate authorizer for mon to mon connections
mon/Monitor: require authorizer from peer monitors

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27798 from neha-ojha/wip-39441

osd/PG: do not use approx_missing_objects pre-nautilus

Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27209 from dzafman/wip-38840

osd: Don't evict after a flush if intersecting scrub range

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27769 from dzafman/wip-39333

osd-backfill-space.sh test failed in TEST_backfill_multi_partial()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27732 into master

* refs/pull/27732/head:
common/assert: include ceph_abort_msg(arg) arg in log output

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #27729 into master

* refs/pull/27729/head:
osd: take heartbeat_lock when calling heartbeat()

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #26176 into master

* refs/pull/26176/head:
os/bluestore: do not check osd_max_object_size in _open_path()

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27786 into master

* refs/pull/27786/head:
qa/suites/upgrade/*-x-singleton: enable bluestore debugging settings

Reviewed-by: Neha Ojha <nojha@redhat.com>

msg/async/ProtocolV1: generate authorizer for mon to mon connections

Signed-off-by: Sage Weil <sage@redhat.com>

mon/Monitor: require authorizer from peer monitors

Mon to mon links use authorizers.  It's only non-mons connecting to mons
that use MAuth messages.

Signed-off-by: Sage Weil <sage@redhat.com>

test: osd-backfill-spsace.sh doesn't matter which PG wins the race

Fixes: http://tracker.ceph.com/issues/39333
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #27789 into master

* refs/pull/27789/head:
os/bluestore: correctly measure deferred writes into new blobs

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #27655 into master

* refs/pull/27655/head:
common/options: flag misc ms_* options STARTUP
common/options: flag misc options STARTUP
common/options: mark cluster log options with FLAG_RUNTIME, use get_val
common/options: mark a bunch of options with FLAG_STARTUP

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27724 from idryomov/wip-rbd-copyup-zeros

librbd: snapshot object maps can go inconsistent during copyup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27755 from trociny/wip-rbd-io-scheduler-leak

librbd: leak on canceling simple io scheduler timer task

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27411 from ivancich/wip-clean-up-logging

rgw: clean up some logging

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #27456 from wjwithagen/wjw-fix-rgw_amqp.cc

rgw: use the compatibilty function for pthread_setname

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27617 from cbodley/wip-39295

rgw: bucket stats report mtime in UTC

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #27503 from dzafman/wip-39099

osd: Give recovery for inactive PGs a higher priority

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

osd/PG: do not use approx_missing_objects pre-nautilus

We changed async recovery cost calculation in nautilus to also take
into account approx_missing_objects in ab241bf7e927cda2d0ed1698383d18dc4a4b601c
This commit depends on https://github.com/ceph/ceph/pull/23663, hence
wasn't backported to mimic.

Mimic only uses the difference in length of logs as the cost. Due to this,
the same OSD might have different costs in a mixed mimic and nautilus(or above)
cluster. This can lead to choose_acting() cycling between OSDs, when trying
to select the acting set and async_recovery_targets.

Fixes: https://tracker.ceph.com/issues/39441
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27764 from neha-ojha/wip-fix-mimic-x-singleton

qa/suites/upgrade: use correct branch names

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

common/options: flag misc ms_* options STARTUP

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: flag misc options STARTUP

Signed-off-by: Sage Weil <sage@redhat.com>

common/options: mark cluster log options with FLAG_RUNTIME, use get_val

We already have observers set up for these; switch to using the
runtime-safe accessor and flag them.

Signed-off-by: Sage Weil <sage@redhat.com>

test: osd-recovery-scrub.sh ignore error from kill_daemons()

Another work around for http://tracker.ceph.com/issues/38195

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add new object_location_counts to pg_stat_t

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: If the requested OSD is down don't trust osd_stat info

If we have a down AND out OSD it may contains osd_stat with num_pgs == 0.
When all PGs aren't active+clean we need an accurate value or we consider
the osd missing stat info.

Fixes: https://tracker.ceph.com/issues/38930
Signed-off-by: David Zafman <dzafman@redhat.com>

mon, mgr: Improve PG count by changing up to up_no_acting in pg_count

By not adding a field to pg_count, we don't have to worry about
versioning or backwards compatibility.  A mixed cluster won't
be any worse than before.

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Improve ok-to-stop by using the avail_no_missing for recovery

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Prevent priority from overflowing in the next base level

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add tests for pool recovery priority conversion

Signed-off-by: David Zafman <dzafman@redhat.com>

osd, mon: New pool recovery priority range -10 to 10

Use OSD_POOL_PRIORITY_MAX and OSD_POOL_PRIORITY_MIN constants
Scale legacy priorities if exceeds maximum

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27781 from theanalyst/rgw-admin-delete-at

rgw: admin: handle delete_at attr in object stat output

Reviewed-By: Casey Bodley <cbodley@redhat.com>

os/bluestore: correctly measure deferred writes into new blobs

Writes into new blobs were all counted as write_small_new, but those can
still be deferred later in _do_alloc_write if they are <= than the
prefer_deferred setting.

See http://tracker.ceph.com/issues/38816

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: the first post-migration snapshot isn't always dirty

Currently, the first post-migration snapshot is always marked EXISTS
(i.e. dirty).  This is wrong, because the data can be inherited from
a pre-migration snapshot, handled by deep copy.

Mark all post-migration snapshots EXISTS_CLEAN in this case.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't update snapshot object maps if copyup data is all zeros

If the data read from the parent is all zeros, deep copyup isn't
performed.  However snapshot object maps are updated unconditionally,
causing inconsistencies where nonexistent objects are marked
OBJECT_EXISTS or OBJECT_EXISTS_CLEAN.

Fixes: http://tracker.ceph.com/issues/39435
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/upgrade/*-x-singleton: enable bluestore debugging settings

We default to bluestore already; use the yaml explicitly so that we get
debugging turned up.

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: admin: handle delete_at attr in object stat output

Fixes: https://tracker.ceph.com/issues/39481
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27574 from iotcg/iwarp_rocev2

msg/async/rdma: fix memory leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: use ibv_device_attr object type in Device class

1. Avoid to do memory management without using pointer to operate the
allocated space.
2. Since member type has been changed in class Device, it use member
domain operator "." to access to the sub-member in object.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

msg/async/rdma: free memory when Infiniband obj is freed

device_list is allocated in Infiniband::init.
When Infiniband obj is destructed, free the allocated
space in case of memory leak.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

msg/async/rdma: check allocated memory immediately before using it

The new "allocation functions" will through exception if it failes to
allocate space. There's no need to assert(devices) here for every
get_device operation.
Refer to: https://en.cppreference.com/w/cpp/memory/new/bad_alloc

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

msg/async/rdma: free member hold memory when destructing obj

device_context_list hold the memory allocated through
ibv_get_device_list. release the memory through calling
ibv_free_device_list.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

msg/sync/rdma: abort rdma operation when ibv_query_device failed

ibv_query_device return none zero when failing to query rdma
device. run ceph_abort in any case when ibv_query_device return
none zero.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

Merge pull request #27741 from tchaikov/doc/dev/wip-corpus.rst

doc/dev/corpus.rst: correct instructions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27626 into master

* refs/pull/27626/head:
mgr/dashboard: Do not default to 'admin' as Admin Resource

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Gabriel Brascher <gabriel@apache.org>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

Merge PR #26184 into master

* refs/pull/26184/head:
mgr/dashboard: Replace IP address validation with Python standard library functions

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #26730 into master

* refs/pull/26730/head:
src/test: remove unused objects to calculate accurate dedup ratio.
src/tools: fix counting total objects
src/tools: fix printing wrong offset.
src/tools: fix dedup tool's name
qa/workunit: add rabin chunk test
src/tool: add rabin fingerprint option to cephdeduptool
src/common: add rabin fingerprint class
Initail work for rabin fingerprint

Reviewed-by: Sage Weil <sage@redhat.com>

test: Remove redundant mon-seesaw.yaml file

mon.yaml runs all scripts in the qa/standalone/mon dir

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: For recovery track OSDs that have 0 missing to know how degraded we are

Add backfill priority log message and remove redundanacy to recovery priority

Fixes: https://tracker.ceph.com/issues/39099
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #27566 into master

* refs/pull/27566/head:
auth: kill AuthUnknownSessionHandler
auth: kill AuthUnknown* (except AuthUnknownSessionHandler)
msg: remove old ms_* auth methods from Dispatcher interface
mon/MonClient: discard old challenge if authorizer is bad
msg/async/ProtocolV1: use AuthServer and AuthClient
mon/Monitor: handle v1 call into handle_auth_request
msg/Connection: add is_msgr2()
mon/MonClient: tolerate lack of authorizer for some dispatchers

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27579 from cbodley/wip-rgw-split-log-trim

rgw: split mdlog/datalog trimming into separate files

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

qa/suites/upgrade: use correct branch names

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26173 from cbodley/wip-rgw-async-putobjprocessor

rgw: add YieldingAioThrottle for async PutObj/GetObj

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

auth: kill AuthUnknownSessionHandler

The only user is ProtocolV1, which passes in an in-use auth_method...
which must be either AUTH_NONE or AUTH_CEPHX since we successfully
authenticated at this point.

Signed-off-by: Sage Weil <sage@redhat.com>

auth: kill AuthUnknown* (except AuthUnknownSessionHandler)

These are unreferenced by any other code.

Signed-off-by: Sage Weil <sage@redhat.com>

msg: remove old ms_* auth methods from Dispatcher interface

In the mon there is an opportunity to further clean up the code around
get_authorizer().

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: discard old challenge if authorizer is bad

If we are a server/accepter, and we get a bad authorizer, discard our old
challenge, since the previous exchange has failed.  This way if the client
retries, their new authorize attempt won't conflict with our old challenge.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV1: use AuthServer and AuthClient

Stop using the old ms_* auth methods and instead use the new interfaces
(like V2).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27279 from dzafman/wip-divergent

Improvements to standalone tests

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27750 from tchaikov/wip-silence-gcc-warning

test/objectstore: silence -Wsign-compare warning

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #27721 into master

* refs/pull/27721/head:
tools: pin the version of breathe that works with Python2

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: avoid repeatedly invoking is_zero() in CopyupRequest

Invoke is_zero() just once, either in handle_read_from_parent() or
handle_deep_copy().

Initialize m_copyup_is_zero to true to preserve existing behavior:
on an empty bufferlist is_zero() returns true.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: leak on canceling simple io scheduler timer task

When the timer task is canceled it is just deleted and the passed
context will not be executed.

Signed-off-by: Mykola Golub <mgolub@suse.com>

test/rgw: rgw throttle tests use mock operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to Aio::librados_op()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to get_obj_data

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to PutObjProcessors

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27221 from LenzGr/update-backportscripts

script: Obtain milestones via github API

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw: read ops use make_throttle() with optional_yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: write ops use make_throttle() with optional_yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add make_throttle() for optionally-async AioThrottle

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add YieldingAioThrottle

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add BlockingAioThrottle

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25692 from Liuchang0812/fix-26835

 rgw: returns LimitExceeded when user creates too many ACLs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25044 from tianshan/fix_36763

rgw: set null version object acl issues

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21154 from mslovy/rgw-admin-api

rgw:  update op_mask of user via admin rest api

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17338 from tianshan/fix_roles_pool

rgw: add roles_pool in RGWZoneParams dump/decode json

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17591 from shashalu/fix-bucket-list-versions

rgw: list_bucket versions return NextVersionIdMarker = "null" if next_marker.instance is empty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #27472 into master

* refs/pull/27472/head:
doc/rados/operations/devices: document device failure prediction

Reviewed-by: Rick Chen <rick.chen@prophetstor.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #27573 from iotcg/corpus

import-generated.sh: use PATH to get ceph-dencoder

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27696 into master

* refs/pull/27696/head:
osd: make use of pg history and past_intervals in pg_create2 messages
mon/OSDMonitor: track history and past_intervals for creating pgs
osd/osd_types: make PastIntervals pi_compact_rep print participants
osd/osd_types: take bare const OSDMap * to check_new_interval
osd/osd_types: add pg_history_t ctor that takes creation epoch+stamp

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27656 into master

* refs/pull/27656/head:
doc/dev/erasure-coded-pool: update
doc/rados/operations/erasure-code*: update default ec profile references
common/options: change default erasure-code-profile to k=2 m=2

Reviewed-by: Neha Ojha <nojha@redhat.com>

test/objectstore: silence -Wsign-compare warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27683 from dillaman/wip-24668

qa/workunits/rbd: use more recent qemu-iotests that support Bionic

Reviewed-by: Mykola Golub <mgolub@suse.com>

script: Obtain milestones via github API

Replace the hard-coded list of milestones with an API request
that obtains this data from github directly.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

import-generated.sh: correct access to ceph-dencoder

align with instructions in doc/dev/corpus.rst.
doc/dev/corpus.rst is under change by kchai@redhat.com
in PR 27552.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

doc/dev/corpus.rst: correct instructions

* should keep all OSD up and running, otherwise some tests will wait for
  healthy cluster for ever.
* should start 2 MGR for an active-standby setting -- better coverage
this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27704 from tchaikov/wip-denc-dump

denc: allow DencDumper to dump OOB buffer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: iSCSI: Limit members to 1 group (#27520)

mgr/dashboard: iSCSI: Limit members to 1 group

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27722 from rhcs-dashboard/39434-prometheus-replace-whitespaces

mgr/prometheus: replace whitespaces in metrics' names

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27678 from tspmelo/wip-iscsi-discovery-read

mgr/dashboard: Fix iSCSI Discovery user permissions

Merge pull request #27609 from tspmelo/wip-iscsi-form-warn

mgr/dashboard: Fix iSCSI target form warning

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27720 from dillaman/wip-39407

rbd-mirror: clear out bufferlist prior to listing mirror images

Reviewed-by: Mykola Golub <mgolub@suse.com>

src/test: remove unused objects to calculate accurate dedup ratio.

Signed-off-by: Myoungwon Oh <ommw@sk.com>

denc: use scope_guard to close(fd)

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush,msg: silence -Wsign-compare warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

denc: allow DencDumper to dump OOB buffer

if encoder encodes buffers into the bufferlist directly, the
contiguous_appender's out_of_band_offset would be greater than zero, in
that case, we should dump the bufferlist passed to appender instead of
just looking at the [pre_pos, post_pos).

Signed-off-by: Kefu Chai <kchai@redhat.com>

denc: rewrite DENC_DUMP_{PRE,POST} using template

* for better readability
* for better debugging experience
* so we can `friend` it in a class in a follow-up change

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27719 from xiexingguo/wip-crush-node-flags

mon/OSDMonitor: crush node flags - two fixes; add tests

Reviewed-by: Sage Weil <sage@redhat.com>