Merge pull request #27755 from trociny/wip-rbd-io-scheduler-leak

librbd: leak on canceling simple io scheduler timer task

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27411 from ivancich/wip-clean-up-logging

rgw: clean up some logging

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #27456 from wjwithagen/wjw-fix-rgw_amqp.cc

rgw: use the compatibilty function for pthread_setname

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27617 from cbodley/wip-39295

rgw: bucket stats report mtime in UTC

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #27503 from dzafman/wip-39099

osd: Give recovery for inactive PGs a higher priority

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27764 from neha-ojha/wip-fix-mimic-x-singleton

qa/suites/upgrade: use correct branch names

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test: osd-recovery-scrub.sh ignore error from kill_daemons()

Another work around for http://tracker.ceph.com/issues/38195

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add new object_location_counts to pg_stat_t

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: If the requested OSD is down don't trust osd_stat info

If we have a down AND out OSD it may contains osd_stat with num_pgs == 0.
When all PGs aren't active+clean we need an accurate value or we consider
the osd missing stat info.

Fixes: https://tracker.ceph.com/issues/38930
Signed-off-by: David Zafman <dzafman@redhat.com>

mon, mgr: Improve PG count by changing up to up_no_acting in pg_count

By not adding a field to pg_count, we don't have to worry about
versioning or backwards compatibility.  A mixed cluster won't
be any worse than before.

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Improve ok-to-stop by using the avail_no_missing for recovery

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Prevent priority from overflowing in the next base level

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add tests for pool recovery priority conversion

Signed-off-by: David Zafman <dzafman@redhat.com>

osd, mon: New pool recovery priority range -10 to 10

Use OSD_POOL_PRIORITY_MAX and OSD_POOL_PRIORITY_MIN constants
Scale legacy priorities if exceeds maximum

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27781 from theanalyst/rgw-admin-delete-at

rgw: admin: handle delete_at attr in object stat output

Reviewed-By: Casey Bodley <cbodley@redhat.com>

rgw: admin: handle delete_at attr in object stat output

Fixes: https://tracker.ceph.com/issues/39481
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #27574 from iotcg/iwarp_rocev2

msg/async/rdma: fix memory leak

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg/async/rdma: use ibv_device_attr object type in Device class

1. Avoid to do memory management without using pointer to operate the
allocated space.
2. Since member type has been changed in class Device, it use member
domain operator "." to access to the sub-member in object.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

msg/async/rdma: free memory when Infiniband obj is freed

device_list is allocated in Infiniband::init.
When Infiniband obj is destructed, free the allocated
space in case of memory leak.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

msg/async/rdma: check allocated memory immediately before using it

The new "allocation functions" will through exception if it failes to
allocate space. There's no need to assert(devices) here for every
get_device operation.
Refer to: https://en.cppreference.com/w/cpp/memory/new/bad_alloc

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

msg/async/rdma: free member hold memory when destructing obj

device_context_list hold the memory allocated through
ibv_get_device_list. release the memory through calling
ibv_free_device_list.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

msg/sync/rdma: abort rdma operation when ibv_query_device failed

ibv_query_device return none zero when failing to query rdma
device. run ceph_abort in any case when ibv_query_device return
none zero.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

Merge pull request #27741 from tchaikov/doc/dev/wip-corpus.rst

doc/dev/corpus.rst: correct instructions

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27626 into master

* refs/pull/27626/head:
mgr/dashboard: Do not default to 'admin' as Admin Resource

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Gabriel Brascher <gabriel@apache.org>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>

Merge PR #26184 into master

* refs/pull/26184/head:
mgr/dashboard: Replace IP address validation with Python standard library functions

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #26730 into master

* refs/pull/26730/head:
src/test: remove unused objects to calculate accurate dedup ratio.
src/tools: fix counting total objects
src/tools: fix printing wrong offset.
src/tools: fix dedup tool's name
qa/workunit: add rabin chunk test
src/tool: add rabin fingerprint option to cephdeduptool
src/common: add rabin fingerprint class
Initail work for rabin fingerprint

Reviewed-by: Sage Weil <sage@redhat.com>

osd: For recovery track OSDs that have 0 missing to know how degraded we are

Add backfill priority log message and remove redundanacy to recovery priority

Fixes: https://tracker.ceph.com/issues/39099
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #27566 into master

* refs/pull/27566/head:
auth: kill AuthUnknownSessionHandler
auth: kill AuthUnknown* (except AuthUnknownSessionHandler)
msg: remove old ms_* auth methods from Dispatcher interface
mon/MonClient: discard old challenge if authorizer is bad
msg/async/ProtocolV1: use AuthServer and AuthClient
mon/Monitor: handle v1 call into handle_auth_request
msg/Connection: add is_msgr2()
mon/MonClient: tolerate lack of authorizer for some dispatchers

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27579 from cbodley/wip-rgw-split-log-trim

rgw: split mdlog/datalog trimming into separate files

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

qa/suites/upgrade: use correct branch names

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26173 from cbodley/wip-rgw-async-putobjprocessor

rgw: add YieldingAioThrottle for async PutObj/GetObj

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

auth: kill AuthUnknownSessionHandler

The only user is ProtocolV1, which passes in an in-use auth_method...
which must be either AUTH_NONE or AUTH_CEPHX since we successfully
authenticated at this point.

Signed-off-by: Sage Weil <sage@redhat.com>

auth: kill AuthUnknown* (except AuthUnknownSessionHandler)

These are unreferenced by any other code.

Signed-off-by: Sage Weil <sage@redhat.com>

msg: remove old ms_* auth methods from Dispatcher interface

In the mon there is an opportunity to further clean up the code around
get_authorizer().

Signed-off-by: Sage Weil <sage@redhat.com>

mon/MonClient: discard old challenge if authorizer is bad

If we are a server/accepter, and we get a bad authorizer, discard our old
challenge, since the previous exchange has failed.  This way if the client
retries, their new authorize attempt won't conflict with our old challenge.

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV1: use AuthServer and AuthClient

Stop using the old ms_* auth methods and instead use the new interfaces
(like V2).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27279 from dzafman/wip-divergent

Improvements to standalone tests

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27750 from tchaikov/wip-silence-gcc-warning

test/objectstore: silence -Wsign-compare warning

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #27721 into master

* refs/pull/27721/head:
tools: pin the version of breathe that works with Python2

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: leak on canceling simple io scheduler timer task

When the timer task is canceled it is just deleted and the passed
context will not be executed.

Signed-off-by: Mykola Golub <mgolub@suse.com>

test/rgw: rgw throttle tests use mock operations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to Aio::librados_op()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to get_obj_data

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add optional_yield to PutObjProcessors

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27221 from LenzGr/update-backportscripts

script: Obtain milestones via github API

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

rgw: read ops use make_throttle() with optional_yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: write ops use make_throttle() with optional_yield

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add make_throttle() for optionally-async AioThrottle

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add YieldingAioThrottle

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add BlockingAioThrottle

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25692 from Liuchang0812/fix-26835

 rgw: returns LimitExceeded when user creates too many ACLs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25044 from tianshan/fix_36763

rgw: set null version object acl issues

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #21154 from mslovy/rgw-admin-api

rgw:  update op_mask of user via admin rest api

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17338 from tianshan/fix_roles_pool

rgw: add roles_pool in RGWZoneParams dump/decode json

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #17591 from shashalu/fix-bucket-list-versions

rgw: list_bucket versions return NextVersionIdMarker = "null" if next_marker.instance is empty

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #27472 into master

* refs/pull/27472/head:
doc/rados/operations/devices: document device failure prediction

Reviewed-by: Rick Chen <rick.chen@prophetstor.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #27573 from iotcg/corpus

import-generated.sh: use PATH to get ceph-dencoder

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27696 into master

* refs/pull/27696/head:
osd: make use of pg history and past_intervals in pg_create2 messages
mon/OSDMonitor: track history and past_intervals for creating pgs
osd/osd_types: make PastIntervals pi_compact_rep print participants
osd/osd_types: take bare const OSDMap * to check_new_interval
osd/osd_types: add pg_history_t ctor that takes creation epoch+stamp

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27656 into master

* refs/pull/27656/head:
doc/dev/erasure-coded-pool: update
doc/rados/operations/erasure-code*: update default ec profile references
common/options: change default erasure-code-profile to k=2 m=2

Reviewed-by: Neha Ojha <nojha@redhat.com>

test/objectstore: silence -Wsign-compare warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27683 from dillaman/wip-24668

qa/workunits/rbd: use more recent qemu-iotests that support Bionic

Reviewed-by: Mykola Golub <mgolub@suse.com>

script: Obtain milestones via github API

Replace the hard-coded list of milestones with an API request
that obtains this data from github directly.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

import-generated.sh: correct access to ceph-dencoder

align with instructions in doc/dev/corpus.rst.
doc/dev/corpus.rst is under change by kchai@redhat.com
in PR 27552.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>

doc/dev/corpus.rst: correct instructions

* should keep all OSD up and running, otherwise some tests will wait for
  healthy cluster for ever.
* should start 2 MGR for an active-standby setting -- better coverage
this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27704 from tchaikov/wip-denc-dump

denc: allow DencDumper to dump OOB buffer

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: iSCSI: Limit members to 1 group (#27520)

mgr/dashboard: iSCSI: Limit members to 1 group

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27722 from rhcs-dashboard/39434-prometheus-replace-whitespaces

mgr/prometheus: replace whitespaces in metrics' names

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #27678 from tspmelo/wip-iscsi-discovery-read

mgr/dashboard: Fix iSCSI Discovery user permissions

Merge pull request #27609 from tspmelo/wip-iscsi-form-warn

mgr/dashboard: Fix iSCSI target form warning

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #27720 from dillaman/wip-39407

rbd-mirror: clear out bufferlist prior to listing mirror images

Reviewed-by: Mykola Golub <mgolub@suse.com>

src/test: remove unused objects to calculate accurate dedup ratio.

Signed-off-by: Myoungwon Oh <ommw@sk.com>

denc: use scope_guard to close(fd)

Signed-off-by: Kefu Chai <kchai@redhat.com>

crush,msg: silence -Wsign-compare warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

denc: allow DencDumper to dump OOB buffer

if encoder encodes buffers into the bufferlist directly, the
contiguous_appender's out_of_band_offset would be greater than zero, in
that case, we should dump the bufferlist passed to appender instead of
just looking at the [pre_pos, post_pos).

Signed-off-by: Kefu Chai <kchai@redhat.com>

denc: rewrite DENC_DUMP_{PRE,POST} using template

* for better readability
* for better debugging experience
* so we can `friend` it in a class in a follow-up change

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27719 from xiexingguo/wip-crush-node-flags

mon/OSDMonitor: crush node flags - two fixes; add tests

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26107 from alimaredia/wip-dpp-rgw-put-obj-processor

rgw: log refactoring for putobj_processor

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: log refactoring for AtomicObjProcessor complete

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge PR #27628 into master

* refs/pull/27628/head:
common: optimize check_utf8

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27702 into master

* refs/pull/27702/head:
osd/PG: fix last_complete re-calculation on splitting

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27691 into master

* refs/pull/27691/head:
os/bluestore: fix out-of-bound access in bmap allocator.
os/bluestore: reproduce out-of-bound access for bmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27707 into master

* refs/pull/27707/head:
common/util: handle long lines in /proc/cpuinfo

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #27537 into master

* refs/pull/27537/head:
mds: better output of 'ceph health detail'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27511 into master

* refs/pull/27511/head:
mds: fix SnapRealm::resolve_snapname for long name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27077 into master

* refs/pull/27077/head:
test: check listattr for snapshot btime entry
test: extend LibCephFS.Xattrs test
client: remove unused vxattr length helpers
client: fix _listxattr() vxattr buffer length calculation
test: add libcephfs snap.btime xattr test
client: add ceph.snap.btime vxattr
mds: carry snapshot creation time with InodeStat

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #27725 from theanalyst/perf-counter-names

rgw: sync counters: drop spaces from counter names

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>

rgw: sync counters: drop spaces from counter names

Since this might break modules like prometheus and general json processing tools
aren't too happy with spaces.

Fixes: https://tracker.ceph.com/issues/39434
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/prometheus: replace whitespaces in metrics' names

Fixes: https://tracker.ceph.com/issues/39434
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #27703 from tchaikov/wip-rbd-replay-denc

rbd_replay: call the member decode() explicitly

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Clean up TableComponent tests and code (#26784)

mgr/dashboard: Clean up TableComponent tests and code

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

tools: pin the version of breathe that works with Python2

Signed-off-by: Alfredo Deza <adeza@redhat.com>

rbd_replay: call the member decode() explicitly

otherwise, the one defined using WRITE_RAW_ENCODER is called instead.
so in this change, rename the the member function which happens to have
the same signature with

decode(type &v, ::ceph::bufferlist::const_iterator& p)

where `type` is `__u8`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: clear out bufferlist prior to listing mirror images

The second call to list mirrored images will fail deep within the
msgr code due to a "bad crc in data" error.

Fixes: http://tracker.ceph.com/issues/39407
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27521 from trociny/wip-rbd-remove-clone_v2-parent

librbd: optionally move parent image to trash on remove

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27484 from majianpeng/rbd-nbd

rbd-nbd: sscanf return 0 mean not-match

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/rados/operations/devices: document device failure prediction

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27697 from cbodley/wip-rgw-bucket-list-unordered

 rgw: cls_bucket_list_unordered lists a single shard

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

osd: make use of pg history and past_intervals in pg_create2 messages

If we get a mismatched epoch and past_intervals, error out early, or else
we'll end up asserting later in the PastIntervals code.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Replace IP address validation with Python standard library functions

Instead of self-written validation methods to validate IPv4 and IPv6 addresses.
Use Python's standard library functions `ipaddress`.

Signed-off-by: Ashish Singh <assingh@redhat.com>

Merge pull request #27713 from tchaikov/wip-24842

doc/rbd/rbd-cloudstack: update disk offering URL to new docs

Reviewed-by: Wido den Hollander <wido@42on.com>