osd/ReplicatedBackend: handle tailing zeros of recovering objects nicer

E.g.,
(1) recovery_info.copy_subset = [0, 4194304]
(2) copy_subset(fiemap) = [0, 8192]
(3) out_op->data_included.span_of(copy_subset, 0, 4194304) = [0, 8192]

We'll set new_progress.data_recovered_to = 8192 as a result, and
fail to complete the push in one go.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27694 from dillaman/wip-39386

qa/suites/rbd: added writearound cache test permutations

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #27682 from dillaman/wip-39031

librbd: async open/close should free ImageCtx before issuing callback

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #24744 from liewegas/wip-stale-prs

.github/stale.yml: warn at 60, close at 90; adjust message

Merge PR #27605 into master

* refs/pull/27605/head:
mon/OSDMonitor: osd add-no{up,down,in,out} - remove state checker

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27695 from dillaman/wip-39072

test/librbd: drop 'ceph_test_librbd_api' target

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #27288 into master

* refs/pull/27288/head:
osd/PG: discover missing objects when an OSD peers and PG is degraded

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27571 into master

* refs/pull/27571/head:
mgr/telemetry: add report_timestamp to sent reports

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27572 into master

* refs/pull/27572/head:
denc: fix build error by calling global snprintf

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27622 into master

* refs/pull/27622/head:
mon: fix INCOMPAT_OCTOPUS feature number
mon/Elector: format mon_release correctly

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27577 from dillaman/wip-rbd-suite-distros

qa/suites/rbd: add random distro selection to librbd tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #27690 from tchaikov/wip-crimson-one-safeguard-is-better-than-two

crimson: pass `Connection*` to Dispatch::ms_dispatch()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #27595 into master

* refs/pull/27595/head:
osd: add 'ceph osd stop <osd.nnn>' command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27619 into master

* refs/pull/27619/head:
mgr/BaseMgrModule: run MonCommandCompletion on the finisher
mgr/BaseMgrModule: fix leak

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #27651 into master

* refs/pull/27651/head:
test/osd/safe-to-destroy.sh: fix typo

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27663 into master

* refs/pull/27663/head:
osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27631 from dmick/wip-telemetry-crasharray

mgr/telemetry: change crash dict to a list

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27599 from gregsfortytwo/wip-stale-prs

Miscellaneous lost fixes

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27615 from ifed01/wip-ifed-verbose-bluefs-replay

os/bluestore: be more verbose doing bluefs log replay.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27509 from Aran85/fix-pg-dout

osd/PG: the warning seems more serious than what it wanna transmit

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27592 from tchaikov/wip-ref

message,mgr: drop MessageFactory and friends and use ref_t<> in mgr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27633 from xiexingguo/wip-39340

osd/osd_types: pool_stat_t::dump - fix 'num_store_stats' field

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27658 from gregsfortytwo/wip-39367-crc-peer

msg: output peer address when detecting bad CRCs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: pass `Connection*` to Dispatch::ms_dispatch()

currently, we use a `with_gate()` in `ProtocolV2::read_message()` for
ensuring that `this` (or `Connection` holding this protocol instance)
will outlive the continuation of `dispatcher.ms_dispatch()` which
references `this->dispatcher`. but we also pass a strong reference of
connection to dispatcher. in short, we have *two* safeguards for the
same purpose.

in this change, one of these safeguards is removed -- to pass the raw
pointer of `Connection` to `Dispatch::ms_dispatch()`. the reason why
the `with_gate()` is kept is that, if we have removed `with_gate()` in
Protocol, we need to

1. let `Dispatcher::ms_dispatch()` return `void`, as it should not block
any succeeding calls.
2. add a `with_gate()` in `Dispatcher::ms_dispatch()` to ensure that
`this` is alive during the lifecycle of the continuation(s) in
`Dispatcher::ms_dispatch()`.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27657 from anthonyeleven/patch-1

doc: operations: improve reweight-by-utilization

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: operations: reweight-by-utilization typo

Add a missing backquote delimiter.

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

test/librbd: drop 'ceph_test_librbd_api' target

This was used for API backwards compatibility testing, but now that
the C++ API will not remain stable, it serves no purpose.

Fixes: http://tracker.ceph.com/issues/39072
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: added writearound cache test permutations

Also tweaked the existing writeback cache test permutations to
specify the use of the writeback cache policy.

Fixes: http://tracker.ceph.com/issues/39386
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/BaseMgrModule: run MonCommandCompletion on the finisher

The completion takes the ActivePyModules::lock, but the caller may be
holding an internal lock for Objecter and/or MonClient.  Avoid deadlock
by putting the completion on our finisher.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/BaseMgrModule: fix leak

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27685 into master

* refs/pull/27685/head:
doc: fix boot transition in mds state diagram

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27686 from dzafman/wip-39383

osd: Increase log level of messages which unnecessarily fill up logs

Reviewed-by: Neha Ojha <nojha@redhat.com>

osd: Increase log level of messages which unnecessarily fill up logs

Caused by: 834d3c19a774f1cc93903447d91d182776e12d18

Fixes: https://tracker.ceph.com/issues/39383
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27037 from alimaredia/wip-rgw-dpp-rest-s3-swift

rgw: log refactoring for rgw_rest_s3/swift ops

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: fix boot transition in mds state diagram

MDSMonitor assigns a booting MDS immediately to standby. From there, the
standby may be promoted to standby-replay or some state leading to active.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27375 into master

* refs/pull/27375/head:
rgw: pass mostly_omap flag when opening/creating pools
rgw/rgw_rados: pass mostly_omap flag when opening/creating pools
rgw: move rgw_init_ioctx() to rgw_tools.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27576 from cbodley/wip-39287

rgw: RGWPeriodPusher uses zone system key for inter-zonegroup messages

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

rbd: fixed memory leak in 'rbd mirror pool' multi-image actions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: async open/close should free ImageCtx before issuing callback

The destructor for ImageCtx attempts to access librados::IoCtx objects
that might have been destroyed immediately after the async open (failure)
and close actions completed.

Fixes: http://tracker.ceph.com/issues/39031
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27518 from jtlayton/vstart

vstart: add new option to pass list of block devices to bluestore

Merge pull request #27567 from cbodley/wip-39283

rgw: multisite log trimming only checks peers that sync from us

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/rbd_fio: fixed missing delimiter between 'cd' and 'configure'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: add random distro selection to librbd tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27275 from Devp00l/issue-39037

mgr/dashboard: Localization for date picker module

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

osd: add 'ceph osd stop <osd.nnn>' command

stop command can be used to force stopping a specified osd daemon, e.g.,
you don't have to pre-figure out where it located.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Fixes: http://tracker.ceph.com/issues/39154
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #27235 into master

* refs/pull/27235/head:
cephfs-shell: support non-octal mode
cephfs-shell: fix rm command
cephfs-shell: fix lls command
cephfs-shell: fix lcd command
cephfs-shell: fix umask command
cephfs-shell: fix cat command
cephfs-shell: fix chmod command
cephfs-shell: fix chdir command
cephfs-shell: fix mv command
cephfs-shell: fix rmdir command
cephfs-shell: fix ls command
cephfs-shell: fix get command
cephfs-shell: fix put command
cephfs-shell: fix mkdir command
cephfs-shell: helper functions and infra changes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

msg/Message: add SafeMessage which forbids put/get

This is an opt-in wrapper around Message inheritance to prevent
undesired get/put calls on the object. Instead, you must/should use a
smart ptr to adjust the reference count.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/MgrClient: use ref_t<M>

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/{Mgr,MgrStandby}: use ref_t<M>

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonState: pass const by reference

DaemonPerfCounters::update() does not change the `MMgrReport` message,
neither does it expect a nullptr. so pass by const reference.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonServer: use ref_t<M>

to avoid explicit use of RefCountedObj interface. and let
intrusive_ptr<> to take care of the life cycle of messages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/Message: remove Message::const_ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: use ref_t<M>

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/Message: drop MessageFactory and friends

we'd better decouple the helpers from the definition of Message types,
to keep the definition Messages simple and concise.

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MOSDPeeringOp: do not include PGPeeringEvent.h

forward delcaration would suffice

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages: do not use MessageSubType anymore

so we can drop it in future commit

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages: #include necessary header

for instance, MLogRec is defined in PGPeeringEvent.h, we should not rely
other header(s) to do this for us. so just #include PGPeeringEvent.h for
its definition.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds/MDSCache: init empty string_view with {}

nullptr is not a valid null-terminated character sequence

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/: use make_message<Foo> instead of Foo::create()

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/Message: remove msgref_cast()

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/Message: add aliases for static_pointer_cast

strictly speaking, they are not "alias" of boost::static_pointer_cast,
but to avoid an extra conversion, instead of using
boost::static_pointer_cast<>, i am using std::static_cast<> directly for
creating boost::intrusive_ptr<> instance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

client,mds: use ref_cast<Foo> instead of Foo::msgref_cast

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/: use ref_t<Message> instead of Message::ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/: use cref_t<Message> instead of Message::const_ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/RefCountedObj: move private members

To follow code-style guidelines.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/RefCountedObj: use type of nref

"int" may narrow the ref count (although very unlikely!).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: add aliases of ref_t and cref_t

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27649 into master

* refs/pull/27649/head:
PendingReleaseNotes: 14.2.1 note on crush required version

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

msg: output peer address when detecting bad CRCs

Fixes: http://tracker.ceph.com/issues/39367
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #27600 from neha-ojha/wip-39006

osd/OSD.cc: make osd bench description consistent with parameters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27481 into master

* refs/pull/27481/head:
qa: add a method to test if a file can be created
qa: add wrappers for vstart's run() with check_status as False
qa: add a method to create an empty file with any user
qa: add stdin parameter to run_shell
qa: command to be ran as a different user should be a single string
qa: looks for quotes in Python shell and get rid of them
qa: move checks on command arguments to a different method
qa: accept args as strings
qa: add a method to run given command as root user
qa: add a method to run a command as a different user
qa: allow keeping sudo in command arguments
qa: make execution abortion optional for command ran through run_shell()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

doc: operations: improve reweight-by-utilization

Add the missing `max_change`, `max_osds`, and `--no-increasing` parameters to `reweight-by-utilization` and `test-reweight-by-utilization`.  Minor adjustments to wording.

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #27282 from mogeb/wip-lvmcache

ceph-volume: add utility functions

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

qa: add a method to test if a file can be created

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa: add wrappers for vstart's run() with check_status as False

Too often in tests it is required to run a command that we want to (or
will probably) fail so that we can check if it failed for expected
reason(s) (eg. a testcase where command must fail with "Permission
denied"). Therefore, add few wrapper methods for vstart_runner.py's
run() with check_status's default value as False so that we don't need
to pass check_status's value at every call to run() or it's other
wrappers.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa: add a method to create an empty file with any user

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa: add stdin parameter to run_shell

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa: command to be ran as a different user should be a single string

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test/osd/safe-to-destroy.sh: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: add async msg perf counter

Signed-off-by: Xinze Chi <xinze@xsky.com>
Updated to account for the new ProtocolV1/ProtocolV2

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

tools/crushtool: add new Option --bucket-tree to show bucket map tree

Signed-off-by: linbing <linbing@t2cloud.net>
Update the crushtool cli test template for new options.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

msg/async: bug fix 23600 by setting _fd = 1.

After listen_socket was closed, it entered the start() processf magically. so
I think we should make '_fd = -1' when listen_socket was closed. And in the
start process, determine the value of listen_socket, if it equal to -1, return
the process.

Fixes: https://tracker.ceph.com/issues/23600
Signed-off-by: shangfufei <shangfufei@inspur.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: be more verbose doing bluefs log replay.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

PendingReleaseNotes: 14.2.1 note on crush required version

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27581 from LenzGr/master-documentation

doc: Improved the dashboard proxy config section

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

osd/osd_types: pool_stat_t::dump - fix 'num_store_stats' field

Fixes: http://tracker.ceph.com/issues/39340
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27593 from smithfarm/wip-39291

cmake: revert librados_tp.so version from 3 to 2

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

mgr/telemetry: change crash dict to a list

For ease of indexing.  There's no value in timestamp keys.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #27514 from tchaikov/wip-crimson-auth-client

crimson: enable cephx for v2 msgr

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #27624 into master

* refs/pull/27624/head:
doc: fixed telemetry module reference link

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/auth: let AuthClient::get_auth_request return a named struct

better readability this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: use const reference for Connection::auth_registry

it is not optional now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth/AuthRegistry: more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: remove "#warning" in v1 implementation

since we've implemented the v2 support, these reminders are not needed
anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: reset promise using `{}`

better readability, less repeating.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: remove unused AuthMethodList

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: use auth_registry for querying auth_methods

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fixed telemetry module reference link

Used :ref: label for telemetry reference in Nautilus.

Fixes: https://tracker.ceph.com/issues/39337
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

mon: fix INCOMPAT_OCTOPUS feature number

It's overlapping with nautilus, which is a problem!

Signed-off-by: Sage Weil <sage@redhat.com>