librbd: the first post-migration snapshot isn't always dirty

Currently, the first post-migration snapshot is always marked EXISTS
(i.e. dirty).  This is wrong, because the data can be inherited from
a pre-migration snapshot, handled by deep copy.

Mark all post-migration snapshots EXISTS_CLEAN in this case.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: don't update snapshot object maps if copyup data is all zeros

If the data read from the parent is all zeros, deep copyup isn't
performed.  However snapshot object maps are updated unconditionally,
causing inconsistencies where nonexistent objects are marked
OBJECT_EXISTS or OBJECT_EXISTS_CLEAN.

Fixes: http://tracker.ceph.com/issues/39435
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: avoid repeatedly invoking is_zero() in CopyupRequest

Invoke is_zero() just once, either in handle_read_from_parent() or
handle_deep_copy().

Initialize m_copyup_is_zero to true to preserve existing behavior:
on an empty bufferlist is_zero() returns true.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #27695 from dillaman/wip-39072

test/librbd: drop 'ceph_test_librbd_api' target

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #27288 into master

* refs/pull/27288/head:
osd/PG: discover missing objects when an OSD peers and PG is degraded

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #27571 into master

* refs/pull/27571/head:
mgr/telemetry: add report_timestamp to sent reports

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27572 into master

* refs/pull/27572/head:
denc: fix build error by calling global snprintf

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27622 into master

* refs/pull/27622/head:
mon: fix INCOMPAT_OCTOPUS feature number
mon/Elector: format mon_release correctly

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27577 from dillaman/wip-rbd-suite-distros

qa/suites/rbd: add random distro selection to librbd tests

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #27690 from tchaikov/wip-crimson-one-safeguard-is-better-than-two

crimson: pass `Connection*` to Dispatch::ms_dispatch()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #27595 into master

* refs/pull/27595/head:
osd: add 'ceph osd stop <osd.nnn>' command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27619 into master

* refs/pull/27619/head:
mgr/BaseMgrModule: run MonCommandCompletion on the finisher
mgr/BaseMgrModule: fix leak

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge PR #27651 into master

* refs/pull/27651/head:
test/osd/safe-to-destroy.sh: fix typo

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #27663 into master

* refs/pull/27663/head:
osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27631 from dmick/wip-telemetry-crasharray

mgr/telemetry: change crash dict to a list

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27599 from gregsfortytwo/wip-stale-prs

Miscellaneous lost fixes

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #27615 from ifed01/wip-ifed-verbose-bluefs-replay

os/bluestore: be more verbose doing bluefs log replay.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27509 from Aran85/fix-pg-dout

osd/PG: the warning seems more serious than what it wanna transmit

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27592 from tchaikov/wip-ref

message,mgr: drop MessageFactory and friends and use ref_t<> in mgr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27633 from xiexingguo/wip-39340

osd/osd_types: pool_stat_t::dump - fix 'num_store_stats' field

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27658 from gregsfortytwo/wip-39367-crc-peer

msg: output peer address when detecting bad CRCs

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson: pass `Connection*` to Dispatch::ms_dispatch()

currently, we use a `with_gate()` in `ProtocolV2::read_message()` for
ensuring that `this` (or `Connection` holding this protocol instance)
will outlive the continuation of `dispatcher.ms_dispatch()` which
references `this->dispatcher`. but we also pass a strong reference of
connection to dispatcher. in short, we have *two* safeguards for the
same purpose.

in this change, one of these safeguards is removed -- to pass the raw
pointer of `Connection` to `Dispatch::ms_dispatch()`. the reason why
the `with_gate()` is kept is that, if we have removed `with_gate()` in
Protocol, we need to

1. let `Dispatcher::ms_dispatch()` return `void`, as it should not block
any succeeding calls.
2. add a `with_gate()` in `Dispatcher::ms_dispatch()` to ensure that
`this` is alive during the lifecycle of the continuation(s) in
`Dispatcher::ms_dispatch()`.

Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27657 from anthonyeleven/patch-1

doc: operations: improve reweight-by-utilization

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: operations: reweight-by-utilization typo

Add a missing backquote delimiter.

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

test/librbd: drop 'ceph_test_librbd_api' target

This was used for API backwards compatibility testing, but now that
the C++ API will not remain stable, it serves no purpose.

Fixes: http://tracker.ceph.com/issues/39072
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/BaseMgrModule: run MonCommandCompletion on the finisher

The completion takes the ActivePyModules::lock, but the caller may be
holding an internal lock for Objecter and/or MonClient.  Avoid deadlock
by putting the completion on our finisher.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/BaseMgrModule: fix leak

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #27685 into master

* refs/pull/27685/head:
doc: fix boot transition in mds state diagram

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27686 from dzafman/wip-39383

osd: Increase log level of messages which unnecessarily fill up logs

Reviewed-by: Neha Ojha <nojha@redhat.com>

osd: Increase log level of messages which unnecessarily fill up logs

Caused by: 834d3c19a774f1cc93903447d91d182776e12d18

Fixes: https://tracker.ceph.com/issues/39383
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #27037 from alimaredia/wip-rgw-dpp-rest-s3-swift

rgw: log refactoring for rgw_rest_s3/swift ops

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: fix boot transition in mds state diagram

MDSMonitor assigns a booting MDS immediately to standby. From there, the
standby may be promoted to standby-replay or some state leading to active.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #27375 into master

* refs/pull/27375/head:
rgw: pass mostly_omap flag when opening/creating pools
rgw/rgw_rados: pass mostly_omap flag when opening/creating pools
rgw: move rgw_init_ioctx() to rgw_tools.cc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27576 from cbodley/wip-39287

rgw: RGWPeriodPusher uses zone system key for inter-zonegroup messages

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #27518 from jtlayton/vstart

vstart: add new option to pass list of block devices to bluestore

Merge pull request #27567 from cbodley/wip-39283

rgw: multisite log trimming only checks peers that sync from us

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/rbd_fio: fixed missing delimiter between 'cd' and 'configure'

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/suites/rbd: add random distro selection to librbd tests

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27275 from Devp00l/issue-39037

mgr/dashboard: Localization for date picker module

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

osd: add 'ceph osd stop <osd.nnn>' command

stop command can be used to force stopping a specified osd daemon, e.g.,
you don't have to pre-figure out where it located.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Fixes: http://tracker.ceph.com/issues/39154
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #27235 into master

* refs/pull/27235/head:
cephfs-shell: support non-octal mode
cephfs-shell: fix rm command
cephfs-shell: fix lls command
cephfs-shell: fix lcd command
cephfs-shell: fix umask command
cephfs-shell: fix cat command
cephfs-shell: fix chmod command
cephfs-shell: fix chdir command
cephfs-shell: fix mv command
cephfs-shell: fix rmdir command
cephfs-shell: fix ls command
cephfs-shell: fix get command
cephfs-shell: fix put command
cephfs-shell: fix mkdir command
cephfs-shell: helper functions and infra changes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

msg/Message: add SafeMessage which forbids put/get

This is an opt-in wrapper around Message inheritance to prevent
undesired get/put calls on the object. Instead, you must/should use a
smart ptr to adjust the reference count.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/MgrClient: use ref_t<M>

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/{Mgr,MgrStandby}: use ref_t<M>

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonState: pass const by reference

DaemonPerfCounters::update() does not change the `MMgrReport` message,
neither does it expect a nullptr. so pass by const reference.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/DaemonServer: use ref_t<M>

to avoid explicit use of RefCountedObj interface. and let
intrusive_ptr<> to take care of the life cycle of messages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/Message: remove Message::const_ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

common: use ref_t<M>

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/Message: drop MessageFactory and friends

we'd better decouple the helpers from the definition of Message types,
to keep the definition Messages simple and concise.

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MOSDPeeringOp: do not include PGPeeringEvent.h

forward delcaration would suffice

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages: do not use MessageSubType anymore

so we can drop it in future commit

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages: #include necessary header

for instance, MLogRec is defined in PGPeeringEvent.h, we should not rely
other header(s) to do this for us. so just #include PGPeeringEvent.h for
its definition.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds/MDSCache: init empty string_view with {}

nullptr is not a valid null-terminated character sequence

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/: use make_message<Foo> instead of Foo::create()

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/Message: remove msgref_cast()

Signed-off-by: Kefu Chai <kchai@redhat.com>

msg/Message: add aliases for static_pointer_cast

strictly speaking, they are not "alias" of boost::static_pointer_cast,
but to avoid an extra conversion, instead of using
boost::static_pointer_cast<>, i am using std::static_cast<> directly for
creating boost::intrusive_ptr<> instance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

client,mds: use ref_cast<Foo> instead of Foo::msgref_cast

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/: use ref_t<Message> instead of Message::ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

src/: use cref_t<Message> instead of Message::const_ref

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/RefCountedObj: move private members

To follow code-style guidelines.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/RefCountedObj: use type of nref

"int" may narrow the ref count (although very unlikely!).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: add aliases of ref_t and cref_t

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #27649 into master

* refs/pull/27649/head:
PendingReleaseNotes: 14.2.1 note on crush required version

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

msg: output peer address when detecting bad CRCs

Fixes: http://tracker.ceph.com/issues/39367
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #27600 from neha-ojha/wip-39006

osd/OSD.cc: make osd bench description consistent with parameters

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #27481 into master

* refs/pull/27481/head:
qa: add a method to test if a file can be created
qa: add wrappers for vstart's run() with check_status as False
qa: add a method to create an empty file with any user
qa: add stdin parameter to run_shell
qa: command to be ran as a different user should be a single string
qa: looks for quotes in Python shell and get rid of them
qa: move checks on command arguments to a different method
qa: accept args as strings
qa: add a method to run given command as root user
qa: add a method to run a command as a different user
qa: allow keeping sudo in command arguments
qa: make execution abortion optional for command ran through run_shell()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

doc: operations: improve reweight-by-utilization

Add the missing `max_change`, `max_osds`, and `--no-increasing` parameters to `reweight-by-utilization` and `test-reweight-by-utilization`.  Minor adjustments to wording.

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #27282 from mogeb/wip-lvmcache

ceph-volume: add utility functions

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

qa: add a method to test if a file can be created

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa: add wrappers for vstart's run() with check_status as False

Too often in tests it is required to run a command that we want to (or
will probably) fail so that we can check if it failed for expected
reason(s) (eg. a testcase where command must fail with "Permission
denied"). Therefore, add few wrapper methods for vstart_runner.py's
run() with check_status's default value as False so that we don't need
to pass check_status's value at every call to run() or it's other
wrappers.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa: add a method to create an empty file with any user

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa: add stdin parameter to run_shell

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa: command to be ran as a different user should be a single string

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test/osd/safe-to-destroy.sh: fix typo

Signed-off-by: Sage Weil <sage@redhat.com>

msg/async: add async msg perf counter

Signed-off-by: Xinze Chi <xinze@xsky.com>
Updated to account for the new ProtocolV1/ProtocolV2

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

tools/crushtool: add new Option --bucket-tree to show bucket map tree

Signed-off-by: linbing <linbing@t2cloud.net>
Update the crushtool cli test template for new options.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

msg/async: bug fix 23600 by setting _fd = 1.

After listen_socket was closed, it entered the start() processf magically. so
I think we should make '_fd = -1' when listen_socket was closed. And in the
start process, determine the value of listen_socket, if it equal to -1, return
the process.

Fixes: https://tracker.ceph.com/issues/23600
Signed-off-by: shangfufei <shangfufei@inspur.com>
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

os/bluestore: be more verbose doing bluefs log replay.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

PendingReleaseNotes: 14.2.1 note on crush required version

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27581 from LenzGr/master-documentation

doc: Improved the dashboard proxy config section

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

osd/osd_types: pool_stat_t::dump - fix 'num_store_stats' field

Fixes: http://tracker.ceph.com/issues/39340
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27593 from smithfarm/wip-39291

cmake: revert librados_tp.so version from 3 to 2

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

mgr/telemetry: change crash dict to a list

For ease of indexing.  There's no value in timestamp keys.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #27514 from tchaikov/wip-crimson-auth-client

crimson: enable cephx for v2 msgr

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #27624 into master

* refs/pull/27624/head:
doc: fixed telemetry module reference link

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/auth: let AuthClient::get_auth_request return a named struct

better readability this way

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: use const reference for Connection::auth_registry

it is not optional now.

Signed-off-by: Kefu Chai <kchai@redhat.com>

auth/AuthRegistry: more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: remove "#warning" in v1 implementation

since we've implemented the v2 support, these reminders are not needed
anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: reset promise using `{}`

better readability, less repeating.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mon/MonClient: remove unused AuthMethodList

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: use auth_registry for querying auth_methods

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: fixed telemetry module reference link

Used :ref: label for telemetry reference in Nautilus.

Fixes: https://tracker.ceph.com/issues/39337
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

mon: fix INCOMPAT_OCTOPUS feature number

It's overlapping with nautilus, which is a problem!

Signed-off-by: Sage Weil <sage@redhat.com>

rgw: refactor remaining rgw_rest_swift logging

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: refactor remaining rgw_rest_s3 logging

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: refactor logging in Swift ObjStore classes in rgw_rest_swift.cc

Signed-off-by: Ali Maredia <amaredia@redhat.com>

rgw: refactor logging in S3 ObjStore classes in rgw_rest_s3.cc

Signed-off-by: Ali Maredia <amaredia@redhat.com>

mon/Elector: format mon_release correctly

Signed-off-by: Sage Weil <sage@redhat.com>