rbd-mirror: admin socket commands to start/stop/restart mirroring

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: make image replayer asok commands available when not started

Initially the asok commands were registered only after the image
replayer start (and unregistered on stop) because their names were
built using remote pool and image names, which became known only after
start.

Now, the asok commands are registered on the image replayer construction
using the temporary name "remote_pool_name/global_image_id". They are
re-registered using "remote_pool_name/remote_image_name" when the
image replayer is started.  Also the commands are not unregistered on
the image replayer stop.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8842 from SUSE/wip-msd-typo-fix

systemd: fix typo in preset file

Reviewed-by: Sage Weil <sage@redhat.com>

ceph_time: add operator<< for more clock types

ceph_time: add operator<< for more clock types

Reviewed-by: Sage Weil <sage@redhat.com>

Revert "Merge pull request #8346 from yuyuyu101/wip-async-clock"

This reverts commit 37117823d108dcd5145421adc0121eea75633851, reversing
changes made to 5ff4988c09ccc2af6ff72f30d053b5864a8c8e14.

This broke the build:

msg/async/Event.cc: In member function 'int EventCenter::process_time_events()':
error: msg/async/Event.cc:280:30: cannot bind 'std::basic_ostream' lvalue to 'std::basic_ostream&&'
ldout(cct, 10) << __func__ << " cur time is " << now << dendl;
^

and elsewhere...

Signed-off-by: Sage Weil <sage@redhat.com>

ceph_time: add operator<< for more clock types

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8838 from cbodley/wip-cmake-rbd-mirror-replay-status

cmake: add missing source file to rbd_mirror/image_replayer

Reviewed-by: Kefu Chai <kefu@redhat.com>

systemd: fix typo in preset file

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #8346 from yuyuyu101/wip-async-clock

msg/async/Event: replace ceph_clock_now with coarse_mono_clock

Reviewed-by: Kefu Chai <kefu@redhat.com>

Merge pull request #8109 from majianpeng/bluestore-cleancode

os/bluestore/BlueStore: cleanup code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8110 from majianpeng/bluestore-aio-log

os/bluestore/KernelDevice: Fio aio_write, it call _aio_log_finish by …

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8416 from vumrao/wip-vumrao-15308

librados: modify Pipe::connect() to return the error code - part 1

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8532 from xiexingguo/xxg-wip-misc

osd: miscellaneous fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8599 from yangdongsheng/on_abort

os: fix submit_transaction return codes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8629 from tchaikov/wip-rados_ping_monitor

librados: always set the outstrlen for rados_ping_monitor()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8677 from tchaikov/wip-graylog-cleanup

log: remove unnecessary src/log/Graylog.*

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8693 from cooboos/perf-reset

common: show success/fail info in json for 'perf reset' command

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8731 from Yan-waller/yj-wip-fixkqueue

msg/async: fix some return values and typo.

Reviewed-by: Sage Weil <sage@redhat.com>

cmake: add missing source file to rbd_mirror/image_replayer

fixes an undefined reference when linking librbd_mirror_internal.a

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8454 from jdurgin/wip-mirror-test2

rbd_mirror.sh: add env vars to allow running under teuthology

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8836 from athanatos/wip-15655

mon/OSDMonitor: avoid underflow in reweight-by-utilization if max_change=1

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8648 from ffilz/master

libcephfs: add ceph_ll_mknod wrapper for ll_mknod

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8820 from ceph/wip-cmake-compression

cmake: fix linking of compressor and plugins

Reviewed-by: Ali Maredia

Merge pull request #8822 from cbodley/wip-cmake-tmp-virtualenv

cmake: put virtualenv directories under /tmp

Reviewed-by: Ali Maredia <amaredia@redhat.com>

cmake: fix for unittest_async_compressor

this test wasn't creating the lib/compressor subdirectory, so it
depended on test_compressor_plugin_zlib or test_compressor_plugin_snappy
being run first

added the same dirty hack to copy libceph_snappy.so into lib/compressor,
and added a dependency on ceph_snappy to make sure it's built

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: fix linking of compressor and plugins

libcommon now includes the compressor objects necessary to load
compressor plugins

also removed extra dependencies on the plugins and their unit tests

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/workunits/rbd: add env vars to adapt rbd_mirror.sh to teuthology

With these set, this script can work in teuthology, where we want to
control the how daemons run and their log locations, valgrind
settings, etc. The workunit task always sets CEPH_ID, and allows
other env vars to be configured optionally.

The RBD_MIRROR_USE_EXISTING_CLUSTER option can also speed up local testing.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: rbd_mirror.sh: change parameters to cluster rather than daemon name

Daemon name is only useful to differentiate asok and pid filenames, so
just inline its use there. Everywhere else replace the 'daemon'
parameter with cluster or local_cluster as appropriate.

Remove the CLUSTER{1,2}_DAEMON vars as well, since they are no longer
used.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8817 from dillaman/wip-15572

librbd: reduce log level for errors encountered disabling object-map while flatten is in-progress

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8819 from dillaman/wip-14540

rbd: disk usage now includes all snapshots by default

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: put virtualenv directories under TMPDIR

this was required for cmake-check to find ceph-disk and ceph-detect-init
in the PATH

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8442 from trociny/wip-rbd-mirror-status

rbd-mirror: replay status now available via the rbd api and cli

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ObjectCacher: don't loop the extents of OSDWrite in map_writeReviewed-by

ObjectCacher: don't loop the extents of OSDWrite in map_write

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7830 from yuyuyu101/wip-14907

IoCtxImpl: delete C_WatchCB2 when free LingerOp

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8635 from runsisi/wip-fix-md-list

cls_rbd: fix loop condition when listing image metadata

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8661 from runsisi/wip-fix-snapc-validation

librbd: put the validation of image snap context earlier

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7800 from renhwsky/renhw-wip-osdmonitor-have_pending_crush

mon/OSDMonitor: _have_pending_crush of return value should be bool

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rbd: test mirror status in pool directory

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: in replay status store number of entries behind master

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8305 from tchaikov/wip-fix-vstart

vstart: make -k with optional mon_num (part 2)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/osdmonitor: _have_pending_crush of return value should be bool

have_pending_crush of return value should be bool, fix it

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #8823 from aderumier/deb-default-tcmalloc-cache-size

etc/default/ceph: set 128MB tcmalloc cache size

Reviewed-by: Sage Weil <sage@redhat.com>

set 128MB tcmalloc cache size by bytes

OSDMonitor: avoid underflow in reweight-by-utilization if max_change=1

Fixes: http://tracker.ceph.com/issues/15655
Signed-off-by: Samuel Just <sjust@redhat.com>

etc/default/ceph: set 128MB tcmalloc cache size

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>

rbd: disk usage now includes all snapshots by default

Added a new "--from-snap" optional to restrict the calculation to
specific subsets of snapshots and the HEAD revision.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8568 from ddiss/osd-librados-writesame

OSD and librados writesame support

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8791 from dillaman/wip-15574

librbd: parent image is closed twice if error encountered while opening

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8790 from dillaman/wip-15630

rbd-mirror: interrupting image replay startup can cause crash

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8785 from tchaikov/wip-librbd-bad-header

librbd: does not crash if image header is too short

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8813 from trociny/fix-test-librbd-aio_write

test: librbd: aio write/discard completion should always return 0 on success

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: librbd: aio write/discard completion should always return 0 on success

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: block RPC requests while updating features

Disabling the journal and object map require acquiring the exclusive
lock locally.  We don't want to start executing long-running ops
for this quick process.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: return -ESHUTDOWN when a refresh is requested on a closed image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: synchronous block_writes should return possible error result

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refresh image before executing maint op

If a maint op is interrupted and restarted, it's possible that the
image will need to be refreshed.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reduce log level when lock owner not detected

This can happen under normal operating conditions and the
lock request is automatically retried.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: reduce log level for interrupted maint ops

Fixes: http://tracker.ceph.com/issues/15572
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8795 from runsisi/wip-fix-librbd-ut

cmake: add library cls_journal for target unittest_librbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8796 from runsisi/wip-fix-invalidate

librbd: fix potential assertion during object map invalidation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8812 from liewegas/wip-dh-systemd

debian/control: dh_systemd_start is in the dh-systemd package

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: dh_systemd_start is in the dh-systemd package

Fixes: http://tracker.ceph.com/issues/15573
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8700 from tchaikov/wip-systemd-deb

debian: package systemd targets

Reviewed-by: Javier M. Mellid <jmunhoz@igalia.com>

cmake: add library cls_journal for target unittest_librbd

otherwise if we want to generate target unittest_librbd we need to
(re)generate cls_journal manually

Signed-off-by: runsisi <runsisi@zte.com.cn>

debian: install systemd target files

* enable it using dh_systemd_enable
* start the target using dh_systemd_start
* move the dh_installinit, dh_systemd_enable, dh_systemd_start calls
  down, so they can identify the service files if they care about them.

Fixes: http://tracker.ceph.com/issues/15573
Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: fix potential assertion for object map invalidation

if update_flags fails we should not go any further, or the async request
will be removed from m_image_ctx.async_requests twice

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #8792 from cbodley/wip-defer-drop-privileges

common: fix value of CINIT_FLAG_DEFER_DROP_PRIVILEGES

Reviewed-by: Sage Weil <sage@redhat.com>

common: fix value of CINIT_FLAG_DEFER_DROP_PRIVILEGES

0x16 (binary 10110) was overlapping two other flags

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: propagate any image refresh errors back to caller

Previously, the saved error code was only propagated when an image
flush was required.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid second close attempt of invalid parent image

Fixes: http://tracker.ceph.com/issues/15574
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: interrupting image replay startup can cause crash

Properly shut down the journaler if a shut down has been requested
while start up is in-progress.

Fixes: http://tracker.ceph.com/issues/15630
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8714 from ceph/wip-14941

rpm: implement scriptlets for the post-split daemon packages

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

librbd: does not crash if image header is too short

if something goes wrong with the object, and returns a chunk shorter
than expected, don't panic

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: Add rpm scripts for ceph-rbd-mirror

We are currently missing the systemd scripts for the new ceph-rbd-mirror
daemons. This patch introduces them.

Signed-off-by: Boris Ranto <branto@redhat.com>

rpm: Start all the targets in %post

We need to have the sub-targets active for the ceph.target to be able
to propagate its calls to the services. If the sub-target is inactive,
the main target won't propagate the stop/restart calls.

Signed-off-by: Boris Ranto <branto@redhat.com>

rpm: implement scriptlets for the post-split daemon packages

This patch gives each of the ceph-{mds,mon,osd,radosgw} packages its own
%post, %preun, and %postun scriptlets dealing with the package's unit files.

The scriptlets of ceph-base are adapted to handle the ceph.target unit file
only.

The scriptlets of ceph-mon handle ceph-create-keys services in addition to ceph-mon.

The scriptlets of ceph-osd handle ceph-disk services in addition to ceph-osd.

Fixes: http://tracker.ceph.com/issues/14941
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Boris Ranto <branto@redhat.com>

systemd: enable all the ceph .target services by default

Some distros, like Fedora and openSUSE, have a policy that all services are
disabled by default.

This patch changes that default for the ceph.target and
ceph-{mds,mon,osd,radosgw}.target services.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Boris Ranto <branto@redhat.com>

librados: modify Pipe::connect() to return the error code

Fixes: http://tracker.ceph.com/issues/15308
Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>
removed ceph-deploy folder, as its been already replaced with ceph-tests.

Merge pull request #8746 from yehudasa/wip-15598

Fixes: http://tracker.ceph.com/issues/15598
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8762 from ceph/wip-15595

rgw: fix bad free in RGWCreateBucket_ObjStore_S3

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix bad free in RGWCreateBucket_ObjStore_S3

get_params() will accept -ERR_LENGTH_REQUIRED from
rgw_rest_read_all_input(), in which case 'data' will not be allocated
and we attempt to free() an uninitialized pointer

Fixes: http://tracker.ceph.com/issues/15595
Signed-off-by: Casey Bodley <cbodley@redhat.com>

msg/async: add numevents statistics for external_eventsReviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: add numevents statistics for external_events

Reviewed-by: Haomai Wang <haomai@xsky.com>

rbd: CLI to retrieve rbd mirror state for a pool / specific image

Fixes: #15144
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: API to retrieve image mirroring status

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: store replay status in mirroring object

Fixes: #14420
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: MockJournaler: add get_cached_client method

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: cls_rbd: methods to store/retrieve mirroring status

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls::rbd:: methods to store/retrieve mirroring status

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

objclass: add method to list watchers

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8683 from Ved-vampir/zlib_fix

compressor: fix of error code control in zlib decompress

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8489 from SUSE/wip-15419

systemd: make Ceph daemons dependent upon time-sync.target

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: James Page <james.page@ubuntu.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #8721 from ibravo/patch-2

doc: update quick-ceph-deploy.rst

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8713 from HeyoQiang/master

osd/PG: remove unused variable have

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

PG:clear an unused variable: have

Signed-off-by: Yongqiang He <he.yongqiang@h3c.com>

Merge pull request #8723 from cooboos/wrong_spell

doc: correct wrong spell 'mnay' to 'many'

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #6764 from kylinstorage/fix-check_become_tier

mon: fix OSDMonitor::_check_become_tier

Reviewed-by: Xinze Chi <xinze@xsky.com>

Merge pull request #8103 from adamemerson/wip-sign-of-the-time-debug

common: signed coarse time

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8407 from dx9/wip-os-release

common/util: add support for distro info from /etc/os-release file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8437 from jjhuo/single_delete

os/kv, os/bluestore: Enable SingleDelete with RocksDB and Bluestore

Reviewed-by: Sage Weil <sage@redhat.com>