cmake: put virtualenv directories under TMPDIR

this was required for cmake-check to find ceph-disk and ceph-detect-init
in the PATH

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8442 from trociny/wip-rbd-mirror-status

rbd-mirror: replay status now available via the rbd api and cli

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ObjectCacher: don't loop the extents of OSDWrite in map_writeReviewed-by

ObjectCacher: don't loop the extents of OSDWrite in map_write

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7830 from yuyuyu101/wip-14907

IoCtxImpl: delete C_WatchCB2 when free LingerOp

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8635 from runsisi/wip-fix-md-list

cls_rbd: fix loop condition when listing image metadata

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8661 from runsisi/wip-fix-snapc-validation

librbd: put the validation of image snap context earlier

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7800 from renhwsky/renhw-wip-osdmonitor-have_pending_crush

mon/OSDMonitor: _have_pending_crush of return value should be bool

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

qa/workunits/rbd: test mirror status in pool directory

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: in replay status store number of entries behind master

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8305 from tchaikov/wip-fix-vstart

vstart: make -k with optional mon_num (part 2)

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/osdmonitor: _have_pending_crush of return value should be bool

have_pending_crush of return value should be bool, fix it

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #8823 from aderumier/deb-default-tcmalloc-cache-size

etc/default/ceph: set 128MB tcmalloc cache size

Reviewed-by: Sage Weil <sage@redhat.com>

set 128MB tcmalloc cache size by bytes

etc/default/ceph: set 128MB tcmalloc cache size

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>

Merge pull request #8568 from ddiss/osd-librados-writesame

OSD and librados writesame support

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8791 from dillaman/wip-15574

librbd: parent image is closed twice if error encountered while opening

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8790 from dillaman/wip-15630

rbd-mirror: interrupting image replay startup can cause crash

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8785 from tchaikov/wip-librbd-bad-header

librbd: does not crash if image header is too short

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8813 from trociny/fix-test-librbd-aio_write

test: librbd: aio write/discard completion should always return 0 on success

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: librbd: aio write/discard completion should always return 0 on success

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8795 from runsisi/wip-fix-librbd-ut

cmake: add library cls_journal for target unittest_librbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8796 from runsisi/wip-fix-invalidate

librbd: fix potential assertion during object map invalidation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8812 from liewegas/wip-dh-systemd

debian/control: dh_systemd_start is in the dh-systemd package

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: dh_systemd_start is in the dh-systemd package

Fixes: http://tracker.ceph.com/issues/15573
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8700 from tchaikov/wip-systemd-deb

debian: package systemd targets

Reviewed-by: Javier M. Mellid <jmunhoz@igalia.com>

cmake: add library cls_journal for target unittest_librbd

otherwise if we want to generate target unittest_librbd we need to
(re)generate cls_journal manually

Signed-off-by: runsisi <runsisi@zte.com.cn>

debian: install systemd target files

* enable it using dh_systemd_enable
* start the target using dh_systemd_start
* move the dh_installinit, dh_systemd_enable, dh_systemd_start calls
  down, so they can identify the service files if they care about them.

Fixes: http://tracker.ceph.com/issues/15573
Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: fix potential assertion for object map invalidation

if update_flags fails we should not go any further, or the async request
will be removed from m_image_ctx.async_requests twice

Signed-off-by: runsisi <runsisi@zte.com.cn>

Merge pull request #8792 from cbodley/wip-defer-drop-privileges

common: fix value of CINIT_FLAG_DEFER_DROP_PRIVILEGES

Reviewed-by: Sage Weil <sage@redhat.com>

common: fix value of CINIT_FLAG_DEFER_DROP_PRIVILEGES

0x16 (binary 10110) was overlapping two other flags

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: propagate any image refresh errors back to caller

Previously, the saved error code was only propagated when an image
flush was required.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid second close attempt of invalid parent image

Fixes: http://tracker.ceph.com/issues/15574
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: interrupting image replay startup can cause crash

Properly shut down the journaler if a shut down has been requested
while start up is in-progress.

Fixes: http://tracker.ceph.com/issues/15630
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8714 from ceph/wip-14941

rpm: implement scriptlets for the post-split daemon packages

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

librbd: does not crash if image header is too short

if something goes wrong with the object, and returns a chunk shorter
than expected, don't panic

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: Add rpm scripts for ceph-rbd-mirror

We are currently missing the systemd scripts for the new ceph-rbd-mirror
daemons. This patch introduces them.

Signed-off-by: Boris Ranto <branto@redhat.com>

rpm: Start all the targets in %post

We need to have the sub-targets active for the ceph.target to be able
to propagate its calls to the services. If the sub-target is inactive,
the main target won't propagate the stop/restart calls.

Signed-off-by: Boris Ranto <branto@redhat.com>

rpm: implement scriptlets for the post-split daemon packages

This patch gives each of the ceph-{mds,mon,osd,radosgw} packages its own
%post, %preun, and %postun scriptlets dealing with the package's unit files.

The scriptlets of ceph-base are adapted to handle the ceph.target unit file
only.

The scriptlets of ceph-mon handle ceph-create-keys services in addition to ceph-mon.

The scriptlets of ceph-osd handle ceph-disk services in addition to ceph-osd.

Fixes: http://tracker.ceph.com/issues/14941
Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Boris Ranto <branto@redhat.com>

systemd: enable all the ceph .target services by default

Some distros, like Fedora and openSUSE, have a policy that all services are
disabled by default.

This patch changes that default for the ceph.target and
ceph-{mds,mon,osd,radosgw}.target services.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
Signed-off-by: Boris Ranto <branto@redhat.com>

Signed-off-by: Tamil Muthamizhan <tmuthami@redhat.com>
removed ceph-deploy folder, as its been already replaced with ceph-tests.

Merge pull request #8746 from yehudasa/wip-15598

Fixes: http://tracker.ceph.com/issues/15598
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8762 from ceph/wip-15595

rgw: fix bad free in RGWCreateBucket_ObjStore_S3

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix bad free in RGWCreateBucket_ObjStore_S3

get_params() will accept -ERR_LENGTH_REQUIRED from
rgw_rest_read_all_input(), in which case 'data' will not be allocated
and we attempt to free() an uninitialized pointer

Fixes: http://tracker.ceph.com/issues/15595
Signed-off-by: Casey Bodley <cbodley@redhat.com>

msg/async: add numevents statistics for external_eventsReviewed-by: Haomai Wang <haomai@xsky.com>

msg/async: add numevents statistics for external_events

Reviewed-by: Haomai Wang <haomai@xsky.com>

rbd: CLI to retrieve rbd mirror state for a pool / specific image

Fixes: #15144
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

librbd: API to retrieve image mirroring status

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: store replay status in mirroring object

Fixes: #14420
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: MockJournaler: add get_cached_client method

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

test: cls_rbd: methods to store/retrieve mirroring status

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls::rbd:: methods to store/retrieve mirroring status

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

objclass: add method to list watchers

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8683 from Ved-vampir/zlib_fix

compressor: fix of error code control in zlib decompress

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8489 from SUSE/wip-15419

systemd: make Ceph daemons dependent upon time-sync.target

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: James Page <james.page@ubuntu.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #8721 from ibravo/patch-2

doc: update quick-ceph-deploy.rst

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8713 from HeyoQiang/master

osd/PG: remove unused variable have

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

PG:clear an unused variable: have

Signed-off-by: Yongqiang He <he.yongqiang@h3c.com>

Merge pull request #8723 from cooboos/wrong_spell

doc: correct wrong spell 'mnay' to 'many'

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #6764 from kylinstorage/fix-check_become_tier

mon: fix OSDMonitor::_check_become_tier

Reviewed-by: Xinze Chi <xinze@xsky.com>

Merge pull request #8103 from adamemerson/wip-sign-of-the-time-debug

common: signed coarse time

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8407 from dx9/wip-os-release

common/util: add support for distro info from /etc/os-release file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8437 from jjhuo/single_delete

os/kv, os/bluestore: Enable SingleDelete with RocksDB and Bluestore

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8451 from tchaikov/wip-signal-hanlder-typo

global/signal_handler: fix typo

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8503 from yuyuyu101/wip-spdk-improve

os/bluestore/NVMEDevice: performance improvements

Merge pull request #8702 from kmroz/wip-fix-run-dir-chown

global-init: fixup chown of the run directory along with log and asok files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8728 from liewegas/wip-notify

osd/osd_types: render notify op properly

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueStore: cleanup overlay using single delete

Overlay entries are also only written once, eligible to use single delete.

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>

os/bluestore/BlueStore: cleanup the wal entries using single delete

By use of single delete, RocksDB should be able to remove deleted wal
entries with only one compaction in theory, when wal entries land on level0.
This should reduce bluestore wal entries incurred WAF.

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>

kv/RocksDBStore: implement rm_single_delete

Use SingleDelete to delete keys which were never overwritten.

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>

kv/KeyValueDB: add a new interface for single delete.

This is useful for log-insert-merge tree based key value store, such as
RocksDB, to avoid more LSM compactions for already deleted key value pairs.

Signed-off-by: Jianjian Huo <jianjian.huo@ssi.samsung.com>

Merge pull request #8474 from xiaoxichen/wip-pycephfs-namespace

pycephfs: using namespace for data isolation

Reviewed-by: John Spray <john.spray@redhat.com>

rgw: fix suffix in RGWZoneParams::fix_pool_names()

Fixes: http://tracker.ceph.com/issues/15598
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph.spec.in: recommend ntp-daemon on SUSE

SUSE has settled on "ntp-daemon" as the generic package name. The "ntp" and
"chrony" etc. packages have "Provides: ntp-daemon" in their respective spec
files.

References: http://tracker.ceph.com/issues/15419
Signed-off-by: Nathan Cutler <ncutler@suse.com>

deb: make ceph-base and radosgw recommend time-daemon

References: http://tracker.ceph.com/issues/15419

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #8715 from runsisi/wip-fix-cmake

cmake: use LINK_PRIVATE instead of PRIVATE for CMake version == 2.8.11

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #8730 from ibravo/patch-3

doc: Updated ceph-deploy osd to choose underlying file system

Merge pull request #8745 from ErwanAliasr1/evelu-deps

install-deps: Adding gcc for debian based systems

Reviewed-by: Sage Weil <sage@redhat.com>

admin-socket: use chown instead of fchown

fchown() returns success, but does not change asok file
ownership. chown() does.

Signed-off-by: Karol Mroz <kmroz@suse.com>

install-deps: Adding gcc for debian based systems

When running install-deps on a minimalistic system, we reach that situation :

dpkg-checkbuilddeps --admindir=/tmp/install-deps.5526 debian/control
    sh: 1: gcc: not found
    dpkg-checkbuilddeps: warning: Couldn't determine gcc system type, falling back to default (native compilation)
    dpkg-checkbuilddeps: error: cannot open /tmp/install-deps.5526/status: No such file or directory

This means that we shall install gcc before calling dpkg-checkbuilddeps.

Signed-off-by: Erwan Velu <erwan@redhat.com>

Merge pull request #8743 from ceph/revert-8502-song20

Revert "crushtool: do not allow to compile crushmap with repeat rules…

Reviewed-by: Sage Weil <sage@redhat.com>

Revert "crushtool: do not allow to compile crushmap with repeat ruleset."

TestRados: add writesame support

Can be tested via "--op writesame". Requests are currently dispatched
*without* a multiplication factor, i.e. data_len == write_len.

Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph_test_rados_api_c_write_ops: add writesame test

Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph_test_rados_api_io: ETOBIG tests for writesame

Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph_test_rados_api_aio: add writesame tests

Write a buffer a number of times using writesame, read the full range
back, and check that it matches. Do this using rados_aio_writesame(),
ioctx.aio_writesame() and ioctx.aio_operate(op.writesame()).

Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph_test_rados_api_misc: add preliminary writesame tests

Test the new ioctx.writesame() and rados_writesame() API functions.

Signed-off-by: David Disseldorp <ddiss@suse.de>

librados: add writesame API

The writesame operation allows callers to write the same data buffer
multiple times to a given object.

Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph osd: add support for new op writesame

This adds a new ceph request writesame that writes a buffer of length
writesame.data_length bytes at writesame.offset over
writesame.length bytes.

This command maps to SCSI's WRITE SAME request, so users like LIO+rbd
can pass this to the OSD. Right now, it only saves having to transfer
writesame.length bytes over the network, but future versions will be
to fully offload it by passing it directly to the FS/devices if they
support it.

v2:
- Fix tab/spaces to matching coding style.
- Allow zero write length. Check for invalid data lengths.

Signed-off-by: Mike Christie <mchristi@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

compressor: fix of error code control in zlib decompress

Signed-off-by: Alyona Kiseleva <akiselyova@mirantis.com>

Merge pull request #8725 from liewegas/wip-make-check

test/encoding/readable.sh: fix make check for autotool

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/encoding/readable.sh: fix make check for autotools

Use ./ if ceph-dencoder is in the local directory.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8719 from SUSE/wip-fix-multidump

multi-dump.sh: fix usage function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Doc: Updated ceph-deploy osd to choose underlying file system

Included paragraph about --fs-type and --bluestore

Signed-off-by: Ignacio Bravo ibravo@hotmail.com

Doc: Update quick-ceph-deploy.rst

Signed-off-by: Ignacio Bravo ibravo@hotmail.com

osd/osd_types: render notify OSDOp properly

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8640 from xiexingguo/xxg-wip-osdsideeffect

osd/ReplicatedPG: allow CMPXATTR ops to trigger side effects too

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8694 from wuxingyi/fixincomplete

osd/PG: PG is only likely to go active when min_size turns smaller.

Reviewed-by: Sage Weil <sage@redhat.com>

osd: honour result code passed in; tidy-up changesReviewed-by: Sage Weil <sage@redhat.com>

osd: honour result code passed in; tidy-up changes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8675 from roidayan/xio-build_deb_with_xio

xio: add MNop.h to dist tarball

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8678 from jack-changtao/pgbackend

osd/PGBackend: if min > max, PGBackend::objects_list_partial will list all objects of that pg

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8680 from ceph/rm-distribute

tools: remove installation for distribute. It is no longer maintained

Reviewed-by: Sage Weil <sage@redhat.com>