rbd: support size suffixes for size-based options

This patch will provide support for size suffixes for
below given options:

--stripe-unit <size in B/K/M>
--io-size <size in B/K/M/G/T>
--io-total <size in B/K/M/G/T>

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Merge pull request #5478 from dachary/wip-test-rados-striper

tests: rados striper tests use 7116 instead of 7113

Reviewed-by: Sebastien Ponce <sebastien.ponce@cern.ch>

Merge pull request #5459 from cxwshawn/master

radosgw-admin: use cout instead of cerr to print help message.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4860 from ceph/wip-nobjectiterator-crash

librados: fix empty NObjectIterator crash on comparisons

Reviewed-by: Sage Weil <sage@redhat.com>

librados: fix empty NObjectIterator crash on comparisons

This would segfault if you did a
NObjectIterator i;
i == ...

Fix comparisons by allowing pointer-wise comparisons
for iterators with impl==NULL, and asserting in
other methods that require a non-null ::impl

Signed-off-by: John Spray <john.spray@redhat.com>

tests: rados striper tests use 7116 instead of 7113

Change src/test/libradosstriper/rados-striper.sh monitor address to 7116
because src/test/mon/osd-crush.sh uses 7113 already.

Signed-off-by: Loic Dachary <ldachary@redaht.com>

Merge pull request #5474 from kylinstorage/wip-sparse_read_bugfix

osd/ReplicatedPG: sparse read should return (extent_map, data_bufferlist)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

osd/ReplicatedPG: sparse read should return (extent_map, data_bufferlist)

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>

Merge remote-tracking branch 'origin/next'

Conflicts:
src/Makefile-client.am (link rbd to libglobal again)

Revert "rbd: remove dependency on non-ABI controlled CephContext"

This reverts commit fa78739933d7b9501116a12a4fbfd58b5ca99cdb.
Original approach broke krbd map/unmap functions and did not fully
address ABI issues.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #5446 from ceph/wip-systemd

systemd support

Reviewed-by: Owen Synge <osynge@suse.com>

Merge pull request #5469 from theanalyst/doc/v0.94.3-release

doc: release notes for v0.94.3

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: release notes for v0.94.3

Signed-off-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

client/Makefile: ship InodeRef

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4753 from ceph/wip-client-inode-refs

client: hold reference for returned inode

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #5466 from ceph/wip-12578

mds: open base inode's snaprealm after decoding snapblob

Reviewed-by: John Spray <john.spray@redhat.com>

mds: open base inode's snaprealm after decoding snapblob

base inode has no parent, so can we mark its snaprealm open
immediately. This change makes sure replica mdsdir's snaplream
is opened.

Fixes: #12578
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #5140 from tchaikov/wip-pg-ls-update-pending-rel-notes

doc: add changes to "pg ls*" commands to pending release notes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add changes to "pg ls*" commands to pending release notes

"pg ls*" commands now accept recovering instead of recovery as
one of its states.

Fixes: #11569
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5337 from branch-predictor/bp-reduce-mallocs

Thread.cc: remove malloc/free pair

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5000 from ceph/wip-da-SCA-20150601

Various SCA and Coverity fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5312 from xinxinsh/wip-sparse-copy

use SEEK_HOLE/SEEK_DATA for sparse copy

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5420 from ceph/wip-12429

osd/OSDMap: handle incrementals that modify+del pool

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

radosgw-admin: use cout not cerr to print help message.

everytime I print radosgw-admin help message, it's too long, and I wanna add | more,
but the help message prints to cerr, so I can only add 2>&1 again, so I think it's
so inconvenient.

Signed-off-by: shawn chen <cxwshawn@gmail.com>

Merge pull request #5431 from kylinstorage/test_cache_pool_script_fix

fix: qa/workunits/rados/test_cache_pool.sh

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5430 from ceph/wip-12537

rgw: url encode exposed bucket

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5397 from ceph/wip-12222

mon: reject over-large values of max_mds

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #5400 from tchaikov/wip-12451

tools/rados: update the help message

Reviewed-by: Sage Weil <sage@redhat.com>

logrotate: fix log rotation with systemd

systemctl does not have a nice way to enumerate (active) units so we can
reload them.  On centos7, the is-active wildcard syntax does not appear to
be supported.  On fedora 22, it prints the state only but not which unit
the state belongs to.

Simply killall -1 instead.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5440 from vumrao/wip-vumrao-12458-doc

doc: change --keyfile description in man page of rbd help

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: change "--keyfile" description in man page of rbd help

"--keyring" has higher priority than "--keyfile".
"--keyfile" option is overridden by "--keyring"
if latter is also specified.

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Merge pull request #5455 from ceph/wip-cephfs-cmake

CMake fixes for CephFS

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5456 from kylinstorage/wip-do_autogen.sh

Allow do_autogen.sh pass configure parameters

Reviewed-by: Sage Weil <sage@redhat.com>

CMake: add crushtool

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: fix librados build

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: fix rbd build

Don't link with 'global' from rbd (to match
autotools update fa78739).  Fixes "g_ceph_context
defined twice" error.

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: libblkid not only needed with rbd

It's linked in libcommon.

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: smalliobenchrbd should dep on rbd

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: set WITH_CEPHFS by default

Option was referenced in CMakeLists but
not actually set.

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: fix cephfs cls build

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: fix cephfs test targets

These had bad lib names.

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: add cephfs tools

cephfs-table-tool, cephfs-data-scan, cephfs-journal-tool

Signed-off-by: John Spray <john.spray@redhat.com>

Makefile: include systemd udev rules in tarball

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5422 from ceph/wip-fix-filestore-getattr

tests: fix the ObjectStore/StoreTest.SimpleCloneTest in "ceph_test_objectstore"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5375 from ceph/wip-config-root-ino

Make UID of new root inos configurable

add help to do_autogen.sh

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

Allow do_autogen.sh to pass configure parameters

Currently do_autogen.sh does not allow to pass parameters
of configure to disable/enable components during building, which is not
flexible, this patch add the ability to it.

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>
Reviewed-by: Li Wang <liwang@ubuntukylin.com>

client: use smart pointer to track temporary inode reference

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: use smart pointer to track 'cwd' and 'root_parents'

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: convert Inode::snapdir_parent to smart pointer

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: convert CapSnap::in to smart pointer

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: convert Fh::inode to smart pointer

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: use smart pointers in MetaRequest

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: convert Dentry::inode to smart pointer

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: hold reference for returned inode

CLient::make_request() returns pointer of target inode, but it does not
increase reference count of the returned inode. The inode may get freed
when CLient::make_request() releases MetaRequest.

he fix is hold a reference for the returned inode. Since there are many
of places use CLient::make_request() directly or indirectly, it's easy
to leak reference. This patch uses intrusive_ptr to track the reference.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

systemd: fix ceph-radosgw@ service

There's no prestart.  Fix the instance name.  Cleanup.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: install the new systemd-based udev rules from ddiss

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in:Fixup renamed ceph-rgw.conf

This is a simple renaming bug.

Signed-off-by: Owen Synge <osynge@suse.com>

ceph.spec.in:fixup bad merge of systemd logic

remove ceph-disk-{activate,prepare} wrappers

These ancient aliases shouldn't be used.  We should have removed them
immediately way back when, honestly.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: ship ceph-disk-* units

Signed-off-by: Sage Weil <sage@redhat.com>

systemd: activate disks via systemd service instead of udev

The udev(7) man page states:
  RUN
  ...
  This can only be used for very short-running foreground tasks. Running
  an event process for a long period of time may block all further
  events for this or a dependent device.

  Starting daemons or other long-running processes is not appropriate
  for udev; the forked processes, detached or not, will be
  unconditionally killed after the event handling has finished.

ceph-disk activate is far from a short-running task:
- check whether path is a block dev, for dirs call through to
  activate_dir()
- call blkid to obtain the filesystem type for the block dev
- pull mount options from hard-coded ceph.conf file
- mount the OSD dev at a temporary path
- check the ceph magic for mounted filesystem
- read cluster uuid and locate corresponding /etc/ceph/{cluster}.conf
  path
- read or generate (if missing) the OSD uuid
- create a file indicating init system usage (systemd)
- mount the device at a second (final) location
- umount (lazy) the temporary mount path
- enable the systemd ceph-osd@{osd_id} service
- start the systemd ceph-osd@{osd_id} service

This logic is therefore best left in a systemd service for execution. As
it is less limited in terms of execution time, and also allows for
improved event handling in future (fsck, dmcrypt mapping etc.).

This change sees 95-ceph-osd.rules.systemd trigger ceph-disk activate or
ceph-disk activate-journal via new ceph-disk-activate-journal@.service,
ceph-disk-activate@.service and ceph-disk-dmcrypt-activate@.service
systemd service files.

ceph-disk-dmcrypt-activate@.service makes use of the newly added
--dmcrypt parameter for ceph-disk activate.

Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph.spec: use sysvinit rbdmap script for now

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-disk: map dmcrypt devices prior to activation

Support mapping of dmcrypt devices during activation via the new
ceph-disk activate[-journal] --dmcrypt and --dmcrypt-key-dir parameters.

Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph-disk: split get_dmcrypt_key_path from key creation

In preparation for mapping dmcrypt cyphertext devs during activation.

Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph-disk: add --log-stdout parameter

This can be used when ceph-disk is run by systemd services, to ensure
that output is captured in the journal.

Signed-off-by: David Disseldorp <ddiss@suse.de>

ceph.spec: no rcceph[-radosgw] with systemd

Signed-off-by: Sage Weil <sage@redhat.com>

systemd: add ceph-create-keys@ service

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: include systemd units files

Signed-off-by: Sage Weil <sage@redhat.com>

system: include all unit files

Signed-off-by: Sage Weil <sage@redhat.com>

Autotools to install systemd unit files.

To simplify the spec file we should install as much using autotools
and as little as possible in the spec file.

Signed-off-by: Owen Synge <osynge@suse.com>

Add unit files to the spec file.

%{_unitdir}/ceph-mds@.service
%{_unitdir}/ceph-mon@.service
%{_unitdir}/ceph-osd@.service
%{_unitdir}/ceph.target

Signed-off-by: Owen Synge <osynge@suse.com>

radosgw systemd prestart moved to systemd

prestart scripts are going to be added to systemd to simplify the switching behavior.

Signed-off-by: Owen Synge <osynge@suse.com>

radosgw systemd support

Added a radosgw systemd support and associated prestart script.
  - With improved checking over first revison.
  - ceph-radosgw-prestart.sh now installed in /usr/lib/ceph-radosgw

Signed-off-by: Owen Synge <osynge@suse.com>

ceph.spec.in: Change redhat defaults for rgw

Changes
--with-rgw-user=$USER \
--with-rgw-group=$GROUP \

to
    USER=root
    GROUP=root

from
    USER=apache
    GROUP=apache

Signed-off-by: Owen Synge <osynge@suse.com>

Added tmpfiles.d for rgw: templated user and group.

tmpfiles.d are part of system.d and define how temporary directories are setup.
rgw needs a socket directory. To do this we template tmpfiles.d user and group
for rgw and fill in the values using autotools.

Note1: Added to spec file.
Note2: Name changed to rgw from radosgw as is preferred name by Sage.
Note3: Adds configure options
    --with-rgw-user=UserName
    --with-rgw-group=GroupName
Note4: Defaults set for debian
Note5: spec file overrides defaults for redhat and suse

Signed-off-by: Owen Synge <osynge@suse.com>

ceph.spec.in:Update rpm hooks and file content

Take rpm hooks from SUSE rpm and move upstream.
These are tested on SLE12 to include the correct files.

Signed-off-by: Owen Synge <osynge@suse.com>

Merge pull request #5445 from ceph/wip-12432

Wip 12432

rgw: set http status in civetweb

Need to set the http status in civetweb so that we report it correctly.
Fixes: #12432
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

civetweb: update submodule to support setting of http status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5258 from zhouyuan/rgw_skip_first_chunk

rgw: skip prefetch first chunk if range get falls to shadow objects

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

doc: krbd supports clones since 3.10

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #5438 from ceph/fix-doc-build

doc: fix the build of placement-groups.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/FileStore: getattr() should return 0 if success

* fix the test of ObjectStore/StoreTest.SimpleCloneTest, the expected
  return value of getattr() should be 0. otherwise it otherwise it fails
  on objectstore backends of "memstore" and "keyvaluestore".
* change FileStore::getattr() to return 0 on success
* tighten related tests in ceph_test_objectstore, to test the return value
  of getattr() using ASSERT_EQ(r, 0) instead of ASSERT_GE(r, 0)

Signed-off-by: Kefu Chai <kchai@redhat.com>

KeyValueStore: Fix incorrect rm_keys behaviors

GenericObjectMap will copyup parents keys when rm keys, but if
caller(KeyValueStore) already buffer "set keys", it shouldn't
copyup reset keys.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

doc: fix the build of placement-groups.rst

add a reference target of "mapping pgs to osds"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5432 from ceph/wip-fix-rbd-build

make: fix the build of "rbd"

Reviewed-by: Loic Dachary <ldachary@redhat.com>

fix: qa/workunits/rados/test_cache_pool.sh

Signed-off-by: Mingxin Liu <mingxinliu@ubuntukylin.com>

Merge pull request #5434 from tobegit3hub/remove_execute_mode

Remove execute mode of source files

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Remove execute mode of source files

Merge pull request #5398 from mingyuez/master

rbd:If the image has any protected snaps,'rbd snap purge image' will return failure without removing any snaps.

Reviewed-by: Josh During <jdurgin@redhat.com>
Reviewed-by: Haomai Wang <haomaiwang@gmail.com>

make: fix the build of "rbd"

`CEPH_GLOBAL` was removed in fa78739, but the linked libraries does not
offer all needed symbols for "rbd". so add them back.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd:'rbd purge image' will return failure without removing any snaps if the image has a protected snap

Signed-off-by: mingyuez <zhao.mingyue@h3c.com>

Merge pull request #5395 from tchaikov/wip-12512

common/syncfs: fall back to sync(2) if syncfs(2) not available

Reviewed-by: Haomai Wang <haomaiwang@gmail.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5402 from ceph/wip-claim-sparse-read-buffer

osd/ReplicatedPG: claim the read buffer instead of copying it

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5428 from ceph/wip-kill-rest-bench

kill rest-bench

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rgw: skip prefetch first chunk if range get falls to shadow objects

Currently the head object will be prefetched in each GET:
 a) This is unnecessary if the Range GET falls to shadow objects.
 b) The GET request would be quite slow if we have a big head object
This patch adds some check on the Range. If it's not in the head
object(>=rgw_max_chunk_size) then skip the prefetch.

Fixes: #12539
Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>

Merge remote-tracking branch 'origin/next'

Merge pull request #5425 from ceph/wip-12479-next

librbd: test crash during client upgrade suite

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: url encode exposed bucket

Fixes: #12537
Don't send the bucket name back without url encoding it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>