librbd: diff_iterate needs to handle holes in parent images

If a clone's parent image snapshot includes a discarded extent,
this was previously causing an assert failure.  Instead, ignore
any discard holes in the parent image.

Fixes: #12885
Backport: hammer
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new test case for librbd diff_iterate over discard extents

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5838 from ceph/wip-12861

qa/workunits/rados/test_alloc_hint.sh: sudo to ls files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5396 from XinzeChi/wip-scrub-promote

bug fix: osd: requeue_scrub when kick_object_context_blocked

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5772 from yuyuyu101/wip-12912

AsyncConnection: Close connection when unregistered connection met WAIT

Merge pull request #5733 from cxwshawn/rgw-fix

rgw: init_rados failed leads to repeated delete

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5809 from rzarzynski/wip-rgw_putobj_debug_improvement

rgw: improve debugs in RGWPutObj and RGWPutObj_ObjStore_SWIFT.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5567 from guce/rgw_check_key_parameter_conflict

rgw: add key parameter conflict check for radosgw-admin command line.

Merge pull request #5819 from rzarzynski/wip-12950

rgw: don't append empty ETag HTTP header.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

qa/workunits/rados/test_alloc_hint.sh: sudo to ls files

The osd data dir is owned by ceph and not readable by other
non-root users.

Fixes: #12861
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5257 from t-miyamae/wip-mshec-fixlogic4

erasure code: shec performance optimization by decoding cache

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: init_rados failed leads to repeated delete

Fixes: #12978
Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

Merge pull request #5463 from kylinstorage/wip-evict_ops

Allow evict operations to be throttled

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5745 from ceph/wip-12904

osd: sparse reads returning invalid extent map

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5580 from xinxinsh/wip-merge-two-setattr

merge two continuous OP_SETATTR ops into one OP_SETATTRS

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5785 from liewegas/wip-12861

qa/workunits/rados/test_alloc_hint.sh: sudo to list files

Merge pull request #5784 from dachary/wip-docker-helper

fix docker-test.sh for CentOS 7

Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: drop docker-tests.sh root and /dev support

Now that ceph-disk.sh activation tests switched to the ceph-disk suite
as found in https://github.com/ceph/ceph-qa-suite/tree/master/suites/ceph-disk,
it does not need for root and /dev support from docker-tests.sh. There
currently is no other use case and since it's generally not a good idea
to run anything as root in a container anyway.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: CentOS 7 needs systemd-container

It is now needed by the Ceph dependencies and won't work unless the fake
version of it is replaced by the actual one.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5728 from trociny/wip-12706-infernalis

ceph-disk: use /sys/dev/block/maj:min/partition to see if partition

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

Merge pull request #4895 from XinzeChi/wip-outdata-set

bug fix: osd: avoid multi set osd_op.outdata in tier pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5707 from athanatos/wip-12809

PG::handle_advance_map: on_pool_change after handling the map change

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #5751 from yuyuyu101/wip-12908

AsyncConnection: Don't use unsafe feature as message encode feature

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/infernalis'

qa/workunits/post-file.sh: sudo

Only root can read the key in /usr/share/...

Signed-off-by: Sage Weil <sage@redhat.com>

osd: dump full map bl at 20 when crc doesn't match

This will help us debug cases where the encoding doesn't match due
to a bug.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5827 from tchaikov/wip-doc-daemon

doc: fix the typo in command example

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: fix the typo in command example

always indent using tab, the rendered html looks good, but it helps with
editor to highlight the codeblock properly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5817 from liewegas/wip-ceph-daemon

doc: 'ceph --admin-daemon ...' -> 'ceph daemon ...'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5193 from SUSE/wip-babeltrace-suse

ceph.spec.in: fix lttng/babeltrace conditionals

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5796 from SUSE/wip-group-cond

ceph.spec.in: put distro conditionals around Group:

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: 'ceph --admin-daemon ...' -> 'ceph daemon ...'

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5624 from vumrao/wip-vumrao-12714

rgw : setting max number of buckets for users via ceph.conf option

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

ceph.spec.in: put distro conditional around Group:

Neither RHEL and Fedora require a Group: line, so retain
only for SLE/openSUSE (and set the right value for these distros)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5798 from SUSE/wip-fix-license

ceph.spec.in: fix License line

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #5801 from dreamhost/wip-12939-header

Fix casing of Content-Type header

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5811 from vumrao/wip-vumrao-12944

doc: Add pgcalc tool link in placement-groups document

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph.spec.in: fix lttng/babeltrace conditionals

lttng and babeltrace are build dependencies for rbd-replay-prep. Make
sure the right package names are used. Enable for SLE12, as well as
for openSUSE 13.1 and higher.

Move the BuildRequires out of the ceph-test subpackage and into the
distro-conditional dependencies section.

Make ordering of BuildRequires a little more alphabetical.

http://tracker.ceph.com/issues/12360 Fixes: #12360

Signed-off-by: Nathan Cutler <ncutler@suse.com>
sqme

Merge pull request #5791 from SUSE/wip-drop-lsb-release

ceph.spec.in: drop lsb-release dependency from ceph-common

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

rgw: don't append empty ETag HTTP header.

Fixes: #12950
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #5808 from ceph/wip-simplify-logrotate

logrotate.conf: Simplify log files reopening after log rotation

Reviewed-by: Milan Broz <mbroz@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

doc: Add pgcalc tool link in placement-groups document

Fixes #12944

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

Merge pull request #5771 from ceph/wip-fix-i386-build

memstore: fix the build on i386

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: improve debugs in RGWPutObj and RGWPutObj_ObjStore_SWIFT.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #5794 from dachary/wip-12933-erasure-code-dir

tests: ceph-erasure-code-corpus must test SIMD variants

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #5804 from tchaikov/wip-fix-build-boost-1.59

mon: fix the build with boost 1.59

Reviewed-by: Loic Dachary <ldachary@redhat.com>

erasure code: shec performance optimization with decoding cache

continued from https://github.com/ceph/ceph/pull/4132

Signed-off-by: Takanori Nakao <nakao.takanori@jp.fujitsu.com>

logrotate.conf: Simplify log files reopening after log rotation

SIGHUP (1) signal makes ceph services reopen its log files so we do not
need to call any init scripts magic, here.

Additionally, the old approach caused problems with rhel-based systems
since it found the service command and tried to reload ceph.service
instead of ceph.target.

This also caused problems with SELinux as processes with context
logrotate_t are not allowed to issue systemctl reload on a process.

This patch is also in sync with other logrotate postrotate scripts which
do exactly the same thing (send SIGHUP to the daemon).

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #5802 from liewegas/wip-coverity

a few coverity fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

mon: fix the build with boost 1.59

cmd_vartype is a
boost::variant<std::string, bool, int64_t, double,
               std::vector<std::string>>, so we can not get a
int8_t from it.

Fixes: #12922
Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/ReplicatedPG: ProxyWriteOp::reqid should not be a ref

CID 1322778 (#1 of 1): Pointer to local outside scope (RETURN_LOCAL)
1. escape_local_addr: Returning, through this->reqid, the address of stack variable _reqid.
    2. return: Returning here.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/osd_types: init coll_t::removal_seq in all ctors

CID 1322784 (#1 of 1): Uninitialized scalar variable (UNINIT)
2. uninit_use_in_call: Using uninitialized value coll.removal_seq when calling coll_t. [show details]

Signed-off-by: Sage Weil <sage@redhat.com>

test_cls_numops: fix iterator use

CID 1322828 (#1 of 1): Wrapper object use after free (WRAPPER_ESCAPE)
28. use_after_free: Using invalidated internal representation of local it.
CID 1322827 (#1 of 1): Wrapper object use after free (WRAPPER_ESCAPE)
25. use_after_free: Using invalidated internal representation of local it.
CID 1322826 (#1 of 1): Wrapper object use after free (WRAPPER_ESCAPE)
31. use_after_free: Using invalidated internal representation of local it.
CID 1322825 (#1 of 1): Wrapper object use after free (WRAPPER_ESCAPE)
31. use_after_free: Using invalidated internal representation of local it.

Signed-off-by: Sage Weil <sage@redhat.com>

Fix casing of Content-Type header

It turns out, despite the HTTP spec declaring that header field names
should be case-insensitive, some clients treat them wrongly, and
consider "Content-type" to not match "Content-Type".

CyberDuck was one of those clients, now fixed upstream in
https://trac.cyberduck.io/ticket/8999

To reduce future occurances of this bug, fix the casing of the
Content-Type header, to strictly comply with the HTTP specification (be
strict about what you send, and generous about what you receive).

Fixes: #12939
Backport: infernalis, hammer, firefly
Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

debian: package radosgw-object-expirer in radosgw deb

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec: package new rgw files

   /usr/bin/radosgw-object-expirer
   /usr/lib64/rados-classes/libcls_timeindex.so

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5799 from liewegas/wip-cmake

cmake: fix build (newstore issues)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

CMakeLists.txt: add newstore files

Signed-off-by: Sage Weil <sage@redhat.com>

.gitignore: ignore build (usually used by cmake)

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/master' into infernalis

rgw/Makefile.am: ship rgw_object_expirer_core.h

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5744 from ceph/wip-12909

cmake: update FUSE_INCLUDE_DIRS to match autoconf

Merge pull request #5610 from ceph/wip-cmake

cmake: make check

ceph.spec: build requires cmake

Signed-off-by: Sage Weil <sage@redhat.com>

debian/control: build requires cmake

Signed-off-by: Sage Weil <sage@redhat.com>

debian/control: build-requires libboost-regex-dev

Signed-off-by: Sage Weil <sage@redhat.com>

tests: ceph-erasure-code-corpus must test SIMD variants

ceph_erasure_code.cc and ceph_erasure_code_benchmark.cc failed to load
the plugins. It went unnoticed when
660ae5bcbb250b06cf88ec7f9a3f37b05c6c8118 was reviewed because

  * ceph_erasure_code_benchmark is not used in make check
  * qa/workunits/erasure-code/encode-decode-non-regression.sh silently
    interpreted the failure as the absence of SIMD variants

http://tracker.ceph.com/issues/12933 Fixes: #12933

Signed-off-by: Loic Dachary <ldachary@redhat.com>

ceph.spec.in: fix License line

This is closer to my reading of
https://github.com/ceph/ceph/blob/master/COPYING than the previous version.

http://tracker.ceph.com/issues/12935 Fixes: #12935

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5692 from ceph/wip-rgw-swift-expiration

Wip rgw swift expiration

Reviewed-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: install crushtool to destdir/bin

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

cmake: add blkid as dependency to libcommon

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cmake: Changed name of crc32 target to crc32c

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Added shell script tests

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Fixed HAVE_BETTER_YASM_ELF64 variable

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Removed trailing spaces from isa .s files

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Uncommented erasure-code/shec conditional

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Removed traces of CDS, minor cmake fixes

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Fixed rbd_replay build issue

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Removed scripts, check_PROGRAMS included

Removed the unittest scripts for the time being.
Built unittests included in check_PROGRAMS target.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: Cleaned up syntax for make check targets

Signed-off-by: Ali Maredia <amaredia@redhat.com>

cmake: check_TESTPROGRAMS tests running

Make check working, accept rocksdb tests. Clean up coming.

Signed-off-by: Ali Maredia <amaredia@redhat.com>

README.md: Add basic CMake instructions

README.md: Fixed spacing, trimmed cmake section

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #5792 from ceph/wip-vstart-rgw

vstart: add -c argument to radosgw-admin commands

Reviewed-by: Sage Weil <sage@redhat.com>

vstart: add -c argument to radosgw-admin commands

Signed-off-by: Casey Bodley <cbodley@redhat.com>

ceph.spec.in: drop redundant centos from conditionals

Since the %{rhel} macro is guaranteed to be defined on all CentOS
installations, the %{centos} macro is superfluous.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: clean up suse_version conditionals

Use 0%{?suse_version} throughout for consistency.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: drop lsb-release dependency from ceph-common

It was there as an equivalent of redhat-lsb-core, but the redhat-lsb-core bits
that ceph-common relies on are included in insserv-compat on SUSE, and
insserv-compat is in base.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #5590 from majianpeng/mds

Mds: add osdmap epoch for setxattr.

Reviewed-by: John Spray <john.spray@redhat.com>

rgw : setting max number of buckets for users via ceph.conf option

This patch adds a new option "rgw_user_max_buckets" for setting
max number of buckets for users via ceph.conf.

Fixes #12714

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>

qa/workunits/rados/test_alloc_hint.sh: sudo to list files

The osd data dir is owned by ceph and not readable by other
non-root users.

Fixes: #12861
Signed-off-by: Sage Weil <sage@redhat.com>

rgw: don't copy delete_at attr, unless it's intra region copy

We don't want to keep the expiration value of a copied object, unless
we're doing a copy within the same zone group.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: objexp shards index by key

Not by time. This should provide better concurrency.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: delete-at and delete-after also on obj put / copy

And potentially later we could use also the S3 api, so it
could work with multipart upload, and POST obj.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5775 from dachary/wip-do-autogen

tools: fix do_autogen.sh -R

Merge pull request #5712 from yuyuyu101/wip-12801

Mon: Make ceph osd metadata support dump all osds

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

tools: fix do_autogen.sh -R

The R letter was missing from the getopts flags. Also sort the flags
alphabetically to make it easier to spot that kind of lossage.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

rocksdb: fix 32-bit build

Signed-off-by: Kefu Chai <kchai@redhat.com>

AsyncConnection: Close connection when unregistered connection met WAIT

Fix #12912
Signed-off-by: Haomai Wang <haomaiwang@gmail.com>