debian: set rest-bench-dbg ceph-test-dbg dependencies

Debian's debug packages ought to depend on their respective binary
packages. This was the case for many of our ceph packages, but it was
not the case for ceph-test-dbg or rest-bench-dbg.

Add the dependencies on the relevant binary packages, pinned to
"= ${binary:Version}" per convention.

http://tracker.ceph.com/issues/11673 Fixes: #11673

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #4682 from SUSE/wip-11629-next

ceph.spec.in: SUSE/openSUSE builds need libbz2-devel

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: SUSE/openSUSE builds need libbz2-devel

http://tracker.ceph.com/issues/11629 Fixes: #11629

Signed-off-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #4686 from SUSE/wip-11638-next

ceph.spec.in: consolidate rhel+centos macros

ceph.spec.in: tweak ceph-common for SUSE/openSUSE

ceph-common needs python-argparse in SUSE/openSUSE and
needs redhat-lsb-core only in RHEL/CentOS/Fedora.

http://tracker.ceph.com/issues/11638 Fixes: #11638

Signed-off-by: Nathan Cutler <ncutler@suse.cz>

ceph.spec.in: consolidate centos/rhel macros

0%{?rhel} || 0%{?centos} can be simplified to just 0%{?rhel}, since
CentOS defines that macro for compatibility with RHEL.

Signed-off-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #4675 from ceph/wip-11622

rgw: merge manifests correctly when there's prefix override

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: merge manifests correctly when there's prefix override

Fixes: #11622
Backport: hammer, firefly

Prefix override happens in a manifest when a rados object does not
conform to the generic prefix set on the manifest. When merging
manifests (specifically being used in multipart objects upload), we need
to check if the rule that we try to merge has a prefix that is the same
as the previous rule. Beforehand we checked if both had the same
override_prefix setting, but that might not apply as both manifests
might have different prefixes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/wip-11613' into next

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4661 from ceph/wip-11604

rgw: restore buffer of multipart upload after EEXIST

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4678 from SUSE/wip-submitting-patches-typo

SubmittingPatches: fix typo

Reviewed-by: Sage Weil <sage@redhat.com>

SubmittingPatches: fix typo

Signed-off-by: Nathan Cutler <ncutler@suse.cz>

rgw: return 412 when bad limit specified on listing buckets

Fixes: #11613
Backport: hammer
For some reason the swift api requires that. This revert a change that
was made in f7b92f9988ea64d248a2b551e0434b3225d296de.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4665 from ceph/wip-11551-next

tests: disable new RBD features for the krbd fsx test

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

tests: disable fast-diff feature for the krbd fsx test

The kernel does not yet support fast-diff.

Fixes: #11551
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: restore buffer of multipart upload after EEXIST

Fixes #11604
Backport: hammer, firefly

When we need to restart a write of part data, we need to revert to
buffer to before the write, otherwise we're going to skip some data.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc: Fix .gitignore for 'doc' directory

caused by: 156e55b7f95bb7f1d73e1131d94aca75fb339315

Fixes: #11574
Signed-off-by: Kefu Chai <kchai@redhat.com>

Added a "ceph hello world" for a simple check for ceph-deploy qa suite

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
(cherry picked from commit 13abae186357f4e9bb40990a7a212f93ec2e1e79)

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #4530 from ceph/wip-librbd-clean-object

librbd: object map is not properly updated when deleting clean objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rados: Tell that pool removal failed. Not that it doesn't exist.

If removing a pool fails it could have various reasons. The pool
might be protected from removal (nodelete flag).

Merge pull request #4542 from liewegas/wip-sepia

sepia

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4387 from wonzhq/async-comp

librbd: don't notify_change on r == -ERESTART when notify async complete

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: don't notify_change on error when notify async complete

Don't need to increment the refresh_seq since the async op fails.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

Merge pull request #4539 from athanatos/wip-11429

OSD: handle the case where we resurrected an old, deleted pg

Reviewed-by: Sage Weil <sage@redhat.com>

doc: link to sepia report for lab info

Signed-off-by: Sage Weil <sage@redhat.com>

9.0.0

Merge pull request #4537 from majianpeng/buffer-fix

bufferptr: Offset should not beyond raw_length rather than _len.

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

man: do not dist man pages if sphinx is not available

Signed-off-by: Kefu Chai <kchai@redhat.com>

configure.ac: do not check for sphinx-build

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/bufferlist: add test case for bufferlist::splice.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

bufferptr: Offset should not beyond raw_length rather than _len.

Fixes: #11526
Commit dfccd3d19c introduce this bug"
./include/buffer.h: 240: FAILED assert(_len >= o)

 ceph version 0.94-1402-g5ac962d
(5ac962d879d920805bd661857592860caae27f2b)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x1640e50]
 2: (ceph::buffer::ptr::set_offset(unsigned int)+0x3a) [0x1647462]
 3: (ceph::buffer::list::splice(unsigned int, unsigned int,
ceph::buffer::list*)+0x238) [0x16458ba]
 4: (JournalStream::read(ceph::buffer::list&, ceph::buffer::list*,
unsigned long*)+0x17c) [0x145588e]
 5: (Journaler::try_read_entry(ceph::buffer::list&)+0x1ef) [0x1453d65]
 6: (MDLog::_replay_thread()+0xc6a) [0x1434200]
 7: (MDLog::ReplayThread::entry()+0x1c) [0x10f0d88]
 8: (Thread::entry_wrapper()+0xa8) [0x16310ee]
 9: (Thread::_entry_func(void*)+0x18) [0x163103c]
 10: /lib64/libpthread.so.0() [0x334c607ee5]
 11: (clone()+0x6d) [0x334bef4d1d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #4514 from majianpeng/rbd

misc bugs in librbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4534 from tchaikov/wip-silence-gcc-warning

osd: silence gcc -Wparentheses warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: silence gcc -Wparentheses warning

otherwise we have following warning:
warning: suggest parentheses around operand of ‘!’ or change ‘|’ to
‘||’ or ‘!’ to ‘~’ [-Wparentheses]

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/CompatSet: mark FeatureSet member vars private

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: For format 2, don't forget set objectcache max objects.

For format 2, before create object_cache, it call init_layout and set
the max object for object cache.
In the later ictx_refresh, it only set for format1.
Now move set logic after creating object cache.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rbd: updated the 'rbd --help'

Now default format is 2. update 'rbd --help'.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: remove the perfcounter l_librbd_aio_rd/write/discard_*.

Now for non-aio read/write/discard, the internal implementation used the
related aio function. So we don't differentiate the aio perfcounter from non-aio for
read/write/discard operations.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Remove the redundant judgement.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Remove the redundant judgement.

For ObjectExtent, if length == object_size, it mean length=0.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Preforker: include acconfig so includers don't have to

Fixes build of systests

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4524 from ceph/wip-remove-fsx-c

test: update CMakefile to sync with c44f8e7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4518 from yuyuyu101/wip-fix-systest-runnable

systest_runnable: Use Preforker to do fork

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4519 from ceph/wip-flush-log

Flush cluster log before MDS::damaged

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4529 from ceph/wip-librbd-open

librbd: invoking RBD::open twice will leak memory

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4511 from yuyuyu101/wip-async-fix-7

Wip async fix 7

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librbd: object map is not properly updated when deleting clean objects

With the new fast diff feature flagging which objects haven't changed
between snapshots, unmodified objects are now flagged with a new state
code in the object map.  The OBJECT_EXISTS_CLEAN state needs to be
treated as if it were OBJECT_EXISTS when deleting.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

atomic_t: change cas to compare_and_swap and return bool

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

mds: in damaged() call flush_log before ending

...so that the clog messages we send before damaged()
actually have a chance to make it to the cluster log.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: add MonClient::flush_log

A simple reasonable-effort call for emitting
a MLog before shutting down.  Used in MDS which
also waits for its beacon to be acked, so due to
message ordering implicitly waits for this to land
too -- no need to implement a waiting log flush
for this use case.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4516 from yuyuyu101/wip-fix-libcephfs-crash

Wip fix libcephfs crash

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librbd: invoking RBD::open twice will leak memory

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4492 from SUSE/wip-suse-firewall

packaging: add SuSEfirewall2 service files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4468 from cernceph/swifty

Swift: Set Content-Length when requesting/checking Keystone tokens

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4008 from SUSE/wip-tcmalloc-conditionals

Conditional-compile against minimal tcmalloc.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3429 from liewegas/wip-precalc-pgtemp

RFC: mon: prime pg_temp when osdmap changes

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>
Tested-by: Zhi (David) Zhang <zhangz@yahoo-inc.com>

Merge branch 'wip-journal-header' of git://github.com/XinzeChi/ceph

Conflicts:
src/gmock

Merge pull request #3474 from majianpeng/fix5

buffer: optimize get_contiguous

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3896 from ceph/wip-auth

auth: optimize crypto key handling, msg sig checks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3975 from wonzhq/skip-cache

osd: flush/evict op fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4362 from XinzeChi/wip-tier-lat

osd: misc latency perf counter for tier pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4393 from majianpeng/copyget-flags

For CEPH_OSD_OP_COPY_GET, add a new filed: flags.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3564 from guangyy/wip-10198

osd: break the handling of PG removal event into multiple iterations

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3004 from guangyy/wip-10183

Add a new tick timer which does not need to hold osd_lock, put sched_scrub to that tick timer

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3659 from SUSE/wip-fix-calamari-health

Always provide summary for non-healthy cluster.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3702 from mslovy/wip-fineGrain-submitManagerLock

FileStore:: queue_transactions, fine-grain submitManager lock

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3919 from guangyy/wip-9558

osd : populate the needs_recovery_map source fast when only one peer has missing

Reviewed-by: Samuel Just <sjust@redhat.com>

OSD: handle the case where we resurrected an old, deleted pg

Prior to giant, we would skip pgs in load_pgs which were not present in
the current osdmap.  Those pgs would eventually refer to very old
osdmaps, which we no longer have causing the assertion failure in 11429
once the osd is finally upgraded to a version which does not skip the
pgs.  Instead, if we do not have the map for the pg epoch, complain to
the osd log and skip the pg.

Fixes: 11429
Signed-off-by: Samuel Just <sjust@redhat.com>

test: update CMakefile to sync with c44f8e7

* remove unused src/test/fsx.c, see
  qa/workunits/suites/fsx.sh

Signed-off-by: Kefu Chai <kchai@redhat.com>

systest_runnable: adjust argument to suite Preforker

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

ceph_mon: output preforker error message

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Preforker: Add child process exit status check

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

systest_runnable: Use Preforker to do fork

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test_libcephfs: Add tests to cover zero length write

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4493 from ceph/wip-use-get-linux-version

use get_linux_version() instead ad-hoc uname().release parsing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Client: Fast return if len is 0

If len == 0,
osdc/Striper.cc: 50: FAILED assert(len > 0)
 2: (Striper::file_to_extents(CephContext*, char const*, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::map<object_t, std::vector<ObjectExtent, std::allocator<ObjectExtent> >, std::less<object_t>, std::allocator<std::pair<object_t const, std::vector<ObjectExtent, std::allocator<ObjectExtent> > > > >&, unsigned long)+0x1c6) [0x7fd1a0f4f522]
 3: (Striper::file_to_extents(CephContext*, char const*, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::vector<ObjectExtent, std::allocator<ObjectExtent> >&, unsigned long)+0x69) [0x7fd1a0f4f315]
 4: (Striper::file_to_extents(CephContext*, inodeno_t, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::vector<ObjectExtent, std::allocator<ObjectExtent> >&)+0x8a) [0x7fd1a0ef9c3c]
 5: (ObjectCacher::file_read(ObjectCacher::ObjectSet*, ceph_file_layout*, snapid_t, long, unsigned long, ceph::buffer::list*, int, Context*)+0x7e) [0x7fd1a0efb186]
 6: (Client::_read_async(Fh*, unsigned long, unsigned long, ceph::buffer::list*)+0x4ca) [0x7fd1a0ec90d4]
 7: (Client::_read(Fh*, long, unsigned long, ceph::buffer::list*)+0x3dc) [0x7fd1a0ec8744]
 8: (Client::read(int, char*, long, long)+0x208) [0x7fd1a0ec8100]
 9: (ceph_read()+0x5e) [0x7fd1a0e80603]

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4461 from ceph/wip-11372

librbd: add valgrind memory checks for unit tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4241 from rzarzynski/wip-11285

rgw: rectify broken statistics on Swift account listing.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4432 from big-henry/wip-rgw-listparts

rgw: fix ListParts response

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: fix valgrind warning on rbd_get_parent_info call

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc: invalid read of freed memory

The bytes not in cache stat was potentially reading the bh length
from a deleted bufferhead.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

krbd: fix incorrect types in the krbd API

The C API functions were referencing the C++ CephContext
instead of the C rados_config_t.  Additionally, the ceph
namespace was missing on the Formatter class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

fsx: cleanup crypto library at exit

Also made small tweaks so that it can be compiled under
a C++ compiler.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

autotools: detect presence of valgrind

Conditionally enable running librbd unit tests under valgrind.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: add support for running API tests under valgrind

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: run librbd valgrind test during 'make check'

Fixes: #11372
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

valgrind: update valgrind suppressions for lttng-ust

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Event: Use atomic cas to avoid missing wakeup

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Atomic: Add CAS primitive support

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4476 from dachary/wip-releases

doc: add giant v0.87.2 to the release timeline

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

config: Document that 'mds_max_file_size' is only read on FS creation

Afterwards you have to change this through the monitors

Merge pull request #4037 from osynge/wip-fix-disk-zap-sgdisk-invocation

Fix "disk zap" sgdisk invocation

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Fix "disk zap" sgdisk invocation

Fixes #11143

If the metadata on the disk is truly invalid, sgdisk would fail to zero
it in one go, because --mbrtogpt apparently tried to operate on the
metadata it read before executing --zap-all.

Splitting this up into two separate invocations to first zap everything
and then clear it properly fixes this issue.

Based on patch by Lars Marowsky-Bree <lmb@suse.com> in ceph-deploy.
Created by Vincent Untz <vuntz@suse.com>

Signed-off-by: Owen Synge <osynge@suse.com>
Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

Merge pull request #4513 from SUSE/wip-suse-run-make-check

run-make-check.sh: fix a copule of SUSE oddities

Reviewed-by: Loic Dachary <ldachary@redhat.com>

run-make-check.sh: stricter check for yum/apt-get/zypper

Believe it or not, it's possible (although not necessarily likely) to
have both yum and zypper installed on openSUSE.  In this case the
previous implementation using `which` would return a string containing
both commands, which doesn't work.

Signed-off-by: Tim Serong <tserong@suse.com>

run-make-check.sh: use /sbin/modprobe

Rationale: In some cases (e.g.: by default on openSUSE), `sudo` won't
add /sbin to $PATH.

Signed-off-by: Tim Serong <tserong@suse.com>

AsyncConnection: Fix connection doesn't exchange in_seq problem

1. osd.a connect to osd.b and osd.b try to connect a
2. osd.b got WAIT then osd.b accpet connection from osd.a
2. osd.b inject error cause each side connection try to reconnect, connect_seq++
3. osd.a try to reconnect
4. osd.b connect to osd.a and cause the new accept connection try to replace
5. osd.b's is_reset_from_peer=true
6. inject error to osd.b
7. osd.b connect to osd.a
8. successfully and is_reset_from_peer still is true
9. long time passed
10. error happen and osd.a connect to osd.b
11. because is_reset_from_peer is true, they don't exchange in_seq

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4509 from wonzhq/miss-arg

librbd: missing an argument when calling invoke_async_request

Reviewed-by: Kefu Chai <kchai@redhat.com>

AsyncConnection: Ornament log output

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>