test/bufferlist: add test case for bufferlist::splice.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

bufferptr: Offset should not beyond raw_length rather than _len.

Fixes: #11526
Commit dfccd3d19c introduce this bug"
./include/buffer.h: 240: FAILED assert(_len >= o)

 ceph version 0.94-1402-g5ac962d
(5ac962d879d920805bd661857592860caae27f2b)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x80) [0x1640e50]
 2: (ceph::buffer::ptr::set_offset(unsigned int)+0x3a) [0x1647462]
 3: (ceph::buffer::list::splice(unsigned int, unsigned int,
ceph::buffer::list*)+0x238) [0x16458ba]
 4: (JournalStream::read(ceph::buffer::list&, ceph::buffer::list*,
unsigned long*)+0x17c) [0x145588e]
 5: (Journaler::try_read_entry(ceph::buffer::list&)+0x1ef) [0x1453d65]
 6: (MDLog::_replay_thread()+0xc6a) [0x1434200]
 7: (MDLog::ReplayThread::entry()+0x1c) [0x10f0d88]
 8: (Thread::entry_wrapper()+0xa8) [0x16310ee]
 9: (Thread::_entry_func(void*)+0x18) [0x163103c]
 10: /lib64/libpthread.so.0() [0x334c607ee5]
 11: (clone()+0x6d) [0x334bef4d1d]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is
needed to interpret this.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Preforker: include acconfig so includers don't have to

Fixes build of systests

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4524 from ceph/wip-remove-fsx-c

test: update CMakefile to sync with c44f8e7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4518 from yuyuyu101/wip-fix-systest-runnable

systest_runnable: Use Preforker to do fork

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4519 from ceph/wip-flush-log

Flush cluster log before MDS::damaged

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4529 from ceph/wip-librbd-open

librbd: invoking RBD::open twice will leak memory

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4511 from yuyuyu101/wip-async-fix-7

Wip async fix 7

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

atomic_t: change cas to compare_and_swap and return bool

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

mds: in damaged() call flush_log before ending

...so that the clog messages we send before damaged()
actually have a chance to make it to the cluster log.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: add MonClient::flush_log

A simple reasonable-effort call for emitting
a MLog before shutting down.  Used in MDS which
also waits for its beacon to be acked, so due to
message ordering implicitly waits for this to land
too -- no need to implement a waiting log flush
for this use case.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4516 from yuyuyu101/wip-fix-libcephfs-crash

Wip fix libcephfs crash

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librbd: invoking RBD::open twice will leak memory

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4492 from SUSE/wip-suse-firewall

packaging: add SuSEfirewall2 service files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4468 from cernceph/swifty

Swift: Set Content-Length when requesting/checking Keystone tokens

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4008 from SUSE/wip-tcmalloc-conditionals

Conditional-compile against minimal tcmalloc.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3429 from liewegas/wip-precalc-pgtemp

RFC: mon: prime pg_temp when osdmap changes

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>
Tested-by: Zhi (David) Zhang <zhangz@yahoo-inc.com>

Merge branch 'wip-journal-header' of git://github.com/XinzeChi/ceph

Conflicts:
src/gmock

Merge pull request #3474 from majianpeng/fix5

buffer: optimize get_contiguous

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3896 from ceph/wip-auth

auth: optimize crypto key handling, msg sig checks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3975 from wonzhq/skip-cache

osd: flush/evict op fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4362 from XinzeChi/wip-tier-lat

osd: misc latency perf counter for tier pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4393 from majianpeng/copyget-flags

For CEPH_OSD_OP_COPY_GET, add a new filed: flags.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3564 from guangyy/wip-10198

osd: break the handling of PG removal event into multiple iterations

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3004 from guangyy/wip-10183

Add a new tick timer which does not need to hold osd_lock, put sched_scrub to that tick timer

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3659 from SUSE/wip-fix-calamari-health

Always provide summary for non-healthy cluster.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3702 from mslovy/wip-fineGrain-submitManagerLock

FileStore:: queue_transactions, fine-grain submitManager lock

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3919 from guangyy/wip-9558

osd : populate the needs_recovery_map source fast when only one peer has missing

Reviewed-by: Samuel Just <sjust@redhat.com>

test: update CMakefile to sync with c44f8e7

* remove unused src/test/fsx.c, see
  qa/workunits/suites/fsx.sh

Signed-off-by: Kefu Chai <kchai@redhat.com>

systest_runnable: adjust argument to suite Preforker

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

ceph_mon: output preforker error message

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Preforker: Add child process exit status check

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

systest_runnable: Use Preforker to do fork

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test_libcephfs: Add tests to cover zero length write

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4493 from ceph/wip-use-get-linux-version

use get_linux_version() instead ad-hoc uname().release parsing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Client: Fast return if len is 0

If len == 0,
osdc/Striper.cc: 50: FAILED assert(len > 0)
 2: (Striper::file_to_extents(CephContext*, char const*, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::map<object_t, std::vector<ObjectExtent, std::allocator<ObjectExtent> >, std::less<object_t>, std::allocator<std::pair<object_t const, std::vector<ObjectExtent, std::allocator<ObjectExtent> > > > >&, unsigned long)+0x1c6) [0x7fd1a0f4f522]
 3: (Striper::file_to_extents(CephContext*, char const*, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::vector<ObjectExtent, std::allocator<ObjectExtent> >&, unsigned long)+0x69) [0x7fd1a0f4f315]
 4: (Striper::file_to_extents(CephContext*, inodeno_t, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::vector<ObjectExtent, std::allocator<ObjectExtent> >&)+0x8a) [0x7fd1a0ef9c3c]
 5: (ObjectCacher::file_read(ObjectCacher::ObjectSet*, ceph_file_layout*, snapid_t, long, unsigned long, ceph::buffer::list*, int, Context*)+0x7e) [0x7fd1a0efb186]
 6: (Client::_read_async(Fh*, unsigned long, unsigned long, ceph::buffer::list*)+0x4ca) [0x7fd1a0ec90d4]
 7: (Client::_read(Fh*, long, unsigned long, ceph::buffer::list*)+0x3dc) [0x7fd1a0ec8744]
 8: (Client::read(int, char*, long, long)+0x208) [0x7fd1a0ec8100]
 9: (ceph_read()+0x5e) [0x7fd1a0e80603]

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4461 from ceph/wip-11372

librbd: add valgrind memory checks for unit tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4241 from rzarzynski/wip-11285

rgw: rectify broken statistics on Swift account listing.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4432 from big-henry/wip-rgw-listparts

rgw: fix ListParts response

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: fix valgrind warning on rbd_get_parent_info call

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc: invalid read of freed memory

The bytes not in cache stat was potentially reading the bh length
from a deleted bufferhead.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

krbd: fix incorrect types in the krbd API

The C API functions were referencing the C++ CephContext
instead of the C rados_config_t.  Additionally, the ceph
namespace was missing on the Formatter class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

fsx: cleanup crypto library at exit

Also made small tweaks so that it can be compiled under
a C++ compiler.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

autotools: detect presence of valgrind

Conditionally enable running librbd unit tests under valgrind.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: add support for running API tests under valgrind

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: run librbd valgrind test during 'make check'

Fixes: #11372
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

valgrind: update valgrind suppressions for lttng-ust

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Event: Use atomic cas to avoid missing wakeup

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Atomic: Add CAS primitive support

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4476 from dachary/wip-releases

doc: add giant v0.87.2 to the release timeline

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

config: Document that 'mds_max_file_size' is only read on FS creation

Afterwards you have to change this through the monitors

Merge pull request #4037 from osynge/wip-fix-disk-zap-sgdisk-invocation

Fix "disk zap" sgdisk invocation

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Fix "disk zap" sgdisk invocation

Fixes #11143

If the metadata on the disk is truly invalid, sgdisk would fail to zero
it in one go, because --mbrtogpt apparently tried to operate on the
metadata it read before executing --zap-all.

Splitting this up into two separate invocations to first zap everything
and then clear it properly fixes this issue.

Based on patch by Lars Marowsky-Bree <lmb@suse.com> in ceph-deploy.
Created by Vincent Untz <vuntz@suse.com>

Signed-off-by: Owen Synge <osynge@suse.com>
Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

Merge pull request #4513 from SUSE/wip-suse-run-make-check

run-make-check.sh: fix a copule of SUSE oddities

Reviewed-by: Loic Dachary <ldachary@redhat.com>

run-make-check.sh: stricter check for yum/apt-get/zypper

Believe it or not, it's possible (although not necessarily likely) to
have both yum and zypper installed on openSUSE.  In this case the
previous implementation using `which` would return a string containing
both commands, which doesn't work.

Signed-off-by: Tim Serong <tserong@suse.com>

run-make-check.sh: use /sbin/modprobe

Rationale: In some cases (e.g.: by default on openSUSE), `sudo` won't
add /sbin to $PATH.

Signed-off-by: Tim Serong <tserong@suse.com>

AsyncConnection: Fix connection doesn't exchange in_seq problem

1. osd.a connect to osd.b and osd.b try to connect a
2. osd.b got WAIT then osd.b accpet connection from osd.a
2. osd.b inject error cause each side connection try to reconnect, connect_seq++
3. osd.a try to reconnect
4. osd.b connect to osd.a and cause the new accept connection try to replace
5. osd.b's is_reset_from_peer=true
6. inject error to osd.b
7. osd.b connect to osd.a
8. successfully and is_reset_from_peer still is true
9. long time passed
10. error happen and osd.a connect to osd.b
11. because is_reset_from_peer is true, they don't exchange in_seq

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4509 from wonzhq/miss-arg

librbd: missing an argument when calling invoke_async_request

Reviewed-by: Kefu Chai <kchai@redhat.com>

AsyncConnection: Ornament log output

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: missing an argument when calling invoke_async_request

This prevents the master branch from compiling.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

packaging: move SuSEfirewall2 templates out of src

Better to have static config like this that needs to be installed in
/etc in a separate subdirectory of the project.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #4186 from ceph/wip-7787

librbd: add fast diff feature

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4381 from ceph/wip-system-retcode

ceph-fuse: check return value on system() invocation

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #4457 from ceph/wip-client-osd-caps

client: check OSD caps before read/write

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-fuse: check return value on system() invocation

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4495 from rzarzynski/wip-11501

rgw: fix broken account listing of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4487 from ceph/wip-11464

Revert "osd: For object op, first check object whether unfound."

Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: fix broken account listing of Swift API.

Fixes: #11501
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

use get_linux_version() instead ad-hoc uname().release parsing

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #4445 from tchaikov/wip-doxygen-comments-in-throttle

doc: doxygen comments in throttle

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4488 from ceph/wip-async-fix-6

Event: Delete driver after cleanup

packaging: add SuSEfirewall2 service files

This adds SuSEfirewall2 service files for Ceph MON, OSD and MDS, for use
on SLES and openSUSE.  The MON template opens port 6789 and the OSD/MDS
template opens the range 6800-7300 as per
http://ceph.com/docs/master/rados/configuration/network-config-ref/

Signed-off-by: Tim Serong <tserong@suse.com>

Event: Delete driver after cleanup

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

rgw: fix ListParts response

The response XML element name should be 'ListPartsResult'.

Fixes: #11494
Signed-off-by: Henry Chang <henry@bigtera.com>

qa/workunits/post-file: pick a dir that's readable by world

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "osd: For object op, first check object whether unfound."

This reverts commit b16b080332616f12852c15cd27e022a83ed86319.

It is normal for objects to be unfound for brief periods of time
after peering--we do not want to EIO in that case.

Fixes: #11464
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4483 from xinxinsh/wip-keyvalue-perfcounter

add perf counter for keyvalue db

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: TaskFinisher should finish all queued tasks

The destructor wasn't waiting for all Finisher tasks
to complete before stopping the thread.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: librados completions are not properly released

Changes associated with rebuilding the object map can result
in the leaking of librados AioCompletions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: fix valgrind errors with librbd unit test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: librbd should release global data before exit

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: cleanup singleton memory allocation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4480 from wonzhq/indent

osd/ReplicatedPG: fix an indent in find_object_context

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4474 from ceph/wip-11482

mds: remove caps from revoking list when caps are voluntarily released

Reviewed-by: John Spray <john.spray@redhat.com>

osd/ReplicatedPG: fix an indent in find_object_context

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

doc: add giant v0.87.2 to the release timeline

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mds: remove caps from revoking list when caps are voluntarily released

Fixes: #11482
Signed-off-by: Yan, Zheng <zyan@redhat.com>

add perf counter for rocksdb to evaluate latency of get and transaction commit

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add perf counter for leveldb to evaluate latency of get&commit

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #4454 from ceph/wip-11447

rgw: use correct oid when creating gc chains

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4471 from ceph/wip-11477

rbd: correct issues with cli integration tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4470 from ceph/wip-11475

librbd: correct issues discovered by Python unit tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/release-notes: v0.87.2

Signed-off-by: Sage Weil <sage@redhat.com>

Swift: Set Content-Length when requesting/checking Keystone tokens

Running Keystone with WSGIChunkedRequest=On is not supported.

We have to make sure that we set the Content-Length header when getting
an admin token and checking revoked tokens, otherwise Keystone returns
a HTTP 411 error.

Same applies when checking revoked tickets.

Fixes: #11473
Backport: Hammer, Firefly
Signed-off-by: Hervé Rousseau <hroussea@cern.ch>

client: check OSD caps before read/write

Signed-off-by: Yan, Zheng <zyan@redhat.com>

tests: rbd cli integration test should explicitly select image format

RBD format 2 is now the default image format, so tests involving the old
format should explicitly request the old format.

Fixes: #11477
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not attempt to retrieve metadata for old format images

An unnecessary error message is being logged due to a failure to retrieve
metadata for old-format images -- which don't support metadata.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: flatten should return -EROFS if image is read-only

Fixes: #11475
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allow snapshots to be created when snapshot is active

The librbd API previously permitted the creation of snapshots while
the image context was associated to another snapshot.  A recent code
cleanup broke that ability, so this re-introduces it.  The code change
also allows minor cleanup with rebuild_object_map.

Fixes: #11475
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4455 from liewegas/wip-civetweb-gitignore

civetweb: add .gitignore