AsyncConnection: Avoid lockdep detect failed

Previously we used a atomic field to avoid deadlock, but it still let lockdep
detect failed. So we remove the atomic field and unlock connection firstly.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

AsyncConnection: Don't dispatch event when connection is stopped

When marking down connection, previously we will call stop which will dispatch event
regardless of the staus of the connection. If this connection is already down and
its events all has cleaned, we will hit NULL event.

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test_async_driver: add dispatch_external_event tests

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4514 from majianpeng/rbd

misc bugs in librbd

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4534 from tchaikov/wip-silence-gcc-warning

osd: silence gcc -Wparentheses warning

Reviewed-by: Loic Dachary <ldachary@redhat.com>

osd: silence gcc -Wparentheses warning

otherwise we have following warning:
warning: suggest parentheses around operand of ‘!’ or change ‘|’ to
‘||’ or ‘!’ to ‘~’ [-Wparentheses]

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/CompatSet: mark FeatureSet member vars private

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: For format 2, don't forget set objectcache max objects.

For format 2, before create object_cache, it call init_layout and set
the max object for object cache.
In the later ictx_refresh, it only set for format1.
Now move set logic after creating object cache.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rbd: updated the 'rbd --help'

Now default format is 2. update 'rbd --help'.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: remove the perfcounter l_librbd_aio_rd/write/discard_*.

Now for non-aio read/write/discard, the internal implementation used the
related aio function. So we don't differentiate the aio perfcounter from non-aio for
read/write/discard operations.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Remove the redundant judgement.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

librbd: Remove the redundant judgement.

For ObjectExtent, if length == object_size, it mean length=0.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Preforker: include acconfig so includers don't have to

Fixes build of systests

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4524 from ceph/wip-remove-fsx-c

test: update CMakefile to sync with c44f8e7

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4518 from yuyuyu101/wip-fix-systest-runnable

systest_runnable: Use Preforker to do fork

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4519 from ceph/wip-flush-log

Flush cluster log before MDS::damaged

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4529 from ceph/wip-librbd-open

librbd: invoking RBD::open twice will leak memory

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4511 from yuyuyu101/wip-async-fix-7

Wip async fix 7

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

atomic_t: change cas to compare_and_swap and return bool

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

mds: in damaged() call flush_log before ending

...so that the clog messages we send before damaged()
actually have a chance to make it to the cluster log.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: add MonClient::flush_log

A simple reasonable-effort call for emitting
a MLog before shutting down.  Used in MDS which
also waits for its beacon to be acked, so due to
message ordering implicitly waits for this to land
too -- no need to implement a waiting log flush
for this use case.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #4516 from yuyuyu101/wip-fix-libcephfs-crash

Wip fix libcephfs crash

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

librbd: invoking RBD::open twice will leak memory

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4492 from SUSE/wip-suse-firewall

packaging: add SuSEfirewall2 service files

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4468 from cernceph/swifty

Swift: Set Content-Length when requesting/checking Keystone tokens

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4008 from SUSE/wip-tcmalloc-conditionals

Conditional-compile against minimal tcmalloc.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3429 from liewegas/wip-precalc-pgtemp

RFC: mon: prime pg_temp when osdmap changes

Reviewed-by: Guang Yang <yguang@yahoo-inc.com>
Tested-by: Zhi (David) Zhang <zhangz@yahoo-inc.com>

Merge branch 'wip-journal-header' of git://github.com/XinzeChi/ceph

Conflicts:
src/gmock

Merge pull request #3474 from majianpeng/fix5

buffer: optimize get_contiguous

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3896 from ceph/wip-auth

auth: optimize crypto key handling, msg sig checks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3975 from wonzhq/skip-cache

osd: flush/evict op fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4362 from XinzeChi/wip-tier-lat

osd: misc latency perf counter for tier pool

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #4393 from majianpeng/copyget-flags

For CEPH_OSD_OP_COPY_GET, add a new filed: flags.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3564 from guangyy/wip-10198

osd: break the handling of PG removal event into multiple iterations

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3004 from guangyy/wip-10183

Add a new tick timer which does not need to hold osd_lock, put sched_scrub to that tick timer

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3659 from SUSE/wip-fix-calamari-health

Always provide summary for non-healthy cluster.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3702 from mslovy/wip-fineGrain-submitManagerLock

FileStore:: queue_transactions, fine-grain submitManager lock

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3919 from guangyy/wip-9558

osd : populate the needs_recovery_map source fast when only one peer has missing

Reviewed-by: Samuel Just <sjust@redhat.com>

test: update CMakefile to sync with c44f8e7

* remove unused src/test/fsx.c, see
  qa/workunits/suites/fsx.sh

Signed-off-by: Kefu Chai <kchai@redhat.com>

systest_runnable: adjust argument to suite Preforker

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

ceph_mon: output preforker error message

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Preforker: Add child process exit status check

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

systest_runnable: Use Preforker to do fork

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

test_libcephfs: Add tests to cover zero length write

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4493 from ceph/wip-use-get-linux-version

use get_linux_version() instead ad-hoc uname().release parsing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Client: Fast return if len is 0

If len == 0,
osdc/Striper.cc: 50: FAILED assert(len > 0)
 2: (Striper::file_to_extents(CephContext*, char const*, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::map<object_t, std::vector<ObjectExtent, std::allocator<ObjectExtent> >, std::less<object_t>, std::allocator<std::pair<object_t const, std::vector<ObjectExtent, std::allocator<ObjectExtent> > > > >&, unsigned long)+0x1c6) [0x7fd1a0f4f522]
 3: (Striper::file_to_extents(CephContext*, char const*, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::vector<ObjectExtent, std::allocator<ObjectExtent> >&, unsigned long)+0x69) [0x7fd1a0f4f315]
 4: (Striper::file_to_extents(CephContext*, inodeno_t, ceph_file_layout const*, unsigned long, unsigned long, unsigned long, std::vector<ObjectExtent, std::allocator<ObjectExtent> >&)+0x8a) [0x7fd1a0ef9c3c]
 5: (ObjectCacher::file_read(ObjectCacher::ObjectSet*, ceph_file_layout*, snapid_t, long, unsigned long, ceph::buffer::list*, int, Context*)+0x7e) [0x7fd1a0efb186]
 6: (Client::_read_async(Fh*, unsigned long, unsigned long, ceph::buffer::list*)+0x4ca) [0x7fd1a0ec90d4]
 7: (Client::_read(Fh*, long, unsigned long, ceph::buffer::list*)+0x3dc) [0x7fd1a0ec8744]
 8: (Client::read(int, char*, long, long)+0x208) [0x7fd1a0ec8100]
 9: (ceph_read()+0x5e) [0x7fd1a0e80603]

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4461 from ceph/wip-11372

librbd: add valgrind memory checks for unit tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4241 from rzarzynski/wip-11285

rgw: rectify broken statistics on Swift account listing.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4432 from big-henry/wip-rgw-listparts

rgw: fix ListParts response

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

pybind: fix valgrind warning on rbd_get_parent_info call

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osdc: invalid read of freed memory

The bytes not in cache stat was potentially reading the bh length
from a deleted bufferhead.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

krbd: fix incorrect types in the krbd API

The C API functions were referencing the C++ CephContext
instead of the C rados_config_t.  Additionally, the ceph
namespace was missing on the Formatter class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

fsx: cleanup crypto library at exit

Also made small tweaks so that it can be compiled under
a C++ compiler.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

autotools: detect presence of valgrind

Conditionally enable running librbd unit tests under valgrind.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: add support for running API tests under valgrind

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: run librbd valgrind test during 'make check'

Fixes: #11372
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

valgrind: update valgrind suppressions for lttng-ust

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Event: Use atomic cas to avoid missing wakeup

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Atomic: Add CAS primitive support

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4476 from dachary/wip-releases

doc: add giant v0.87.2 to the release timeline

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@ril.com>

config: Document that 'mds_max_file_size' is only read on FS creation

Afterwards you have to change this through the monitors

Merge pull request #4037 from osynge/wip-fix-disk-zap-sgdisk-invocation

Fix "disk zap" sgdisk invocation

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Fix "disk zap" sgdisk invocation

Fixes #11143

If the metadata on the disk is truly invalid, sgdisk would fail to zero
it in one go, because --mbrtogpt apparently tried to operate on the
metadata it read before executing --zap-all.

Splitting this up into two separate invocations to first zap everything
and then clear it properly fixes this issue.

Based on patch by Lars Marowsky-Bree <lmb@suse.com> in ceph-deploy.
Created by Vincent Untz <vuntz@suse.com>

Signed-off-by: Owen Synge <osynge@suse.com>
Signed-off-by: Thorsten Behrens <tbehrens@suse.com>

Merge pull request #4513 from SUSE/wip-suse-run-make-check

run-make-check.sh: fix a copule of SUSE oddities

Reviewed-by: Loic Dachary <ldachary@redhat.com>

run-make-check.sh: stricter check for yum/apt-get/zypper

Believe it or not, it's possible (although not necessarily likely) to
have both yum and zypper installed on openSUSE.  In this case the
previous implementation using `which` would return a string containing
both commands, which doesn't work.

Signed-off-by: Tim Serong <tserong@suse.com>

run-make-check.sh: use /sbin/modprobe

Rationale: In some cases (e.g.: by default on openSUSE), `sudo` won't
add /sbin to $PATH.

Signed-off-by: Tim Serong <tserong@suse.com>

AsyncConnection: Fix connection doesn't exchange in_seq problem

1. osd.a connect to osd.b and osd.b try to connect a
2. osd.b got WAIT then osd.b accpet connection from osd.a
2. osd.b inject error cause each side connection try to reconnect, connect_seq++
3. osd.a try to reconnect
4. osd.b connect to osd.a and cause the new accept connection try to replace
5. osd.b's is_reset_from_peer=true
6. inject error to osd.b
7. osd.b connect to osd.a
8. successfully and is_reset_from_peer still is true
9. long time passed
10. error happen and osd.a connect to osd.b
11. because is_reset_from_peer is true, they don't exchange in_seq

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

Merge pull request #4509 from wonzhq/miss-arg

librbd: missing an argument when calling invoke_async_request

Reviewed-by: Kefu Chai <kchai@redhat.com>

AsyncConnection: Ornament log output

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

librbd: missing an argument when calling invoke_async_request

This prevents the master branch from compiling.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

packaging: move SuSEfirewall2 templates out of src

Better to have static config like this that needs to be installed in
/etc in a separate subdirectory of the project.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #4186 from ceph/wip-7787

librbd: add fast diff feature

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #4381 from ceph/wip-system-retcode

ceph-fuse: check return value on system() invocation

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #4457 from ceph/wip-client-osd-caps

client: check OSD caps before read/write

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

ceph-fuse: check return value on system() invocation

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #4495 from rzarzynski/wip-11501

rgw: fix broken account listing of Swift API.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #4487 from ceph/wip-11464

Revert "osd: For object op, first check object whether unfound."

Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: fix broken account listing of Swift API.

Fixes: #11501
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

use get_linux_version() instead ad-hoc uname().release parsing

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #4445 from tchaikov/wip-doxygen-comments-in-throttle

doc: doxygen comments in throttle

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #4488 from ceph/wip-async-fix-6

Event: Delete driver after cleanup

packaging: add SuSEfirewall2 service files

This adds SuSEfirewall2 service files for Ceph MON, OSD and MDS, for use
on SLES and openSUSE.  The MON template opens port 6789 and the OSD/MDS
template opens the range 6800-7300 as per
http://ceph.com/docs/master/rados/configuration/network-config-ref/

Signed-off-by: Tim Serong <tserong@suse.com>

Event: Delete driver after cleanup

Signed-off-by: Haomai Wang <haomaiwang@gmail.com>

rgw: fix ListParts response

The response XML element name should be 'ListPartsResult'.

Fixes: #11494
Signed-off-by: Henry Chang <henry@bigtera.com>

qa/workunits/post-file: pick a dir that's readable by world

Signed-off-by: Sage Weil <sage@redhat.com>

Revert "osd: For object op, first check object whether unfound."

This reverts commit b16b080332616f12852c15cd27e022a83ed86319.

It is normal for objects to be unfound for brief periods of time
after peering--we do not want to EIO in that case.

Fixes: #11464
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #4483 from xinxinsh/wip-keyvalue-perfcounter

add perf counter for keyvalue db

Reviewed-by: Sage Weil <sage@redhat.com>

librbd: TaskFinisher should finish all queued tasks

The destructor wasn't waiting for all Finisher tasks
to complete before stopping the thread.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: librados completions are not properly released

Changes associated with rebuilding the object map can result
in the leaking of librados AioCompletions.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: fix valgrind errors with librbd unit test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: librbd should release global data before exit

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: cleanup singleton memory allocation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #4480 from wonzhq/indent

osd/ReplicatedPG: fix an indent in find_object_context

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #4474 from ceph/wip-11482

mds: remove caps from revoking list when caps are voluntarily released

Reviewed-by: John Spray <john.spray@redhat.com>

osd/ReplicatedPG: fix an indent in find_object_context

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

doc: add giant v0.87.2 to the release timeline

Signed-off-by: Loic Dachary <ldachary@redhat.com>

mds: remove caps from revoking list when caps are voluntarily released

Fixes: #11482
Signed-off-by: Yan, Zheng <zyan@redhat.com>

add perf counter for rocksdb to evaluate latency of get and transaction commit

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add perf counter for leveldb to evaluate latency of get&commit

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

Merge pull request #4454 from ceph/wip-11447

rgw: use correct oid when creating gc chains

Reviewed-by: Josh Durgin <jdurgin@redhat.com>