bluestore/BitAllocator: Fix deadlock with musl libc

pthread_rwlock_wrlock() behaviour is undefined when calling thread holds lock. Added unlock() in shutdown() to avoid deadlock in destructor.

http://pubs.opengroup.org/onlinepubs/7908799/xsh/pthread_rwlock_wrlock.html

Signed-off-by: John Coyle <dx9err@gmail.com>

Merge pull request #10579 from dreamhost/wip-16928

rgw: Fix civetweb IPv6

verified:  autobuild

Merge pull request #10597 from pritha-srivastava/wip_ldap_fix

rgw: Fix for incorrect access key id in reworked ldap auth code.

verified non-regression (MS AD)

rgw: Fix for incorrect access key id reworked ldap auth code.

The reworked ldap code does not use the correct access key id in get_policy.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #10595 from tchaikov/wip-rocksdb

rocksdb: revert the change introduced by dc41731

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rocksdb: revert the change introduced by dc41731

dc41731 reverts the submodule reference of rocksdb, and point it to an
older version which fails to support ppc64le and AArch64.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10262 from dachary/wip-16580-ceph-disk-timeout

ceph-disk: timeout ceph-disk to avoid blocking forever

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Ken Dreyer (Red Hat) <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.cz>

Merge pull request #10491 from xiexingguo/xxg-wip-osdmon-2016-07-28

osd/OSDMonitor: misc. fixes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9259 from xiexingguo/xxg-wip-fix-monclientpinger

mon: MonClient may hang on pinging an unresponsive monitor

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10593 from linuxbox2/ldap-bindv3-again

rgw ldap: enforce simple_bind w/LDAPv3

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9718 from songbaisen/a8

mon: clear list in better way

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10392 from stiopaa1/mon_monclient_moveCCancelMonCommand

mon/MonClient: move C_CancelMonCommand to cc file

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10470 from xiexingguo/xxg-wip-osd-2016-07-28

osd: minor performance improvements

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10539 from jcsp/wip-mds-counters

mds: add perf counters for MDLog replay and SessionMap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw ldap: enforce simple_bind w/LDAPv3

Found by Harald Klein <hklein@redhat.com>.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10542 from ktdreyer/doc-rbd-libvirt-logging

doc: describe libvirt client logging

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10488 from lande1234/wip-lan-testing

rbd-mirror: add additional test scenarios

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10573 from renhwztetecs/renhw-wip-doc-cephfs

doc/mds: fixup mds doc

Reviewed-by: John Spray <john.spray@redhat.com>

mds: add perf counter for rate of replay

Useful for watching the progress of a daemon
in standby-replay or replay.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: expose perf counters for SessionMap

So that we can see how many sessions exist via
the counters rather than only by looking at status.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #10512 from renhwztetecs/renhw-wip-mds-balancer

mds/MDBalancer: cleanup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10485 from gregsfortytwo/wip-client-dump-complete

client: include COMPLETE and ORDERED states in cache dump

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10533 from gregsfortytwo/wip-mds-dispatch-refcount

mds: fix up _dispatch ref-counting semantics

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10540 from jcsp/wip-mds-multimds-standby

mds: fix daemon selection when starting ranks

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rgw: Fix civetweb IPv6

Commit c38e3cbb6f7c6221209f2b512cba18c564c895a4 introduced a second
compiled version of src/civetweb/src/civetweb.c, but did not pass the
configuration header (civetweb/include/civetweb_conf.h).

As a result, USE_IPV6 was not defined when it was compiled, and that
copy was included into the radosgw binary. This caused breakage for the
civetweb frontend when used with IPv6:
  rgw frontends = civetweb port=[::]:7480

Reintroduce the header so that civetweb is compiled correctly again.

Fixes: http://tracker.ceph.com/issues/16928
Backport: jewel
Signed-off-by: Robin H. Johnson <robin.johnson@dreamhost.com>

Merge pull request #10565 from dzafman/wip-16672

OSD crash with Hammer to Jewel Upgrade: void FileStore::init_temp_collections()

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10545 from trociny/fixup-rbd-mirror-test

qa/workunits/rbd: wait for image deleted before checking health

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: describe libvirt client logging

Describe how to configure logging and the admin socket for debugging RBD
libvirt clients.

Thanks to Jason Dillman <jdillama@redhat.com> on ceph-devel for this
information.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10566 from batrick/mds-daemon-unused

mds: remove unused MDSDaemon::objecter

Reviewed-by: John Spray <john.spray@redhat.com>

doc/mds: add generated files into the /etc/ceph/

Fixes: 16906
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mds: remove unused MDSDaemon::objecter

Leftover from a2682edb76ceac428c686e0ac21c06511e27693e.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10307 from pritha-srivastava/wip_s3_auth_rework

rgw: Rework of s3 LDAP Authentication code.

verified: MS AD

os: Fix HashIndex::recursive_remove() to remove everything but original path

Fixes: http://tracker.ceph.com/issues/16672
Signed-off-by: David Zafman <dzafman@redhat.com>

filestore: Clear objects before calling collection_list() again

Signed-off-by: David Zafman <dzafman@redhat.com>

filestore: Improve logging

Signed-off-by: David Zafman <dzafman@redhat.com>

mds/MDBalancer: change OPT_FLOAT to OPT_DOUBLE

utime_t convert double in MDBalancer::tick(),
so g_conf->mds_bal_sample_interval should be set OPT_DOUBLE

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mds/MDBalancer: redundance cleanup

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mds/MDBalancer: some types of definitions are cleanup

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mds/MDBalancer: add err info when ifstream open failure

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge remote-tracking branch 'upstream/pull/10543/head'

* upstream/pull/10543/head:
  vstart.sh: only delete conf if -n

Merge pull request #10492 from chhabaramesh/master

os/bluestore: remove cmake warning from extent alloc functions

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

vstart.sh: only delete conf if -n

src/test/ceph_objectstore_tool.py depends on an implicit "-k" when not
creating a new cluster. (Deleting the conf file without -n makes no
sense anyway.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10462 from dillaman/wip-16478

rbd-mirror: non-primary image is recording journal events during image sync

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10484 from dillaman/wip-16538

rbd-mirror: image deleter should use pool id + global image uuid for key

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: wait for image deleted before checking health

This is a fixup to the previous commit.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10458 from gregsfortytwo/wip-doc-troubleshooting

doc: extend the CephFS troubleshooting guide

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9695 from ktdreyer/doc-tarballs-url

doc: update canonical tarballs URL

Reviewed-by: Alfredo Deza <adeza@redhat.com>

os/bluestore: remove cmake warning from extent alloc functions.
link: http://tracker.ceph.com/issues/16766
Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10386 from ceph/wip-jlayton-nlink

Fix attribute handling at lookup time

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10541 from tchaikov/wip-cmake

cmake: install ceph_test_rados_striper_api_*

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: install ceph_test_rados_striper_api_*

so we can test them in ceph-qa-suite

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: unit test cases for disabling librbd journal by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: utilize factory methods to create AioObjectRequest objects

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: convert AioObjectRequest/AioObjectRead classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move read callback helpers to narrowest scope

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: convert AioImageRequest derived classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: removed namespace indentation from legacy classes

Better follows the Ceph C++ style guide

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not record journal events if append is disabled by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove unused journal replay canceled callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: optionally support disabling journal appends via policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10505 from wjwithagen/wip-wjw-dns-resolv

src/common/dns_resolve.cc: reorder the includes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mds: fix daemon selection when starting ranks

Previously, if "standby for rank" was set, but
"standby for fscid" wasn't, the daemon wouldn't
be recognised as the right one for the rank,
and we would end up promoting some arbitrary
standby.

Signed-off-by: John Spray <john.spray@redhat.com>

ceph: don't fudge the ctime in stat() unless it's really older than the mtime

The current code just looks at the sec field, but the difference may
only be evident in the nsec field.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: only skip querying the MDS in _lookup when we have the necessary caps

If we don't have the caps we'll need later, then we must reissue the
lookup to get them regardless of whether we have a lease on the
dentry.

Fixes: http://tracker.ceph.com/issues/16668
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: plumb a mask argument into _lookup

...and attempt to pass in a sane value there, based on what we'll do
with the resulting inode.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: add mask parameter to _do_lookup

We need to allow callers to specify caps to acquire during a lookup, as
they may need to scrape certain info out of the inode later. Allow them
to pass in a mask. For now, _lookup just passes in 0 for the mask,
but verify_reply_trace passes in the regetattr_mask to match what we'd
request in the _getattr request if there were a traceless reply.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

test: add a link count test

...and use the lowlevel calls to emulate what ganesha would do. In
particular, it dumps its inode from the cache after an unlink. So
do a second lookup afterward and verify that the nlink field in the
struct stat is valid.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

doc: add a new document on capabilities

Signed-off-by: Jeff Layton <jlayton@redhat.com>

cephfs: remove some unused constants

The CEPH_CAP_SFLOCK shift value (and the other constants derived from
it) are entirely unused. Ditto for CEPH_CAP_BITS. Just remove them.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

doc/mds: consistent with "../doc/rados/deployment/index.rst"

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

Merge pull request #10535 from dzafman/wip-cot-fix

test: Removeall merged after print_function commit needs a fix

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Removeall merged after print_function commit needs a fix

Signed-off-by: David Zafman <dzafman@redhat.com>

mds: fix up _dispatch ref-counting semantics

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10344 from xiexingguo:xxg-wip-pipe-2016-07-19-02

msg/simple: apply prefetch policy more precisely

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Conflicts:
src/msg/simple/Pipe.cc (removed unneeded cast)

Merge pull request #10532 from linuxbox2/rgw-ldap-recon

rgw: add reinit/rebind logic (ldap)

Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: add reinit/rebind logic (ldap)

Gracefully handle stale LDAP connections by rebinding--verified
with MS AD.  Rebind is attempted once per request until rebound--
not worse than misconfiguring the environment.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10365 from majianpeng/osd-perfcounter

osd/ReplicatedPG: for sync-read it don't cacl l_osd_op_r_prepare_lat.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10450 from xiexingguo/xxg-wip-cleanup-options

osd: cleanup options and other redundancies

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10471 from xiexingguo/xxg-wip-fix-memdb-assert

kv/MemDB: fix assert triggerred by m_total_bytes underflow

Merge pull request #9679 from xiexingguo/xxg-wip-fix-cephx

auth/cephx: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9734 from xiexingguo/xxg-wip-cleanup-omap

os/filestore: add sanity checks and cleanups for mount() process

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10231 from yonghengdexin735/wip-zzz-paxosserver-type

mon/PaxosService: make the return value type inconsistent

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10340 from yuyuyu101/wip-16715

msg/async/AsyncConnection: replace Mutex with std::mutex for peformance

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10531 from scuttlemonkey/master

Add UK mirror and update copyright

doc: added new UK Ceph mirror to doc and mirroring

Signed-off-by: Patrick McGarry <pmcgarry@redhat.com>

Merge remote-tracking branch 'origin/master'

Merge pull request #10207 from prallabh/wip-16388

rgw: Fix incorrect content length and range for zero sized objects during range requests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10510 from yehudasa/wip-16861

test_rgw_bencode: null terminate strings before checking

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9944 from sileht/master

python-rados: extends ReadOp/WriteOp API

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10098 from dzafman/wip-16568

tools: New "removeall" used to remove head with snapshots

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10085 from chengweiv5/fix-doc

doc: ceph-deploy mon add doesn't take multiple nodes

Reviewed-by: Kefu Chai <kchai@redhat.com>

For range requests, let the range be dumped before the error is honored.

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

For zero sized objects, let the range adhere as per the RFC

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

For zero sized objects, return 416 during range requests

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

Merge pull request #10480 from ceph/lifecycle-cleanup

Clean up lifecycle thread

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Rework of s3 LDAP Authentication code.

The LDAP authentication code has been reworked based
on the new authentication infrastructure.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

doc/cephfs: fix up mds_standby_replay describe

Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>

mon/OSDMonitor: fix CEPH_FEATURE_SERVER_JEWEL of preprocess_boot() check

We shall check CEPH_FEATURE_SERVER_JEWEL carried by osd_features of
MOSDBoot message, as it will be properly filled by every OSD during
OSD::_send_boot() process.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: expose require_kraken_osds flag to user

For newly created cluster the CEPH_OSDMAP_REQUIRE_KRAKEN will be
automatically set, while for existing clusters it will not.

This change add "require_jewel_osds" to white list, so user
can access it by the "ceph osd set *" command family.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge remote-tracking branch 'upstream/pull/10514/head'

* upstream/pull/10514/head:
  src/vstart.sh: stop script aborting when ceph.cfg is not there

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>