os/MemStore: move OmapIteratorImpl to cc file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10566 from batrick/mds-daemon-unused

mds: remove unused MDSDaemon::objecter

Reviewed-by: John Spray <john.spray@redhat.com>

mds: remove unused MDSDaemon::objecter

Leftover from a2682edb76ceac428c686e0ac21c06511e27693e.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10307 from pritha-srivastava/wip_s3_auth_rework

rgw: Rework of s3 LDAP Authentication code.

verified: MS AD

Merge remote-tracking branch 'upstream/pull/10543/head'

* upstream/pull/10543/head:
  vstart.sh: only delete conf if -n

Merge pull request #10492 from chhabaramesh/master

os/bluestore: remove cmake warning from extent alloc functions

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

vstart.sh: only delete conf if -n

src/test/ceph_objectstore_tool.py depends on an implicit "-k" when not
creating a new cluster. (Deleting the conf file without -n makes no
sense anyway.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10462 from dillaman/wip-16478

rbd-mirror: non-primary image is recording journal events during image sync

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10484 from dillaman/wip-16538

rbd-mirror: image deleter should use pool id + global image uuid for key

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10458 from gregsfortytwo/wip-doc-troubleshooting

doc: extend the CephFS troubleshooting guide

Reviewed-by: Zack Cerza <zack@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9695 from ktdreyer/doc-tarballs-url

doc: update canonical tarballs URL

Reviewed-by: Alfredo Deza <adeza@redhat.com>

os/bluestore: remove cmake warning from extent alloc functions.
link: http://tracker.ceph.com/issues/16766
Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10386 from ceph/wip-jlayton-nlink

Fix attribute handling at lookup time

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10541 from tchaikov/wip-cmake

cmake: install ceph_test_rados_striper_api_*

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: install ceph_test_rados_striper_api_*

so we can test them in ceph-qa-suite

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: unit test cases for disabling librbd journal by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: utilize factory methods to create AioObjectRequest objects

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: convert AioObjectRequest/AioObjectRead classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move read callback helpers to narrowest scope

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: convert AioImageRequest derived classes to templates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: removed namespace indentation from legacy classes

Better follows the Ceph C++ style guide

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: do not record journal events if append is disabled by policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove unused journal replay canceled callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: optionally support disabling journal appends via policy

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10505 from wjwithagen/wip-wjw-dns-resolv

src/common/dns_resolve.cc: reorder the includes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph: don't fudge the ctime in stat() unless it's really older than the mtime

The current code just looks at the sec field, but the difference may
only be evident in the nsec field.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: only skip querying the MDS in _lookup when we have the necessary caps

If we don't have the caps we'll need later, then we must reissue the
lookup to get them regardless of whether we have a lease on the
dentry.

Fixes: http://tracker.ceph.com/issues/16668
Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: plumb a mask argument into _lookup

...and attempt to pass in a sane value there, based on what we'll do
with the resulting inode.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: add mask parameter to _do_lookup

We need to allow callers to specify caps to acquire during a lookup, as
they may need to scrape certain info out of the inode later. Allow them
to pass in a mask. For now, _lookup just passes in 0 for the mask,
but verify_reply_trace passes in the regetattr_mask to match what we'd
request in the _getattr request if there were a traceless reply.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

test: add a link count test

...and use the lowlevel calls to emulate what ganesha would do. In
particular, it dumps its inode from the cache after an unlink. So
do a second lookup afterward and verify that the nlink field in the
struct stat is valid.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

doc: add a new document on capabilities

Signed-off-by: Jeff Layton <jlayton@redhat.com>

cephfs: remove some unused constants

The CEPH_CAP_SFLOCK shift value (and the other constants derived from
it) are entirely unused. Ditto for CEPH_CAP_BITS. Just remove them.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #10535 from dzafman/wip-cot-fix

test: Removeall merged after print_function commit needs a fix

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Removeall merged after print_function commit needs a fix

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #10344 from xiexingguo:xxg-wip-pipe-2016-07-19-02

msg/simple: apply prefetch policy more precisely

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Conflicts:
src/msg/simple/Pipe.cc (removed unneeded cast)

Merge pull request #10532 from linuxbox2/rgw-ldap-recon

rgw: add reinit/rebind logic (ldap)

Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: add reinit/rebind logic (ldap)

Gracefully handle stale LDAP connections by rebinding--verified
with MS AD.  Rebind is attempted once per request until rebound--
not worse than misconfiguring the environment.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10365 from majianpeng/osd-perfcounter

osd/ReplicatedPG: for sync-read it don't cacl l_osd_op_r_prepare_lat.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10450 from xiexingguo/xxg-wip-cleanup-options

osd: cleanup options and other redundancies

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10471 from xiexingguo/xxg-wip-fix-memdb-assert

kv/MemDB: fix assert triggerred by m_total_bytes underflow

Merge pull request #9679 from xiexingguo/xxg-wip-fix-cephx

auth/cephx: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9734 from xiexingguo/xxg-wip-cleanup-omap

os/filestore: add sanity checks and cleanups for mount() process

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10231 from yonghengdexin735/wip-zzz-paxosserver-type

mon/PaxosService: make the return value type inconsistent

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10340 from yuyuyu101/wip-16715

msg/async/AsyncConnection: replace Mutex with std::mutex for peformance

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10531 from scuttlemonkey/master

Add UK mirror and update copyright

doc: added new UK Ceph mirror to doc and mirroring

Signed-off-by: Patrick McGarry <pmcgarry@redhat.com>

Merge remote-tracking branch 'origin/master'

Merge pull request #10207 from prallabh/wip-16388

rgw: Fix incorrect content length and range for zero sized objects during range requests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10510 from yehudasa/wip-16861

test_rgw_bencode: null terminate strings before checking

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9944 from sileht/master

python-rados: extends ReadOp/WriteOp API

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10098 from dzafman/wip-16568

tools: New "removeall" used to remove head with snapshots

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10085 from chengweiv5/fix-doc

doc: ceph-deploy mon add doesn't take multiple nodes

Reviewed-by: Kefu Chai <kchai@redhat.com>

For range requests, let the range be dumped before the error is honored.

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

For zero sized objects, let the range adhere as per the RFC

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

For zero sized objects, return 416 during range requests

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

Merge pull request #10480 from ceph/lifecycle-cleanup

Clean up lifecycle thread

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: Rework of s3 LDAP Authentication code.

The LDAP authentication code has been reworked based
on the new authentication infrastructure.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge remote-tracking branch 'upstream/pull/10514/head'

* upstream/pull/10514/head:
  src/vstart.sh: stop script aborting when ceph.cfg is not there

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

src/vstart.sh: stop script aborting when ceph.cfg is not there

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10432 from dillaman/wip-16708

rbd-mirror: potential IO stall when using asok flush request

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10341 from dillaman/wip-16223

rbd-mirror: reduce memory footprint during journal replay

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

python-rados: Implements new write_ops

* rados_write_op_create
* rados_write_op_append
* rados_write_op_write_full
* rados_write_op_write
* rados_write_op_remove
* rados_write_op_truncate
* rados_write_op_zero

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

python-rados: add some read_op/write_op methods

This change adds python API for the following C methods:
* rados_aio_write_op_operate
* rados_write_op_omap_set
* rados_read_op_set_flags
* rados_aio_read_op_operate

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #10489 from ceph/wip-cmake

cmake changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

test_rgw_bencode: null terminate strings before checking

Fixes: http://tracker.ceph.com/issues/16861
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc: add some links to CephFS troubleshooting guide

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

doc: extend the CephFS troubleshooting guide

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

common/dns_resolve.h: Make sure the .h file has al required includes.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

src/common/dns_resolve.cc: reorder the includes

 - dns_resolv.h should only be include after all the std-includes are
   made. Otherwise it will result into undefined structs/variables on
   FreeBSD/Clang

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

cmake: link libcommon against pthread

as Thread.cc lives in libcommon, and global is not using libpthread
directly. so moving libpthread linkage from libglobal to libcommon.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10177 from BlaXpirit/ceph-tests-py3

pybind: Port Python-based tests and remaining Python bindings to Python 3

Reviewed-by: Case Bodley <cbodley@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Clean up lifecycle thread

Make sure that the lifecycle thread is terminated and all memory is
cleaned up on shutdown.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge remote-tracking branch 'upstream/pull/10312/head'

* upstream/pull/10312/head:
  vstart: refactor and fix forced config changes
  vstart: cleanup echoed commands and sh issues

cmake: cleanup Findfuse.cmake

* remove unused stuff, we do not pass/use FUSE_VERSION in cmake.
  the FUSE_VERSION we are using is defined by "fuse_common.h".
* use FindPackageHandleStandardArgs for handling find_package()
  arguments.
* also detect osxfuse on MacOS.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not reference undefined variables

* AIO_LIBS is now named AIO_LIBARIES, and there is not point to print
  out its path
* USE_NSS is not defined if NSS is not checked, so if(USE_NSS) is better
  here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated linked lib for libcommon

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use fullpaths of libraries

better off using the detected path, instead of using the default path
for finding the linked libaries.

see "cmake  --help-policy CMP0065" for more details.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: set policies

this makes sure we are compatible with latest and future cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove uuid

as it is replaced by the boost implementation, and not detected in
cmake anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link ec_jerasure_generic against crush

crush is not used in erasure plugins at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: exclude private symbols in librbd

missed it in 0edfad0.
* librbd: pass '--exclude-libs,ALL' to linker
* rbd: link against used libraries explicitly, rbd is accessing some
  internal symbols not exposed by librbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10483 from trociny/fixup-test-rbd-mirror

qa/workunits/rbd: small fixup and improvements for rbd-mirror tests

Reviewed-by: Jason Dillaman <dillaman@hotmail.com>

Merge pull request #10374 from ktdreyer/wip-doc-debug-ceph-fuse

doc: add troubleshooting steps for ceph-fuse

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10332 from dillaman/wip-16689

librbd: optimize away unnecessary object map updates

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

doc: add troubleshooting steps for ceph-fuse

Thanks to John Spray <jspray@redhat.com> on ceph-devel for this
information.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10479 from batrick/standby-notes

doc: small standby doc edits

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10434 from batrick/i16743

doc: add client config ref

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10409 from weiqiaomiao/wqm-wip-rgw-reboot

rgw:fix rgw boot failed after upgrade to master latest version

Merge pull request #10379 from xiexingguo/xxg-wip-bluestore-2016-07-21

os/bluestore: fix bitmap allocating failure if max_alloc_size is 0

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10140 from ifed01/wip-bluestore-fsck-update

os/bluestore: replace interval_set with a bitset for used_block conta…

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests, however no fsck specific performance or memory tests were performed.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

librbd: support deleting image by id instead of name

The rbd-mirror daemon will use this API to delete images instead
of attempting to use the local image name.

Fixes: http://tracker.ceph.com/issues/16227
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: use pool id + global image id as deletion primary key

Fixes: http://tracker.ceph.com/issues/16538
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: add historical warning for mds_standby_replay

When testing < v10.2.1 with standby replay, I was often confused why the
MDS daemons would go into replay even when this was false.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: fix config separator/title size.

This makes the setting consistent with the others.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10060 from trociny/wip-16386

librbd: discard hangs when 'rbd_skip_partial_discard' is enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10472 from tchaikov/wip-cmake

cmake: exclude non-public symbols in shared libraries

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10273 from yonghengdexin735/wip-zzz-pip-delete-no_casting_required

msg/simple/Pipe: eliminating casts for the comparing of len and recv_max_prefetch

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10477 from cbodley/wip-16839

radosgw-admin: zone[group] modify can change realm id
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9988 from ifed01/wip-bluestore-reduce-blob4

os/bluestore: get rid off blob's ref_map for non-shared objects

Mark's Comments:

This passed Jenkins checks.
This PR did not appear to have a significant impact on performance tests.
This initially failed "ceph_test_objectstore --gtest_filter=*/2" segfaulting in one of the tests.  See the attached log in the comments for the PR for details.  Subsequent reruns passed, and a run through valgrind's memcheck also passed.  Igor indicated that he has seen this particular segfault previously and believes it is related to running out of memory.  Given that the PR has passed the object store tests multiple times and the segfault has been observed in the past, I believe this PR is unrelated.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10439 from weiqiaomiao/wqm-wip-rgw-zonedump

rgw: add lc_pool when decode or encode struct RGWZoneParams

Reviewed-by: Casey Bodley <cbodley@redhat.com>