msg/async/Event: remove event wakeup flag

Now only dispatch external event will wakeup event thread(previously
delete_time_event will call wakeup), we only need to use
"external_num_events" to indicate whether we have extra events.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: debug event address when dispatching and executing

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: support NetworkStack api instead of posix

1. replace sd to ConnectedSocket
2. Replace WorkerPool with Stack
3. Use Stack worker

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/net_handler: let set_socket_options return value

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: add PosixStack support

Add default posix backend support, it should be the full replacement for
the original AsyncMessenger IO logics.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Stack: add abstract Stack

Stack is a network IO framework which encapsulates all necessary basic network
interface, then it manages threads to work.

Different network backend like posix, dpdk even RDMA need to inherit Stack
class to implement necessary interfaces. So it will make ease for other
network backend to integrated into ceph. Otherwise, each backend need to
implement the whole Messenger logics like reconnect, policy handle, session
maintain...

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10535 from dzafman/wip-cot-fix

test: Removeall merged after print_function commit needs a fix

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Removeall merged after print_function commit needs a fix

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #10344 from xiexingguo:xxg-wip-pipe-2016-07-19-02

msg/simple: apply prefetch policy more precisely

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Conflicts:
src/msg/simple/Pipe.cc (removed unneeded cast)

Merge pull request #10532 from linuxbox2/rgw-ldap-recon

rgw: add reinit/rebind logic (ldap)

Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: add reinit/rebind logic (ldap)

Gracefully handle stale LDAP connections by rebinding--verified
with MS AD.  Rebind is attempted once per request until rebound--
not worse than misconfiguring the environment.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #10365 from majianpeng/osd-perfcounter

osd/ReplicatedPG: for sync-read it don't cacl l_osd_op_r_prepare_lat.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10450 from xiexingguo/xxg-wip-cleanup-options

osd: cleanup options and other redundancies

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10471 from xiexingguo/xxg-wip-fix-memdb-assert

kv/MemDB: fix assert triggerred by m_total_bytes underflow

Merge pull request #9679 from xiexingguo/xxg-wip-fix-cephx

auth/cephx: misc fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #9734 from xiexingguo/xxg-wip-cleanup-omap

os/filestore: add sanity checks and cleanups for mount() process

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10231 from yonghengdexin735/wip-zzz-paxosserver-type

mon/PaxosService: make the return value type inconsistent

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10340 from yuyuyu101/wip-16715

msg/async/AsyncConnection: replace Mutex with std::mutex for peformance

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10531 from scuttlemonkey/master

Add UK mirror and update copyright

doc: added new UK Ceph mirror to doc and mirroring

Signed-off-by: Patrick McGarry <pmcgarry@redhat.com>

Merge remote-tracking branch 'origin/master'

Merge pull request #10207 from prallabh/wip-16388

rgw: Fix incorrect content length and range for zero sized objects during range requests

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10510 from yehudasa/wip-16861

test_rgw_bencode: null terminate strings before checking

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9944 from sileht/master

python-rados: extends ReadOp/WriteOp API

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10098 from dzafman/wip-16568

tools: New "removeall" used to remove head with snapshots

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10085 from chengweiv5/fix-doc

doc: ceph-deploy mon add doesn't take multiple nodes

Reviewed-by: Kefu Chai <kchai@redhat.com>

For range requests, let the range be dumped before the error is honored.

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

For zero sized objects, let the range adhere as per the RFC

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

For zero sized objects, return 416 during range requests

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>

Merge pull request #10480 from ceph/lifecycle-cleanup

Clean up lifecycle thread

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge remote-tracking branch 'upstream/pull/10514/head'

* upstream/pull/10514/head:
  src/vstart.sh: stop script aborting when ceph.cfg is not there

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

src/vstart.sh: stop script aborting when ceph.cfg is not there

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #10432 from dillaman/wip-16708

rbd-mirror: potential IO stall when using asok flush request

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10341 from dillaman/wip-16223

rbd-mirror: reduce memory footprint during journal replay

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

python-rados: Implements new write_ops

* rados_write_op_create
* rados_write_op_append
* rados_write_op_write_full
* rados_write_op_write
* rados_write_op_remove
* rados_write_op_truncate
* rados_write_op_zero

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

python-rados: add some read_op/write_op methods

This change adds python API for the following C methods:
* rados_aio_write_op_operate
* rados_write_op_omap_set
* rados_read_op_set_flags
* rados_aio_read_op_operate

Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

Merge pull request #10489 from ceph/wip-cmake

cmake changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

test_rgw_bencode: null terminate strings before checking

Fixes: http://tracker.ceph.com/issues/16861
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: link libcommon against pthread

as Thread.cc lives in libcommon, and global is not using libpthread
directly. so moving libpthread linkage from libglobal to libcommon.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10177 from BlaXpirit/ceph-tests-py3

pybind: Port Python-based tests and remaining Python bindings to Python 3

Reviewed-by: Case Bodley <cbodley@redhat.com>
Reviewed-by: John Spray <jspray@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Clean up lifecycle thread

Make sure that the lifecycle thread is terminated and all memory is
cleaned up on shutdown.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge remote-tracking branch 'upstream/pull/10312/head'

* upstream/pull/10312/head:
  vstart: refactor and fix forced config changes
  vstart: cleanup echoed commands and sh issues

cmake: cleanup Findfuse.cmake

* remove unused stuff, we do not pass/use FUSE_VERSION in cmake.
  the FUSE_VERSION we are using is defined by "fuse_common.h".
* use FindPackageHandleStandardArgs for handling find_package()
  arguments.
* also detect osxfuse on MacOS.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not reference undefined variables

* AIO_LIBS is now named AIO_LIBARIES, and there is not point to print
  out its path
* USE_NSS is not defined if NSS is not checked, so if(USE_NSS) is better
  here.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove duplicated linked lib for libcommon

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: use fullpaths of libraries

better off using the detected path, instead of using the default path
for finding the linked libaries.

see "cmake  --help-policy CMP0065" for more details.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: set policies

this makes sure we are compatible with latest and future cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: remove uuid

as it is replaced by the boost implementation, and not detected in
cmake anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not link ec_jerasure_generic against crush

crush is not used in erasure plugins at this moment.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: exclude private symbols in librbd

missed it in 0edfad0.
* librbd: pass '--exclude-libs,ALL' to linker
* rbd: link against used libraries explicitly, rbd is accessing some
  internal symbols not exposed by librbd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10483 from trociny/fixup-test-rbd-mirror

qa/workunits/rbd: small fixup and improvements for rbd-mirror tests

Reviewed-by: Jason Dillaman <dillaman@hotmail.com>

Merge pull request #10374 from ktdreyer/wip-doc-debug-ceph-fuse

doc: add troubleshooting steps for ceph-fuse

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10332 from dillaman/wip-16689

librbd: optimize away unnecessary object map updates

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

doc: add troubleshooting steps for ceph-fuse

Thanks to John Spray <jspray@redhat.com> on ceph-devel for this
information.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #10479 from batrick/standby-notes

doc: small standby doc edits

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10434 from batrick/i16743

doc: add client config ref

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10409 from weiqiaomiao/wqm-wip-rgw-reboot

rgw:fix rgw boot failed after upgrade to master latest version

Merge pull request #10379 from xiexingguo/xxg-wip-bluestore-2016-07-21

os/bluestore: fix bitmap allocating failure if max_alloc_size is 0

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10140 from ifed01/wip-bluestore-fsck-update

os/bluestore: replace interval_set with a bitset for used_block conta…

Mark's Comments:

This passed Jenkins checks.
This passed "ceph_test_objectstore --gtest_filter=*/2".
This PR did not appear to have a significant impact on performance tests, however no fsck specific performance or memory tests were performed.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

doc: add historical warning for mds_standby_replay

When testing < v10.2.1 with standby replay, I was often confused why the
MDS daemons would go into replay even when this was false.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: fix config separator/title size.

This makes the setting consistent with the others.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10060 from trociny/wip-16386

librbd: discard hangs when 'rbd_skip_partial_discard' is enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10472 from tchaikov/wip-cmake

cmake: exclude non-public symbols in shared libraries

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10273 from yonghengdexin735/wip-zzz-pip-delete-no_casting_required

msg/simple/Pipe: eliminating casts for the comparing of len and recv_max_prefetch

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #10477 from cbodley/wip-16839

radosgw-admin: zone[group] modify can change realm id
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #9988 from ifed01/wip-bluestore-reduce-blob4

os/bluestore: get rid off blob's ref_map for non-shared objects

Mark's Comments:

This passed Jenkins checks.
This PR did not appear to have a significant impact on performance tests.
This initially failed "ceph_test_objectstore --gtest_filter=*/2" segfaulting in one of the tests.  See the attached log in the comments for the PR for details.  Subsequent reruns passed, and a run through valgrind's memcheck also passed.  Igor indicated that he has seen this particular segfault previously and believes it is related to running out of memory.  Given that the PR has passed the object store tests multiple times and the segfault has been observed in the past, I believe this PR is unrelated.

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #10439 from weiqiaomiao/wqm-wip-rgw-zonedump

rgw: add lc_pool when decode or encode struct RGWZoneParams

Reviewed-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: zone[group] modify can change realm id

allows the default zone and zonegroup (created with empty realm_id) to
be later added to a realm. the 'modify' command now accepts either
--realm_id=id or --rgw-realm=name

Fixes: http://tracker.ceph.com/issues/16839
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10476 from yonghengdexin735/wip-zzz-CDir

mds/CDir: remove the part of judgment for _next_dentry_on_set

Reviewed-by: John Spray <john.spray@redhat.com>

mds/CDir: remove the part of judgment for _next_dentry_on_set

Signed-off-by: zhang.zezhu <zhang.zezhu@zte.com.cn>

Merge pull request #10428 from jcsp/wip-16592

mon: validate states transmitted in beacons

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mon: validate states transmitted in beacons

Since FSMap was added, the state of a daemon can lead
to an entirely invalid map, but we were letting daemons
send any state they wanted.

Especially, we must not allow standby daemons to set
any state other than STANDBY.

Fixes: http://tracker.ceph.com/issues/16592
Signed-off-by: John Spray <john.spray@redhat.com>

qa/workunits/rbd: wait for image deleted before checking health

When a primiry image is being deleted, the mirrored image might
temporary be reported in error state, before deletion is propagated.

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

qa/workunits/rbd: small fixup and improvements for rbd-mirror tests

- log to stderr;
- log status if a `wait_for` function failed;
- don't needlessly sleep in `wait_for` functions after the last
  unsuccessful iteration;
- make `wait_for_pool_images` work for image removal case;
- fix `wait_for_pool_images` reset timeout (last_count set).

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cmake: link rgw_a against cls_lock_client

rgw_lc.cc is using it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10429 from jcsp/wip-16807-assertion

mds: add assertion in handle_slave_rename_prep

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

cmake: exclude private symbols in public libs

we should avoid exposing non-public symbols from user facing dynamic
libraries. so pass '--exclude-libs' with appropriate argument to linker.
as libcephfs does not add the ((visibility ("default"))) specifier to
the exported symbols, we can not set the default visiblity to hidden for
libcephfs, and "-export-symbols-regex" is a libtool option, we need to
develop a way for cmake to fix the visibility of libcephfs. it's still a
TODO.

* librados
  - pass '--exclude-libs=ALL' to linker
  - add buffer.cc to librados, so we can use '--exclude-libs=ALL'.
* libcephfs: pass '--exclude-libs=libcommon.a,libclient.a,libosdc.a' to
  linker
* libcommon
  - extract common_buffer_obj from libcommon, to avoid compilation
    this source file repeatly.
* tests:
  - link against common_internal_objs explicitly if the test in
    question is using the internal symbols.
* ceph-client-debug:
  - link against client explicitly, and do not link against librados
    anymore, as it is not used in this tool.

Fixes: http://tracker.ceph.com/issues/16556
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: move common_utf8 closer to where it is used

for example, json_spirit uses common_utf8, so link json_spirit against common_utf8.
because libcommon includes json_spirit, this change does not impact the
consumers of libcommon.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9925 from fullerdj/wip-djf-11171

mds: Add path filtering for dump cache

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10281 from vishalkanaujia/wip_inotable_repair

cephfs: Inotable repair during forward scrub

Reviewed-by: John Spray <john.spray@redhat.com>

kv/MemDB: allow _load() to passed out result to caller

The _load() process can fail. By detecting there error
cases earlier, we can avoid other weird failure cases
later.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test: fix librbd tests for rbd_skip_partial_discard

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

kv/MemDB: fix load/save process of kv pairs

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

librbd: discard hangs when 'rbd_skip_partial_discard' is enabled

Fixes: http://tracker.ceph.com/issues/16386
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

rgw:fix rgw boot failed after upgrade to master latest version

Signed-off-by: weiqiaomiao <wei.qiaomiao@zte.com.cn>

msg/simple/Pipe: eliminating casts for the comparing of len and recv_max_prefetch

Signed-off-by: zhang.zezhu <zhang.zezhu@zte.com.cn>

osd: drop redundant existence check of osd

The is_up() check implicitly coveres the exist() check,
so below here the exist() check is not necessary.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix wrong tip if no-recover flag is unset

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: drop redundant ping of tp thread

Because the lock_suspend_timeout() process of PG does the same thing
for each loop.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

common/config: kill osd_compression_plugins option

It is not used anywhere and removing the unnecessary code
is always the preferred option.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: add client config ref

Fixes: http://tracker.ceph.com/issues/16743
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #10457 from SUSE/wip-15497

doc: add Backporting section to Essentials chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add Backporting section to Essentials chapter

... and fix some broken links.

Fixes: http://tracker.ceph.com/issues/15497
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10414 from SUSE/wip-doc-dev

doc: add Upload logs to archive server section...

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: add Upload logs to archive server section...

...to Testing in the cloud testing chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #10456 from cbodley/wip-16834

rgw: use endpoints from master zone instead of zonegroup
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

rgw: use endpoints from master zone instead of zonegroup

Fixes: http://tracker.ceph.com/issues/16834
Signed-off-by: Casey Bodley <cbodley@redhat.com>

test: Use --head option instead of grep

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: New "removeall" used to remove head with snapshots

Use --force to allow remove to only remove head object
Adding testing to unit test

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Don't compact json so errors are easier to identify

Signed-off-by: David Zafman <dzafman@redhat.com>