cmake: Fix mismatched librgw VERSION / SOVERSION

Without this, rpmlint (on openSUSE Tumbleweed) fails with:

librgw2.x86_64: E: shlib-policy-name-error (Badness: 10000) librgw1
Your package contains a single shared library but is not named
after its SONAME.

It seems that the VERSION/SOVERSION mismatch results in the
creation of librgw.so.1 and librgw.so.2.0.0, whereas it should
be librgw.so.2 and librgw.so.2.0.0.

Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #10749 from mikulely/fix-rgw-client-cb

rgw: fix flush_read_list() error msg
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10747 from oliveiradan/master

gcc 6.1.1 complains about missing include: <random>. 4.8.3 does not c…

rgw: fix flush_read_list() error msg

Signed-off-by: Jiaying Ren <jiaying.ren@umcloud.com>

test: gcc 6.1.1 complains about missing include: <random>. 4.8.3 does not complain about it

Signed-off-by: Daniel Oliveira <doliveira@suse.com>

Merge pull request #10264 from yuyuyu101/wip-async-abstract-backend

msg/async: allow other async backend implementations

Reviewed-by: Kefu Chai <kchai@redhat.com>

test_msgr: ensure wait for ms_dispatch executed

Otherwise in slow runner, ms_dispatch may not executed when assert

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: remove global_centers assert

In tests we allow to reset EventCenter instance in the same CephContext,
so it may let global_centers->centers to set the same position multi times.

Signed-off-by: Haomai Wang <haomai@xsky.com>

test_async_networkstack: add networkstack tests

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: fix _conn_prefix racing when stopped

When the connection is lossy and enter fault, it will dispatch reset event.
If cleanup handler is executed as well as ms_handle_reset call mark_down,
it may exists racing for "cs". cleanup handler will reset "cs" but
_conn_prefix in mark_down will access "cs".

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: readd set_priority call

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: allow EventCenter::set_owner reentry

If daemonlize we need to respawn event threads, it need to allow set_owner again

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Stack: register fork watcher

Signed-off-by: Haomai Wang <haomai@xsky.com>

common/ceph_context: add prefork/postfork wathcher support

Because daemon() will termniate all existing threads, it will make something go
wrong.

So we want to add hook at CephContext, do something before/after fork.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Stack: disable smart thread spawn now

New async msgr runtime need to spawn threads when binding, but ceph-osd will
call daemon() after binding port. So we need to respawn threads if forked.

Then thread spawn delay will increase complexity for this change and it's
really a simple strategy which help less, we disable auto spawn now.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/PosixStack: increase log level

AsyncMessenger will try to loop the bind port range, so it will produce
some addr inuse errors which is not abnormal.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: remove event wakeup flag

Now only dispatch external event will wakeup event thread(previously
delete_time_event will call wakeup), we only need to use
"external_num_events" to indicate whether we have extra events.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Event: debug event address when dispatching and executing

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/AsyncConnection: support NetworkStack api instead of posix

1. replace sd to ConnectedSocket
2. Replace WorkerPool with Stack
3. Use Stack worker

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/net_handler: let set_socket_options return value

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async: add PosixStack support

Add default posix backend support, it should be the full replacement for
the original AsyncMessenger IO logics.

Signed-off-by: Haomai Wang <haomai@xsky.com>

msg/async/Stack: add abstract Stack

Stack is a network IO framework which encapsulates all necessary basic network
interface, then it manages threads to work.

Different network backend like posix, dpdk even RDMA need to inherit Stack
class to implement necessary interfaces. So it will make ease for other
network backend to integrated into ceph. Otherwise, each backend need to
implement the whole Messenger logics like reconnect, policy handle, session
maintain...

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10691 from ceph/wip-jlayton-dotdot

Clean up handling of "/.." in ceph client

Merge pull request #10733 from dillaman/wip-16608

rbdmap: specify bash shell interpreter

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10735 from dillaman/wip-17030

librbd: always respond to "release lock" request if lock owner

Reviewed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #10712 from dillaman/wip-16740

librbd: permit disabling journaling if in corrupt state

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10711 from dillaman/wip-16984

librbd: fix possible inconsistent state when disabling mirroring fails

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10708 from dillaman/wip-16967

rbd: bench io-size should not be larger than image size

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10714 from dillaman/wip-16529

qa/workunits: support filtering cls_rbd unit test cases

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9688 from shylesh/wip-16280

ceph-disk: change ownership of initfile to ceph:ceph

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10376 from onyb/wip-rbd-pypi

pybind: Rework rbd/setup.py for PyPI

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: always respond to "release lock" request if lock owner

Fixes: http://tracker.ceph.com/issues/17030
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10709 from cgwalters/doc-use-https-for-repos

doc/install: Use https:// for download.ceph.com

Reviewed-by: Dan Mick <dmick@redhat.com>

doc: added rbdmap man page to RBD restructured index

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbdmap: specify bash shell interpreter

Fixes: http://tracker.ceph.com/issues/16608
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10596 from cbodley/wip-rgw-period-puller

rgw multisite: fixes for period puller

Merge pull request #10330 from rzarzynski/wip-rgw-objdeler-fixes

rgw: multiple fixes for Swift's object expiration
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10619 from chengweiv5/fix-doc

doc: fix rados/configuration/osd-config-ref.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10138 from Yan-waller/yj-wip-rgwrados-0705

rgw: fix the error return variable in log message and cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10729 from SUSE/wip-rnotes-script-regex

script: improve ceph-release-notes regex

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10705 from chengweiv5/doc-image-format

doc: format 2 now is the default image format

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

script: improve ceph-release-notes regex

Tolerate multiple spaces after ':', remove duplicate "fs", add "log" and
"msgr".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: fix possible memory leak when image open fails

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10118 from weiqiaomiao/wqm-wip-bucketcreate

rgw: for the create_bucket api, if the input creation_time is zero, we should set it to 'now"
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10431 from ifed01/wip-bluestore-fix-csum-disable2

os/bluestore: proper handling for csum enable/disable settings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10719 from stiopaa1/mon_pgmonitor_moveC_StatsStr

mon/PGMonitor: move C_Stats struct to cc file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10208 from BlaXpirit/cmake-py3

Include Python 3 bindings into the cmake build and make packages for them

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

client: add a testcase for "/.." handling

Signed-off-by: Jeff Layton <jlayton@redhat.com>

client: don't use special faked-up inode for /..

The CEPH_INO_DOTDOT thing is quite strange. Under most OS (Linux
included), the parent of the root is itself. IOW, at the root, '.' and
'..' refer to the same inode.

Change the ceph client to do the same, as this allows users to get
valid stat info for '..', as well as elimnating some special-casing.

Also in several places, we're checking dn_set.empty as an indicator
of being the root. While that is true for the root, it's also true
for unlinked directories.

This patch has treats them the same. An unlinked directory will
be reparented to itself, effectively acting as a root of its own.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

mon/PGMonitor: move C_Stats struct to cc file

Signed-off-by: Michal Jarzabek <stiopa@gmail.com>

Merge pull request #10685 from liewegas/wip-bluefs-extent

os/bluestore/bluefs_types: fix extent operator<<

Merge pull request #10676 from Yan-waller/yj-wip-rgwdocmultisite

rgw/doc: fix indent

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/workunits: support filtering cls_rbd unit test cases

Fixes: http://tracker.ceph.com/issues/16529
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: permit disabling journaling if in corrupt state

Fixes: http://tracker.ceph.com/issues/16740
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new journal policy to disable initializing the journal

This will be used in the case where the journal is being disabled.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10448 from pritha-srivastava/wip_s3_v2_localauth_rework

rgw: Rework of s3 v2 local authentication code.

verfied:  f23, subset of s3tests

Merge pull request #10449 from ifed01/wip-bluestore-bluestore-perfcnt

os/bluestore: add perf counters for compression effectiveness and space utilization measurements

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10193 from yehudasa/wip-16435

rgw: fix issue 16435
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

librbd: fix possible inconsistent state when disabling mirroring

Fixes: http://tracker.ceph.com/issues/16984
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10706 from xiexingguo/xxg-wip-fix-mondbstore

mon/MonDBStore: fix assert which never fires

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10704 from chhabaramesh/master

os/bluestore/BitAllocator: batch is_allocated bit checks

Reviewed-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

Merge pull request #10689 from liewegas/wip-bluestore-hex-extents

os/bluestore: print bluefs_extents in hex

Reviewed-by: Varada Kari <varada.kari@sandisk.com>

doc/install: Use https:// for download.ceph.com

While the Ceph RPMs etc. are GPG signed, there are various
attacks against signed RPMs without signed/trusted *metadata*
that are elaborated on in https://theupdateframework.github.io/

Since download.ceph.com handles it today, let's make it the default.
(Might want to run this by the server admins though).

Signed-off-by: Colin Walters <walters@verbum.org>

rbd: bench io-size should not be larger than image size

Fixes: http://tracker.ceph.com/issues/16967
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: Rework of s3 v2 local authentication code.

The v2 local auth code has been reworked based on the new new authentication infrastructure.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

doc: format 2 now is the default image format

Signed-off-by: Chengwei Yang <yangchengwei@qiyi.com>

cmake: Move Python detection back outside of pybind

Signed-off-by: Oleh Prypin <oleh@pryp.in>

batch is_allocated bit checks

Signed-off-by: Ramesh Chander <Ramesh.Chander@sandisk.com>

doc: fix command format

Signed-off-by: Chengwei Yang <yangchengwei@qiyi.com>

Merge pull request #10621 from tchaikov/wip-doc

doc: silence sphinx warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

doc: fix typo maximum -> minimal

Signed-off-by: Chengwei Yang <yangchengwei@qiyi.com>

doc: replace cluster name with cluster-name to avoid confusing

Signed-off-by: Chengwei Yang <yangchengwei@qiyi.com>

Merge pull request #10701 from dx9/wip-run-make-cmake-args

run-make-check.sh: pass args to do_cmake.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/MonDBStore: fix assert which never fires

The condition is always true and thus the assert never fires,
which is obvious not our plan.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw/doc: fix typo and indent.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

run-make-check.sh: pass args to do_cmake.sh

Signed-off-by: John Coyle <dx9err@gmail.com>

Lowered third example to keep PG count lower.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #10638 from linuxbox2/rgw-nfs-31

rgw file: remove busy-wait in RGWLibFS::gc()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw file: remove busy-wait in RGWLibFS::gc()

This is a background thread.  However, CPU is wasted.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

os/bluestore: print bluefs_extents in hex

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10670 from ceph/wip-cmake

cmake: cleanup Findgperftools.cmake

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #10632 from ryneli/fix_ceph_buffer_list

librados: use bufferlist instead of buffer::list in public header

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #10633 from dx9/wip-sig

global/signal_handler: use sig_str instead of sys_siglist

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/bluefs_types: fix extent operator<<

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10634 from dx9/wip-bit-allocator

bluestore/BitAllocator: Fix deadlock with musl libc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10675 from chiehtu/fix-16996

doc: fix broken link in SHEC erasure code plugin

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: fix whitespace

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #10215 from ifed01/wip-bluestore-blob-release-fix

os/bluestore: refactor dirty blob tracking along with some related fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10680 from ifed01/wip-bluestore-somefix

os/bluestore: fix improper local var variable in collection_list meth…

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore: fix improper local var variable in collection_list method causing StoreTest.Synthetic to fail.

Signed-off-by: Igor Fedotov <ifed@mirantis.com>

Merge pull request #10493 from Yan-waller/yj-wip-librados-0728

rados/client: fix typo

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10669 from cbodley/wip-rgw-ldap-auth

rgw: fix for s3tests failure when ldap auth is not applied

rpm: Add packages for Python 3 bindings

Signed-off-by: Oleh Prypin <oleh@pryp.in>

debian: Add packages for Python 3 bindings

Signed-off-by: Oleh Prypin <oleh@pryp.in>

pybind: Make ceph_volume_client's syntax compatible with Python 3
(just so it passes the bytecompilation step)

Signed-off-by: Oleh Prypin <oleh@pryp.in>

cmake: Also build Python bindings for Python 3

Signed-off-by: Oleh Prypin <oleh@pryp.in>

rgw: don't overwrite auth result if ldap is not applicable

this was causing the following s3tests failure:

======================================================================
FAIL: s3tests.functional.test_s3.test_list_buckets_invalid_auth
----------------------------------------------------------------------
AssertionError: u'AccessDenied' != 'InvalidAccessKeyId'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10667 from dillaman/wip-16980

rbd-mirror: potential race condition during failure shutdown

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

ceph-disk: change ownership of init file to ceph:ceph

Fixes: http://tracker.ceph.com/issues/16280
Signed-off-by: Shylesh Kumar <shmohan@redhat.com>

os/bluestore: collect 'dirty' blobs on per-transaction basis to avoid onode::blob_map enumeration

Signed-off-by: Igor Fedotov <ifedotov@mirantis.com>