rgw: Add zone group admin commands

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: zonegroup json encoding/decoding with backward compatability

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Rename Region to ZoneGroup only internal structures and class

User visuable are not changed: admin commands, jsons and rados objects names

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_admin.cc
src/rgw/rgw_rados.cc

rgw: Add period admin commands and rest api to handle period

the new commands are prepare,delete, get, activate, pull and push

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_admin.cc

rgw: Add RGWPeriod

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Add RGWRealm class and admin commands to handle it

admin realm commands: create, get, delete, rename, set-default and get-default

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: Use switch for raw storage opcodes

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: use string for default realm

Conflicts:
src/rgw/rgw_rados.cc

rgw: Add RGWSystemMetaObj class

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Conflicts:
src/rgw/rgw_rados.cc

rgw: Add RGWNameToId and RGWDefaultSystemMetaObjInfo

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: fix test_rgw_admin_log.cc

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

cmake: add rgw_sync.cc

Signed-off-by: Orit Wasserman <owasserm@redhat.com>

rgw: async ops can generate new stacks for parallel execution

Can now have an op generating another op that will be called at
a different stack. The current op will halt execution until new
op finishes. We can call this multiple times so that multiple ops
will be executed concurrently. This can be useful in the case we
want to generate a single op that will read from multiple objects,
after it had read some meta information. So the root op will read
the meta information and will generate multiple request that will
execute concurrently (albeit on a single thread).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: a few bug fixes related to async operations

fix uninitialized state, trim stacks, etc.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add a generic op to pull data off metadata obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add another state for simple async ops

so that these ops can chain other ops and handle their
response

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more simplification

create a helper method in the ops manager to run a single op, it
builds the needed env

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more simplification, no need to take reference to op

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: some cleanup, keep async env structure

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create simple async op handler

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: async rados requests through a work queue

Tie in the async rados requests to the async framework.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: retrieve a specific sync shard info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: command to show mdlog sync status

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: stackable ops infrastructure

ops can now call other ops, so that we could simplify
the different states each ops need to keep.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial work on sync status marker

Also, rename radosgw-admin mdlog sync to radosgw-admin mdlog-fetch.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: read mdlog marker

Use our local marker as a starting point for fetching the mater mdlog.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clean up mdlog clone states, other fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't reference msg after finish_request()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: hide the completion manager

some more layer separation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: create some abstraction around async ops infrastructure

This will be reused later for other operations.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: mdlog sync error reporting on failed request

Basic error reporting. We'll need to dump this to the backend and
add admin commands to read.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

some error handling work, not ready yet

rgw: md log cloning completely async

hook librados completions into the rgw completion notifier, split the
md log write into two different states.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework metadata log sync to use async capabilities

We now have a window of operations. The next step will be to
asynchronously write to the local mdlog, so that we don't
just hold the work on a synchronous request.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: can now wait on async rest requests

In addition to waiting on a specific request, added a
completion manager that we can wait on so that we get
which request is ready.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add new class to create resource request

RGWRESTReadResource holds the request input and output. Previously
we were using stack variables that couldn't be used in the async
path.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more http aio stuff

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: threaded http manager groundwork

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more work on cleaning up http client interfaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rework http client to handle multiple async requests

Currently we use the curl multi handler using only a single handle. Rework
things a bit to prepare for multiple handlers.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: paginated mdlog clone

still need to iterate by going through over older entries first

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial mdlog clone implementation

naive handling, not paging, unbalanced.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: can list mdlog entries

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move json helper into RGWRESTConn

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

ceph_json: make sure header doesn't clobber _ASSERT_H

_ASSERT_H is needed for dout() compilation

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: decode number of mdlog shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix use of zone name

zone_name doesn't really hold what we need

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw-admin: initial mdlog sync work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initial work on mdlog sync

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't do set_attr on meta objects

only complete object writes (and attributes) is now available. This helps
managing the meta heap and simplifies the states. Since we cannot use rm_attrs
anymore, need to reset meta object before writing it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: store immutable metadata entries in heap pool

Define a new pool for immutable metadata, store new meta objects
in that pool.
(still need to work on the set_attrs case)

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rename default pool names

Name default pools a bit better

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: don't require system pools to start with a period

Make sure cache operations are being done on system objects.
Separate calls where needed to make sure that separate pool/bucket
handling is done correctly. When separating calls, don't use objv_tracker
for data objects, don't use data constructs for system objects.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cmake: add cls_timeindex to rgw targets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7496 from xiexingguo/xxg-wip-fixnvme

osd: BlueStore/NVMEDevice: fix compiling and fd leak

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7349 from yuyuyu101/add-pwritev

osd: FileStore: use pwritev instead of lseek+writev

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7504 from renhwsky/renhw-wip-mon-shutdown

mon: unregister command on shutdown

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #7550 from tchaikov/wip-osd-cleanup

osd: a fix for HeartbeatDispatcher and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7553 from XinzeChi/wip-osdmap-affinity

osd: OSDMap: reset osd_primary_affinity shared_ptr when deepish_copy_from

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7560 from tchaikov/wip-fix-icl

crush/CrushTester: workaround a bug in boost::icl

Merge pull request #7568 from liewegas/wip-bluestore-mutex

osd: bluestore/blockdevice: use std::mutex et al

Merge pull request #7578 from ceph/fix-stripper

Stripper: Fix incorrect push_front -> append_zero change

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Stripper: Fix incorrect push_front -> append_zero change

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7444 from dreamhost/wip-rgw-static-website-robbat2

rgw: approximate AmazonS3 HostId error field.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/jewel'

v10.0.3

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7572 from ceph/wip-cmake-2-8-16

cmake: Added new unittests to make check

Merge pull request #7574 from SUSE/wip-spec-license

ceph.spec.in: add license declaration

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: add license declaration

Signed-off-by: Nathan Cutler <ncutler@suse.com>

cmake: Added new unittests to make check

Added unittest_shunique_lock & unittest_mutex_debug
to make check, and made them link ALLOC_LIBS

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #7569 from SUSE/wip-copyright2

ceph.spec.in: add copyright notice

Reviewed-by: Sage Weil <sage@redhat.com>

ceph.spec.in: add copyright notice

http://tracker.ceph.com/issues/14694 Fixes: #14694

Signed-off-by: Nathan Cutler <ncutler@suse.com>

os/bluestore/BlockDevice: std::atomic_int

Signed-off-by: Sage Weil <sage@redhat.com>

os/bluestore/BlockDevice: use std::mutex and std::condition_variable

Pull aio wait code into an IOContext method too.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7513 from athanatos/wip-ceph-test-rados-2

ceph_test_rados: use less CPU

Merge pull request #5931 from ceph/wip-cxx11concurrency

librados: move to c++11 concurrency types

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge branch 'print-blocked-op' of git://github.com/majianpeng/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6608 from esnyder/wip-6634

osd: remove unused SnapContext member snapc from MOSDSubOp message

Reviewed-by: Xiaoxi Chen <xiaoxi.chen@intel.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'master' of git://github.com/joconcepts/ceph

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7181 from songbaisen/song5

crush: reply quickly from get_immediate_parent

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #7437 from Ved-vampir/wip_zlib_plugin

common: add zlib compression plugin

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7527 from liewegas/wip-no-rgw-prestart

systemd/ceph-radosgw-prestart.sh: remove

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7547 from yuyuyu101/wip-14679

AsyncConnection: avoid debug log in cleanup_handler

crush/CrushTester: workaround a bug in boost::icl

the boost::icl in boost v1.55 has a bug when working with libc++,

> Assertion failed: (this->_map.find(inter_val) == this->_map.end()),
> function gap_insert, file
> /usr/local/include/boost/icl/interval_base_map.hpp, line 555.

see https://svn.boost.org/trac/boost/ticket/9501 and
    https://svn.boost.org/trac/boost/ticket/9987

the bug was fixed in boost v1.56. as a workaround we use the the
`set` from `boost::container` instead of the `std::set` from libc++.
we should drop this workaround once we bump the supported boost
version from 1.42 to 1.56.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7554 from tchaikov/wip-ceph-disk-test

tests: ceph-disk.sh should compare with the resolved path

Reviewed-by: Loic Dachary <ldachary@redhat.com>

OSDMap: reset osd_primary_affinity shared_ptr when deepish_copy_from

Base on current logic, OSDMonitor may call create_pending and
encode_pending twice for the some epoch.

In encode_pending:

tmp.deepish_copy_from(osdmap);
tmp.apply_incremental(pending_inc);

This Op would change the tmp osd_primary_affinity, but the osd_primary_affinity
is declared as ceph::shared_ptr, so this would change the osdmap too. When this
round encode_pending is proposed fail. We may call encode_pending again, but the
osdmap is changed last round, so the pending_inc would be wrong.

Fixes: #14686
Signed-off-by: Xinze Chi <xinze@xsky.com>

Merge pull request #7551 from JevonQ/update-libcephfs-version

libcephfs: update LIBCEPHFS_VERSION to indicate the interface was changed

Reviewed-by: Sage Weil <sage@redhat.com>

tests: ceph-disk.sh should compare with the resolved path

Signed-off-by: Kefu Chai <kchai@redhat.com>

libcephfs: Update LIBCEPHFS_VERSION to indicate the interface was changed

Signed-off-by: Jevon Qiao <scaleqiao@gmail.com>

AsyncMessenger: remove unused variables

* remove AsyncMessenger::listen_sd
* remove `ee` in KqueueDriver::del_event()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crypto: remove unused Digest::sec_type

Signed-off-by: Kefu Chai <kchai@redhat.com>

journal: remove unused ObjectPlayer::m_fetch_interval

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: remove unused C_DiffObject::m_request_num

Signed-off-by: Kefu Chai <kchai@redhat.com>

librbd: replace atomic_t with std::atomic<int>

it fails to compile with clang, and turns out it should be
`ceph::atomic_t`, but I take this chance to replace it
with std::atomic<> to phase out atomic_t.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ECUtil: remove unused stripe_info_t::stripe_size

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: init HeartbeatDispatcher using OSD->cct

its base class 'Dispatcher' is uninitialized when used here to access
'Dispatcher::cct', so we'd better use OSD::cct to initialize it instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7549 from tchaikov/wip-fix-ceph-disk-tests

ceph-disk/test: fix test_prepare.py::TestPrepare tests

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ceph-disk/test: fix test_prepare.py::TestPrepare tests

actually we are verifying if the variable is an instance of
specified class. for example, the `prepare.data` should be
a `PrepareFilestoreData` if `--bluestore` is not specified.

Signed-off-by: Kefu Chai <kchai@redhat.com>

AsyncConnection: avoid debug log in cleanup_handler

local connection will be stop and call cleanup_handler after messenger
is down
introduced in
commit(https://github.com/ceph/ceph/commit/9da2fffd31562ed5d0b795d7862b3ebec66aba40)

Fix #14679

Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7540 from dillaman/wip-librados-test-stub-unwatch

librados_test_stub: protect against notify/unwatch race

Reviewed-by: Josh Durgin <jdurgin@redhat.com>