rgw: data sync includes instance in rgw_obj_index_key

this was breaking the versioned object test in test_multi.py, because we
were trying to create delete markers instead of unlinking a specific
object instance

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13722 from dachary/wip-crush-doc

crush: document tunables and rule step set_

Reviewed-by: Sage Weil <sage@redhat.com>

crush: document tunables and rule step set_

Signed-off-by: Loic Dachary <loic@dachary.org>

Merge pull request #13781 from SUSE/wip-19184

build/ops: make package groups comply with openSUSE guidelines

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #13926 from trociny/wip-19260

librbd: fix clang compilation error

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

librbd: make build with clang

Fixes: http://tracker.ceph.com/issues/19260
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #13694 from liupan1111/wip-fix-no-error-map

rbd-nbd: warn when kernel parameters are ignored

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rbd-nbd: only try to load nbd module once when open nbd device, during do_map operation.

Signed-off-by: Pan Liu <liupan1111@gmail.com>

rbd-nbd: warn when kernel parameters are ignored

Fixes: http://tracker.ceph.com/issues/19108
Signed-off-by: Pan Liu <liupan1111@gmail.com>

Merge pull request #13801 from xiexingguo/wip-fix-func-name

osd: fix func name in log produced by handle_pg_peering_evt()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13856 from zhangsw/cleanup-radosgw-admin-tier_type

rgw: remove useless --tier_type in radosgw-admin.

Reviewed-by: Jiaying Ren <mikulely@gmail.com>

Merge pull request #13920 from cbodley/wip-rgw-fix-cleanup-meta-sync

rgw: fix transition from full to incremental meta sync

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #11586 from cbodley/wip-rgw-fetchallmeta-yield-break

rgw: fix break inside of yield in RGWFetchAllMetaCR

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10407 from yonghengdexin735/wip-zzz-perf-counters

common/perf_counters.: Remove unnecessary judgment

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: fix transition from full to incremental meta sync

when transitioning to incremental sync, we need to write temp_marker
instead of sync_marker, because temp_marker has replaced the full sync
marker (which is a metadata key, rather than an log marker) with
next_step_marker

this was breaking the metadata checkpoint in test_multi.py, because it
was comparing metadata keys (i.e. 'user:zone.user') against mdlog
markers

Signed-off-by: Casey Bodley <cbodley@redhat.com>

build/ops: extend rpm spec file copyright period to 2017

Signed-off-by: Nathan Cutler <ncutler@suse.com>

build/ops: make package groups comply with openSUSE guidelines

. . . and put all Group: lines in SUSE conditional blocks.

Fixes: http://tracker.ceph.com/issues/19184
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13737 from dillaman/wip-19104

test: support blacklisting within librados_test_stub

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rbd-mirror: correct race conditions within leader watcher

It was possible for multiple, concurrent acquire attempts to be
running for a single instance due to notification messages. Now
only a single acquire state machine can be running.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: improve debug log messages for managed lock

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: potential shutdown race in TestMockImageReplayer.DecodeError

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: avoid duplicating librados IoCtx objects if not needed

This introduces the potential for shutdown race conditions within
the unit tests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: support blacklisting of connections

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: watch/notify should be connection-aware

Utilize the AIO completion thread of the local connection to best
simulate how librados actually performs when connected a cluster.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librados_test_stub: separate mock cluster from connection

The plan to support independent connections sharing the same
view of the mocked cluster.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13857 from liupan1111/wip-rename-onack-to-oncommit

librados: replace the var name from onack to complete

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11485 from yehudasa/wip-bucket-cleanup

rgw structures rework

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13758 from dillaman/wip-18748

rbd: demote/promote all mirrored images in a pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rgw: fix crash when listing objects via swift

Fixes: http://tracker.ceph.com/issues/19249
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

qa/tasks/radosgw_admin: adjust test to new bucket structure

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: new rest api to retrieve object layout

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get_obj_state() checks for empty oids

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: metadata put of bucket instance sets bucket_id

Need to parse the bucket id off the entry and then set it on the
bucket struct.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw_admin: remove broken check

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

mrgw.sh: fix script

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix async cr operation

Fix crash due to code cleanup. Changes scope of obj ref.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: modify default pools to use namespaces

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: handle pools namespace

Use rgw_pool all around, and replace librados::create_ioctx() with
helper that also sets the namespace.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add namespace to rgw_pool

add a namespace field to the rgw_pool struct

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: compilation and other fixes following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move placement rule out of rgw_bucket

Bucket's placement rule is in the bucket instance's info. Object's
placement rule is in the manifest

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls/version: add more useful logging

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix ECANCELED handling in rgw_get_system_obj()

Fixes: http://tracker.ceph.com/issues/17996
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: keep rgw_obj key info in rgw_obj_key field

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add rgw_obj_index_key, kill RGWObjEnt

Use rgw_obj_index_key to represent entries in bucket index (typedef of
cls_rgw_obj_key). Get rid of RGWObjEnt, it was duplicate of rgw_bucket_dir_entry
anyway.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: clean rgw_obj

Instead of storing the oid and the name, just store the name
and calculate it when needed (same goes to locator). Also give more
coherent names to the various fields.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix bucket overwrite

got broken through the rgw_bucket cleanup related work

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test/rgw: extend obj, manifest unitests

Test rgw_raw_obj and upgrade of old rgw_obj, rgw_bucket and
old manifest.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: implicit rgw_bucket placement, manifest uses rgw_raw_obj

Two main changes here:
1. Newly created rgw_bucket does not have a predetermined placement pools
assigned to it. The placement_id param in the objects themselves points
at where the data is located. This affects object's tail location, head
is located where the bucket instance's placement rule points at.
2. Modify object manifest to use rgw_raw_obj instead of rgw_obj.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: simple manifest compaction

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use rgw_raw_obj in manifest code

This drags in multiple related changes that are needed in order to
support that.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: decode rgw_raw_obj as rgw_obj when it's old object

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unneeded virtual declarations

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more fixes and adjustments following rgw_pool, rgw_raw_obj

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: separate RGWObjState

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: introduce rgw_pool, rgw_raw_obj

Pools are represented by rgw_pool (and not rgw_bucket anymore),
and we use rgw_raw_obj to reference rados objs and all 'system'
objects (vs rgw_obj that is used for rgw objects).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

librados: replace the var name from onact to oncomplete.

Signed-off-by: Pan Liu <pan.liu@istuary.com>

Merge pull request #13901 from Adirl/default

msg/async/rdma: compile with rdma as default

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13760 from zhangsw/fix-rgw-torrent-uri

rgw: get torrent request's parameter is not the same as amazon s3.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13660 from shashalu/rgw-read-sync-status

rgw: use separate http_manager for read_sync_status

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13878 from smithfarm/wip-jewel-release-notes-fixup

doc: mention ENXIO change in the 10.2.6 release notes

Reviewed-by: Sage Weil <sweil@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

msg/async/rdma: compile with rdma as default

Issue: 992583
Issue: 992580

Change-Id: I24128b87294d3083c44c934f7d4bed554dd1f8a4
Signed-off-by: DanielBar-On <danielbo@mellanox.com>

Merge pull request #13896 from optimistyzy/39_1

Bluestore, NVMeDevice: fix the core id for rte_remote_launch

Reviewed-by: Haomai Wang <haomai@xsky.com>

Bluestore, NVMeDevice: fix the core id for rte_remote_launch

Previously, we use the id. The id value will be 1 always since
we will only support one NVMe device per osd. Also we have
coremask conf , the default is 3, and we will use core 1.
It is correct. However if we specify another core mask, e.g.,
0xC, still passing 1 to start the dpdk_thread is wrong. we
need to pass core id = 4.

Since for each shareddata, we only use one cpu core, so
just passing rte_get_next_lcore(-1, 0, 0), which selects
the first slave core.

Signed-off-by: optimistyzy <optimistyzy@gmail.com>

Merge pull request #13891 from yehudasa/wip-vstart-not-new

vstart: don't create cluster by default

rgw: get torrent request's parameter is not the same as amazon s3.

Fixes: http://tracker.ceph.com/issues/19136
Signed-off-by: Zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

Merge pull request #13556 from tangwenjun3/wip-fix-coredump

os/bluestore: fix coredump in register_ctrlr()

Reviewed-by: Haomai Wang <haomai@xsky.com>

rgw: use separate http_manager for read_sync_status

concurrent users of read_sync_status() use different cr managers, when get_resource must
use http_manager related to the cr manager.

Fixes: http://tracker.ceph.com/issues/19236
Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>

Merge pull request #13892 from dmick/wip-19232

qa/suites/upgrade/kraken-x: do not thrash cluster full during upgrade

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/suites/upgrade/kraken-x: do not thrash cluster full during upgrade

Same thing as 39fdc53fe5f33678fbbd00cf8810b6d523d0040c

Fixes: http://tracker.ceph.com/issues/19232
Signed-off-by: Dan Mick <dan.mick@redhat.com>

vstart: don't create cluster by default

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

doc: mention ENXIO change in the 10.2.6 release notes

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #13880 from cbodley/wip-qa-submodule

build: remove ceph-qa-suite directory

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #13217 from liewegas/wip-bluestore-prefer-wal-size

os/bluestore: add bluestore_prefer_wal_size option

Reviewed-by: Igor Fedotov <ifedotov@mirantis.com>

build: remove ceph-qa-suite directory

after commit 3afb889 "qa: add supported distros for ceph-ansible", git
submodule update commands are failing with:

No submodule mapping found in .gitmodules for path 'ceph-qa-suite'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #13785 from liewegas/wip-cleanup-features

msg: no need to pass supported features to Messenger::Policy ctor

Reviewed-by: Haomai Wang <haomai@xsky.com>

doc: update RBD mirroring instructions for pool demotion/promotion

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: new rbd mirror pool promote/demote actions

Fixes: http://tracker.ceph.com/issues/18748
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add aio_mirror_image_* API methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: additional asynchronous mirror state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: asynchronous journal demote state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move group and diff API functions

The group and diff_iterate API functions are already in separate
units, so it's trivial to move them to the new 'api' namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move mirror-related API functions

Also ensure that the API helper methods can be tested via
mock test cases.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13078 from zhangsw/fix-rgw-setacl-bug

rgw: put object's acl can't work well on the latest object

Reviewed-by: Casey Bodley <cbodley@redhat.com>

librbd: rename parent_spec/parent_info type to match code style specs

This also includes a consolidation of SnapInfo, ParentSpec, and ParentInfo
into a single header.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13852 from liewegas/wip-19133

qa/suites: fix upgrade tests vs cluster full thrashing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #13711 from tmuthamizhan/wip-add-supported-distros

qa: add supported distros for ceph-ansible

Reviewed-by: Yuri Weinstein <yweinstei@redhat.com>

Merge pull request #13776 from Liuchang0812/wip-add-override-in-tool-headers

tools: add override in tool submodule

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13791 from runsisi/wip-rbd-ack-cleanup

librbd: rbd ack cleanup

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13775 from Liuchang0812/wip-add-override-librados-headers

librados: add override in headers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13774 from Liuchang0812/wip-add-override-in-common-headers

common: add override in header file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13773 from Liuchang0812/wip-add-override-in-test-headers

test: add override in test submodule

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13771 from Liuchang0812/wip-add-override-in-msg-headers

common: add override in msg subsystem

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #13782 from vshankar/wip-19034

rbd: use concurrent writes for imports

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13752 from trociny/wip-19130

librbd: relax "is parent mirrored" check when enabling mirroring for pool

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #13444 from dachary/wip-18943-device-class

crush: add devices class that rules can use as a filter

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #13835 from theanalyst/doc/releases/10.2.6

doc: Jewel 10.2.6 release notes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #13453 from xiaoxichen/drop_mallinfo

common/MemoryModel: Bump int to long and drop mallinfo

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #13580 from ukernel/wip-18914

client: wait for lastest osdmap when handling set file/dir layout

Reviewed-by: John Spray <john.spray@redhat.com>