cmake: fix rbd compile errors

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Revert "rgw: RGWDataSyncStatusManager::finalize() cleanup"

This reverts commit 365e8334b4d376ba2d57c2d7d904462fdcc35059.

rgw: RGWDataSyncStatusManager::finalize() cleanup

rgw: RGWDataSyncStatusManager::finalize() cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9055 from cbodley/wip-rgw-period-commit-master

rgw: 'period commit' sends to new master zone by default

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8605 from theanalyst/fix/15455

rgw: return 503 for requests when master zone is down

Reviewed-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'period commit' sends to new master zone by default

if no --remote or --url is given, set --remote = master_zone_id to
override the rest_master_conn (which refers to the old master zone)

Fixes: http://tracker.ceph.com/issues/15828
Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: allow --remote to specify zone or zonegroup id

--remote was only looking for zonegroup connections. extended to check
for zone connections as well

updated the usage to clarify that it expects zone/zonegroup ids

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8909 from theanalyst/rgw/zone-eexist-msg

rgw: remove EEXIST error msg for ZoneCreate

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #9035 from tchaikov/wip-cmake-venv

cmake: pass CEPH_BUILD_VIRTUALENV to test as an env variable

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8548 from weiqiaomiao/wq

rgw:remove an unnecessary assignment of the variable 'marker'

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #8301 from tchaikov/wip-fortify

cmake: enable FORTFY flags if supported

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cmake: enable FORTFY flags if supported

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8945 from cbodley/wip-rgw-admin-output

radosgw-admin: added missing json output to some commands

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #8951 from ceph/wip-15745

rgw: handle stripe transition when flushing final pending_data_bl

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: 'period delete' cleans up all period objectsReviewed-by: Yehuda Sadeh

rgw: 'period delete' cleans up all period objects

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9045 from ceph/wip-selinux-noprio

rpm: Drop SELinux priority setting

Reviewed by: Nathan Cutler <ncutler@suse.com>

rpm: Drop SELinux priority setting

Older versions of semodule binary that are in Centos/RHEL do not
support priority setting, dropping it.

Fixes: #15822
Signed-off-by: Boris Ranto <branto@redhat.com>

Merge pull request #8921 from dillaman/wip-15110

rbd-mirror: image sync needs to handle snapshot size and protection status

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Conflicts:
src/test/librbd/test_mock_Journal.cc (moved and edited MockJournaler - trivial)

Merge pull request #8937 from dillaman/wip-15642

librbd: client-side handling for incompatible object map sizes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8958 from dillaman/wip-15750

librbd: constrain size of AioWriteEvent journal entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: potential concurrent event processing during journal replayReviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: potential concurrent event processing during journal replay

Merge pull request #8999 from dillaman/wip-15765

journal: incorrectly computed object offset within set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #9034 from dillaman/wip-rbd-cli-integration-test

test: update rbd integration cram test to remove format 1 warning

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

cmake: pass CEPH_BUILD_VIRTUALENV to test as an env variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9030 from trociny/wip-15771

librbd: fix get/list mirror image status API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test: update rbd integration cram test to remove format 1 warning

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: fixup to get/list mirror image status API

- embed name and rbd_mirror_image_info_t into rbd_mirror_image_status_t;
- index image status list by image id to make 'start' work correctly;
- provide rbd_mirror_image_status_list_cleanup function.

Fixes: #15771
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #8197 from H3C/wip-rgw-bugfixa

rgw:response information is error when geting token of swift account
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #8944 from vshankar/rbd-map-errno

rbd: helpful error message on map failure

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

test: add missing header dependency for MockJournaler

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8918 from ceph/wip-rgw-leak-3

rgw: fix memory leaks

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWDataSyncStatusManager::finalize() cleanup

cleanup following code review

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9012 from cbodley/wip-cmake-freelist

cmake: add missing FreelistManager sources

rgw: RGWHTTPClient requests can unregister themselves early

No need to wait for req_state to complete anymore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #9013 from cbodley/wip-rgw-test-mtime

test/rgw: enable mtime comparison in check_object_eq()

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: move around sync_env.init()

call sync_env.init() even if we're initalized in RGWRemoteDataLog::init()

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rados crs, explicit cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWHTTPManager, avoid referring to req_data->client when completing

this is past calling req_data->finish(), client can be destroyed by now

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: civetweb_callback() refer to store inside lock

pe->store might change when reconfiguring

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWPeriodPusher, stop http manager explicitly

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWHTTPManager, can call stop() more than once

destructor calls stop(), but it can be also called explicitly, if caller
wants to ensure requests are not being handled anymore.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWReadRESTResourceCR, fix refcounting

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWReadRemoteDataLogShardCR fix destructor

was missing destructor, constructor was broken

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: RGWDataSyncStatusManager, cleanup if failing init

init can be called more than once, ended up with leaking some resources

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: rest crs, explicitly call cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: more leaks fixes

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: drop a reference to http op

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8766 from yehudasa/wip-15597

rgw: upgrade default zonegroup and set correct zone[group] id

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: code cleanup

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #8772 from yehudasa/wip-15625

rgw: sync fixes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/rgw: enable mtime comparison in check_object_eq()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8880 from cbodley/wip-rgw-zonegroup-remove

radosgw-admin: add missing 'zonegroup remove'

Reviewed-by: Yehuda Sadeh <yehudasa@gmail.com>

cmake: add missing FreelistManager sources

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #8166 from kmroz/wip-rgw-rest-conn-use-vector

rgw: rest and http client code to use param vectors

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common/ShardedOpWQ: make timeout when calling WaitInterval configurabletest run http://pulpito.ceph.com/yuriw-2016-05-07_09:11:04-rados-wip-yuri-testing---basic-smithi/

common/ShardedOpWQ: make timeout when calling WaitInterval configurable

test run http://pulpito.ceph.com/yuriw-2016-05-07_09:11:04-rados-wip-yuri-testing---basic-smithi/

Merge pull request #8810 from xiexingguo/xxg-wip-fixgotfullmap

osd: fix problematic got_full_map() logic

test run http://pulpito.ceph.com/yuriw-2016-05-07_09:11:04-rados-wip-yuri-testing---basic-smithi/

Merge pull request #8857 from liewegas/wip-15678

osd: handle boot racing with NOUP set

test run http://pulpito.ceph.com/yuriw-2016-05-07_09:11:04-rados-wip-yuri-testing---basic-smithi/

Merge pull request #8881 from liewegas/wip-14952

osd: fix pg stuck messages while PGs are being created

test run http://pulpito.ceph.com/yuriw-2016-05-07_09:11:04-rados-wip-yuri-testing---basic-smithi/

Merge pull request #8798 from majianpeng/buffer

common/buffer: clean up alignment checks

test run http://pulpito.ceph.com/yuriw-2016-05-08_08:12:51-rados-wip-yuri-testing2---basic-smithi/

Merge pull request #8953 from xiexingguo/xxg-wip-prettydump

mon/PGMap: pretty output for "pg dump osds" command

Merge pull request #8986 from stiopaa1/osd_prepublishmap_move

osd/OSD.h: move shared_ptr instead of copying it

Reviewed-by: Adam Emerson <aemerson@redhat.com
Reviewed-by: Sage Weil <sage@redhat.com>

rbd: helpful error message on map failure

Failure to map an rbd image provides pretty much nothing to
debug (or rectify) the nature of failure. This change tries
to improve the same by providing helpful error messages (by
pointing to dmesg, etc..).

Fixes: http://tracker.ceph.com/issues/15721
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #8981 from ceph/wip-rgw-test-fix

test/rgw: fix bucket checkpoint

Reviewed-by: Casey Bodley <cbodley@redhat.com>

journal: incorrectly computed object offset within set

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #8786 from tchaikov/wip-btrfs-sudo

test: sudo to rm btrfs subvol

Reviewed-by: Erwan Velu <erwan@redhat.com>

Merge pull request #8994 from theanalyst/fix/15776

 rgw: log name instead of id for SystemMetaObj on failure

Merge pull request #8919 from stiopaa1/log_moveToPrivateOsdService

osd/OSD.h: change some data members to private

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8989 from flyd1005/wip-fix-python-warnings

cleanup: python: remove warnings of 'trailing whitespace' and 'new blank line at EOF'

Merge pull request #8992 from runsisi/wip-fix-dup-keygen

cls_journal: remove duplicated key generation

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8991 from emenguy/doc_test-reweight-by-utilization

doc: adding test-reweight-by-utilization documentation

Merge pull request #8984 from stiopaa1/osd_removeUnorderedSet

osd/OSD.h: remove unneeded include file

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8097 from aclamk/crushtool-pool-id

crushtool: add ability of precise testing of placement group calculation.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8832 from stiopaa1/log_graylogmove

common/Graylog.cc: use std move to avoid copy

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8920 from XinzeChi/wip-race-shutdown

osd: fix deadlock in OSD::_committed_osd_maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8922 from runsisi/wip-fix-lockdep-assert

lockdep: fix assertion expression if ran out of lock ids

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #8967 from liewegas/wip-15760

osdc/Objecter: upper bound watch_check result

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8357 from liewegas/wip-osd-prestart

osd: update crush_location from ceph-osd on startup

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #8194 from tanghaodong25/fix_4

os/bluestore/KernelDevice.cc: small fix in buffer flush

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #8349 from Aran85/prepare_entry_cleanup

os/filestore: prepare entry cleanup

Reviewed-by: Sage Weil <sage@redhat.com>

osd: create osd id if it does not exist

Most tools create the osd id before trying to start ceph-osd.  Notably,
teuthology does not.  We could fix that, but we would be changing behavior,
as the osd boot will happily create the osd id on the fly for us.  Other
provisioning tools might rely on that behavior.  Instead, just allocate
the id sooner in the process (if necessary).

Signed-off-by: Sage Weil <sage@redhat.com>

osd: change osd_crush_initial_weight = 0 to mean weight to 0

Negative now means auto-weight, 0 means weight to 0.  Change the
default accordingly.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: update crush_location on startup from ceph-osd

Update the crush location from ceph-osd instead of relying on
kludgey bash in ceph-osd-prestart.sh.  Among other things, this
lets us get accurate statfs information from the ObjectStore
implementation instead of relying on 'df'.

Fixes: http://tracker.ceph.com/issues/15213
Signed-off-by: Sage Weil <sage@redhat.com>

osdc/Objecter: use cct's crush_location

Keep the observer so that we refresh our copy of the multimap.

Signed-off-by: Sage Weil <sage@redhat.com>

global: init crush_location on daemon start

Signed-off-by: Sage Weil <sage@redhat.com>

common/ceph_context: add CrushLocation to cct

Signed-off-by: Sage Weil <sage@redhat.com>

crush/CrushLocation: add class to manage crush_location

The crush_location can come from an explicitly set config,
a hook, or a simple fabricated default (root=default host=...).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #8684 from vuhuong/wip-radosaio-copyout-data-xio

librados: copy out data to users' buffer for xio

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #8759 from xiexingguo/xxg-wip-fixobjecterrace

osdc/Objecter: fix race condition for sortbitwise flag detection

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #8826 from liewegas/wip-bluestore-bitmap-freelist

os/bluestore: bitmap-based freelist using merge operator

Reviewed-by: Allen Samuels <allen.samuels@sandisk.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

rgw: drop unnecessary spacing in rgw zg init log

Dropping unneeded space when we're printing the failed reading zg info
message

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: log name instead of id for SystemMetaObj on failure

Currently if we fail to read a SystemMetaObj we try to log the
MetaObject id, however this will not be set mostly as read_id has
failed, so we end up logging an empty id, changing this to log
the object name instead

Fixes: http://tracker.ceph.com/issues/15776
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

adding test-reweight-by-utilization documentation

Signed-off-by: Etienne Menguy <etienne.menguy@corp.ovh.com>

cls_journal: remove duplicated key generation

Signed-off-by: runsisi <runsisi@zte.com.cn>

cleanup: python: remove warnings of 'new blank line at EOF'

When applying ceph patches, some warnings reported, e.g.
qa/workunits/mon/caps.py:367: new blank line at EOF.

Signed-off-by: Li Peng <lip@dtdream.com>

cleanup: python: remove warnings of 'trailing whitespace'

When applying ceph patches, some warnings reported, e.g.
doc/scripts/gen_state_diagram.py:99: trailing whitespace.

Signed-off-by: Li Peng <lip@dtdream.com>

os/filestore/FileJournal: optimize align_bl.

Using is_aligned_size_and_memory replace is_aligned &&
is_n_aligned_size.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/buffer: For bufferlist add func is_aligned_size_and_memory.

For directIO requirement, we need check bufferpt whether size aligned
and the address aligned. To do this, we should call is_aligned &&
is_n_align_sized. Every func also list all ptr of bufferlist.
To reduce one list, we add is_aligned_size_and_memroy(align_size,
align_memory) which only need list once.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

aos/bluestore/KernelDevice: optimize rebuild for aio_write.

For the requirement of directio, the content maybe rebuild.
In fact, rebuild_aligned_size_and_memory first check is_n_align_sized &&
is_algined and if need it rebuild.
So using rebuild_aligned_size_and_memory can remove the check.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common/buffer: Change rebuild_aligned_* & rebuild_page_aligned API.

Make those func return value from void to bool. Using the return value
we can know whether really rebuild content in order to optimize .

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

os/bluestore/BlueFS: remove the duplicate bufferlist.rebuild().

In the later BlockDevice::aio_write(), it will rebuild again if
bufferlist need rebuild.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>