rgw: make RGWMetadataLog::get_shard_oid public and const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add RGWStatObjCR for async raw_obj_stat

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: only write meta sync shard markers if can_adjust_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: make RGWMetaSyncCR wakeups thread-safe

now that the shards will be coming and going after startup, they need to
be reference counted and protected by a mutex

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw-admin: add read_current_period_id() helper function

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7869 from rzarzynski/wip-rgw-aws4-puthash

rgw: calculate payload hash in RGWPutObj_ObjStore only when necessary.

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #7612 from liewegas/wip-buffer-combined

buffer: raw_combined allocations buffer and ref count together

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #7690 from xiexingguo/xxg-wip-14804

common/TrackedOp: fix inaccurate counting for slow requests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7704 from xiexingguo/xxg-wip-14814

osd: filestore: fix wrong scope of result code for error cases during mkfs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7812 from xiexingguo/xxg-wip-14872

osd: fix wrong return type of find_osd_on_ip()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7815 from branch-predictor/bp-conditional-crc

msg/async: don't calculate msg header crc when not needed

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7711 from xiexingguo/xxg-wip-14817

osd: filestore: fix result code overwritten for clone

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7790 from tchaikov/wip-log-segv

log: segv in a portable way

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #7842 from xiexingguo/xxg-wip-14921

osd: fix overload of '==' operator for pg_stat_t

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7390 from badone/wip-13829

common: Allow OPT_INT settings with negative values

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #7854 from yehudasa/wip-authors-update

AUTHORS: update email

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7846 from dillaman/wip-14551

rbd: update default image features

Conflicts:
PendingReleaseNotes (trivial)

Merge pull request #7841 from dillaman/wip-14550

rbd: deprecate image format 1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7839 from dillaman/wip-14809

librbd: remove last synchronous librados calls from open/close state machine

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7785 from dillaman/wip-14822

librbd: journal replay needs to support re-executing maintenance ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7776 from dillaman/wip-librbd-journal-tweaks

journal: improve commit position tracking

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

AUTHORS: update email

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: calculate payload hash in RGWPutObj_ObjStore only when necessary.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #7767 from athanatos/wip-sam-journal-throttle-4

osd: filestore: restructure journal and op queue throttling

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #7823 from SUSE/wip-14906

packaging: add lsb-release dependency for SUSE and Debian

Reviewed-by: David Disseldorp <ddiss@suse.de>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #7837 from rldleblanc/wpq-documentation

doc: osd-config Add Configuration Options for op queue.

Merge pull request #7851 from jupiturliu/obj-bencher-bug

rados: fix bug for write bench

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7729 from yuyuyu101/wip-spdk2

osd: bluestore: Revert NVMEDevice task cstor and refresh interface changes

packaging: lsb_release build and runtime dependency

The lsb_release executable is being run in multiple places, not least in
src/common/util.cc, which calls it via shell in the collect_sys_info() code
path.

This patch addresses this issue on SUSE- and Debian-derivatives, as well
as reinstating the dependency for RHEL/Fedora after it was dropped in
15600572265bed397fbd80bdd2b7d83a0e9bd918.

http://tracker.ceph.com/issues/14906 Fixes: #14906

Signed-off-by: Nathan Cutler <ncutler@suse.com>

buffer: use alignof for raw_combined allocation arithmetic

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: clean up raw_combined construction

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: size append_buffer so that it fits into page-multiple allocations

We drop some unittest assertions about alloc buffer size.  Sorry!

Signed-off-by: Sage Weil <sage@redhat.com>

rbd-replay: s/CEPH_BUFFER_APPEND_SIZE/CEPH_PAGE_SIZE/

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: alloc right-sized buffer from read_fd

This may as well fit the input; this doesn't relate to the
append buffer.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: use raw_combined for certain allocations

If the alignment is on a page boundary, or the allocation is big,
a separate buffer::raw goes faster.  The rest of the time,
a raw_combined does.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: align unspecified allocations to a word

This will let us put policy create_aligned.

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: combine data and buffer::raw into single allocation

Signed-off-by: Sage Weil <sage@redhat.com>

buffer: add front(), back(), get_num_buffers() methods

These eliminate most callers of buffers(), which exposes the
internal list<ptr>.

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: benchmark some allocations

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: fix append_bench

- fix source
- include larger sizes

Signed-off-by: Sage Weil <sage@redhat.com>

unittest_bufferlist: fix ptr move test

Do not assume there is a trailing null the terminate the string.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7739 from xiexingguo/xxg-wip-14387

osd: fix race condition for heartbeat_need_update

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7703 from xiexingguo/xxg-wip-statfs

osd: filestore: fast abort if statfs encounters ENOENT

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7766 from adamemerson/wip-portable-mutex

common: Do not use non-portable constants in mutex_debug

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #7799 from xiexingguo/xxg-wip-fixnvme

osd: bluestore: NVMEDevice: fix error handling

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7804 from jjhuo/fixes

osd: kstore: fix a race condition in _txc_finish()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7816 from marcan/build-fixes

configure.ac: boost_iostreams is required, not optional

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: use aio watch/unwatch during open/close state machines

Removed the place holders in the open/close async state machines with
the newly added aio watch/unwatch methods.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: added aio_watch/aio_unwatch/aio_watch_flush

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: adjust rbd test case guards to handle new defaults

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: use default RBD features in test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: permit deep-flatten to be dynamically disabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: update default image features

Exclusive lock, object map, fast-diff, and deep-flatten have been
enabled by default for all new images.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: apply orphaned maintenance ops after journal replay

If a client recorded a maintenance op to the journal but crashed
before writing the op finish event, the image will be in an
inconsistent state.  Therefore, once the end of the journal is
reached, attempt to apply all queued ops.

Fixes: #14822
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: only cancel replay of journal op events upon error

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: delay commit of op start event

If the start event is flagged as committed before the op is
actually executed, librbd won't be able to replay the event
should a crash occur.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: permit fire-and-forget aio commit position flush

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support replay of maintenance ops

Ignore errors that are to be expected when replaying
a maintenance op two or more times.

Fixes: #14822
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7858 from jdurgin/wip-rbd-formatted-output-progress

test/cli-integration/rbd: disable progress output

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7745 from sileht/sileht/rados-cython

pybind: move cephfs to Cython

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7831 from yuyuyu101/wip-14912

AsyncMessenger: fix several bugs

Merge pull request #7852 from xiexingguo/xxg-wip-14928

msg: async: start over after failing to bind a port in specified range

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kefu@redhat.com>

msg: async: start over after failing to bind a port in specified range

The original report associated with this fix can be found at
http://tracker.ceph.com/issues/13002. I reopen one in case it
needs to be backported or something alike.

Fixes: #14928
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/cli-integration/rbd: disable progress output

This test is intended to check machine-readable output and usage, not
human-readable output.

Fixes: #14931
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7778 from marcan/pybind-fixes

pybind/rados: fix object lifetime issues and other bugs in aio

Reviewed-by: Mehdi Abaakouk <sileht@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7855 from tchaikov/wip-fix-ftbfs

test/TestPGLog: fix the FTBFS

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test/TestPGLog: fix the FTBFS

the new pure virtual method was introduced by e7edf20.

Fixes: #14930
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7781 from ceph/wip-fix-rbd-cli-tests

rbd/run_cli_tests.sh: Reflect test failures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

fixed a bug for write bench with rados

Signed-off-by: James Liu <james.liu@ssi.samsung.com>

Merge pull request #7826 from SUSE/wip-move-requires

RPM: move runtime dependencies to ceph-base and fix other packaging issues

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #7769 from somnathr/wip-make-shared

Use make_shared while creating shared_ptr

Reviewed-by: Samuel Just <sjust@redhat.com>

test_filejournal: reserve throttle as needed

This is awkward, if there are later users other than this test
and JournalingObjectStore, it'll be worth making it a bit less
clunky and error prone.

Signed-off-by: Samuel Just <sjust@redhat.com>

config_opts: make filestore_queue_max_(ops|bytes) U64

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #7765 from athanatos/wip-lost-unfound-repop

Repop and lost-unfound overhaul

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: deprecate image format 1

The rbd cli will warn about the deprecation when attempting to create
image format 1 images.  librbd will log an error message when opening
a format 1 RBD image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

AsyncConnection: log cleanup, remove redundant state display

Signed-off-by: Haomai Wang <haomai@xsky.com>

Event: complete all pending events

Fix #14925
Signed-off-by: Haomai Wang <haomai@xsky.com>

RPM: drop duplicate /var/lib/ceph/* directories

As of a recent commit, we had /var/lib/ceph/{mds,mon,osd} directories
packaged twice, once with %attr and once without. Drop the latter.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

RPM: drop duplicate udev rules from ceph-base

The udev rules should be packaged in ceph-osd only.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

packaging: rados-classes libraries in ceph-base

Drop duplicate rados-classes libraries from ceph-osd and align the RPM
packaging with Debian by using a wildcard to package everything that
autotools puts in the rados-classes directory.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

AsyncConnection: dispatch write handler when accept done

Fix #12912
Signed-off-by: Haomai Wang <haomai@xsky.com>

osd: fix typo

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: fix overload of '==' operator for pg_stat_t

The newly added fields -- 'last_peered' and 'last_became_peered'
are not compared.

Fixes: #14921
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #7824 from dillaman/wip-aio-watch-notify

librados: fix test failure with new aio watch/unwatch API

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #7832 from trociny/wip-journal-register-async

journal: async methods to (un)register and update client

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7829 from xiexingguo/xxg-wip-doc

doc: fix typo, indention etc.

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix typo, indention etc.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind: update ceph_volume_client for cython

No need to explicitly touch the (no-longer-existing)
load_libcephfs method during module load, as with
the cython version we already get an ImportError
if the C library is unavailable.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: fix error handling on getxattr

The ctypes bindings returned empty string
instead of raising exception.  This was a bug,
because it made it impossible to detect the
difference between missing xattr and empty
xattr.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: move cephfs to Cython

This change moves cephfs binding to Cython.

Closes-bug: #14818
Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

doc: osd-config Add Configuration Options for op queue.

Signed-off-by: Robert LeBlanc <robert.leblanc@endurance.com>

Merge pull request #7747 from vsjulu/master

xio: fix incorrect ip being assigned in case of multiple RDMA ports

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7630 from rzarzynski/wip-rgw-admin-token-cache

rgw: add support for caching of Keystone admin token.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7825 from ceph/wip-rgw-openssl

rgw: link civetweb with openssl (Sage, Marcus Watts)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

packaging: pkg_resources.py runtime dependency

ceph-detect-init uses pkg_resources.py

http://tracker.ceph.com/issues/14864 Fixes: #14864

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: move ceph-disk runtime dependency to ceph-osd

ceph-disk uses sgdisk, which resides in the gdisk and gptfdisk packages on
RHEL and SUSE derivatives, respectively.

ceph-disk is included in the ceph-osd package, so the sgdisk dependency
belongs there.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph.spec.in: drop support for ancient SUSE versions

In Jewel, there is no reason to support anything less than openSUSE 13.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

packaging: move python-flask runtime dependency to ceph-mon, radosgw

Flask is used by MON (in the Ceph REST API) and by RGW (powerdns).
Therefore, it only needs to be in the ceph-mon and ceph-radosgw packages.

Also, this commit encloses the specification of the python-flask runtime
dependency in distro-conditional blocks to account for a minor difference
in the package name between RHEL and SUSE.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

RPM: align runtime dependencies with Debian packaging

The ceph package is now a metapackage, so move all its runtime dependencies
to ceph-base as we already have done in the Debian packaging.

Signed-off-by: Nathan Cutler <ncutler@suse.com>