[MON] Fixed calculation of %USED. Now it is shows (space used by all replicas)/(raw space available on OSDs). Before it was (size of pool)/(raw space available on OSDs).

Signed-off-by: Adam Kupczyk akupczyk@mirantis.com

Merge pull request #7854 from yehudasa/wip-authors-update

AUTHORS: update email

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7846 from dillaman/wip-14551

rbd: update default image features

Conflicts:
PendingReleaseNotes (trivial)

Merge pull request #7841 from dillaman/wip-14550

rbd: deprecate image format 1

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7839 from dillaman/wip-14809

librbd: remove last synchronous librados calls from open/close state machine

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7785 from dillaman/wip-14822

librbd: journal replay needs to support re-executing maintenance ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7776 from dillaman/wip-librbd-journal-tweaks

journal: improve commit position tracking

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

AUTHORS: update email

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7767 from athanatos/wip-sam-journal-throttle-4

osd: filestore: restructure journal and op queue throttling

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #7837 from rldleblanc/wpq-documentation

doc: osd-config Add Configuration Options for op queue.

Merge pull request #7851 from jupiturliu/obj-bencher-bug

rados: fix bug for write bench

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7729 from yuyuyu101/wip-spdk2

osd: bluestore: Revert NVMEDevice task cstor and refresh interface changes

Merge pull request #7739 from xiexingguo/xxg-wip-14387

osd: fix race condition for heartbeat_need_update

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7703 from xiexingguo/xxg-wip-statfs

osd: filestore: fast abort if statfs encounters ENOENT

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7766 from adamemerson/wip-portable-mutex

common: Do not use non-portable constants in mutex_debug

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #7799 from xiexingguo/xxg-wip-fixnvme

osd: bluestore: NVMEDevice: fix error handling

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7804 from jjhuo/fixes

osd: kstore: fix a race condition in _txc_finish()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7816 from marcan/build-fixes

configure.ac: boost_iostreams is required, not optional

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: use aio watch/unwatch during open/close state machines

Removed the place holders in the open/close async state machines with
the newly added aio watch/unwatch methods.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: added aio_watch/aio_unwatch/aio_watch_flush

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: adjust rbd test case guards to handle new defaults

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: use default RBD features in test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: permit deep-flatten to be dynamically disabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: update default image features

Exclusive lock, object map, fast-diff, and deep-flatten have been
enabled by default for all new images.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: apply orphaned maintenance ops after journal replay

If a client recorded a maintenance op to the journal but crashed
before writing the op finish event, the image will be in an
inconsistent state.  Therefore, once the end of the journal is
reached, attempt to apply all queued ops.

Fixes: #14822
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: only cancel replay of journal op events upon error

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: delay commit of op start event

If the start event is flagged as committed before the op is
actually executed, librbd won't be able to replay the event
should a crash occur.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: permit fire-and-forget aio commit position flush

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support replay of maintenance ops

Ignore errors that are to be expected when replaying
a maintenance op two or more times.

Fixes: #14822
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7858 from jdurgin/wip-rbd-formatted-output-progress

test/cli-integration/rbd: disable progress output

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #7745 from sileht/sileht/rados-cython

pybind: move cephfs to Cython

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #7831 from yuyuyu101/wip-14912

AsyncMessenger: fix several bugs

Merge pull request #7852 from xiexingguo/xxg-wip-14928

msg: async: start over after failing to bind a port in specified range

Reviewed-by: Haomai Wang <haomai@xsky.com>
Reviewed-by: Kefu Chai <kefu@redhat.com>

msg: async: start over after failing to bind a port in specified range

The original report associated with this fix can be found at
http://tracker.ceph.com/issues/13002. I reopen one in case it
needs to be backported or something alike.

Fixes: #14928
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

test/cli-integration/rbd: disable progress output

This test is intended to check machine-readable output and usage, not
human-readable output.

Fixes: #14931
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7778 from marcan/pybind-fixes

pybind/rados: fix object lifetime issues and other bugs in aio

Reviewed-by: Mehdi Abaakouk <sileht@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7855 from tchaikov/wip-fix-ftbfs

test/TestPGLog: fix the FTBFS

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test/TestPGLog: fix the FTBFS

the new pure virtual method was introduced by e7edf20.

Fixes: #14930
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7781 from ceph/wip-fix-rbd-cli-tests

rbd/run_cli_tests.sh: Reflect test failures

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

fixed a bug for write bench with rados

Signed-off-by: James Liu <james.liu@ssi.samsung.com>

Merge pull request #7769 from somnathr/wip-make-shared

Use make_shared while creating shared_ptr

Reviewed-by: Samuel Just <sjust@redhat.com>

test_filejournal: reserve throttle as needed

This is awkward, if there are later users other than this test
and JournalingObjectStore, it'll be worth making it a bit less
clunky and error prone.

Signed-off-by: Samuel Just <sjust@redhat.com>

config_opts: make filestore_queue_max_(ops|bytes) U64

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #7765 from athanatos/wip-lost-unfound-repop

Repop and lost-unfound overhaul

Reviewed-by: Sage Weil <sage@redhat.com>

rbd: deprecate image format 1

The rbd cli will warn about the deprecation when attempting to create
image format 1 images.  librbd will log an error message when opening
a format 1 RBD image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

AsyncConnection: log cleanup, remove redundant state display

Signed-off-by: Haomai Wang <haomai@xsky.com>

Event: complete all pending events

Fix #14925
Signed-off-by: Haomai Wang <haomai@xsky.com>

AsyncConnection: dispatch write handler when accept done

Fix #12912
Signed-off-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7824 from dillaman/wip-aio-watch-notify

librados: fix test failure with new aio watch/unwatch API

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #7832 from trociny/wip-journal-register-async

journal: async methods to (un)register and update client

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7829 from xiexingguo/xxg-wip-doc

doc: fix typo, indention etc.

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix typo, indention etc.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind: update ceph_volume_client for cython

No need to explicitly touch the (no-longer-existing)
load_libcephfs method during module load, as with
the cython version we already get an ImportError
if the C library is unavailable.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: fix error handling on getxattr

The ctypes bindings returned empty string
instead of raising exception.  This was a bug,
because it made it impossible to detect the
difference between missing xattr and empty
xattr.

Signed-off-by: John Spray <john.spray@redhat.com>

pybind: move cephfs to Cython

This change moves cephfs binding to Cython.

Closes-bug: #14818
Signed-off-by: Mehdi Abaakouk <sileht@redhat.com>

doc: osd-config Add Configuration Options for op queue.

Signed-off-by: Robert LeBlanc <robert.leblanc@endurance.com>

Merge pull request #7747 from vsjulu/master

xio: fix incorrect ip being assigned in case of multiple RDMA ports

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7630 from rzarzynski/wip-rgw-admin-token-cache

rgw: add support for caching of Keystone admin token.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7825 from ceph/wip-rgw-openssl

rgw: link civetweb with openssl (Sage, Marcus Watts)

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test: add tests to (un)register and update journal client

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

Revert "AsyncConnection: Close connection when unregistered connection met WAIT"

This reverts commit 9cc1055eb6e18d55084d5af2b2f93eb8da5ab7da.

Fix #14912
Signed-off-by: Haomai Wang <haomai@xsky.com>

journal: async methods to (un)register and update client

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cls::journal: async client_unregister

Signed-off-by: Mykola Golub <mgolub@mirantis.com>

cmake fix: build civetweb with the right include path.

In an earlier commit, civetweb.c got moved into a separate cmake 'object'
library civetweb_common_objs so that it could be built just once for use
with several different targets.  At the time, there was a separate global
"include_directory" to give the right include path for including "civetweb.h".
A later commit in master created an rgw_a library, and restricted civetweb.h's
include path for only building rgw_a objects.  So, as the product of
these these two commits, the target_include_directories command for the
civetweb include path now needs to be applied to civetweb_common_objs
and not rgw_a.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Merge pull request #7650 from avnerbh/fix1_rand

msg: add thread safety for "random" Messenger + fix wrong usage of random functions

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #7651 from xiexingguo/xxg-wip-fixtypecheck

osd: bluestore: improve fs-type verification and tidy up

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7654 from rldleblanc/wip-wpq-optimization

osd: WeightedPriorityQueue: move to intrusive containers

Merge pull request #7821 from rzarzynski/wip-rgw-14903

rgw: fix wrong handling of limit=0 during listing of Swift account.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: change naming of utility functions in rgw_keystone.cc.

Slightly improve code formatting and const-correctness as well.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Use fixed version of civetweb to avoid warning.

The warning about SSLv23_client_method is because when using dynamic
linking, this should always be non-zero.  That's inside of civetweb,
so this commit is to pull the submodule up to the fix.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Fix ssl link error.

When linking against -lssl, should also include -lcrypto.

// fixup merge with previous

Signed-off-by: Marcus Watts <mwatts@redhat.com>

Use ld.so to link in ssl crypto (cmake changes).

Using dlopen() to load in ssl crypto makes me cringe.
Also it requires either installing the devel package,
or appending library versioning to the library name
civetweb tries to load, which is ugly.  This is only
a license problem for people whose system distribution
does not include openssl.

Signed-off-by: Marcus Watts <mwatts@redhat.com>

rgw: fix wrong handling of limit=0 during listing of Swift account.

Fixes: #14903
Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

Merge pull request #7777 from theanalyst/rgw/insecure-keystone

rgw: allow authentication keystone with self signed certs

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7809 from cbodley/wip-rgw-includes

rgw: use pimpl pattern for RGWPeriodHistory

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7822 from rzarzynski/wip-rgw-keystone-proper-parse-ordering

rgw: try to parse Keystone token in order appropriate to configuration.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #7803 from oritwas/wip-14826

rgw: reset return code in when iterating over the bucket the objects

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

test: remove doubled unwatch API call

This error was unmasked by the fix included in commit 8caa2e4.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: aio watch/unwatch/notify now uses completion objver

A local variable was used for an async callback operation,
resulting in the potential for memory corruption.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

XIO: Changes to fix incorrect ip being assigned in case of multiple RDMA ports.remove rdma local and return error instead to avoid issues with using rdma_local incase of multiple ports.

Signed-off-by: Subramanyam Varanasi <s.varanasi@ssi.samsung.com>

tests: update rbd-mirror tests for updated commit handling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: fix delimiter in ObjectSetPosition formater

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: update JournalTrimmer to support new commit tracking

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: active and minimum set should always grow

Prevent async reload from canceling out an in-flight update.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: update JournalPlayer to support new commit tracking

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: update JournalMetadata to support new commit handling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: track commit position for each object splay offset

It's possible, when delaying appends to the journal, that the
current commit position might be in object set X while future
events for a different offset might be in an object set < X.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: differentiate corruption vs missing entry errors

librbd should treat the corruption of the journal differently from
missing journal entries.  If entries are missing, it might be the
result of a crash and the journal should just be replayed through
the most recent, consistent entry.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: try to parse Keystone token in order appropriate to configuration.

Signed-off-by: Radoslaw Zarzynski <rzarzynski@mirantis.com>

rgw: add insecure option to the http client

This allows the http client to turn off ssl certificate peer checking,
which is turned on by default. This is useful in cases like when
Keystone is SSL terminated with a self signed certificate.
The option `rgw_keystone_verify_ssl` (default true) can be toggled if
self signed certs are used, so that swift and s3 apis using keystone
authentication can work.

Fixes: #14583
Reported-by: Karol Mroz <mroz.karol@gmail.com>
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: use pimpl pattern for RGWPeriodHistory

this removes the dependency on RGWPeriod from rgw_rados.h, which breaks
a circular dependency between rgw_rados.h -> rgw_metadata.h ->
rgw_period_history.h -> rgw_rados.h

Reported-by: Willem Jan Withagen <wjw@digiware.nl>
Signed-off-by: Casey Bodley <cbodley@redhat.com>

doc: final v9.2.1 release notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7694 from Abhishekvrshny/wip-infernalis-release-notes

release-notes: draft v9.2.1 release notes

rbd/run_cli_tests.sh: Reflect test failures

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #7813 from xiexingguo/xxg-wip-doc

doc: small fixes

doc: batch small fixes, including typo, syntax etc.

Signed-off-by: xiexingguo <xie.xingguo@zte.com.cn>

doc: fix typo

Signed-off-by: xiexingguo <xie.xingguo@zte.com.cn>

os/bluestore/NVMEDevice: remove unused variables ref

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/NVMEDevice: fix error handling for try_get()

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

os/bluestore/NVMEDevice: fix error handling for open()

According to Linux man page, the system call read() may return zero to indicate
that end of file reached on success. So below here we need to convert the result
code to a customized one in this case as errno is not set and we actually want
let caller know something is going wrong.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>