journal: playback should notify when complete

When not performing a live-playback, the client should be
notified when there are no more records to read.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: add default constructor to Future

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: added additional callbacks for async ops

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: cls_journal: get_immutable_metadata is now async

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_journal_client: get_immutable_metadata is now asynchronous

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_journal: permit empty client id string

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: updated tests for journal trimmer modifications

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_journal: new convenience functions for ObjectSetPosition

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: add support for trimming committed journal objects

As clients periodically mark payloads as committed, the trimmer will
determine the new minimum object set and delete all objects before the
new minimum.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: move rados_ctx_callback to utils namespace

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: initial unit tests for new generic journal library

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/encoding/types.h: add journal::Entry to types

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: new generic journal implementation

The initial use-case for this journal is for librbd and its mirroring
feature. This journal is different from the current MDS journal in that
it organizes journal entries across journal objects w/o striping
individual entries.  It also allows multiple clients to read and write
from a single journal concurrently.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: added support for append op

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: add support for new aio_notify API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new test for librados AIO notify API

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: new AIO version of notify API

Allow watch/notify notifications to be sent asynchronously so
that they can be safely sent from a librados op callback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/encoding/types.h: add cls::journal types

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: new cls_journal_client test cases

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_journal_client: initial implementation of the cls journal client library

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_journal: initial implementation of journal class methods

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5645 from xinxinsh/wip-snap-rename

implement snapshot rename

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #5856 from runsisi/wip-misc-remove-dup-test

librbd: remove duplicate read_only test in librbd::async_flatten

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6339 from xinxinsh/wip-13558

librbd : fix enable objectmap feature issue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6445 from ceph/wip-init-rbdmap

Init rbdmap patchset.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6468 from cxwshawn/wip-mail-fix

modify member info in mailmap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

[mailmap] add member info.
Signed-off-by: Xiaowei Chen <chen.xiaowei@h3c.com>

Merge tag 'v9.2.0'

v9.2.0

librbd : fix enable objectmap feature issue

Fixes: #13558
Signed-off-by: xinxin shu <xinxin.shu@intel.com>

doc/release-notes: edits from Nathan Cutler

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: final infernalis notes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6334 from xinxinsh/wip-13533

rbd : fix clone isssue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #6465 from oritwas/wip-13596

radosgw-admin: metadata list user should return an empty list when user pool is empty

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

radosgw-admin: metadata list user should return an empty list when pool is empty

Fixes: #13596
Signed-off-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #6340 from joaquimrocha/rados_write_benchmark_new_options

Rados write benchmark new options

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test/test_rados_tool.sh: Add tests for the new bench's write options

Those are the combinations of --write-omap, --write-xattr and --write-object.

Signed-off-by: Joaquim Rocha <joaquim.rocha@cern.ch>

tools/rados/rados.cc: Add options to choose the benchmark's write destination

Signed-off-by: Joaquim Rocha <joaquim.rocha@cern.ch>

tools/rados/rados.cc: Write to different destinations

The write benchmark only writes the objects' contents, not the omap
nor the extended attributes, which is something also interesting to
measure depending on the cluster's configuration.

These changes add a way to specify where the contents should be
written, allowing to combine any of the destinations mentioned above.

Signed-off-by: Joaquim Rocha <joaquim.rocha@cern.ch>

Merge pull request #6400 from ceph/wip-12997

rgw: Add default quota config

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5738 from dwj192/dwj192_bkt_index

rgw: fix objects can not be displayed which object name does not cont…

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

9.2.0

Merge pull request #6452 from dachary/wip-13664-test-ec

tests: test/librados/test.cc must create profile

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6410 from Vicente-Cheng/ceph-disk-list-plain-single-failure

ceph-disk: get Nonetype when ceph-disk list with --format plain on single device.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rbd: fix clone issue when we specify image feature

Fixes: #13553
Signed-off-by: xinxin shu <xinxin.shu@intel.com>

tests: test/librados/test.cc must create profile

Now that the create_one_ec_pool function removes the testprofile each
time it is called, it must create the testprofile erasure code profile
again for the test to use.

http://tracker.ceph.com/issues/13664 Refs: #13664

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6446 from dachary/wip-13664-test-ec

tests: destroy testprofile before creating one

Reviewed-by: Samuel Just <sjust@redhat.com>

tests: destroy testprofile before creating one

The testprofile erasure code profile is destroyed before creating a new
one so that it does not fail when another testprofile erasure code
profile already exists with different parameters.

This must be done when creating erasure coded pools with the C++
interface, in the same way it's done with the C interface.

http://tracker.ceph.com/issues/13664 Fixes: #13664

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: add destroy_ec_profile{,_pp} helpers

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6444 from liewegas/wip-pg-key

osd/PG: tolerate missing epoch key

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6406 from xiexingguo/xxg-wip-13626

FileJournal:_fdump wrongly returns if journal is currently unreadable.

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

ceph.spec.in: We no longer need redhat-lsb-core

Drop the redhat-lsb-core dependency as it is no longer necessary on
fedora/rhel.

The other two init scripts do not use redhat-lsb-core either. The
init-ceph.in conditionally requires /lib/lsb/init-functions and does not
use any of the functions defined in that file (at least not directly).
The init-radosgw file includes /etc/rc.d/init.d/functions on non-debian
platforms instead of /lib/lsb/init-functions file so it does not require
redhat-lsb-core either.

Signed-off-by: Boris Ranto <branto@redhat.com>

init-rbdmap: Rewrite to use logger + clean-up

This patch rewrites the init-rbdmap init script so that it uses logger
instead of the log_* functions. The patch also fixes various smaller
bugs like:
* MAP_RV was undefined if mapping already existed
* UMNT_RV and UMAP_RV were almost always empty (if they succeeded) ->
  removed them
* use of continue instead RET_OP in various places (RET_OP was not being
  checked after the switch to logger messages)
* removed use of DESC (used only twice and only one occurrence actually
  made sense)

Signed-off-by: Boris Ranto <branto@redhat.com>

osd/PG: tolerate missing epoch key

An orphan PG may have an info attr but no epoch key.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #5133 from stiopaa1/ThreadMoveCopyConstr

Thread.h: disable copy constructor and assignment op

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6344 from dillaman/wip-13559-infernalis

librbd: potential assertion failure during cache read

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #6329 from dzafman/wip-zafman-ceph

Wip zafman ceph

Reviewed-by: Dan Mick <dmick@redhat.com>

ceph.in: Notify user that 'tell' can't be used in interactive mode

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #6435 from ceph/docs-new-url

updates all http://ceph.com/ to download.ceph.com

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: update rpm links to download.ceph.com

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: update debian links to download.ceph.com

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #5783 from dzafman/wip-12738

Fix scrub error handling and log output
Add set-size and remove-clone-metadata to ceph-objectstore-tool
Fix trim_object() to not crash on corrupt snapset
Add scrub clones test and add clones to ceph-objectstore-tool test

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

test: osd-scrub-snaps.sh uses ceph-helpers.sh and added to make check

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Use boost::optional instead of snap 0 for "all_clones"

Signed-off-by: David Zafman <dzafman@redhat.com>

osd, test: When head missing a snapset, clones not an error

Fix log message for this case and make into "info"

Signed-off-by: David Zafman <dzafman@redhat.com>

osd, test: Keep missing count and log number of missing clones

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Eliminate check for bogus "obj13/head snaps empty" error

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add new remove-clone-metadata object op

Fixes: #12740
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix trim_object() to not crash on corrupt snapset

Add test case to check for trim_object() osd crash

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Improve object spec error handling

Add test cases

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add undocumented clear-snapset command for testing

Add test cases using new feature to corrupt SnapSet

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add set-size command for objects

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Enhanced dump command replaces dump-info

Show object stat information from objectstore
Show SnapSet if present
Add --head for search by object name

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add some clones to ceph-objectstore-tool test

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: For corrupt objectstores, don't abort listing on errors

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Improve some error messages

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: White space fixes

Signed-off-by: David Zafman <dzafman@redhat.com>

tools/rados: Improve xattr import handling so future internal xattrs ignored

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test scrubbing of snapshot problems

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Don't crash if OI_ATTR attribute is missing or corrupt

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Additional _scrub() check for snapset inconsistency

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Better SnapSet scrub checking (find issues instead of asserting)

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Make the _scrub routine produce good output and detect errors properly

Catch decode errors so osd doesn't crash on corrupt OI_ATTR or SS_ATTR
Use boost::optional<> to make current state clearer
Create next_clone as needed using head/curclone
Add equivalent logic after getting to end of scrubmap.objects

Fixes: #12738
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5865 from dzafman/wip-13016

osd: Add config option osd_read_ec_check_for_errors for testing

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6045 from Sandy4999/wip-13133

rgw: refuse to calculate digest when the s3 secret key is empty

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6424 from badone/curl_easy_perform_fix

rgw: Fix typo in RGWHTTPClient::process error message

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6363 from ceph/wip-init-rbdmap

Drop redhat-lsb-core dependency

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: Fix typo in RGWHTTPClient::process error message

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #6358 from dx9/wip-12406-cmake-arch-detection

cmake: Use uname instead of arch.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6310 from dx9/wip-12406-test-libcephfs-flock

test/libcephfs/flock: add sys/file.h include for flock operations

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6377 from xiexingguo/wip-xxg-13597

FileStore: potential memory leak if _fgetattrs fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6414 from dzafman/wip-13625

messages/MOSDOp: Cast in assert to eliminate warnings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6418 from tchaikov/wip-fix-ceph-monstore-update-crush

tools: ceph-monstore-update-crush: add "--test" when testing crushmap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tools: ceph-monstore-update-crush: add "--test" to crushtool

in addition to testing the references to unknown name/type, we should
perform necessary test the crushmap when looking for a good crushmap

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: ceph-disk: coverage list_format_dev_plain() new behavior.

    modify the unittest to coverage the list_format_dev_plain()
    new behavior.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

FileJournal:_fdump wrongly returns if journal is currently unreadable.

_fdump wrongly returns if journal is currently unreadable, and fix some other errors also.
Fixes: #13626
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #5875 from Aran85/parsemetadatakey

rgw:modify the conditional statement in parse_metadata_key method.

Merge pull request #6409 from dachary/wip-mailmap

mailmap: Xie Xingguo affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5879 from H3C/wip-rgw-bugfix1

rgw/rgw_admin: Checking the legality of the parameters

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

messages/MOSDOp: Cast in assert to eliminate warnings

Fixes: #13625
Caused by 0bf2a79e

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5262 from jmunhoz/wip-12352

rgw: remove extra check in RGWGetObj::execute

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>