mds: properly update scrubs_in_progress

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: re-use C_KickOffScrubs context

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: avoid potenial double get CDentry::PIN_SCRUBQUEUE

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: call CDentry::scrub_finished() when skipping special dentry

Also set the on finish context when starting scrub a dentry, this
avoids deleting context for skipped dentry.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: remove dir dentry from scrubstack after finishing scrube dirfrags

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: properly call CDir::scrub_finished()

call CDir::scrub_finished() after finishing scrub a dirfrag

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: skip scrubbing remote linkage

Signed-off-by: Yan, Zheng <zyan@redhat.com>

mds: apply validate_disk_state to dirs too

Signed-off-by: John Spray <john.spray@redhat.com>

mds: tidy up cdir scrub_initialize in scrubstack

...now that I grok how long the dir is meant to be complete

Signed-off-by: John Spray <john.spray@redhat.com>

mds: write scrub tag during validation

Signed-off-by: John Spray <john.spray@redhat.com>

mds: Hook ScrubStack into CInode::validate_disk_state

Signed-off-by: John Spray <john.spray@redhat.com>

mds: implement ScrubHeader

A place for scrubbed dentries to learn about the params
that started the scrub off, such as the tag.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add "tag path" command in MDSRank

Signed-off-by: John Spray <john.spray@redhat.com>

mds: implement enqueue scrub in MDCache

Signed-off-by: John Spray <john.spray@redhat.com>

CMake: update for ScrubStack

Signed-off-by: John Spray <john.spray@redhat.com>

mds: initial ScrubStack implementation

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: CInode: create scrub_info_t and surrounding infrastructure

Like the CDir stuff, this has data structures and helpers to track its
own internal scrub state, and -- in the case of directory inodes -- to
select the next dirfrag you need to handle.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: CDentry: create scrub_info_t and surrounding infrastructure

The CDentry is going to be the point of entry for scrubbing code, and
is how we maintain our cache size. Therefore, it gets a scrub_parent pointer
and a new PIN_SCRUBQUEUE in addition to the basic scrub_info_t struct and
scrub_initialize() and scrub_finish() functions.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: CDir: implement scrub_local() and call it in CInode::validate_disk_state()

Add another set of scrub stamps for the local scrub, which map onto the fnode
local scrub members.  Use it in CInode::validate_disk_state() instead of the
call to check_rstats, because this way we bump the scrub stats, and also
in scrub_initialize() because we know we'll have everything we need in
memory at that time, while we might not later on.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: CDir: Implement scrub_dentries_scrubbing()

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: CDir: create scrub_info_t and surrounding scrub infrastructure

This is set up so that the coming-shortly scrub system can initialize scrubbing
on a directory and perform the actual scrubs, but rely on the directory to
keep track of its internal scrub state and selection of dentries.

Note that initialize_scrub() and scrub_local()  can only be called while the
directory is complete.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: frag_t: add scrub stamp and version for recursive and local scrubs

The local scrub bits are just for the data stored in the dirfrag, like
checking rstats match the actual directory contents and such. The recursive
stats include actual scrubs on all the descendant files and directories too!

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: inode_t: add scrub stamp and version for latest complete scrub

Signed-off-by: Greg Farnum <greg@inktank.com>

MDSContinuation: remove expectation that it's using an MDR

The ValidationContinuation is wrapping an MDR, but other MDSContinuations
might not. Remove that bit of the interface.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mdstypes: dentry_key_t: add an is_valid() function

Wrap up checks for whether the dentry has actually been initialized
to contain any data.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

mds: CDir: rearrange constructor

Use colon initialization for all the members we can, instead of assignment
in the function body. The split is confusing and doesn't seem helpful.

Signed-off-by: Greg Farnum <greg@inktank.com>

Merge pull request #6400 from ceph/wip-12997

rgw: Add default quota config

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #5738 from dwj192/dwj192_bkt_index

rgw: fix objects can not be displayed which object name does not cont…

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6452 from dachary/wip-13664-test-ec

tests: test/librados/test.cc must create profile

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6410 from Vicente-Cheng/ceph-disk-list-plain-single-failure

ceph-disk: get Nonetype when ceph-disk list with --format plain on single device.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: test/librados/test.cc must create profile

Now that the create_one_ec_pool function removes the testprofile each
time it is called, it must create the testprofile erasure code profile
again for the test to use.

http://tracker.ceph.com/issues/13664 Refs: #13664

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6446 from dachary/wip-13664-test-ec

tests: destroy testprofile before creating one

Reviewed-by: Samuel Just <sjust@redhat.com>

tests: destroy testprofile before creating one

The testprofile erasure code profile is destroyed before creating a new
one so that it does not fail when another testprofile erasure code
profile already exists with different parameters.

This must be done when creating erasure coded pools with the C++
interface, in the same way it's done with the C interface.

http://tracker.ceph.com/issues/13664 Fixes: #13664

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: add destroy_ec_profile{,_pp} helpers

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6406 from xiexingguo/xxg-wip-13626

FileJournal:_fdump wrongly returns if journal is currently unreadable.

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #5133 from stiopaa1/ThreadMoveCopyConstr

Thread.h: disable copy constructor and assignment op

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6329 from dzafman/wip-zafman-ceph

Wip zafman ceph

Reviewed-by: Dan Mick <dmick@redhat.com>

ceph.in: Notify user that 'tell' can't be used in interactive mode

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #6435 from ceph/docs-new-url

updates all http://ceph.com/ to download.ceph.com

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

doc: update rpm links to download.ceph.com

Signed-off-by: Alfredo Deza <adeza@redhat.com>

doc: update debian links to download.ceph.com

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #5783 from dzafman/wip-12738

Fix scrub error handling and log output
Add set-size and remove-clone-metadata to ceph-objectstore-tool
Fix trim_object() to not crash on corrupt snapset
Add scrub clones test and add clones to ceph-objectstore-tool test

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

test: osd-scrub-snaps.sh uses ceph-helpers.sh and added to make check

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Use boost::optional instead of snap 0 for "all_clones"

Signed-off-by: David Zafman <dzafman@redhat.com>

osd, test: When head missing a snapset, clones not an error

Fix log message for this case and make into "info"

Signed-off-by: David Zafman <dzafman@redhat.com>

osd, test: Keep missing count and log number of missing clones

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Eliminate check for bogus "obj13/head snaps empty" error

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add new remove-clone-metadata object op

Fixes: #12740
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Fix trim_object() to not crash on corrupt snapset

Add test case to check for trim_object() osd crash

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Improve object spec error handling

Add test cases

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add undocumented clear-snapset command for testing

Add test cases using new feature to corrupt SnapSet

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Add set-size command for objects

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Enhanced dump command replaces dump-info

Show object stat information from objectstore
Show SnapSet if present
Add --head for search by object name

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Add some clones to ceph-objectstore-tool test

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: For corrupt objectstores, don't abort listing on errors

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: Improve some error messages

Signed-off-by: David Zafman <dzafman@redhat.com>

ceph-objectstore-tool: White space fixes

Signed-off-by: David Zafman <dzafman@redhat.com>

tools/rados: Improve xattr import handling so future internal xattrs ignored

Signed-off-by: David Zafman <dzafman@redhat.com>

test: Test scrubbing of snapshot problems

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Don't crash if OI_ATTR attribute is missing or corrupt

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Additional _scrub() check for snapset inconsistency

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Better SnapSet scrub checking (find issues instead of asserting)

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Make the _scrub routine produce good output and detect errors properly

Catch decode errors so osd doesn't crash on corrupt OI_ATTR or SS_ATTR
Use boost::optional<> to make current state clearer
Create next_clone as needed using head/curclone
Add equivalent logic after getting to end of scrubmap.objects

Fixes: #12738
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5865 from dzafman/wip-13016

osd: Add config option osd_read_ec_check_for_errors for testing

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #6045 from Sandy4999/wip-13133

rgw: refuse to calculate digest when the s3 secret key is empty

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6424 from badone/curl_easy_perform_fix

rgw: Fix typo in RGWHTTPClient::process error message

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Fix typo in RGWHTTPClient::process error message

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #6358 from dx9/wip-12406-cmake-arch-detection

cmake: Use uname instead of arch.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6310 from dx9/wip-12406-test-libcephfs-flock

test/libcephfs/flock: add sys/file.h include for flock operations

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #6377 from xiexingguo/wip-xxg-13597

FileStore: potential memory leak if _fgetattrs fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #6414 from dzafman/wip-13625

messages/MOSDOp: Cast in assert to eliminate warnings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #6418 from tchaikov/wip-fix-ceph-monstore-update-crush

tools: ceph-monstore-update-crush: add "--test" when testing crushmap

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tools: ceph-monstore-update-crush: add "--test" to crushtool

in addition to testing the references to unknown name/type, we should
perform necessary test the crushmap when looking for a good crushmap

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: ceph-disk: coverage list_format_dev_plain() new behavior.

    modify the unittest to coverage the list_format_dev_plain()
    new behavior.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

FileJournal:_fdump wrongly returns if journal is currently unreadable.

_fdump wrongly returns if journal is currently unreadable, and fix some other errors also.
Fixes: #13626
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #5875 from Aran85/parsemetadatakey

rgw:modify the conditional statement in parse_metadata_key method.

Merge pull request #6409 from dachary/wip-mailmap

mailmap: Xie Xingguo affiliation

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #5879 from H3C/wip-rgw-bugfix1

rgw/rgw_admin: Checking the legality of the parameters

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

messages/MOSDOp: Cast in assert to eliminate warnings

Fixes: #13625
Caused by 0bf2a79e

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #5262 from jmunhoz/wip-12352

rgw: remove extra check in RGWGetObj::execute

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Add default quota config

Add a default quota config that is applied when no specific quota is
enabled.  There are both per-bucket and per-user defaults.  Defaults are
enabled if either max count or max size is enabled.

This fixes issues 12912 and 12997.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

ceph-disk: get Nonetype when ceph-disk list with --format plain on single device.

    with dmcrypt device, ceph-disk list will get Nonetype with the real device
    i.e. /dev/sdb1 for /dev/dm-1, `ceph-disk --list /dev/sdb1` will get Nonetype

    that because the holder we query is `dm-x`, the devices includ the path
    is `sdX`, it cannot match.

    if we list format plain and meet dmcrypt device, query again with the
    holder (dm-x).

    Also change the list_devices() parameter to let other function to use it
    easily.

Signed-off-by: Vicente Cheng <freeze.bilsted@gmail.com>

mailmap: Xie Xingguo affiliation

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6320 from JiYou/mything

Fix examples/librados/Makefile error.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6381 from xiexingguo/xxg-wip-fs-cleanup

FileStore: remove unused local variable 'handle'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6384 from ktdreyer/wip-doc-download-gpg-key

doc: download GPG key from download.ceph.com

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6246 from liewegas/wip-13477

crush: validate bucket id before indexing buckets array

Reviewed-by: Loic Dachary <ldachary@redhat.com>

crush/mapper: ensure take bucket value is valid

Ensure that the take argument is a valid bucket ID before indexing the
buckets array.

Signed-off-by: Sage Weil <sage@redhat.com>

crush/mapper: ensure bucket id is valid before indexing buckets array

We were indexing the buckets array without verifying the index was within
the [0,max_buckets) range.  This could happen because a multistep rule
does not have enough buckets and has CRUSH_ITEM_NONE
for an intermediate result, which would feed in CRUSH_ITEM_NONE and
make us crash.

Fixes: #13477
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #6239 from zhouyuan/swift_testing_access

vstart.sh: grant full access to Swift testing account

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6303 from dx9/wip-12406-use-freopen

test_rgw_admin:  Use freopen for output redirection.

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #6387 from dachary/wip-test-ports

tests: port uniqueness reminder

Reviewed-by: Vicente Cheng <freeze.bilsted@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

tests: port uniqueness reminder

When copy/pasting a tests, it is easy to forget (or not know) that the
port used must be unique to allow for multiple tests to run in
parallel (make -j8). Add a reminder next to each port.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6379 from tchaikov/wip-13554

rocksdb: remove rdb source files from dist tarball

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #6258 from swamireddy/wip-v910-mailmap

mailmap updates

Reviewed-by: Loic Dachary <ldachary@redhat.com>

FileStore: remove unused local variable 'handle'

Remove unused local variable 'handle' in FileStore::collection_empty
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

FileStore: potential memory leak if _fgetattrs fails

Memory leak happens if _fgetattrs encounters some error and simply returns.
Fixes: #13597
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

doc: download GPG key from download.ceph.com

Fixes: #13603
Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

doc/release-notes: v0.94.5

Signed-off-by: Sage Weil <sage@redhat.com>

release-notes: draft v0.94.5 release notes

Signed-off-by: Loic Dachary <ldachary@redhat.com>