librbd: enforce write ordering with snapshot

The md_lock is now held for reading when scheduling write/discards.
Since snap_create now holds the lock for writing and flushes all
pending IO, write/discard operations will now be consistent for a
given request across objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: use separate files for snapshot object maps

Instead of relying on the built-in object snapshot support,
create a separate object map object for each image snapshot.
This will allow a future repair utility to rebuild the object
map for an image's snapshots.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados_test_stub: added IoCtx::write_full implementation

Mapped IoCtx::write_full to existing test method used by the
ObjectWriteOperation::write_full API method. Also added missing
cls_log implementation for debugging.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge branch 'wip-librbd-object-map-crc' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: skip invalidating an invalid object map

If the object map is already flagged as invalid, avoid
attempting to set the invalid object map flag again.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: added CRC validation to object map

Added a footer to the object map which stores a header CRC and
and data CRCs for each 4KB chunk.  Updates to the object map only
require recomputing the CRC to the affected 4KB chunk.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3739 from athanatos/wip-10840

PG,ReplicatedPG: make missing_digests local to scrub_compare_maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3738 from athanatos/wip-10830

osd/: include version_t in extra_reqids with promote

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3737 from athanatos/wip-10768

ReplicatedPG::on_change: requeue in_progress_async_reads close to last

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3694 from ceph/wip-10784

Fix 10784: librbd: image has watchers - not removing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3720 from dachary/wip-10800-shec-experimental

erasure-code: mark the shec plugin as experimental

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: mark the shec plugin as experimental

Reject the creation of an erasure code profile including plugin=shec
unless --enable-experimental-unrecoverable-data-corrupting-features=shec
is set.

http://tracker.ceph.com/issues/10800 Fixes: #10800

Signed-off-by: Loic Dachary <ldachary@redhat.com>

tests: fix unused variable warning

Signed-off-by: Loic Dachary <ldachary@redhat.com>

common: capture check_experimental_feature_enabled message

Implement check_experimental_feature_enabled so that it returns the
message instead of unconditionally displaying it via derr. It allows the
caller to display it in another context.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3717 from ceph/wip-decoding-oldest-snap

mds: fix decoding of InodeStore::oldest_snap

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

PG,ReplicatedPG: make missing_digests local to scrub_compare_maps

We don't want/need to carry this state between scrub stages, so lets
remove the tempation by simply making it a local to scrub_compare_maps.

This caused a bug where we would attempt to write out the
missing_digests from the previous chunk during all future chunks, which
caused trouble.

Fixes: 10840
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'origin/wip-10785' into hammer

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librbd/ImageWatcher.cc

mds: fix decoding of InodeStore::oldest_snap

There is no ENCODE_START/FINISH block when encoding inode that
embedded in dentry. So we can't use encoding version to check
if the buffer contains InodeStore::oldest_snap. Instead, we check
if the buffer iterator reaches end of buffer.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge remote-tracking branch 'gh/master' into hammer

Merge pull request #3522 from ceph/wip-8600

mon: OSDMonitor: tolerate catastrophically bad crush maps

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3703 from vuhuong/wip-xio-hammer

Wip xio hammer

Reviewed-by: Matt Benjamin <matt@cohortfs.com>

osd/: include version_t in extra_reqids with promote

Otherwise, we can't return the correct user version on a dup request.

Note: This patch does not handle compatilibity with the variant which
does not include the version_t field.  Since it's been less than 2 weeks
and we haven't had a release, I think that's ok since handling
compatilibity would require some overhead in the encode/decode
methods.

Fixes: 10830
Introduced (merge): b79c349067ab82b1ccb5d6cdac702f651b27a435
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3664 from liewegas/wip-10695

mon: fix 'nan' from osd df output

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3704 from ceph/wip-10761

librbd: blacklist clients before breaking locks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3513 from rzarzynski/wip-10425

librados: Wip 10425

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3515 from rzarzynski/wip-10458

librados: Wip 10458 - rectify the pool name caching in IoCtxImpl.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3674 from jdurgin/wip-python-leak

rados.py: keep reference to python callbacks

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3676 from majianpeng/misc

Misc

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3678 from tchaikov/fix-pg-leak

osd: fix PG leak in SnapTrimWQ._clear()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge remote-tracking branch 'ceph/hammer'

Merge pull request #3683 from dachary/wip-10845-python-flask

tests: no need for python-flask at build time

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

tests: no need for python-flask at build time

http://tracker.ceph.com/issues/10845 Refs: #10845

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3699 from ceph/wip-early-adopters

doc: Early adopter notes for CephFS

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3617 from ceph/wip-10737

mds: disallow layout changes for files with data

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #3683 from ceph/wip-10791-stopping

MDSMonitor: do not allow MDS to transition from STATE_STOPPING

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #3599 from ceph/wip-traceless-reply

Wip traceless reply

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3700 from ceph/wip-librbd-maint-tests

tests: add simple tests for proxied maintenance operations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3706 from dachary/wip-10839-shec-uninitialized-hammer

erasure-code: fix uninitialized data members on SHEC

Reviewed-by: Sage Weil <sage@redhat.com>

erasure-code: fix uninitialized data members on SHEC

Also fix the tests that verify the result when init is not called and
factorize the two tests that were almost identical for simplicity.

http://tracker.ceph.com/issues/10839 Fixes: #10839

Backport: hammer
Signed-off-by: Loic Dachary <ldachary@redhat.com>

ReplicatedPG::on_change: requeue in_progress_async_reads close to last

Fixes: 10768
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3692 from athanatos/wip-8753

ReplicatedPG::on_change: clean up callbacks_for_degraded_object

Reviewed-by: Sage Weil <sage@redhat.com>

ReplicatedPG::on_change: clean up callbacks_for_degraded_object

Backport: dumpling, firefly, giant
Fixes: 8753
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3643 from athanatos/wip-10734

ObjectStore: fix Transaction encoding version number

Reviewed-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/master' into hammer

Merge pull request #3092 from abartlet/ceph-dmcrypt-plain

Ceph dmcrypt with LUKS (split into 4 patches)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge branch 'hammer' of github.com:ceph/ceph into hammer

mon bits
Reviewed-by: Sage Weil <sage@redhat.com>
cli bits
Reviewed-by: Dan Mick <dmick@redhat.com>

mon/PGMonitor: drop pg ls debug line

Signed-off-by: Sage Weil <sage@redhat.com>

doc: add a page for CephFS early adopters

This is where we should point people who are
trying out CephFS in evaluation environments.

Signed-off-by: John Spray <john.spray@redhat.com>

doc/cephfs: Ceph FS -> CephFS

Signed-off-by: Sage Weil <sage@redhat.com>

doc/cephfs: clarify the 'not production ready' warning

Continue to run into people who see this and run away (occasionally to
less robust solutions) without understanding why.

Signed-off-by: Sage Weil <sage@redhat.com>

build-doc: package checks for fedora

Signed-off-by: John Spray <john.spray@redhat.com>

build-doc: don't ignore errors

It's hard to see what went wrong when e.g. a pip
install fails, if it just steams right past.

Signed-off-by: John Spray <john.spray@redhat.com>

xio: Fix out-of-order responses

Without this clients received out-of-order responses from xio messenger

Signed-off-by: Raju.Kurunkad@sandisk.com

cmake: Fix cmakelist to compile librbd

Signed-off-by: Vu Pham <vu@melanox.com>

osd: Update object state after removing watch from object info

Fixes: #10784
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Simplify handle_watch_timeout() using existing support functions

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #3691 from athanatos/wip-6003

FileJournal: fix journalq population in do_read_entry()

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3690 from athanatos/wip-10817

Wip 10817

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3689 from athanatos/wip-10771

ReplicatedPG::eval_repop: check waiting_for_* even if !m

Reviewed-by: Sage Weil <sage@redhat.com>

Merge branch 'wip-7701' of git://github.com/xinxinsh/ceph into hammer

Conflicts:
src/osd/osd_types.h

Merge branch 'dachary-wip-10728-fedora-junit'

ceph.spec.in: junit always except for EPEL 6

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

ceph.spec.in: junit always except for EPEL 6

The package was renamed a long time ago (around the Fedora 15
timeframe). The "junit4" name is only relevant for EPEL 6. For EPEL 7
and Fedora 20, the "junit" package has "Provides: junit4". And most
recently, in the junit package that ships in Fedora 21 and 22, the
package maintainer dropped the old Provides: line.

http://tracker.ceph.com/issues/10728 Fixes: #10728

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
Signed-off-by: Loic Dachary <ldachary@redhat.com>

MDSMonitor: do not allow MDS to transition from STATE_STOPPING

They can only go into STATE_STOPPED. I was a little concerned
that some of the rejoin code might inadvertently allow a stopping
MDS to end up active, but after a little auditing it looks like
that all behaves properly and this is an invariant we maintain.

Fixes: #10791
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

rgw: Swift API. The second way of specifying desirable response format.

OpenStack Object Storage API v1 defines two ways for the client to tell which response format it understands.
Until now RGW looked at query variable 'format' only. This commit implements the second way of setting the format by
providing 'Accept' HTTP-header with the desirable response mime-type.

Backport: hammer
Fixes: #10746
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>
Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>

Merge pull request #3682 from ceph/wip-10719

Wip 10719

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3680 from ceph/wip-10827

osdc/Objecter: linger_register now acquires rwlock

Reviewed-by: Sage Weil <sage@redhat.com>

rgw: remove multipart entries for bucket index when aborting

Fixes #10719

Since we now let the gc clean the aborter multipart uploads, we need to
clear the multipart entries from the bucket index when cleaning up the
meta object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: encode rgw_obj::orig_obj

We weren't encoding orig_obj, however, it's now needed so that we can
call get_index_key() on decoded objects. Only encode/decode it if ns or
instance are not empty.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3672 from ceph/wip-old-gcc

fix build and warnings on older gcc

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge branch 'wip-diurchenko-master' into hammer

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: Swift API. Support for X-Remove-Container-Meta-{key} header.

Fixes: #10475
Backport: hammer, firefly
Reported-by: Josh Durgin <jdurgin@redhat.com>
Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>

rgw: fix doc, A typo in command line argument name

Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>

rgw: Swift API. Dump container's custom metadata.

Fixes: #10665
Backport: hammer, firefly
Reported-by: Ahmad Faheem <ahmad.faheem@ril.com>
Signed-off-by: Dmytro Iurchenko <diurchenko@mirantis.com>

Merge pull request #3525 from qiushanggao/master

Fix bug: When run Test_filejournal  testcase with gtest argument, all of testcases is failed.

Reviewed-by: Loic Dachary <ldachary@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rados: add 'watch/notify' in rados --help command.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

rados: Using readable format to print object mtime when call 'stat obj'.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3677 from tchaikov/fix-sign-unsigned-comparison-warnings

erasure-code: fix compilation warnings

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Fix bug: When run Test_filejournal testcase with gtest argument, all of testcases is failed.

when run testcase with gtest argument, for example, with argument --gtest_output=xml:/root/reports/ceph_test_cls_version.xml,
then the test result is failed, because of the test program use the first argument as the journal file name.
Signed-off-by: shanggao qiu <qiushanggao@qq.com>

erasure-code: fix compilation warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: fix PG leak in SnapTrimWQ._clear()

Fixes: #10421
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #3673 from tchaikov/cosmetic-init-with-false-no-0

osd/PG.c: use boolean to init `bool` variables

Reviewed-by: Loic Dachary <ldachary@redhat.com>

fsync-tester: print info about PATH and locations of lsof lookup

We're seeing the lsof invocation fail (as not found) in testing and nobody can
identify why. Since attempting to reproduce the issue has not worked, this
patch will gather data from a genuinely in-vitro location.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

test_libcephfs: test we can't set layouts on existing files

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

rados.py: keep reference to python callbacks

If we don't keep a reference to these, the librados aio calls will
segfault since the python-level callbacks will have been garbage
collected. Passing them to aio_create_completion() does not take a
reference to them. Keep a reference in the python Completion object
associated with the request, since they need the same lifetime.

This fixes a regression from 60b019f69aa0e39d276c669698c92fc890599f50.

Fixes: #10775
Backport: dumpling, firefly, giant
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ShardedThreadPool: make wait timeout on empty queue configurable

Fixes: 10818
Backport: giant
Signed-off-by: Samuel Just <sjust@redhat.com>

osd/PG.c: use boolean to init `bool` variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

WorkQueue: make wait timeout on empty queue configurable

Fixes: 10817
Backport: giant, firefly, dumpling
Signed-off-by: Samuel Just <sjust@redhat.com>

add tests for ceph pg ls-by-pool

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add CLI ceph pg ls-by-pool

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add tests for ceph pg ls-by-osd

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add CLI ceph pg ls-by-osd

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

add tests for ceph pg ls-by-primary

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

fix error : ceph pg ls 0

no valid command found; 10 closest matches:
pg ls-by-primary <osdname (id|osd.id)> {<int>} {active|clean|down|replay|splitting|scrubbing|scrubq|degraded|inconsistent|peering|repair|recovery|backfill_wait|incomplete|stale|remapped|deep_scrub|backfill|backfill_toofull|recovery_wait|undersized [active|clean|down|replay|splitting|scrubbing|scrubq|degraded|inconsistent|peering|repair|recovery|backfill_wait|incomplete|stale|remapped|deep_scrub|backfill|backfill_toofull|recovery_wait|undersized...]}
Error EINVAL: invalid command

this is due to partial compare of prefix

Signed-off-by: xinxin shu <xinxin.shu@intel.com>

locally disable pragma warnings on gcc < 4.6

diagnostic push|pop are not present before 4.6, so each use gives a
warning. Temporarily ignore these by disabling/reenabling -Wpragmas.
Note that this means the other disabled warnings are not scoped on
old gcc. I don't think this is worth fixing, since newer gcc will
respect the scope and show us the warnings outside of it.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Pipe: conditionally compile IPTOS setting

This doesn't exist on older platforms.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

librbd: consolidate all async operation flush logic

librbd has three different methods for flushing asynchronous
operations. These have all been consolidated down to a single,
shared method to fix an existing issue and simplify maintenance
going forward.

Fixes: #10783
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

xlist: added missing include

The xlist header references NULL without including
the header which defines it.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ReplicatedPG::eval_repop: check waiting_for_* even if !m

There can be client ops waiting on a promote now.

Fixes: 10771
Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #3602 from dachary/wip-10730-fedora-docker

tests: add Dockerfile for fedora

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3656 from dachary/wip-osd-scrub-repair

tests: run osd-scrub-repair.sh with make check

Reviewed-by: Sebastien Ponce <Sebastien.Ponce@cern.ch>