rgw: decouple object name from index representation

We can't have a direct mapping between the object name to the way we
index it in the bucket index. Have rgw_obj provide different
representations appropriately and use the proper ones where needed.
Introduce the instance element which will be used to refer to versioned
objects.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: remove unused code

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: add accounted_size for object metadata entry

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: enable s3 get/set versioning ops

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: restful op to set bucket versioning

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: get bucket versioning status op

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: add versioning_enabled field to bucket info

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3342 from ceph/wip-10311

rgw: only keep track for cleanup of rados objects that were written

Reviewed-by: Ray Lv <xiangyulv@gmail.com>

os/FileJournal: Fix journal write fail, align for direct io

when config journal_zero_on_create true, osd mkfs will fail when zeroing journal.
journal open with O_DIRECT, buf should align with blocksize.

Backport: giant, firefly, dumpling
Signed-off-by: Xie Rui <875016668@qq.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mon: encode stashed monmap with all features

latest_monmap that we stash is only used locally--the encoded bl is never shared. Which means we should just use CEPH_FEATURES_ALL all of the time.

Fixes: #5203
Backport: giant, firefly
Signed-off-by: Xie Rui <875016668@qq.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge remote-tracking branch 'gh/next'

PendingReleaseNotes: make a note about librados flag changes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3360 from mattrichards/bump_rados_version

librados: bump rados version number

Reviewed-by: Sage Weil <sage@redhat.com>

0.91

Merge pull request #2697 from ceph/wip-8900

RBD image watcher and new exclusive lock handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3254 from trociny/feature-10036

osd: osd tree to show primary-affinity value

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3281 from ceph/wip-10441-b

osd: fix watch ordering bug 10441 option b

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3290 from ceph/wip-da-SCA-20150102

Coverity and SCA fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3302 from ceph/wip-9956

os/FileStore: verify kernel is new enough before using extsize ioctl

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3305 from majianpeng/fix5

fix bugs about sync_filesystem

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3364 from ceph/wip-quota-test

qa: set -e explicitly in quota test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa: set -e explicitly in quota test

Previously was set in hashbang, which meant
that "./quota.sh" was OK, but "sh ./quota.sh" would
just run through ignoring errors.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3336 from ceph/wip-fs-reset

mon: implement `fs reset`

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3343 from dachary/wip-10505-centos-parted

tests: install parted in centos Dockerfile

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

librbd: flush pending AIO requests under all existing flush scenarios

AIO requests that are waiting on the image lock should be flushed
during all existing RBD flush scenarios.  A few flush cases were
missed in the original implementation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AIO requests should retry lock requests

Added a timer to support retrying AIO lock requests until
they are successful.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: differentiate between R/O vs R/W RBD features

The new RBD exclusive lock feature should be treated as a
feature that is only applied when the image is opened in
R/W mode.

Older clients will need to handle the updated
cls_rbd::get_features method in order to properly determine
the incompatible features for an image depending on the
current mode.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Add internal unit test cases

The new unit tests cover the modifications made to integrate
the internal librbd functionality with the new ImageWatcher.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Add ImageWatcher unit test cases

Directly unit test the new ImageWatcher class to complement
the existing librbd integration tests of exclusive lock
handling.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Add convenience library to support unit tests

Unit tests need access to the private symbols of librbd no
longer exported from librbd.so.  A new librbd_internal
convenience library was created to allow access.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: Allow CLI to optionally create shared images

Images that are flagged as shared cannot use the RBD
object map nor RBD mirroring features.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Integrate librbd with new exclusive lock feature

Operations that update the image now require the exclusive lock
if the feature is enabled.  AIO write and discard operations will
automatically request the exclusive lock from the current leader
to support live-migration.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: bump rados version number

As a follow-on to 49d114f1fff90e5c0f206725a5eb82c0ba329376,
increment the "extra" version field so clients can easily
determine if they have a version of librados that properly
translates C API operation flags.

Signed-off-by: Matthew Richards <mattjrichards@gmail.com>

Merge pull request #3316 from ceph/wip-10471

rgw: index swift keys appropriately

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: Create image exclusive lock watch/notify handler

The new watch/notify handler replaces the existing header
update watch/notify handler and adds support for managing
image exclusive lock leadership.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: enable filestore_extsize by default

Note that this will only get used if the kernel is new enough; if it is
older than 3.5 the option will get disabled and extsize will not be used
even if the option is set to true.

This partially reverts 01cd3cdc726a3e838bce05b355a021778b4e5db1.

Fixes: #9956
Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: verify kernel is new enough before using extsize ioctl

Old kernels have an XFS bug that exposes uninitialized data when the
extsize hint is set and only partially written.  This is fixed by Linux
commit aff3a9edb7080f69f07fe76a8bd089b3dfa4cb5d, documented in XFS bug
http://oss.sgi.com/bugzilla/show_bug.cgi?id=874, and tested by XFS
test xfs/229 to prevent regressions.

Notably the original bug affects kernel 3.2, which is widely deployed with
ubuntu precise 12.04.

Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3352 from kylinstorage/fix-10503

Fix bug 10503: http://tracker.ceph.com/issues/10503

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3203 from majianpeng/fix1

avoid memcopy from librados to caller buffer

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3034 from dachary/wip-10017-erasure-code-repair

erasure code repair when there are two failures

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3148 from mslovy/optimazation_wbthrottle

os: WBThrottle: optimize the map to unordered_map

Reviewed-by: Samuel Just <sjust@redhat.com>

mon/MDSMonitor: add confirm flag to fs reset

This was already in the command map but was not
being checked.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add `fs reset` to cephtool tests

This is just a superficial "I can call it" test,
it's actual behaviour is checked elsewhere.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement `fs reset`

This is for use in CephFS disaster recovery.  When
the metadata pool has been forcibly reset to a single-MDS
metadata tree, we would like to reset the MDSMap to match.

Signed-off-by: John Spray <john.spray@redhat.com>

Fix bug 10503: http://tracker.ceph.com/issues/10503
ceph-fuse: quota code is not 32-bit safe for vxattr output

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>

Merge pull request #2948 from ceph/wip-promote

osd: promote_object separation; proxy read

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_rados: add some debug output

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: improve proxy read cancelation

Avoid taking the PG lock for a canceled read op (if we are lucky).  Recheck
after the lock is taken for good measure.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: put proxy read completion on finisher

We can't use the synchronous completion callbacks (in fast dispatch
context) do to the proxy read completion work.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: tiering: avoid duplicate promotion on proxy read

Do not promote if it is already undergoing in maybe_handle_cache.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: proxy instead of redirect read in writeback mode when the
cache pool is full

To preserve read op order

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: cancel and requeue proxy read when needed

Cancel and requeue proxy read on the following cases:
1) on_shutdown
2) on_change
3) background promotion is done

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

osd/ReplicatedPG: allow reads to proxy etc even if blocked

If we are not write ordered, continue with cache checks so that we can
(among other things) proxy reads while promoting.

Note that this may reorder reads for clients, but we've decided that's okay.

Signed-off-by: Sage Weil <sage@redhat.com>

test: add proxy read test

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: proxy reads during promote

wip 9980. Do proxy read and async promotion for writeback.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: add cache mode READPROXY

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: add proxy read support

wip 9979

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd/ReplicatedPG: separate promotion from the triggering op

Remove the triggering op from the internal promote machinery.

We keep the optional op arg to promote_object() only because we may
block on an object other than the original obc.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: pass promote error to all blocked operations

This isn't the most elegant strategy, but it is the best we can do
right now.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: drop unnecessary cache_mode checks

This currently enumerates all cache modes except none, and we don't
arrive in this function when caching is disabled.  And creating a whiteout
is not cache_mode dependent.  Simplify!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatdPG: adjust braces (no semantic change)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: factor out must_promote case from all cache modes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: factor out common exists case from all cache modes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: make op argument to promote_object optional

For now, we still always pass it.  In preparation, however, we modify
promote_object() so that it will work when op is null.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3309 from trociny/wip-9483

OSD: add a get_latest_osdmap command to the admin socket

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

OSD: add a get_latest_osdmap command to the admin socket

The command blocks and ensures we have the latest map from the
mon. This is useful in testing and to "unstick" clusters in some
odd situations.

Fixes: #9483, #9484 (maybe)
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: Fix PHP librados documentation

Merge pull request #3348 from ceph/wip-mon-wishlist

doc: mon janitorial list is now a wishlist

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: mon janitorial list is now a wishlist

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3327 from ceph/wip-peeringqueue

osd: fix peering queue bug

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3344 from ceph/wip-librbd-snap-unprotect

librbd: shadow variable in snap_unprotect

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: return InvalidAccessKeyId instead of AccessDenied

Fixes: #10334
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 56af795b1046a4c1bfba59e1fefde272bb0e5c1e)

rgw: return SignatureDoesNotMatch instead of AccessDenied

Fixes: #10329
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit ef75d720f289ce2e18c0047380a16b7688864560)

tests: install parted in centos Dockerfile

It is needed to run root ceph-disk tests.

http://tracker.ceph.com/issues/10505 Fixes: #10505

Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc: Clean up pool usage.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

doc: Cleanup RGW pool usage.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3341 from liewegas/wip-10504

client: add ceph version to metadata

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: include ceph and git version in client metadata

Fixes: #10504
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3325 from ceph/wip-nits

allow 'ops' instead of 'dump_ops_in_flight'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3250 from ceph/wip-10372

osdc/Objecter: improve pool deletion detection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert "rgw: switch to new watch/notify API"

This reverts commit dc67cd69604ec4e4df846b818ec739dc7b09a537.

Conflicts:
src/rgw/rgw_rados.cc

Merge pull request #3313 from ceph/wip-asok-get-subtrees

Add MDS "get subtrees" asok command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3312 from ceph/wip-mdscacheobject-const

mds: support constness in MDSCacheObjects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3248 from dachary/wip-table-formatter

add table to Formatter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Added section to install priorities/preferences.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

rgw: only keep track for cleanup of rados objects that were written

Fixes: #10311
We're keeping track of rados objects that we've written so that we could
clean them up if needed. Earlier we weren't too accurate about it and
were also setting the head object that is yet to be written. This now
only applies to the tail data, and a bit clearer.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3324 from mattrichards/rados_translate_op_flag

librados: Translate operation flags from C APIs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3328 from xiaoxichen/memstore_size_u64

Bump memstore_device_bytes from U32 to U64

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3337 from dachary/wip-10494-disable-unittest-msgr

tests: temporarily disable unittest_msgr

tests: temporarily disable unittest_msgr

http://tracker.ceph.com/issues/10494 Refs: #10494

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3331 from dachary/wip-10493-async-port

msg: initialize AsyncConnection::port

Reviewed-by: Haomai Wang<haomaiwang@gmail.com>

mds: add asok command for getting subtreemap

For when we want to inspect this from a test or
during debugging.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: give CDir a dump() method for JSON output

Useful when listing subtrees via admin socket.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: support constness in MDSCacheObjects

So that one can have const CInode and CDir references
from time to time.

Signed-off-by: John Spray <john.spray@redhat.com>

librbd: shadow variable in snap_unprotect and list_children

The shadow variable prevented snap_unprotect from returning the
correct error return code.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: Add Librados PHP documentation

Merge pull request #3332 from dachary/wip-9570-filejournal

journal related cleanups

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

common: Formatter: cosmetic re-indent

Signed-off-by: Andreas Peters <andreas.joachim.peters@cern.ch>

common: Formatter: add TableFormatter class

For more human readable and shell parsable output.

Signed-off-by: Andreas Peters <andreas.joachim.peters@cern.ch>