rgw: finalize perfcounters after shutting down storage

Fixes: #10572
Backport: giant, firefly
First disable the storage subsystem, then disable perfcounters.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3504 from ceph/wip-10553

rgw: fix partial GET in swift

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd image_read.sh: disable exclusive locking

Until the kernel supports RBD exclusive locking, this test
has been updated to create shared images (exclusive locking
disabled).

Fixes: #10613
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rbd: ensure aio_write buffer isn't invalidated during image import

The buffer provided to aio_write shouldn't be invalidated until
after aio_write has indicated that the operation has completed.

Fixes: #10590
Backport: giant
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3426 from jdurgin/wip-10592

qa: disable automatic locking for manual locking test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #3427 from jdurgin/wip-cram

test: fix rbd cli tests for new feature bit

Reviewed-by: Sage Weil <sage@redhat.com>

test: fix rbd cli tests for new feature bit

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

qa: disable automatic locking for manual locking test

Automatic locking hides the ESHUTDOWN from the caller, which is how
this test detects that blacklisting works.

Fixes: #10592
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Revert "Merge remote-tracking branch 'origin/wip-bi-sharding-3' into next"

This reverts commit f79d8f24e9c0bf0d0b37270eba2745a878f2caed, reversing
changes made to 896c8899ac28eb0403bfaa20454f3756f3705c51.

Merge remote-tracking branch 'origin/wip-bi-sharding-3' into next

Merge remote-tracking branch 'origin/wip-10271' into next

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: bilog marker related fixes

Fix the way we parse the marker. Instead of specifying whether it's a
sharded or not sharded bucket, we pass a shard_id. If string itself
points to a singe shard, we'll use the passed shard_id, otherwise we'll
parse the string and determine the shard id by that. In this way when
referencing a single shard we can get the marker with either shard id
specified or not. This works with the non-shard case too.
Adjust the bilog listing function, set it to work with the new
interface. It was broken before, and there are multiple fixes to it.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #3342 from ceph/wip-10311

rgw: only keep track for cleanup of rados objects that were written

Reviewed-by: Ray Lv <xiangyulv@gmail.com>

rgw: fix partial GET in swift

Fixes: #10553
backport: firefly, giant

Don't set the ret code to reflect partial download, just set the
response status when needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

os/FileJournal: Fix journal write fail, align for direct io

when config journal_zero_on_create true, osd mkfs will fail when zeroing journal.
journal open with O_DIRECT, buf should align with blocksize.

Backport: giant, firefly, dumpling
Signed-off-by: Xie Rui <875016668@qq.com>
Reviewed-by: Sage Weil <sage@redhat.com>

mon: encode stashed monmap with all features

latest_monmap that we stash is only used locally--the encoded bl is never shared. Which means we should just use CEPH_FEATURES_ALL all of the time.

Fixes: #5203
Backport: giant, firefly
Signed-off-by: Xie Rui <875016668@qq.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

rgw: wait for completion only if not completion available

In a bucket aio operation, wait for completions only if there are no
completions available. Otherwise we might wait forever, as everything
already complete.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/next'

rgw: bi list, update marker only if result not empty

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix memory leak

We were iterating on both completion_objs, and completions assuming that
they follow each other. They don't do it. While at it, index completions
by id, so that we could update the completed objs correctly.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: initialize RGWBucketInfo::num_shards

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: call ioctx->aio_operate() under lock

close a race window

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: fix linkage following rebase

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: update calls to handle bucket sharding

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: only keep track for cleanup of rados objects that were written

Fixes: #10311
We're keeping track of rados objects that we've written so that we could
clean them up if needed. Earlier we weren't too accurate about it and
were also setting the head object that is yet to be written. This now
only applies to the tail data, and a bit clearer.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

test: fix test_cls_rgw

Adjust to new api calls.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: remove incorrect function declaration

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: max shards configuration is part of the zone config

The zone config params are set in the region configuration. Also,
there's a ceph.conf configurable (rgw_override_bucket_index_max_shards)
for overriding this per rgw.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: pass num shards on bucket initialization

Need to pass the actual num shards that are going to be used for this
specific bucket. Bucket may be created by applying metadata from
different zone, so num shards might be different.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: write multi shard markers on replica_log appropriately

When getting a list of shard_id#marker, iterate through the shards and
write each as needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: extend shards marker api

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw, cls_rgw: keep shard ids with oids

Instead of just having the list of oids, keep the shard ids together, so
that we can know on which shard the operation happened.
Bucket markers are just using the shard numeric id, instead of the
bucket instance shard id. This makes it easier to parse the markers
appropriately.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: clean up CLSRGWConcurrentIO

Class is no longer a template, and keeps a map of oids by shard_id. Call
issue_op() using both shard_id and oids. Shard id is used for mapping
the results in the derived classes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: modify bucket instance shard marker ids

Instead of having the markers prefixed by the oids, use the bucket
instance id.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket replica log, handle shard ids

bucket replica log can now save entries by the specified shard id

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw: list bi log should not return marker entry

The marker should be served as a lower bound, but should not be
returned.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bucket_index_shard_hash_type fixes

add initializations, json encode /decode.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: decode the req_state bucket instance id if needed

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: improve bucket sharding hashing

Amplify small source changes.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: data changes log, log info by bucket shard id

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: use new BucketShard structure for index manipulation calls

Instead of recalculating the hash every call, do it once, and pass this
structure around. Also, will be used for logging changes into the data
log.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: bi log list/trim can get specific bucket shard

bucket shard can be specified on the bucket instance param. It can be
added like this: <bucket-instance>[:shard-id]

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Fix the multipart uploads functional test failures due to bucket index sharding.

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

Fix get_bucket_instance_info, only build the oid if it is empty.

Signed-off-by: Guang Yang <yguang@yahoo-inc.com>

Adjust bi log trim implementation to work with multiple bucket shards.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Adjust bi log listing to work with multiple bucket shards.
Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

cls_rgw, rgw: switch different ops to new concurrent infrastructure

Make all the relevant ops use the CLSRGWConcurrentIO infrastructure,
which simplifies things.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: generalize container type for concurrent IO base class

Turned the ConcurrentIO class a template, so that we could use different
kind of containers that are needed for the different operations.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

cls_rgw, rgw: create base class for common bucket shard operations

Instead of copy pasting the same code all over again, create a base
class for the needed concurrent IO operations.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

Adjust bucket stats/index checking/index rebuild/tag timeout implementation to work with multiple shards.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Adjust bucket listing to work with multiple shards.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Adjust rgw bucket prepare/complete OP to work with multiple bucket index shards.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Implement sharding for bucket creation.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

Add a new field to bucket info indicating the number of shards of this bucket and make it configurable.

Signed-off-by: Guang Yang (yguang@yahoo-inc.com)

PendingReleaseNotes: make a note about librados flag changes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3360 from mattrichards/bump_rados_version

librados: bump rados version number

Reviewed-by: Sage Weil <sage@redhat.com>

0.91

Merge pull request #2697 from ceph/wip-8900

RBD image watcher and new exclusive lock handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3254 from trociny/feature-10036

osd: osd tree to show primary-affinity value

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3281 from ceph/wip-10441-b

osd: fix watch ordering bug 10441 option b

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3290 from ceph/wip-da-SCA-20150102

Coverity and SCA fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3302 from ceph/wip-9956

os/FileStore: verify kernel is new enough before using extsize ioctl

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3305 from majianpeng/fix5

fix bugs about sync_filesystem

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3364 from ceph/wip-quota-test

qa: set -e explicitly in quota test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa: set -e explicitly in quota test

Previously was set in hashbang, which meant
that "./quota.sh" was OK, but "sh ./quota.sh" would
just run through ignoring errors.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3336 from ceph/wip-fs-reset

mon: implement `fs reset`

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3343 from dachary/wip-10505-centos-parted

tests: install parted in centos Dockerfile

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

librbd: flush pending AIO requests under all existing flush scenarios

AIO requests that are waiting on the image lock should be flushed
during all existing RBD flush scenarios.  A few flush cases were
missed in the original implementation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AIO requests should retry lock requests

Added a timer to support retrying AIO lock requests until
they are successful.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: differentiate between R/O vs R/W RBD features

The new RBD exclusive lock feature should be treated as a
feature that is only applied when the image is opened in
R/W mode.

Older clients will need to handle the updated
cls_rbd::get_features method in order to properly determine
the incompatible features for an image depending on the
current mode.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Add internal unit test cases

The new unit tests cover the modifications made to integrate
the internal librbd functionality with the new ImageWatcher.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Add ImageWatcher unit test cases

Directly unit test the new ImageWatcher class to complement
the existing librbd integration tests of exclusive lock
handling.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Add convenience library to support unit tests

Unit tests need access to the private symbols of librbd no
longer exported from librbd.so.  A new librbd_internal
convenience library was created to allow access.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: Allow CLI to optionally create shared images

Images that are flagged as shared cannot use the RBD
object map nor RBD mirroring features.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Integrate librbd with new exclusive lock feature

Operations that update the image now require the exclusive lock
if the feature is enabled.  AIO write and discard operations will
automatically request the exclusive lock from the current leader
to support live-migration.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: bump rados version number

As a follow-on to 49d114f1fff90e5c0f206725a5eb82c0ba329376,
increment the "extra" version field so clients can easily
determine if they have a version of librados that properly
translates C API operation flags.

Signed-off-by: Matthew Richards <mattjrichards@gmail.com>

Merge pull request #3316 from ceph/wip-10471

rgw: index swift keys appropriately

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: Create image exclusive lock watch/notify handler

The new watch/notify handler replaces the existing header
update watch/notify handler and adds support for managing
image exclusive lock leadership.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: enable filestore_extsize by default

Note that this will only get used if the kernel is new enough; if it is
older than 3.5 the option will get disabled and extsize will not be used
even if the option is set to true.

This partially reverts 01cd3cdc726a3e838bce05b355a021778b4e5db1.

Fixes: #9956
Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: verify kernel is new enough before using extsize ioctl

Old kernels have an XFS bug that exposes uninitialized data when the
extsize hint is set and only partially written.  This is fixed by Linux
commit aff3a9edb7080f69f07fe76a8bd089b3dfa4cb5d, documented in XFS bug
http://oss.sgi.com/bugzilla/show_bug.cgi?id=874, and tested by XFS
test xfs/229 to prevent regressions.

Notably the original bug affects kernel 3.2, which is widely deployed with
ubuntu precise 12.04.

Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3352 from kylinstorage/fix-10503

Fix bug 10503: http://tracker.ceph.com/issues/10503

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3203 from majianpeng/fix1

avoid memcopy from librados to caller buffer

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3034 from dachary/wip-10017-erasure-code-repair

erasure code repair when there are two failures

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3148 from mslovy/optimazation_wbthrottle

os: WBThrottle: optimize the map to unordered_map

Reviewed-by: Samuel Just <sjust@redhat.com>

mon/MDSMonitor: add confirm flag to fs reset

This was already in the command map but was not
being checked.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add `fs reset` to cephtool tests

This is just a superficial "I can call it" test,
it's actual behaviour is checked elsewhere.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement `fs reset`

This is for use in CephFS disaster recovery.  When
the metadata pool has been forcibly reset to a single-MDS
metadata tree, we would like to reset the MDSMap to match.

Signed-off-by: John Spray <john.spray@redhat.com>

Fix bug 10503: http://tracker.ceph.com/issues/10503
ceph-fuse: quota code is not 32-bit safe for vxattr output

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>

Merge pull request #2948 from ceph/wip-promote

osd: promote_object separation; proxy read

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_rados: add some debug output

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: improve proxy read cancelation

Avoid taking the PG lock for a canceled read op (if we are lucky).  Recheck
after the lock is taken for good measure.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: put proxy read completion on finisher

We can't use the synchronous completion callbacks (in fast dispatch
context) do to the proxy read completion work.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: tiering: avoid duplicate promotion on proxy read

Do not promote if it is already undergoing in maybe_handle_cache.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: proxy instead of redirect read in writeback mode when the
cache pool is full

To preserve read op order

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: cancel and requeue proxy read when needed

Cancel and requeue proxy read on the following cases:
1) on_shutdown
2) on_change
3) background promotion is done

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

osd/ReplicatedPG: allow reads to proxy etc even if blocked

If we are not write ordered, continue with cache checks so that we can
(among other things) proxy reads while promoting.

Note that this may reorder reads for clients, but we've decided that's okay.

Signed-off-by: Sage Weil <sage@redhat.com>

test: add proxy read test

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: proxy reads during promote

wip 9980. Do proxy read and async promotion for writeback.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: add cache mode READPROXY

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: add proxy read support

wip 9979

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>