Merge pull request #3379 from ceph/wip-mon-drop-conversion

mon: drop store conversion code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3377 from ceph/wip-fail-idempotent

mon/MDSMonitor: make 'mds fail' idempotent for IDs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mon: Monitor: drop StoreConverter code

We no longer convert stores on upgrade.  Users coming from bobtail or
before sould go through an interim version such as cuttlefish, dumpling,
firefly or giant.

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

ceph_mon: no longer attempt store conversion on start

People upgrading from bobtail or previous clusters should first go
through an interim version (quite a few to pick from: cuttlefish,
dumpling, firefly, giant).

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3376 from dachary/wip-10547-formatter

common: restore format fallback semantic

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mon/MDSMonitor: make 'mds fail' idempotent for IDs

Was returning ENOENT, should succeed for 'fail' on
a non-existent name, as the fail operation makes
it cease to exist.

Signed-off-by: John Spray <john.spray@redhat.com>

tests: adapt to new json-pretty format

The json-pretty format was modified for readability and now includes
additional newlines / spaces. Either switch to json to avoid dealing
with space changes or modify the expected output to include them.

http://tracker.ceph.com/issues/10547 Fixes: #10547

Signed-off-by: Loic Dachary <ldachary@redhat.com>

test: rename test_activate_osd

It was incorrectly shadowing test_run_osd.

Signed-off-by: Loic Dachary <ldachary@redhat.com>

common: restore format fallback semantic

When Formatter::create replaced new_formatter, the handling of an
invalid format was also incorrectly changed. When an invalid format (for
instance "plain") was specified, new_formatter returned a NULL pointer
which was sometime handled by creating a json-pretty formatter and
sometimes differently.

A new Formatter::create prototype with a fallback argument is added and
is used if it is not the empty string and that the format is not
known. This prototype is used where new_formatter returning NULL was
replaced by a json-pretty formatter.

http://tracker.ceph.com/issues/10547 Fixes: #10547

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3370 from ceph/wip-10382

mds: handle heartbeat_reset during shutdown

Merge pull request #3366 from ceph/wip-formatter

formatter: improve pretty output, rename factory method

Reviewed-by: Dan Mick <dan.mick@redhat.com>

doc/release-notes: v0.91

Signed-off-by: Sage Weil <sage@redhat.com>

doc/release-notes: typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3372 from ceph/wip-10539

qa: fail_all_mds between fs reset and fs rm

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa: fail_all_mds between fs reset and fs rm

Because fs reset opens a brief window for the previously
failed MDSs to spring back into life.

Fixes: #10539
Signed-off-by: John Spray <john.spray@redhat.com>

doc/release-notes: v0.80.8

Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'gh/next'

mds: handle heartbeat_reset during shutdown

Because any thread might grab mds_lock and call heartbeat_reset
immediately after a call to suicide() completes, this needs
to be handled as a special case where we tolerate MDS::hb having
already been destroyed.

Fixes: #10382
Signed-off-by: John Spray <john.spray@redhat.com>

PendingReleaseNotes: make a note about librados flag changes

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3360 from mattrichards/bump_rados_version

librados: bump rados version number

Reviewed-by: Sage Weil <sage@redhat.com>

0.91

Merge pull request #2697 from ceph/wip-8900

RBD image watcher and new exclusive lock handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3254 from trociny/feature-10036

osd: osd tree to show primary-affinity value

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3281 from ceph/wip-10441-b

osd: fix watch ordering bug 10441 option b

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3290 from ceph/wip-da-SCA-20150102

Coverity and SCA fixes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3302 from ceph/wip-9956

os/FileStore: verify kernel is new enough before using extsize ioctl

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3305 from majianpeng/fix5

fix bugs about sync_filesystem

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

common/Formatter: new_formatter -> Formatter::create

Also make the default formatter selection more explicit.

Signed-off-by: Sage Weil <sage@redhat.com>

common/Formatter: improve json-pretty whitespace

The whitespace for the current pretty mode is awkward and weird.  Move
to a more standard approach.  It's more newlines, but *much* more
readable.

Signed-off-by: Sage Weil <sage@redhat.com>

common/Formatter: add newline to flushed output if m_pretty

This applies to json-pretty and xml-pretty modes.

Signed-off-by: Sage Weil <sage@inktank.com>

Merge pull request #3364 from ceph/wip-quota-test

qa: set -e explicitly in quota test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa: set -e explicitly in quota test

Previously was set in hashbang, which meant
that "./quota.sh" was OK, but "sh ./quota.sh" would
just run through ignoring errors.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3336 from ceph/wip-fs-reset

mon: implement `fs reset`

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3343 from dachary/wip-10505-centos-parted

tests: install parted in centos Dockerfile

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

librbd: flush pending AIO requests under all existing flush scenarios

AIO requests that are waiting on the image lock should be flushed
during all existing RBD flush scenarios.  A few flush cases were
missed in the original implementation.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: AIO requests should retry lock requests

Added a timer to support retrying AIO lock requests until
they are successful.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: differentiate between R/O vs R/W RBD features

The new RBD exclusive lock feature should be treated as a
feature that is only applied when the image is opened in
R/W mode.

Older clients will need to handle the updated
cls_rbd::get_features method in order to properly determine
the incompatible features for an image depending on the
current mode.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Add internal unit test cases

The new unit tests cover the modifications made to integrate
the internal librbd functionality with the new ImageWatcher.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Add ImageWatcher unit test cases

Directly unit test the new ImageWatcher class to complement
the existing librbd integration tests of exclusive lock
handling.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Add convenience library to support unit tests

Unit tests need access to the private symbols of librbd no
longer exported from librbd.so.  A new librbd_internal
convenience library was created to allow access.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: Allow CLI to optionally create shared images

Images that are flagged as shared cannot use the RBD
object map nor RBD mirroring features.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: Integrate librbd with new exclusive lock feature

Operations that update the image now require the exclusive lock
if the feature is enabled.  AIO write and discard operations will
automatically request the exclusive lock from the current leader
to support live-migration.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librados: bump rados version number

As a follow-on to 49d114f1fff90e5c0f206725a5eb82c0ba329376,
increment the "extra" version field so clients can easily
determine if they have a version of librados that properly
translates C API operation flags.

Signed-off-by: Matthew Richards <mattjrichards@gmail.com>

Merge pull request #3316 from ceph/wip-10471

rgw: index swift keys appropriately

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

librbd: Create image exclusive lock watch/notify handler

The new watch/notify handler replaces the existing header
update watch/notify handler and adds support for managing
image exclusive lock leadership.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd: enable filestore_extsize by default

Note that this will only get used if the kernel is new enough; if it is
older than 3.5 the option will get disabled and extsize will not be used
even if the option is set to true.

This partially reverts 01cd3cdc726a3e838bce05b355a021778b4e5db1.

Fixes: #9956
Signed-off-by: Sage Weil <sage@redhat.com>

os/FileStore: verify kernel is new enough before using extsize ioctl

Old kernels have an XFS bug that exposes uninitialized data when the
extsize hint is set and only partially written.  This is fixed by Linux
commit aff3a9edb7080f69f07fe76a8bd089b3dfa4cb5d, documented in XFS bug
http://oss.sgi.com/bugzilla/show_bug.cgi?id=874, and tested by XFS
test xfs/229 to prevent regressions.

Notably the original bug affects kernel 3.2, which is widely deployed with
ubuntu precise 12.04.

Backport: giant, firefly
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3352 from kylinstorage/fix-10503

Fix bug 10503: http://tracker.ceph.com/issues/10503

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3203 from majianpeng/fix1

avoid memcopy from librados to caller buffer

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3034 from dachary/wip-10017-erasure-code-repair

erasure code repair when there are two failures

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3148 from mslovy/optimazation_wbthrottle

os: WBThrottle: optimize the map to unordered_map

Reviewed-by: Samuel Just <sjust@redhat.com>

mon/MDSMonitor: add confirm flag to fs reset

This was already in the command map but was not
being checked.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add `fs reset` to cephtool tests

This is just a superficial "I can call it" test,
it's actual behaviour is checked elsewhere.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement `fs reset`

This is for use in CephFS disaster recovery.  When
the metadata pool has been forcibly reset to a single-MDS
metadata tree, we would like to reset the MDSMap to match.

Signed-off-by: John Spray <john.spray@redhat.com>

Fix bug 10503: http://tracker.ceph.com/issues/10503
ceph-fuse: quota code is not 32-bit safe for vxattr output

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>

Merge pull request #2948 from ceph/wip-promote

osd: promote_object separation; proxy read

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_rados: add some debug output

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: improve proxy read cancelation

Avoid taking the PG lock for a canceled read op (if we are lucky).  Recheck
after the lock is taken for good measure.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: put proxy read completion on finisher

We can't use the synchronous completion callbacks (in fast dispatch
context) do to the proxy read completion work.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: tiering: avoid duplicate promotion on proxy read

Do not promote if it is already undergoing in maybe_handle_cache.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: proxy instead of redirect read in writeback mode when the
cache pool is full

To preserve read op order

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: cancel and requeue proxy read when needed

Cancel and requeue proxy read on the following cases:
1) on_shutdown
2) on_change
3) background promotion is done

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

osd/ReplicatedPG: allow reads to proxy etc even if blocked

If we are not write ordered, continue with cache checks so that we can
(among other things) proxy reads while promoting.

Note that this may reorder reads for clients, but we've decided that's okay.

Signed-off-by: Sage Weil <sage@redhat.com>

test: add proxy read test

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: proxy reads during promote

wip 9980. Do proxy read and async promotion for writeback.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: add cache mode READPROXY

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: add proxy read support

wip 9979

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd/ReplicatedPG: separate promotion from the triggering op

Remove the triggering op from the internal promote machinery.

We keep the optional op arg to promote_object() only because we may
block on an object other than the original obc.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: pass promote error to all blocked operations

This isn't the most elegant strategy, but it is the best we can do
right now.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: drop unnecessary cache_mode checks

This currently enumerates all cache modes except none, and we don't
arrive in this function when caching is disabled.  And creating a whiteout
is not cache_mode dependent.  Simplify!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatdPG: adjust braces (no semantic change)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: factor out must_promote case from all cache modes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: factor out common exists case from all cache modes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: make op argument to promote_object optional

For now, we still always pass it.  In preparation, however, we modify
promote_object() so that it will work when op is null.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3309 from trociny/wip-9483

OSD: add a get_latest_osdmap command to the admin socket

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

OSD: add a get_latest_osdmap command to the admin socket

The command blocks and ensures we have the latest map from the
mon. This is useful in testing and to "unstick" clusters in some
odd situations.

Fixes: #9483, #9484 (maybe)
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: Fix PHP librados documentation

Merge pull request #3348 from ceph/wip-mon-wishlist

doc: mon janitorial list is now a wishlist

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: mon janitorial list is now a wishlist

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3327 from ceph/wip-peeringqueue

osd: fix peering queue bug

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3344 from ceph/wip-librbd-snap-unprotect

librbd: shadow variable in snap_unprotect

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: return InvalidAccessKeyId instead of AccessDenied

Fixes: #10334
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 56af795b1046a4c1bfba59e1fefde272bb0e5c1e)

rgw: return SignatureDoesNotMatch instead of AccessDenied

Fixes: #10329
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit ef75d720f289ce2e18c0047380a16b7688864560)

tests: install parted in centos Dockerfile

It is needed to run root ceph-disk tests.

http://tracker.ceph.com/issues/10505 Fixes: #10505

Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc: Clean up pool usage.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

doc: Cleanup RGW pool usage.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3341 from liewegas/wip-10504

client: add ceph version to metadata

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: include ceph and git version in client metadata

Fixes: #10504
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3325 from ceph/wip-nits

allow 'ops' instead of 'dump_ops_in_flight'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3250 from ceph/wip-10372

osdc/Objecter: improve pool deletion detection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert "rgw: switch to new watch/notify API"

This reverts commit dc67cd69604ec4e4df846b818ec739dc7b09a537.

Conflicts:
src/rgw/rgw_rados.cc

Merge pull request #3313 from ceph/wip-asok-get-subtrees

Add MDS "get subtrees" asok command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3312 from ceph/wip-mdscacheobject-const

mds: support constness in MDSCacheObjects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3248 from dachary/wip-table-formatter

add table to Formatter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Added section to install priorities/preferences.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3324 from mattrichards/rados_translate_op_flag

librados: Translate operation flags from C APIs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3328 from xiaoxichen/memstore_size_u64

Bump memstore_device_bytes from U32 to U64

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3337 from dachary/wip-10494-disable-unittest-msgr

tests: temporarily disable unittest_msgr

tests: temporarily disable unittest_msgr

http://tracker.ceph.com/issues/10494 Refs: #10494

Signed-off-by: Loic Dachary <ldachary@redhat.com>