Merge pull request #3305 from majianpeng/fix5

fix bugs about sync_filesystem

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3364 from ceph/wip-quota-test

qa: set -e explicitly in quota test

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

qa: set -e explicitly in quota test

Previously was set in hashbang, which meant
that "./quota.sh" was OK, but "sh ./quota.sh" would
just run through ignoring errors.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #3336 from ceph/wip-fs-reset

mon: implement `fs reset`

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3343 from dachary/wip-10505-centos-parted

tests: install parted in centos Dockerfile

Reviewed-by: Joao Eduardo Luis <joao@redhat.com>

Merge pull request #3352 from kylinstorage/fix-10503

Fix bug 10503: http://tracker.ceph.com/issues/10503

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3203 from majianpeng/fix1

avoid memcopy from librados to caller buffer

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3034 from dachary/wip-10017-erasure-code-repair

erasure code repair when there are two failures

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3148 from mslovy/optimazation_wbthrottle

os: WBThrottle: optimize the map to unordered_map

Reviewed-by: Samuel Just <sjust@redhat.com>

mon/MDSMonitor: add confirm flag to fs reset

This was already in the command map but was not
being checked.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add `fs reset` to cephtool tests

This is just a superficial "I can call it" test,
it's actual behaviour is checked elsewhere.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement `fs reset`

This is for use in CephFS disaster recovery.  When
the metadata pool has been forcibly reset to a single-MDS
metadata tree, we would like to reset the MDSMap to match.

Signed-off-by: John Spray <john.spray@redhat.com>

Fix bug 10503: http://tracker.ceph.com/issues/10503
ceph-fuse: quota code is not 32-bit safe for vxattr output

Signed-off-by: Yunchuan Wen <yunchuanwen@ubuntukylin.com>

Merge pull request #2948 from ceph/wip-promote

osd: promote_object separation; proxy read

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_rados: add some debug output

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: improve proxy read cancelation

Avoid taking the PG lock for a canceled read op (if we are lucky).  Recheck
after the lock is taken for good measure.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: put proxy read completion on finisher

We can't use the synchronous completion callbacks (in fast dispatch
context) do to the proxy read completion work.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: tiering: avoid duplicate promotion on proxy read

Do not promote if it is already undergoing in maybe_handle_cache.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: proxy instead of redirect read in writeback mode when the
cache pool is full

To preserve read op order

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: cancel and requeue proxy read when needed

Cancel and requeue proxy read on the following cases:
1) on_shutdown
2) on_change
3) background promotion is done

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

osd/ReplicatedPG: allow reads to proxy etc even if blocked

If we are not write ordered, continue with cache checks so that we can
(among other things) proxy reads while promoting.

Note that this may reorder reads for clients, but we've decided that's okay.

Signed-off-by: Sage Weil <sage@redhat.com>

test: add proxy read test

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: proxy reads during promote

wip 9980. Do proxy read and async promotion for writeback.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: add cache mode READPROXY

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: add proxy read support

wip 9979

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd/ReplicatedPG: separate promotion from the triggering op

Remove the triggering op from the internal promote machinery.

We keep the optional op arg to promote_object() only because we may
block on an object other than the original obc.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: pass promote error to all blocked operations

This isn't the most elegant strategy, but it is the best we can do
right now.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: drop unnecessary cache_mode checks

This currently enumerates all cache modes except none, and we don't
arrive in this function when caching is disabled.  And creating a whiteout
is not cache_mode dependent.  Simplify!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatdPG: adjust braces (no semantic change)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: factor out must_promote case from all cache modes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: factor out common exists case from all cache modes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: make op argument to promote_object optional

For now, we still always pass it.  In preparation, however, we modify
promote_object() so that it will work when op is null.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3309 from trociny/wip-9483

OSD: add a get_latest_osdmap command to the admin socket

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

OSD: add a get_latest_osdmap command to the admin socket

The command blocks and ensures we have the latest map from the
mon. This is useful in testing and to "unstick" clusters in some
odd situations.

Fixes: #9483, #9484 (maybe)
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: Fix PHP librados documentation

Merge pull request #3348 from ceph/wip-mon-wishlist

doc: mon janitorial list is now a wishlist

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: mon janitorial list is now a wishlist

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3327 from ceph/wip-peeringqueue

osd: fix peering queue bug

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3344 from ceph/wip-librbd-snap-unprotect

librbd: shadow variable in snap_unprotect

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: return InvalidAccessKeyId instead of AccessDenied

Fixes: #10334
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 56af795b1046a4c1bfba59e1fefde272bb0e5c1e)

rgw: return SignatureDoesNotMatch instead of AccessDenied

Fixes: #10329
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit ef75d720f289ce2e18c0047380a16b7688864560)

tests: install parted in centos Dockerfile

It is needed to run root ceph-disk tests.

http://tracker.ceph.com/issues/10505 Fixes: #10505

Signed-off-by: Loic Dachary <ldachary@redhat.com>

doc: Clean up pool usage.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

doc: Cleanup RGW pool usage.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3341 from liewegas/wip-10504

client: add ceph version to metadata

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: include ceph and git version in client metadata

Fixes: #10504
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3325 from ceph/wip-nits

allow 'ops' instead of 'dump_ops_in_flight'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3250 from ceph/wip-10372

osdc/Objecter: improve pool deletion detection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert "rgw: switch to new watch/notify API"

This reverts commit dc67cd69604ec4e4df846b818ec739dc7b09a537.

Conflicts:
src/rgw/rgw_rados.cc

Merge pull request #3313 from ceph/wip-asok-get-subtrees

Add MDS "get subtrees" asok command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3312 from ceph/wip-mdscacheobject-const

mds: support constness in MDSCacheObjects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3248 from dachary/wip-table-formatter

add table to Formatter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Added section to install priorities/preferences.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3324 from mattrichards/rados_translate_op_flag

librados: Translate operation flags from C APIs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3328 from xiaoxichen/memstore_size_u64

Bump memstore_device_bytes from U32 to U64

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3337 from dachary/wip-10494-disable-unittest-msgr

tests: temporarily disable unittest_msgr

tests: temporarily disable unittest_msgr

http://tracker.ceph.com/issues/10494 Refs: #10494

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3331 from dachary/wip-10493-async-port

msg: initialize AsyncConnection::port

Reviewed-by: Haomai Wang<haomaiwang@gmail.com>

mds: add asok command for getting subtreemap

For when we want to inspect this from a test or
during debugging.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: give CDir a dump() method for JSON output

Useful when listing subtrees via admin socket.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: support constness in MDSCacheObjects

So that one can have const CInode and CDir references
from time to time.

Signed-off-by: John Spray <john.spray@redhat.com>

librbd: shadow variable in snap_unprotect and list_children

The shadow variable prevented snap_unprotect from returning the
correct error return code.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: Add Librados PHP documentation

Merge pull request #3332 from dachary/wip-9570-filejournal

journal related cleanups

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

common: Formatter: cosmetic re-indent

Signed-off-by: Andreas Peters <andreas.joachim.peters@cern.ch>

common: Formatter: add TableFormatter class

For more human readable and shell parsable output.

Signed-off-by: Andreas Peters <andreas.joachim.peters@cern.ch>

erasure-code: test repair when file is removed

Add tests for when files disappear from the file system :

  * file is missing from the primary OSD
  * file is missing from an OSD that is not the primary
  * files are missing from two OSDs that are not the primary
  * files are missing from two OSDs, one of which is the primary

http://tracker.ceph.com/issues/10017 Refs: #10017

Signed-off-by: Loic Dachary <ldachary@redhat.com>

osd: accumulate authoritative peers during recovery

When PGBackend::be_compare_scrubmaps finds multiple good peers, it only
keeps the last one. This is fine for replication but erasure coding
needs to know all good peers for recovery.

PGBackend::be_compare_scrubmaps is modified to accumulate all good peers
and return them to PGBackend::be_compare_scrubmaps and indirectly to
PG::scrub_compare_maps.

PG::scrub_compare_maps will dispatch the good peers to authmap and
good_peers. In the case of authmap, the data structure is not modified
and only the last good peer is set. The ReplicatedPG::_scrub uses
authmap in a non trivial way and it should probably be modified to use
information from multiple good peers instead of just the last one. This
could be the focus of another change.

The scrubber.authoritative data structure is changed to include a list
of pair<ScrubMap::object, pg_shard_t> instead of a single
pair<ScrubMap::object, pg_shard_t> to pass to PG::repair_object and
allow it to add all good peers to the missing_loc locations if the
primary has a missing object. It could be just a list of pg_shard_t
instead because the ScrubMap::object is not used but makes more sense to
keep both and it will presumably be useful when / if the logic changes.

http://tracker.ceph.com/issues/10017 Fixes: #10017

Signed-off-by: Loic Dachary <ldachary@redhat.com>

os: fix confusing indentation in FileJournal::corrupt

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

os: remove debug message leftover in FileJournal

The len of the buffer shows in the message above anyway.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

msg: initialize AsyncConnection::port

http://tracker.ceph.com/issues/10493 Fixes: #10493

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Bump memstore_device_bytes from U32 to U64

U32 limit the max size of memstore to a few GB, which
block our test on memstore performance(as a phototype).

Bump it to U64 will suit for more widely usage

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

osd: assert there is a peering event

This became conditional way back in 12e22b3d44eba51a70d8babebc2684f0c46575a7
for unclear reasons.  It probably predates the in_use checks.  In any case,
at this point, we should only arrive here if the PG was queued, implying
that there will always be an event to process.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: requeue PG when we skip handling a peering event

If we don't handle the event, we need to put the PG back into the peering
queue or else the event won't get processed until the next event is
queued, at which point we'll be processing events with a delay.

The queue_null is not necessary (and is a waste of effort) because the
event is still in pg->peering_queue and the PG is queued.

Note that this only triggers when we exceeed osd_map_max_advance, usually
when there is a lot of peering and recovery activity going on.  A
workaround is to increase that value, but if you exceed osd_map_cache_size
you expose yourself to crache thrashing by the peering work queue, which
can cause serious problems with heavily degraded clusters and bit lots of
people on dumpling.

Backport: giant, firefly
Fixes: #10431
Signed-off-by: Sage Weil <sage@redhat.com>

librados: Translate operation flags from C APIs

The operation flags in the public C API are a distinct enum
and need to be translated to Ceph OSD flags, like as happens in
the C++ API. It seems like the C enum and the C++ enum consciously
use the same values, so I reused the C++ translation function.

Signed-off-by: Matthew Richards <mattjrichards@gmail.com>

Merge pull request #3321 from cernceph/wip-nobarrier-doc

doc: don't suggest mounting xfs with nobarrier

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3322 from dachary/wip-10426-test-directories

tests: group clusters in a single directory

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3306 from ceph/wip-10041

client: fix mount timeout

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: allow 'ops' as shorthand for 'dump_ops_in_flight'

This is an extremely annoying thing to type when working with a
production cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: allow 'ops' as shorthand for 'dump_ops_in_flight'

This is an extremely annoying thing to type when working with a
production cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

tests: group clusters in a single directory

Group all test directories used for mini clusters into a single
sub-directory (testdir). This is easier to cleanup manually and less
error prone.

http://tracker.ceph.com/issues/10426 Fixes: #10426

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3215 from dachary/wip-10384-ceph-test-helper-races

tests: resolve ceph-helpers races

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: don't suggest mounting xfs with nobarrier

nobarrier is dangerous, so stop suggesting it as an example osd mount
options xfs.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

mon, os: check the result of sync_filesystem.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #3310 from ceph/wip-mdsmonitor-fixes

MDSMonitor fixes

Merge pull request #3167 from ceph/wip-10307

rgw: use s->bucket_attrs instead of trying to read obj attrs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3036 from dachary/wip-make-check

packages: add python-virtualenv and xmlstarlet

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3269 from ceph/wip-10387

client: close dirfrag whem rmdir

Reviewed-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: fix `mds fail` for standby MDSs

This command takes a gid, rank or name, but
in the name case it would previously only work if
the named daemon had a rank assigned (mds_info->rank >= 0),
otherwise it would fail silently.

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: respect MDSMAP_DOWN when promoting standbys

Previously, a standby could become active even if 'cluster_down'
had been run.  This was awkward, because it would get you a
"laggy or crashed" mds for the standby that was actually
up and running, just being ignored because of cluster_down.

Signed-off-by: John Spray <john.spray@redhat.com>

init-ceph: stop returns before daemons are dead

The existence of the pidfile must be checked outside of the loop to send
a signal to the daemon. Otherwise the daemon will remove the pidfile and
stop can return before the process is dead because it only checks
/proc/$pid if the pidfile exists.

http://tracker.ceph.com/issues/10389 Fixes: #10389

Signed-off-by: Loic Dachary <ldachary@redhat.com>

client: fix mount timeout

implement a simple timeout mechanism for make_request()

Fixes: #10041
Signed-off-by: Yan, Zheng <zyan@redhat.com>

common: Don't call ioctl(BTRFS_IOC_SYNC) in sync_filesystem.

From the manual of syncfs, it first appeared in Linux 2.6.39. At this
point, btrfs didn't as a stable production. So remove this.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

common: Directly return the result of syncfs().

In commit 808c644248e4, it will try sync() if syncfs() return error.
No evidence prove this way can work. And sync() don't return result
so make this function always return zero which cause filestore omit the
error.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge remote-tracking branch 'origin/wip-10270' into master

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librados/IoCtxImpl.cc
src/librados/IoCtxImpl.h

Merge branch 'wip-fast-txn'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

osd/ECBackend: make sure localt uses tbl is ec txn does

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-object-corpus: drop compat with old ObjectStore::Transaction

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix Transaction::get_data_offset bug when map layout used

add following offset:
  sizeof(__u8) +      // encode struct_v
  sizeof(__u8) +      // encode compat_v
  sizeof(__u32);      // encode len

Change-Id: I5b6662eb42aeeae64baa8699da6ce65e0b1d58c3
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>