mon/MDSMonitor: add confirm flag to fs reset

This was already in the command map but was not
being checked.

Signed-off-by: John Spray <john.spray@redhat.com>

qa: add `fs reset` to cephtool tests

This is just a superficial "I can call it" test,
it's actual behaviour is checked elsewhere.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: implement `fs reset`

This is for use in CephFS disaster recovery.  When
the metadata pool has been forcibly reset to a single-MDS
metadata tree, we would like to reset the MDSMap to match.

Signed-off-by: John Spray <john.spray@redhat.com>

Merge pull request #2948 from ceph/wip-promote

osd: promote_object separation; proxy read

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph_test_rados: add some debug output

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: improve proxy read cancelation

Avoid taking the PG lock for a canceled read op (if we are lucky).  Recheck
after the lock is taken for good measure.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: put proxy read completion on finisher

We can't use the synchronous completion callbacks (in fast dispatch
context) do to the proxy read completion work.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: tiering: avoid duplicate promotion on proxy read

Do not promote if it is already undergoing in maybe_handle_cache.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: proxy instead of redirect read in writeback mode when the
cache pool is full

To preserve read op order

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: cancel and requeue proxy read when needed

Cancel and requeue proxy read on the following cases:
1) on_shutdown
2) on_change
3) background promotion is done

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>
Conflicts:
src/osd/ReplicatedPG.cc
src/osd/ReplicatedPG.h

osd/ReplicatedPG: allow reads to proxy etc even if blocked

If we are not write ordered, continue with cache checks so that we can
(among other things) proxy reads while promoting.

Note that this may reorder reads for clients, but we've decided that's okay.

Signed-off-by: Sage Weil <sage@redhat.com>

test: add proxy read test

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: proxy reads during promote

wip 9980. Do proxy read and async promotion for writeback.

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: add cache mode READPROXY

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd: tiering: add proxy read support

wip 9979

Signed-off-by: Zhiqiang Wang <zhiqiang.wang@intel.com>

osd/ReplicatedPG: separate promotion from the triggering op

Remove the triggering op from the internal promote machinery.

We keep the optional op arg to promote_object() only because we may
block on an object other than the original obc.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: pass promote error to all blocked operations

This isn't the most elegant strategy, but it is the best we can do
right now.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: drop unnecessary cache_mode checks

This currently enumerates all cache modes except none, and we don't
arrive in this function when caching is disabled.  And creating a whiteout
is not cache_mode dependent.  Simplify!

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatdPG: adjust braces (no semantic change)

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: factor out must_promote case from all cache modes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: factor out common exists case from all cache modes

Signed-off-by: Sage Weil <sage@redhat.com>

osd/ReplicatedPG: make op argument to promote_object optional

For now, we still always pass it.  In preparation, however, we modify
promote_object() so that it will work when op is null.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3309 from trociny/wip-9483

OSD: add a get_latest_osdmap command to the admin socket

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

OSD: add a get_latest_osdmap command to the admin socket

The command blocks and ensures we have the latest map from the
mon. This is useful in testing and to "unstick" clusters in some
odd situations.

Fixes: #9483, #9484 (maybe)
Signed-off-by: Mykola Golub <mgolub@mirantis.com>

doc: Fix PHP librados documentation

Merge pull request #3348 from ceph/wip-mon-wishlist

doc: mon janitorial list is now a wishlist

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: mon janitorial list is now a wishlist

Signed-off-by: Joao Eduardo Luis <joao@redhat.com>

Merge remote-tracking branch 'gh/next'

Merge pull request #3327 from ceph/wip-peeringqueue

osd: fix peering queue bug

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #3344 from ceph/wip-librbd-snap-unprotect

librbd: shadow variable in snap_unprotect

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: return InvalidAccessKeyId instead of AccessDenied

Fixes: #10334
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 56af795b1046a4c1bfba59e1fefde272bb0e5c1e)

rgw: return SignatureDoesNotMatch instead of AccessDenied

Fixes: #10329
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit ef75d720f289ce2e18c0047380a16b7688864560)

doc: Clean up pool usage.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

doc: Cleanup RGW pool usage.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3341 from liewegas/wip-10504

client: add ceph version to metadata

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: include ceph and git version in client metadata

Fixes: #10504
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #3325 from ceph/wip-nits

allow 'ops' instead of 'dump_ops_in_flight'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3250 from ceph/wip-10372

osdc/Objecter: improve pool deletion detection

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert "rgw: switch to new watch/notify API"

This reverts commit dc67cd69604ec4e4df846b818ec739dc7b09a537.

Conflicts:
src/rgw/rgw_rados.cc

Merge pull request #3313 from ceph/wip-asok-get-subtrees

Add MDS "get subtrees" asok command

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #3312 from ceph/wip-mdscacheobject-const

mds: support constness in MDSCacheObjects

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #3248 from dachary/wip-table-formatter

add table to Formatter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: Added section to install priorities/preferences.

Signed-off-by: John Wilkins <jowilkin@redhat.com>

Merge pull request #3324 from mattrichards/rados_translate_op_flag

librados: Translate operation flags from C APIs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #3328 from xiaoxichen/memstore_size_u64

Bump memstore_device_bytes from U32 to U64

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3337 from dachary/wip-10494-disable-unittest-msgr

tests: temporarily disable unittest_msgr

tests: temporarily disable unittest_msgr

http://tracker.ceph.com/issues/10494 Refs: #10494

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3331 from dachary/wip-10493-async-port

msg: initialize AsyncConnection::port

Reviewed-by: Haomai Wang<haomaiwang@gmail.com>

mds: add asok command for getting subtreemap

For when we want to inspect this from a test or
during debugging.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: give CDir a dump() method for JSON output

Useful when listing subtrees via admin socket.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: support constness in MDSCacheObjects

So that one can have const CInode and CDir references
from time to time.

Signed-off-by: John Spray <john.spray@redhat.com>

librbd: shadow variable in snap_unprotect and list_children

The shadow variable prevented snap_unprotect from returning the
correct error return code.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc: Add Librados PHP documentation

Merge pull request #3332 from dachary/wip-9570-filejournal

journal related cleanups

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

common: Formatter: cosmetic re-indent

Signed-off-by: Andreas Peters <andreas.joachim.peters@cern.ch>

common: Formatter: add TableFormatter class

For more human readable and shell parsable output.

Signed-off-by: Andreas Peters <andreas.joachim.peters@cern.ch>

os: fix confusing indentation in FileJournal::corrupt

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

os: remove debug message leftover in FileJournal

The len of the buffer shows in the message above anyway.

Signed-off-by: Loic Dachary <loic-201408@dachary.org>

msg: initialize AsyncConnection::port

http://tracker.ceph.com/issues/10493 Fixes: #10493

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Bump memstore_device_bytes from U32 to U64

U32 limit the max size of memstore to a few GB, which
block our test on memstore performance(as a phototype).

Bump it to U64 will suit for more widely usage

Signed-off-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

osd: assert there is a peering event

This became conditional way back in 12e22b3d44eba51a70d8babebc2684f0c46575a7
for unclear reasons.  It probably predates the in_use checks.  In any case,
at this point, we should only arrive here if the PG was queued, implying
that there will always be an event to process.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: requeue PG when we skip handling a peering event

If we don't handle the event, we need to put the PG back into the peering
queue or else the event won't get processed until the next event is
queued, at which point we'll be processing events with a delay.

The queue_null is not necessary (and is a waste of effort) because the
event is still in pg->peering_queue and the PG is queued.

Note that this only triggers when we exceeed osd_map_max_advance, usually
when there is a lot of peering and recovery activity going on.  A
workaround is to increase that value, but if you exceed osd_map_cache_size
you expose yourself to crache thrashing by the peering work queue, which
can cause serious problems with heavily degraded clusters and bit lots of
people on dumpling.

Backport: giant, firefly
Fixes: #10431
Signed-off-by: Sage Weil <sage@redhat.com>

librados: Translate operation flags from C APIs

The operation flags in the public C API are a distinct enum
and need to be translated to Ceph OSD flags, like as happens in
the C++ API. It seems like the C enum and the C++ enum consciously
use the same values, so I reused the C++ translation function.

Signed-off-by: Matthew Richards <mattjrichards@gmail.com>

Merge pull request #3321 from cernceph/wip-nobarrier-doc

doc: don't suggest mounting xfs with nobarrier

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3322 from dachary/wip-10426-test-directories

tests: group clusters in a single directory

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3306 from ceph/wip-10041

client: fix mount timeout

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

mds: allow 'ops' as shorthand for 'dump_ops_in_flight'

This is an extremely annoying thing to type when working with a
production cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

osd: allow 'ops' as shorthand for 'dump_ops_in_flight'

This is an extremely annoying thing to type when working with a
production cluster.

Signed-off-by: Sage Weil <sage@redhat.com>

tests: group clusters in a single directory

Group all test directories used for mini clusters into a single
sub-directory (testdir). This is easier to cleanup manually and less
error prone.

http://tracker.ceph.com/issues/10426 Fixes: #10426

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3215 from dachary/wip-10384-ceph-test-helper-races

tests: resolve ceph-helpers races

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc: don't suggest mounting xfs with nobarrier

nobarrier is dangerous, so stop suggesting it as an example osd mount
options xfs.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #3310 from ceph/wip-mdsmonitor-fixes

MDSMonitor fixes

Merge pull request #3167 from ceph/wip-10307

rgw: use s->bucket_attrs instead of trying to read obj attrs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3036 from dachary/wip-make-check

packages: add python-virtualenv and xmlstarlet

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #3269 from ceph/wip-10387

client: close dirfrag whem rmdir

Reviewed-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: fix `mds fail` for standby MDSs

This command takes a gid, rank or name, but
in the name case it would previously only work if
the named daemon had a rank assigned (mds_info->rank >= 0),
otherwise it would fail silently.

Signed-off-by: John Spray <john.spray@redhat.com>

mon/MDSMonitor: respect MDSMAP_DOWN when promoting standbys

Previously, a standby could become active even if 'cluster_down'
had been run.  This was awkward, because it would get you a
"laggy or crashed" mds for the standby that was actually
up and running, just being ignored because of cluster_down.

Signed-off-by: John Spray <john.spray@redhat.com>

init-ceph: stop returns before daemons are dead

The existence of the pidfile must be checked outside of the loop to send
a signal to the daemon. Otherwise the daemon will remove the pidfile and
stop can return before the process is dead because it only checks
/proc/$pid if the pidfile exists.

http://tracker.ceph.com/issues/10389 Fixes: #10389

Signed-off-by: Loic Dachary <ldachary@redhat.com>

client: fix mount timeout

implement a simple timeout mechanism for make_request()

Fixes: #10041
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge remote-tracking branch 'origin/wip-10270' into master

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
src/librados/IoCtxImpl.cc
src/librados/IoCtxImpl.h

Merge branch 'wip-fast-txn'

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

osd/ECBackend: make sure localt uses tbl is ec txn does

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-object-corpus: drop compat with old ObjectStore::Transaction

Signed-off-by: Sage Weil <sage@redhat.com>

osd: fix Transaction::get_data_offset bug when map layout used

add following offset:
  sizeof(__u8) +      // encode struct_v
  sizeof(__u8) +      // encode compat_v
  sizeof(__u32);      // encode len

Change-Id: I5b6662eb42aeeae64baa8699da6ce65e0b1d58c3
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: add feature CEPH_FEATURE_OSD_TRANSACTION_MAY_LAYOUT (1ULL<<47)>>)

This feature determine whether we use tbl encode for transaction of use
the new map layout.

The primary uses peer_features to determine whether transaction should
use tbl, while the replica just follow the primary.

Change-Id: I92ca6e5b59bd1acde6007ad0dffc085be17accab
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: build fields for Transaction::iterator when tbl is used

When tbl is used (for compatibility), the Transaction::begin method need
to build all fields used by iterator. That includes: coll_index,
object_index, data_bl, op_bl, etc.)

Change-Id: I48ea74fec8d052f50da254a726a9c0dffead19bc
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

ceph_perf_objectstore: fix warning

Signed-off-by: Sage Weil <sage@redhat.com>

osd: Transaction::append & Transaction::swap

Finish append and swap for new Transaction encode/decode layout.

Since append will modify the op_bl now, we changed the order of append
and swap in ReplicatedBackend::sub_op_modify and
ReplicatedBackend::submit_transaction to avoid append call on op_t, so
the op_t can be encode in message.

Change-Id: I6fb421e0defdb092fb9732eef818e90291b039f5
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: new Transaction::iterator interface

This patch add new Transaction::iterator interface according to new
encode/decode layout. The new iterator give the whole Op struct in a
single decode_op method.

All ObjectStore Impl (FileStore/MemStore/KeyValueStore) is also changed
to use the new interface.

Change-Id: I1900a6ec302890df2c4357b071e4966c26d7f037
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: add encode/deocde impl for new layout

When use_tbl is true, Transaction::encode will give the same result as
before, while when use_tbl is false, Transaction::encode will use new
field and logic to encode and all related methods such as
get_encoded_bytes, get_data_offset will do the same.

Change-Id: Ia5864e489d47f37cf496fe3fb825b21977d2d938
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: new format for Transaction encode/decode

This patch add a new fixed size struct Transaction::Op to represent
all actions.

All coll and ghobject used by the transaction are keeped in two maps:
  coll:   map<coll_t, __le32> coll_index;
  object: map<ghobject_t, __le32> object_index;

And the Op struct use the map value(__le32) to refer coll and object,
so each coll and object is only need to encode once in the transaction.

Other variable-size fields(key/value/data) is encoded in bufferlist
data_bl.

Change-Id: I52b2fcd3217a6cb35de7b309a6dd74a99478feb2
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: Add Transaction::TransactionData for fast encode/decode

TransactionData wrap the following fields:
      __le64 ops;
      __le32 largest_data_len;
      __le32 largest_data_off;
      __le32 largest_data_off_in_tbl;
      __le32 pad; //make TransactionData multiple of uint64_t

This struct can encode/decode just by a single memcpy instead of many
encode/decode operations.

Change-Id: I56df78def43bd2b80b77be0825756e133434a6e6
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

osd: remove unused Transaction fields

We don't need sobject and pool_override anymore since we don't need to
support anything older than dumpling.

Change-Id: I22c01d4b5c6bf99765bf6bc13aecadc997d6750c
Signed-off-by: Dong Yuan <yuandong1222@gmail.com>

Merge pull request #3272 from trociny/tell-mon-version

cli: make ceph tell mon.* version work

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3196 from tchaikov/shadow-cct

client, librados, osdc: do not shadow Dispatcher::cct

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3298 from majianpeng/fix5

TestLFNIndex.cc: For root, dont do permission operations.

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: resolve ceph-helpers races

Some tests were racing against the monitor. On a fast machine it worked
but slower machines (or sometime when running in parallel), the monitor
is lagging behind. Use wait_for_clean to make sure the monitor is in the
desired state for the test to succeed.

http://tracker.ceph.com/issues/10384 Fixes: #10384

Signed-off-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #3283 from dachary/wip-make-debs

debian: create a repository from sources

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3301 from ktdreyer/rm-tiobench

#10152: qa: drop tiobench suite

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #3300 from ceph/wip-10412

client: fix use-after-free bug in unmount()

Reviewed-by: Sage Weil <sage@redhat.com>