mds: cleanup MDCache::open_snaprealms()

don't retry if all snaprealms' parent are open

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: make sure snaptable version > 0

version 0 snaptable confuses snapshot cache code

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't consider CEPH_INO_LOST_AND_FOUND as base inode

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: replace MAX() with std::max()

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

tools/cephfs: make cephfs-data-scan create snaprealm for base inodes

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/cephfs: don't run TestSnapshots.test_kill_mdstable on kclient

this test forcibly kills mount. Killing kernel mount requires reboot,

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/cephfs: adjust check of 'cephfs-table-tool all show snap' output

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: don't warn unconnected snaplrealms in cluster log

When mds recovers, clients add all snaprealms they have to reconnect
message. In multimds setup, it's common that the recovering mds only
have a portion of these snaprealms

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: update CInode/CDentry's first according to global snapshot seq

This simplifies case that inode gets moved into different snaprealm

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/cephfs: add tests for snapclient cache

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/cephfs: add tests for snaptable transaction

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: add asok command that dumps cached snap infos

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/cephfs: add tests for multimds snapshot

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

client: don't mark snap directory complete when its dirstat is empty

MDS has trouble in tracking dirstat for snap inode. Snap directory
inode's dirstat can be inaccuracy.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/workunits/snaps: add snaprealm split test

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: make sure mds has uptodate mdsmap before checking 'allows_snaps'

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

client: fix incorrect snaprealm when adding caps

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

qa/workunits/snaps: add hardlink snapshot test

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: add incompat feature and bump protocol for snapshot changes

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: detach inode with single hardlink from global snaprealm

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: record hardlink snaps in inode's snaprealm

Inode with multiple hardlinks is attached to global snaprealm.
Before modifying a hardlink, record snaps that reference the
the hardlink. When all hardlinks are removed, stray inode gets
moved into normal snaprealm. By checking the recorded snaps,
mds knows if there still are snaps reference the stray inode.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: attach inode with multiple hardlinks to dummy global snaprealm

The dummy global snaprealm includes all snapshots in the filesystem.
For any later snapshot, mds will COW the inode and preserve snap data.
These snap data will cover any possible snapshot on remote linkages
of the inode.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: cleanup rename code

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: ensure xlocker has uptodate lock state

This simplifies trans-authority rename. Master can prepare new snaplream
for source inode even it's not auth mds.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: simplify SnapRealm::build_snap_{set,trace}

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: record global last_created/last_destroyed in snaptable

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: pop projected snaprealm before inode's parent changes

When creating new snaprealm, we need to split its parent snaprealm's
inodes_with_caps. If new snaprealm is newly created during rename,
inode's original snaprealm's inodes_with_caps should be split. So in
rename/rmdir cases, we should pop projectd snaprealm before inode's
parent changes

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: keep isnap lock in sync state

unlike locks of other types, isnap lock and dentry lock in unreadable
state can block path traverse, so it should be in sync state as much
as possible.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: handle mksnap vs resolve_snapname race

In multimds setup, it's possible that mds receives snap update message
after receiving client requests that lookup the newly created snapshot.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: cleanup snaprealm past parents open check

For new format snaprealm, there is no need to open past parent,
SnapRealm::have_past_parents_open() always return true, In multimds
setup, mds may use snaprealm whithout opening past parents. So the
assertion in SnapRealm::check_cache() is wrong.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: rollback snaprealms when rolling back slave request

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: send updated snaprealms along with slave requests

rmdir and rename may create/update snaprealms. If snaprealms are
created/updated, encode the updated snaprealms in slave requests
and dentry unlink messages. So that when rmdir or rename finishes,
snaprealms in different mds are in sync.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: explict notification for snap update

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: send snap related messages centrally during mds recovery

sending CEPH_SNAP_OP_SPLIT and CEPH_SNAP_OP_UPDATE messages to
clients centrally in MDCache::open_snaprealms()

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: synchronize snaptable caches when mds recovers

The basic idea is:

1. For recovering mds:
 Learn other mds' pending snaptable commits from resolve messages.
 Load snaptable cache from snapserver when resolve done.

2. For survivor mds:
  Refresh snaptable cache from snapserver when cluster is in resolving
  state.
  Learn recovering mds' pending snaptable commits from resolve messages.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: introduce MDCache::maybe_finish_slave_resolve()

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: notify all mds about prepared snaptable update

After snaptable update get prepared, push the update preparation to
all active snaptable clients, then send reply to update initiator.
By this way, the initiator know that all mds have record the update
preparation in their cache. When committing the snaptable update,
the initiator notifies all mds about the commit. Bystander mds'
snaptable cache get synchronized when it receives the notification.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: record snaps in old snaprealm when moving inode into new snaprealm

To get effective snaps in past snaprealms, we just need to filter out
deleted snaps by using global snap infos. This avoids the complexity
of opening 'past parents'

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: cache snaptable in snapclient

The idea is caching both snap infos and pending updates in snapclient.
The snapclient also tracks updates that are being committed, it applies
these commits to its cached snap infos. Steps to update snaptable are:

 - mds.x acquire locks (xlock on snaplock of affected snaprealm inode)
 - mds.x prepares snaptable update. (send preare to snapserver and waits
   for 'agree' reply)
 - snapserver sends notification about the update to all mds and waits
   for ACKs. (not implemented by this patch)
 - snapserver send 'agree' reply to mds.x
 - mds.x journals corresponding
 - mds.x commits the snaptable update and notifies all mds that it
   commits that update. then mds drops locks.

When receiving committing notification, mds applies the committing
update to its cached snap infos. By this way, cached snap infos get
synchronized before snaplock become readable.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: recover snaptable client when mds enters resolve state

this is preparetion for later change that caches snaptable in
snapclient and sync the cached snaptable between mds.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #20382 from tchaikov/wip-fix-ftbfs-store-test

test/store_test: fix FTBFS as Sequencer is removed

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

test/store_test: fix FTBFS as Sequencer is removed

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20294 from rzarzynski/wip-bs-drop-std_function

os/bluestore: avoid overhead of std::function in blob_t.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19232 from socketpair/precision

mgr: increase time resolution of Commit/Apply OSD latencies.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #18804 from majianpeng/bluestore-collection-prealloc

os/bluestore: Prealloc memory avoid realloc in list_collection.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18494 from ifed01/wip-stupidalloc-fix2

os/bluestore: do not assert if BlueFS rebalance is unable to allocate sufficient space

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18343 from shinobu-x/sk-remove-osdmap

mon/PGMap: Remove unnecessary header

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #19954 into master

* refs/pull/19954/head:
test/encoding: refactor to avoid escaping shell magic
mds: minor refactor of SimpleLock
mds: track Capability in mempool
mds: move CInode container members to mempool
mds: move CDentry container members to mempool
mds: move CDir container members to mempool
mds: put MDSCacheObject compact_map in mempool
common: use size_t for object size
mds: convert to allocator agnostic string_view
mds: simplify initialization
compact_*: support mempool allocated containers

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge pull request #20007 from mogeb/steady-clock-librbd

librbd: use steady clock to measure elapsed time in AioCompletion

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20008 from mogeb/steady-clock-tools-rbd

tools/rbd: use steady clock in bencher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20218 from shun-s/wip-speedup-diskusage-resize

librbd: speed up object map disk usage and resize

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20311 from Songweibin/wip-group-snap-ls

rbd: do not show title if there is no group snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20349 from trociny/wip-22932

rbd-mirror: fix potential infinite loop when formatting status message

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #20310 into master

* refs/pull/20310/head:
qa: adjust cephfs full test for kclient

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20148 into master

* refs/pull/20148/head:
mds: reset connection's priv when marking down connection
mds: fix session reference leak

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20155 into master

* refs/pull/20155/head:
osdc/Journaler: make sure flush() writes enough data

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #20190 into master

* refs/pull/20190/head:
mon: allow removal of tier of ec overwritable pool

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #20200 into master

* refs/pull/20200/head:
client: add cap_dirtier_uid/gid to CapSnap

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #20246 into master

* refs/pull/20246/head:
mds: remove extra 0x in ino prints
mds: print inode number not CInode ptr

Merge pull request #20367 from ceph/simple-custom-cluster

ceph-volume: adds custom cluster name support to simple

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: use a custom cluster name in simple functional tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20305 from xiexingguo/wip-more-balancer-fixes

pybind/mgr/balancer: more specific command outputs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20355 from luomuyao/typo

rgw: fix a typo in comment

Reviewed-by: Jos Collin <jcollin@redhat.com>

test/encoding: refactor to avoid escaping shell magic

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: minor refactor of SimpleLock

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: track Capability in mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: move CInode container members to mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

os/bluestore: avoid overhead of std::function in blob_t.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: fix a typo in comment

Signed-off-by: luomuyao <luo.muyao@zte.com.cn>

Merge pull request #20161 from tchaikov/wip-20086

test/librados: be more tolerant with timed lock tests

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20173 from liewegas/wip-os-ch

os: unify Sequencer and CollectionHandle

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: set conf.cluster name during simple activate

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20144 from adamemerson/wip-cache-register

Cache Register!

Provide the ability to examine and delete elements from the cache.

rgw: Add cache introspection and manipulation

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #20176 from dillaman/wip-librbd-clone-v2-1

librbd: initial hooks for clone v2 support

Reviewed-by: Mykola Golub <mgolub@suse.com>

rgw: Add cache introspection and manipulation

Provide the ability to examine and delete elements from the cache.

Fixes: http://tracker.ceph.com/issues/22603
Fixes: http://tracker.ceph.com/issues/22604
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd-mirror: fix potential infinite loop when formatting status message

The improvements include:

- tag_tid values should always be increasing, so loop only if
  master.tag_tid > mirror_tag_tid in calculate_behind_master_or_send_update;
- in send_update_tag_cache don't refetch a tag if it is already in the
  cache;
- make fake tags with tag_data.predecessor.tag_tid set to zero;
- make sure the new tag is inserted to the cache if an old entry
  with this id happens to exist.

Fixes: http://tracker.ceph.com/issues/22932
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20318 from ovh/bp-limit-normalize-key

common/config: limit calls to normalize_key_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19958 from renhwztetecs/renhw-wip-osd-sighup

osd: Sign in early SIGHUP signal

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: move CDentry container members to mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: move CDir container members to mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: put MDSCacheObject compact_map in mempool

Partial-fix: http://tracker.ceph.com/issues/21402

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: use size_t for object size

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: convert to allocator agnostic string_view

This is necessary to allow many interfaces to take mempool allocated strings.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20264 from ceph/wip-rm22620

ceph-volume dmcrypt support for simple

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #20307 from tchaikov/wip-std-filesystem

test/admin_socket_output: switch to std::experimental::filesystem

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #20324 from tchaikov/wip-cmake-build-boost

cmake: fix the include dir for building boost::python

Reviewed-by: Piotr Dałek <piotr.dalek@corp.ovh.com>

pybind/mgr/balancer: cancel plan if distribution is already perfect

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/balancer: allow customized optimization threshold

In practice it is hard to achieve a 100% (perfect) balanced distribution.
This patch introduces a user visiable threshold configuration here, which
allow user to define the minimal deviation to trigger an optimization.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

pybind/mgr/balancer: be more specific on creating an optimization plan

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #20347 from Songweibin/wip-osd-cleanup

osd: drop redundant comment

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

osd: drop redundant comment

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge PR #20288 into master

* refs/pull/20288/head:
ptl-tool: open githubmap with utf-8 code

Reviewed-by: Jos Collin <jcollin@redhat.com>

mds: remove extra 0x in ino prints

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

test/objectstore/TestObjectStoreState: make objects match collection

- start pool at 1
- make hash fall within pg (just match it for now)

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: remove void *osr from Transaction

No longer used!  (It was only needed for an ugly FileStore hack.)

Signed-off-by: Sage Weil <sage@redhat.com>

os/filestore: do not rely on ObjectStore::Transaction::osr

This was used only to name the osr for the purposes of the tracepoint. We
can do that ourselves!

Signed-off-by: Sage Weil <sage@redhat.com>

osd: drop explicit calls to osr->flush

These are no longer needed as the implementations do it themselves.

Signed-off-by: Sage Weil <sage@redhat.com>

osd/PrimaryLogPG: drop unnecessary flush() on shutdown

The ObjectStore implementation should handle this.

Signed-off-by: Sage Weil <sage@redhat.com>

os: make implementation flush before collection_{list,empty}

This takes the (awkward) burden off the caller.

Signed-off-by: Sage Weil <sage@redhat.com>

os/ObjectStore: make all read operations CollectionHandle-based

Drop the coll_t shims, and convert the last few that weren't already
handle-based.

Signed-off-by: Sage Weil <sage@redhat.com>