tools/cephfs: update for FSMap

Everywhere we used to take a rank, we now
take a role (<fscid:rank>)

Signed-off-by: John Spray <john.spray@redhat.com>

mon: update for multiple filesystems

This is a big logic change throughout MDSMonitor,
and additonally some new commands to handle
per-fs variants of what used to be global things.

Signed-off-by: John Spray <john.spray@redhat.com>

messages: add MFSMap

For mon clients that subscribe to 'fsmap'.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: add FSMap

FSMap encapsulates a collection of MDSMaps, one
for each filesystem, and a collection of standby
MDS daemons that haven't been assigned to
a filesystem yet.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: introduce fs_cluster_id_t type

Unique identifier for a filesystem within
the ceph cluster.  Abbreviated to `fscid`,
deliberately chosen to not conflict
with legacy "fsid" identifiers.

Additionally add mds_role_t type which is
a more general form of rank: ranks are
now only unique within a particular fscid.
Roles are a fscid plus a rank.

Signed-off-by: John Spray <john.spray@redhat.com>

mds: handle get_mds_info as const

Signed-off-by: John Spray <john.spray@redhat.com>

qa: update cephtool test for CephFS multi filesystems

Includes adding test for the enable_multiple flag.

Signed-off-by: John Spray <john.spray@redhat.com>

doc: cephfs admin command list

Creating this skeleton to include the new
"ceph fs ..." commands.  Needs filling out
with prose about purpose and result of
commands.

Signed-off-by: John Spray <john.spray@redhat.com>

messages: fix strings for MMonGetVersionReply

It previously described itself as "mon_check_map_ack"

Signed-off-by: John Spray <john.spray@redhat.com>

java: update libcephfs tests to learn pool name

Don't assume the data/metadata pools are called
"data" and "metadata".

Signed-off-by: John Spray <john.spray@redhat.com>

java: add O_DIRECTORY to libcephfs interface

Signed-off-by: John Spray <john.spray@redhat.com>

include/CompatSet: trivial const-ness fix

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: don't set max_mds to use all daemons

This is only useful if you want to test a
single namespace with no standbys.  It's easier
to initially have max_mds=1 and increase it
later than it is to go through the ritual
of decreasing max_mds and then deactivating
a daemon and waiting for it to stop.

Signed-off-by: John Spray <john.spray@redhat.com>

vstart: don't call "fs new" repeatedly

It was doing it once for each MDS, which is safe
(because it's idempotent) but silly.

Signed-off-by: John Spray <john.spray@redhat.com>

mon: initialize last_* timestamps on new pgs to creation time

Currently, when you create a pool, until the PGs peer you generate a health
error like

            8 pgs are stuck inactive for more than 300 seconds

which is inaccurate and misleading.  Instead, set the timestamps to the
creation time so that warnings don't appear until it's clear they're stuck.

Fixes: #14952
Signed-off-by: Sage Weil <sage@redhat.com>

Merge remote-tracking branch 'origin/jewel'

Conflicts:
src/client/Client.h
Signed-off-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7495 from ukernel/jewel-10436

client: flush kernel pagecache before creating snapshot

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7798 from ukernel/wip-mds-snap

A few more snapshot fixes, mostly around snapshotted inode/dentry tracking

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7994 from ukernel/jewel-14996

Merge: Fix client handling of "lost" open directories on shutdown

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge remote-tracking branch 'origin/greg-fs-testing-3-7-damage'

Merge pull request #7686 from david-z/wip-fix-fh-ref-leak

ceph-fuse: Fix potential filehandle ref leak at umount

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7685 from david-z/wip-fix-put-free-fd-back

[cephfs] fix free fds being exhausted eventually because freed fds are never put back

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7003 from jcsp/wip-cephfs-tmap-migrate

tools/cephfs: add tmap_upgrade

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8014 from ukernel/wip-15038

mds: fix inode_t::compare()

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #8012 from zmc/jewel

man/8/ceph-disk: fix formatting issue

Reviewed-by: Dan Mick <dmick@redhat.com>

mds: fix inode_t::compare()

Fixes: #15038
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7978 from dillaman/wip-14908

librbd: refresh image if required before replaying journal ops

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8009 from dillaman/wip-14988

test: reproducer for writeback CoW deadlock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8005 from dillaman/wip-rbd-lock-fence

pybind: flag an RBD image as closed regardless of result code

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8001 from dillaman/wip-rbd-cram

test: update rbd integration cram tests for new default features

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

test: reproducer for writeback CoW deadlock

Refs: #14988

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

man/8/ceph-disk: fix formatting issue

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:39: Unexpected indentation.
WARNING: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:40: Block quote ends without a blank line; unexpected unindent.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 8dcaacbb0549a188ebe4b598d5eccf5584a27e6f)

Merge pull request #7906 from dillaman/wip-14869

journal: re-use common threads between journalers

Conflicts:
src/journal/JournalPlayer.cc
src/librbd/Journal.cc
src/test/rbd_mirror/image_replay.cc
src/tools/rbd_mirror/ImageReplayer.h
src/tools/rbd_mirror/Mirror.cc

(merged interface changes to ImageReplayer, and reduced scope for
change to JournalPlayer due to pr #7884 (wip-14663)).

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7958 from trociny/wip-rbd-mirror-fixup

rbd-mirror: minor fix-ups for initial skeleton implementation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7884 from dillaman/wip-14663

librbd: allocate new journal tag after acquiring exclusive lock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7772 from xinxinsh/wip-14789

librbd: truncate does not need to mark the object as existing in the object map

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #8003 from liewegas/wip-ceph-disk-doc

man/8/ceph-disk: fix formatting issue

Merge pull request #7927 from oritwas/wip-rgw-fix-default

rgw: RGWZoneParams::create should not handle -EEXIST error

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge remote-tracking branch 'gh/jewel'

Merge pull request #7768 from xinxinsh/wip-fix-zero

osd: filestore: FALLOC_FL_PUNCH_HOLE must be used with FALLOC_FL_KEEP_SIZE

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7844 from xiexingguo/xxg-wip-14922

osd: replicatedpg: break out loop if we encounter fatal error during do_pg_op()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7864 from dillaman/wip-librados-aio-notify

librados: race condition on aio_notify completion handling

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #7928 from branch-predictor/bp-radostool-reduce-mem

tools/rados: reduce "rados put" memory usage by op_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7967 from tchaikov/wip-si-cast-coverity

common/strtol.cc: fix the coverity warnings

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #7948 from avnerbh/fix_avoid_enqueue_dup_code

msg: remove duplicated code - local_delivery will now call 'enqueue'

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #7890 from liewegas/wip-reweight

mon: osd [test-]reweight-by-{pg,utilization} command updates

Reviewed-by: Samuel Just <sjust@redhat.com>

pybind: flag an RBD Image as closed regardless of result code

The image will always be closed -- the error code just alerts the
user to any possible error encountered when closing the image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: permit watch flush to return error code

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mon: remove range=100 from reweight-by-* commands

Specifying the range makes it skip instead of error when you give
it a bad value.

Signed-off-by: Sage Weil <sage@redhat.com>

man/8/ceph-disk: fix formatting issue

ERROR: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:39: Unexpected indentation.
WARNING: /srv/autobuild-ceph/gitbuilder.git/build/doc/man/8/ceph-disk.rst:40: Block quote ends without a blank line; unexpected unindent.

Signed-off-by: Sage Weil <sage@redhat.com>

test: update rbd integration cram tests for new default features

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7847 from wido/mirroring-fixes

Updated scripts and documentation for mirrors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Add script to test connectivity and availability of Ceph mirrors

Signed-off-by: Wido den Hollander <wido@42on.com>

mirror: Add examle rsync daemon configuration

This can be used by mirrors to offer a rsync service to end-users

Signed-off-by: Wido den Hollander <wido@42on.com>

doc: Updated docs with additional mirror information

Signed-off-by: Wido den Hollander <wido@42on.com>

Merge pull request #7552 from dachary/wip-14669-key-management

ceph-disk: key management support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7995 from majianpeng/bluestore-dump-onode

osd: bluestore: allow _dump_onode dynamic accept log level

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: Allow _dump_onode dynamic accept log level.

Make _dump_onode dynamic accept different log level in different case.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #7996 from majianpeng/bluestore-fullextent-overwrite

osd: bluestore: for overwrite a extent, allocate new extent on min_alloc_size write

Reviewed-by: Sage Weil <sage@redhat.com>

os/bluestore/BlueStore: For overwrite a extent, allocate new extent rather than WAL.

This bug introduce by commit:b0b4b6de362f0edf. This commit forget this
case which overwrite(0, bluestore_min_alloc_size).
For this case it  need a new extent rather than WAL.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

client: pass proper old snap context to queue_cap_snap()

the snap realm for split point can be newly created, its snap context
is empty in this case.

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: flush kernel pagecache before creating snapshot

Fixes: #10436
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7960 from SUSE/wip-docdev-mon

doc/dev: continue writing Testing in the cloud chapter

Reviewed-by: Loic Dachary <ldachary@redhat.com>

doc/dev: Continue writing Testing in the cloud chapter

Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph_test_libcephfs: shutdown without closing file/dir

Signed-off-by: Yan, Zheng <zyan@redhat.com>

client: close opened dirs when umounting

Fixes: #14996
Signed-off-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #7986 from liewegas/wip-xxhash

xxhash: use clone of xxhash.git; add .gitignore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

xxHash: add .gitignore for build artifacts

Signed-off-by: Sage Weil <sage@redhat.com>

.gitmodules: use github.com/ceph/xxHash clone

This lets us add our own commits, if necessary.

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: refresh image if required before replaying journal ops

Fixes: #14908
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/release-notes: v10.0.4

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #7966 from dachary/wip-release-notes

release-notes: draft v10.0.4 release notes

Merge pull request #7880 from SUSE/wip_keyring_permisions_for_mon_deamon_14950

mon: fix keyring permissions

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7121 from efirs/ef_eversion_t_no_sprintf

osd: Replace snprintf with faster implementation in eversion_t::get_key_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7141 from jecluis/wip-mon-clockskew

mon: make clock skew checks sane

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #7899 from liewegas/wip-osd-markdown

test/osd: Relax the timing intervals in osd-markdown.sh

Merge pull request #7900 from liewegas/wip-bluefs-alignment

osd: bluefs: fix alignment for odd page sizes

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #7904 from liewegas/wip-log-delete

log: use delete[]

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

journal: possible race condition during fetch playback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: fix errors introduced by rebase to master

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: clean up playback notification handling

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

tests: updated test cases for librbd journal tag allocation

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: properly handle tag transition

Now that the tag concept has been re-used for delineating epochs for
librbd, we need playback to properly handle the cases where the active
playback tag abruptly ends and a newer tag is inserted in the first
splay offset object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: additional debug messages surrounding commit updates

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: improved debugging for journal client / tag processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allocate new journal tag when acquiring exclusive lock

The journal tag tracks a write epoch within the image, so before
allowing writes to proceed, ensure the journal entries will be
stored under a new epoch.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add tag handling to journal state machine

The journal will not retrieve the tag class for the image within
the journal in addition to the most recently allocated tag. Also
added helper methods to allocate new tags.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: helper method to retrieve in-memory client data

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: local pool id no longer stored in client registration

The remote pool name should match the remote pool name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update journal client and tag data structures

The mirror peer now holds remote's image id as well as a collection
of bootstrap sync points.  The tag now contains uuids to map back
to the owning pool.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: allocate new uuid when enabling mirroring

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls_rbd: support for uuid to represent a mirrored pool

This uuid will be used as the client id for remote peer image
journals.  This will allow the uuid to be also utilized within
the librbd journal tag for tracking epoch ownership.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7742 from ErwanAliasr1/evelu-spec

Adding build requires

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

common/strtol.cc: fix the coverity warnings

* promote the compared types properly to address the signed/unsigned
  comparison warnings. this also fixes the potential problems of
  slicing a compared type down to a "smaller" type before the
  comparison.
* check for the width of resulting type and shift bits caused by SI
  prefix to avoid the -Wshift-count-overflow warnings. this again
  is a potential issue, as shifting n bits of an integer of m bits
  width, where n >= m, leads to undefined behaviour.
* add a test for the 2nd fixed issue.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: integrate single thread pool for all processing

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: own the lifecycle of the journaler's threads

This is the first step in merging the journal threads into the
librbd singleton thread pool.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: use provided work queue and timer

This avoids the need to open two threads per journaler.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: pass metadata instead of finisher to FutureImpl

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #7972 from badone/wip-CMakeLists-dup-line

cmake: Remove duplicate find_package libcurl line.

Reviewed-by: Kefu Chai <kchai@redhat.com>

librbd: skip truncate request if object does not exist

Signed-off-by: xinxin shu <xinxin.shu@intel.com>