doc: fill keyring with caps before passing it to ceph-monstore-tool

to make sure the recovered monitor store is ready for use.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit af8e21163735377071b6832d8a81b035bb835257)

tools/ceph_monstore_tool: bail out if no caps found for a key

we take it as an error if no caps is granted to an entity in the
specified keyring file when rebuilding the monitor db.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b4bd4004a836121c11b0bb97d8123df54c271f04)

tools/ceph_monstore_tool: update pgmap_meta also when rebuilding store.db

we should rebuild pgmap_meta table from the collected osdmaps

Fixes: http://tracker.ceph.com/issues/17400
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit cdfa7a69f63d047205dcfccd63b5d58ab0d4695b)

tools/rebuild_mondb: kill compiling warning

As follow:

[ 72%] Building CXX object src/tools/CMakeFiles/ceph-objectstore-tool.dir/RadosDump.cc.o
/home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/rebuild_mondb.cc: In function ‘int update_mon_db(ObjectStore&, OSDSuperblock&, const string&, const string&)’:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/rebuild_mondb.cc:289:22: warning: ‘crc’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         if (have_crc && osdmap.get_crc() != crc) {
                      ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/rebuild_mondb.cc:238:14: note: ‘crc’ was declared here
     uint32_t crc;

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit f16a31476a3f9b44a7c3dabf0dfd2a0d015b11b9)

tools/rebuild_mondb: avoid unnecessary result code cast

In general we return negative codes for error cases, so there is
no need perform the cast here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 6a1c01d334fe65124043aa68a6e0cfaea43836b5)

doc: add rados/operations/disaster-recovery.rst

document the process to recover from leveldb corruption.

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 79a9f299253e24d20547131b3c9c9e0667e3b869)
Conflicts:
        src/tools/rebuild_mondb.cc:
remove the code change in this file from this commit.
and the code gets removed is added in anther commit.

tools/ceph_monstore_tool: add "rebuild" command

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d909fa035c8fbbded786b2ca072acc10ea6b6052)

tools/ceph-objectstore-tool: add "update-mon-db" command

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts:
src/tools/CMakeLists.txt: this file was added in master, so
update src/CMakeLists.txt instead
src/tools/Makefile-server.am: jewel is still using autotools,
so update this file also.
        src/tools/rebuild_mondb.cc: move the code spilled into
                doc/rados/troubleshooting/troubleshooting-mon.rst
                by accident back to this commit.
(cherry picked from commit 24faea7ce446bbf09cbd4a9d3434dd5444a6c295)

mon/AuthMonitor: make AuthMonitor::IncType public

so ceph-objectstore-tool is able to use it when rebuilding monitor
db.

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 19ef4f16b3aba04119ac647cd6261c74a57ce829)

Merge pull request #10885 from dachary/wip-17145-jewel

jewel: PG::choose_acting valgrind error or ./common/hobject.h: 182: FAILED assert(!max || (*this == hobject_t(hobject_t::get_max())))

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10883 from dachary/wip-17141-jewel

jewel: PG::_update_calc_stats wrong for CRUSH_ITEM_NONE up set items

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11420 from dachary/wip-17556-jewel

jewel: librbd::Operations: update notification failed: (2) No such file or directory

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10857 from dachary/wip-16984-jewel

jewel: Disabling pool mirror mode with registered peers results orphaned mirrored images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10645 from dachary/wip-16458-jewel

jewel: Potential crash during journal::Replay shut down

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10652 from dachary/wip-16951-jewel

jewel: ceph 10.2.2 rbd status on image format 2 returns (2) No such file or directory

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11337 from SUSE/wip-17060-jewel

jewel: librbd: cannot disable journaling or remove non-mirrored, non-primary image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10650 from dachary/wip-16868-jewel

jewel: Prevent the creation of a clone from a non-primary mirrored image

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11433 from dillaman/wip-17416-jewel

jewel: rbd-mirror: improve resiliency of stress test case

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10796 from dillaman/wip-17059-jewel

jewel: rbd: bench io-size should not be larger than image size

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11414 from dachary/wip-17477-jewel

jewel: Crash in Client::_invalidate_kernel_dcache when reconnecting during unmount

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11412 from dachary/wip-17479-jewel

jewel: Duplicate damage table entries

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11415 from dachary/wip-17476-jewel

jewel: Failure in snaptest-git-ceph.sh

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11416 from dachary/wip-17474-jewel

jewel: Failure in dirfrag.sh

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11418 from dachary/wip-17246-jewel

jewel: Log path as well as ino when detecting metadata damage

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11419 from dachary/wip-17244-jewel

jewel: Failure in snaptest-git-ceph.sh

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10877 from dachary/wip-16946-jewel

jewel: client: nlink count is not maintained correctly

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10758 from dachary/wip-17007-jewel

jewel: ceph-disk should timeout when a lock cannot be acquired

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10864 from dachary/wip-17131-jewel

jewel: Jewel: segfault in ObjectCacher::FlusherThread

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11417 from dachary/wip-17347-jewel

jewel: ceph-create-keys: sometimes blocks forever if mds allow is set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11272 from aiicore/wip-17402-jewel

jewel: OSDMonitor: Missing nearfull flag set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11193 from SUSE/wip-17377-jewel

jewel: LIBRADOS modify Pipe::connect() to return the error code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10861 from dachary/wip-17135-jewel

jewel: ceph mon Segmentation fault after set crush_ruleset ceph 10.2.2

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10855 from dachary/wip-16657-jewel

jewel: i386 tarball gitbuilder failure on master

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11409 from dachary/wip-17341-jewel

jewel: librados memory leaks from ceph::crypto (WITH_NSS)

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11311 from dzafman/wip-scrub-boundary-jewel

jewel: osd: adjust scrub boundary to object without SnapSet

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11231 from badone/wip-17376

jewel: common: Log.cc: Assign LOG_INFO priority to syslog calls

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9872 from odivlad/fix-init-el7-jewel

jewel: remove SYSTEMD_RUN from initscript

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10891 from dachary/wip-16793-jewel

jewel: rgw: upgrade from old multisite to new multisite fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10889 from dachary/wip-17143-jewel

jewel: rgw: rgw file uses too much CPU in gc/idle thread

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10868 from dachary/wip-17064-jewel

jewel: rgw: radosgw daemon core when reopen logs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10867 from dachary/wip-17118-jewel

jewel: rgw： period commit return error when the current period has a zonegroup which doesn't have a master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11367 from linuxbox2/jewel-s3-versioning

jewel: rgw: S3 object versioning fails when applied on a non-master zone

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11342 from SUSE/wip-17505-jewel

jewel: rgw: doc: description of multipart part entity is wrong

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11330 from cbodley/wip-17073

jewel: rgw: RGWDataSyncCR fails on errors from RGWListBucketIndexesCR

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11139 from cbodley/wip-radosgw-admin-man-jewel

jewel: add orphan options to radosgw-admin --help and man page

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10832 from ceph/jewel-default-quota

jewel: rgw - default quota fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10107 from xiaoxichen/wip-16313-jewel

jewel: client: FAILED assert(root_ancestor->qtree == __null)

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

rgw: Add documentation for RGW default quota

Fixes http://tracker.ceph.com/issues/16447

This was modified to use size in kB, rather than size in bytes, since
Jewel has not yet been converted to use size in bytes.

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
(cherry picked from commit e3ab3476146cf0fe604783e9e5a0d63237271735)

rgw: Do not bother to check against conf values during quota handling

Fixes http://tracker.ceph.com/issues/16447

This was modified to use size in kB, rather than size in bytes, since
Jewel has not yet been converted to use size in bytes.

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
(cherry picked from commit 17d2c1712a5b72315a47ab0f8380331bfd478c0b)

rgw: Let the default quota settings take effect during user creation

Fixes http://tracker.ceph.com/issues/16447

Signed-off-by: Pavan Rallabhandi <PRallabhandi@walmartlabs.com>
Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
(cherry picked from commit 400d7c982f18efd7cf96acfc3a63078791d1ec0a)

journal: ensure in-flight ops are complete destroying journaler

Fixes: http://tracker.ceph.com/issues/17446
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a25b1d7cbebb7b19cebee4cfb362cd744bbb4607)

journal: complete action only after notification completed

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 55762cca2bf586d037cb9f32775ec158dc3287c1)

qa/workunits: reduce amount of client IO for rbd-mirror stress

Journal IO is faster now, resulting in OSDs quickly filling up
under the stress test case.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a6dd6b50d632aa6995a0f4b6fc5e1a1bda23e2a0)

rbd-mirror: potential race allocating tag during shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8c2ff9bce61a8af024150b8be9dee484f339f7df)

librbd: ignore attempts to request lock if already lock owner

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 355c79cb7c955e62e3153cf28046a9b8a6d0d25c)

journal: clean up object recorder closed/overflow callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 24faead086a50ea1b9614268d4dd5f3ea7bbe445)

journal: delay moving overflowed buffers until quiesced

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4483531aa3aa3d0f298778062e2b9a339ad05075)

journal: use reverse iterator search to find flush record

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a96065815558e50361af4c701c23e5248962dfe0)

journal: avoid holding lock while sending journal append

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dc77a629ed353d586b63f0bd8e20f54a7595afba)

rbd-mirror: wait for in-flight event commit notifications

Fixes: http://tracker.ceph.com/issues/17355
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f810c106ad4e9ae94494801fea6c580d81b9156b)

journal: fix AsyncOpTracker compiler error

The master branch has an async JournalTrimmer which shouldn't be
backported to the Jewel branch yet. This change addresses the
missing sync AsyncOpTracker::wait_for_ops method from the backported
class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: move AsyncOpTracker to common library

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 72d8992f054a7e36f92fdd2e01278ce3b9ede2eb)

Conflicts:
src/journal/CMakeLists.txt: doesn't exist in Jewel

librbd: support deleting image by id instead of name

The rbd-mirror daemon will use this API to delete images instead
of attempting to use the local image name.

Fixes: http://tracker.ceph.com/issues/16227
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 87b32d15914207f61595c1c943817d983faceacd)

rbd-mirror: use pool id + global image id as deletion primary key

Fixes: http://tracker.ceph.com/issues/16538
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 25203a8a9d59ff025d223ec1afaeb14946d54993)

rbd-mirror: concurrent access of event might result in heap corruption

Fixes: http://tracker.ceph.com/issues/17283
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ac9ad37ab80db8913f9f1149707bd0009f8235c4)

rbd-mirror: test: Fixed timeout problem in rbd_mirror_stress.sh

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 5c737038dd6d44bd76605587486ddf9457bc2d96)

rbd: Fix race between journal flush and append events

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit aa959e71fe5a8cec43de75007fc9cef8de5ee3a5)

journal: make librados call async in ObjectRecorder

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 7b740f5b4ac1c66ac3c80782d2d34e846d00fddd)

journal: increase concurrency of journal recorder

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 5c88edd68a1ee7c77f11e4113251fbe5768b8d99)

journal: send update notification when tag allocated

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f1cd613e3d8ccb6d05b9adfe1956927991a4f4fe)

librbd: new journal listener event for force promotion

Fixes: http://tracker.ceph.com/issues/16974
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit fd005490e95d7fca85be4cad34344a58986f64d6)

librbd: helper class for quiescing in-flight async ops

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 39d9e5cc9b38de2ee9ad2faf8e04253314160811)

librbd: unify journal event callbacks into single interface

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit dbbcecf4a289ca36b734b7bda9530cc0a59f84ac)

journal: expose ability to retrieve partial tag list

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 277b6dd9f13a8390cbf7c9ac7a313813ecad4d27)

qa/workunits/rbd: fix remove mirrored image race conditions

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a0f65b968b350629dfad978c191fc878ca26d093)

qa/workunits/rbd: new mirroring forced promotion test

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a43268a4a30c06233152d531cbf2550224fb8a15)

test/rbd: fix possible mock journal race conditions

Fixes: http://tracker.ceph.com/issues/17317
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 471898392372ba4c404376410fb56f3af5287c80)

qa/workunits/rbd: before removing image make sure it is not bootstrapped

If an image is being bootstrapped, it implies that the rbd-mirror
daemon currently has the image open. The removal API will prevent the
removal of any image that is opened by another client.

Works-around: http://tracker.ceph.com/issues/16555
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 9737a8d6cbaf1b79bbc6008249d39acbae883941)

librbd: fix journal create compilation error

The synchronous journal create method no longer exists on the master
branch and the associated change to introduce an asynchronous journal
creation state machine should be delayed to provide more testing time
on the master branch before being backported to jewel.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test: fixed unused function warnings in unittest_rbd_mirror

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3545d9ed19be8d3956f0db901ea9d3bb8b10d13d)

rbd-mirror: prevent syncing to non-primary image after failover

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit beaef377d69f555277b706afff944a15086da28e)

rbd-mirror: demote/promote in same cluster results in split-brain

Fixes: http://tracker.ceph.com/issues/16855
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a6901ca1a065419426b3ad704e27e43ba8d591b8)

librbd: normalize journal promotion/demotion events

A non-primary image's commit possition won't accurately reflect
the current demotion/promotion chain. Therefore, directly specify
the predecessor for promotion events.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ac590e85a472533005cad73c82b2d61db161ab7a)

librbd: block RPC requests when demoting the image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 8b195e1fc8fe70a0e5417934302d5831b1f8dfb3)

librbd: separate journal::TagData predecessor data into new struct

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 718befdae711141ef4a1e2e9f5e9aca97f1b5513)

rbd-mirror: include tag tid in bootstrap debug log messages

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 47d1e625006d554164f020e496a847735240ec95)

qa/workunits/rbd: demote/promote image on same cluster

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e6aa18ea0df6dc4d1add597bc2d972c79699bf4c)

rbd-mirror: option to automatically resync after journal client disconnect

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 77fd6a1c2016262d734b0bb5387e6b6a41232e8b)

Conflicts:
src/common/config_opts.h: trivial resolution

rbd-mirror: stop replay when client is disconnected

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 330dba00ba3153ba2862eef52714e0dceae05192)

rbd-mirror: resync was possible only when image replayer start had succeeded

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 4bf6912f3c75560b89324fc29286028750f122c1)

rbd-mirror: decode_client_meta should return false on error

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit cd5eb36e98f46a1d84bddeafa4e7dcad415aa4a2)

rbd: new command to disconnect journal client

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit fc3ba54b3c101498a08a3f34ac8f7eab0152ad7c)

librbd: optionally flag "laggy" journal clients disconnected

Fixes: http://tracker.ceph.com/issues/14738
Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit b8eafefba9f2221a0ce927568795cb1c3ac0fa9c)

journal: allow to trim journal for "laggy" clients

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 0b8b1aaedc10f7f46e91bf6ad809414feb770c8d)

cls/journal: add async client_update_state method

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 58b8c66d5bfa60e6dd3ad2ec79360c2eca165c58)

Merge pull request #11389 from jcsp/wip-17466-jewel

jewel: mon: don't crash on invalid standby_for_fscid

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10997 from batrick/i17105-backport

jewel: multimds: allow_multimds not required when max_mds is set in ceph.conf at startup

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10958 from ukernel/jewel-16764

jewel: client: fix shutdown with open inodes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10921 from jcsp/wip-client-lock-backport

jewel: client: add missing client_lock for get_root

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11400 from ceph/wip-rbd-cli-jewel

jewel: krbd-related CLI patches

Reviewed-by: Loic Dachary <ldachary@redhat.com>