Merge pull request #11478 from dachary/wip-17312-jewel

jewel: build/ops: allow building RGW with LDAP disabled

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11642 from tchaikov/wip-17685-jewel

jewel: mon: fix missing osd metadata (again)

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11467 from dachary/wip-17262-jewel

jewel: rbd-nbd IO hang

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mon: fix missing osd metadata (again)

The JSON output was getting broken by continuing
in the wrong place.

Fixes: http://tracker.ceph.com/issues/17685
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit c5700ce4b45b3a385fe4c2111da852bea7d86da2)

Merge pull request #10757 from dachary/wip-17056-jewel

jewel: mon/osdmonitor: decouple adjust_heartbeat_grace and min_down_reporters

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10759 from dachary/wip-16866-jewel

jewel: OSD: ceph osd df does not show summarized info correctly if one or more OSDs are out

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11590 from dachary/wip-17642-jewel

jewel: TestJournalReplay: sporadic assert(m_state == STATE_READY || m_state == STATE_STOPPING) failure

Reviewed-by: Loic Dachary <ldachary@redhat.com>

test: TestJournalReplay test cases need to wait for committed journal event

Fixes: http://tracker.ceph.com/issues/17566
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 1bdf7a425eb7436838c9445ea2c8ab42dfd3a1b6)

librbd: ignore cache busy errors when shrinking an image

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4ce663845679dc35f2f15b893c6f988c4a60b25b)

Conflicts:
     src/test/librbd/operation/test_mock_ResizeRequest.cc:
     when_resize does not have the allow_shrink argument because
     d1f2c557b2c039730baca9efa3f5244bc19dcb1a has not been
     backported

librbd: invalidate cache before trimming image

Any potential writeback outside the extents of a shrunk image
would result in orphaned objects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3f93a1917437ba7b69b306e4ff971b79e8b79c89)

rbd-nbd: mask out-of-bounds IO errors caused by image shrink

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c6cfb616a846959a2cd1c0e540b14668c61a2afd)

rbd-nbd: fix kernel deadlock during teuthology testing

Fixes: http://tracker.ceph.com/issues/16921
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ce7c1520a8019f011fa34dd898af317f78974577)

Merge pull request #11519 from dachary/wip-17576-jewel

jewel: RGW loses realm/period/zonegroup/zone data: period overwritten if somewhere in the cluster is still running Hammer

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11471 from dachary/wip-17511-jewel

jewel: s3tests-test-readwrite failing with 500

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11469 from dachary/wip-17538-jewel

jewel: rgw:user email can modify to empty when it has values

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11466 from dachary/wip-17290-jewel

jewel: ImageWatcher: use after free within C_UnwatchAndFlush

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11464 from dachary/wip-17373-jewel

jewel: image.stat() call in librbdpy fails sometimes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11463 from dachary/wip-17384-jewel

jewel: helgrind: TestLibRBD.TestIOPP potential deadlock closing an image with read-ahead enabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11462 from dachary/wip-17404-jewel

jewel: update_features API needs to support backwards/forward compatibility

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11459 from dachary/wip-17483-jewel

jewel: RBD should restrict mirror enable/disable actions on parents/clones

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11460 from dachary/wip-17482-jewel

jewel: Enable/Disable of features is allowed even the features are already enabled/disabled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11461 from dachary/wip-17481-jewel

jewel: Proxied operations shouldn't result in error messages if replayed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11431 from ceph/wip-jewel-tarball

jewel: build/ops: include more files in "make dist" tarball

Reviewed-by: Nathan Cutler <ncutler@suse.cz>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #11474 from dachary/wip-17350-jewel

jewel: rgw:response information is error when geting token of swift account

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11492 from SUSE/wip-17575-jewel

jewel: aarch64: Compiler-based detection of crc32 extended CPU type is broken

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11475 from dachary/wip-17349-jewel

jewel: Modification for TEST S3 ACCESS section in INSTALL CEPH OBJECT GATEWAY page

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11473 from dachary/wip-17509-jewel

jewel: Config parameter rgw keystone make new tenants in radosgw multitenancy does not work

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11201 from ceph/wip-backport-logrotate-jewel

jewel: build/ops: backport 'logrotate: Run as root/ceph'

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11126 from tchaikov/wip-17179-jewel

jewel: add a tool to rebuild mon store from OSD

Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #11563 from ceph/wip-jewel-acl-underscore

rgw: fix regression with handling double underscore

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: set correct instance on the object

Fixes: http://tracker.ceph.com/issues/17443
This was broken by commit bc840afafdfe5e528e5c1b711e71420ac3cb5a67

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d2ada35c8c0d938a704ecf9974988ea30a9c0105)

Merge pull request #11548 from dachary/wip-17609-jewel

jewel: tests: ceph-disk must ignore debug monc

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11411 from dachary/wip-17245-jewel

jewel: tests: scsi_debug fails /dev/disk/by-partuuid

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10884 from dachary/wip-17149-jewel

jewel: ceph-disk: expected systemd unit failures are confusing

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: fix regression with handling double underscore

Fixes: http://tracker.ceph.com/issues/16856
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
(cherry picked from commit bc840afafdfe5e528e5c1b711e71420ac3cb5a67)

Merge pull request #11408 from dachary/wip-17345-jewel

jewel: Ceph Status - Segmentation Fault

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11407 from dachary/wip-17360-jewel

jewel: ceph-objectstore-tool crashes if --journal-path <a-directory>

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #11158 from dillaman/wip-rbdmap-jewel

jewel: systemd: add install section to rbdmap.service file

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10862 from dachary/wip-17095-jewel

jewel: rpm: ceph installs stuff in %_udevrulesdir but does not own that directory

Reviewed-by: Loic Dachary <ldachary@redhat.com>

tests: ceph-disk: force debug monc = 0

The sh function will collect both stderr and stdout and debug
will mess the json parsing.

Fixes: http://tracker.ceph.com/issues/17607
Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit 469a53a4adc335ea4ff8e34a958b12f0f222a6c1)

Merge pull request #10860 from dachary/wip-17121-jewel

jewel: the %USED of ceph df is wrong

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10886 from dachary/wip-17144-jewel

jewel: mark_all_unfound_lost() leaves unapplied changes

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #10784 from dachary/wip-17067-jewel

jewel: Request exclusive lock if owner sends -ENOTSUPP for proxied maintenance op

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

doc: fill keyring with caps before passing it to ceph-monstore-tool

to make sure the recovered monitor store is ready for use.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit af8e21163735377071b6832d8a81b035bb835257)

tools/ceph_monstore_tool: bail out if no caps found for a key

we take it as an error if no caps is granted to an entity in the
specified keyring file when rebuilding the monitor db.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b4bd4004a836121c11b0bb97d8123df54c271f04)

tools/ceph_monstore_tool: update pgmap_meta also when rebuilding store.db

we should rebuild pgmap_meta table from the collected osdmaps

Fixes: http://tracker.ceph.com/issues/17400
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit cdfa7a69f63d047205dcfccd63b5d58ab0d4695b)

tools/rebuild_mondb: kill compiling warning

As follow:

[ 72%] Building CXX object src/tools/CMakeFiles/ceph-objectstore-tool.dir/RadosDump.cc.o
/home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/rebuild_mondb.cc: In function ‘int update_mon_db(ObjectStore&, OSDSuperblock&, const string&, const string&)’:
/home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/rebuild_mondb.cc:289:22: warning: ‘crc’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         if (have_crc && osdmap.get_crc() != crc) {
                      ^
/home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/rebuild_mondb.cc:238:14: note: ‘crc’ was declared here
     uint32_t crc;

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit f16a31476a3f9b44a7c3dabf0dfd2a0d015b11b9)

tools/rebuild_mondb: avoid unnecessary result code cast

In general we return negative codes for error cases, so there is
no need perform the cast here.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 6a1c01d334fe65124043aa68a6e0cfaea43836b5)

doc: add rados/operations/disaster-recovery.rst

document the process to recover from leveldb corruption.

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 79a9f299253e24d20547131b3c9c9e0667e3b869)
Conflicts:
        src/tools/rebuild_mondb.cc:
remove the code change in this file from this commit.
and the code gets removed is added in anther commit.

tools/ceph_monstore_tool: add "rebuild" command

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d909fa035c8fbbded786b2ca072acc10ea6b6052)

tools/ceph-objectstore-tool: add "update-mon-db" command

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts:
src/tools/CMakeLists.txt: this file was added in master, so
update src/CMakeLists.txt instead
src/tools/Makefile-server.am: jewel is still using autotools,
so update this file also.
        src/tools/rebuild_mondb.cc: move the code spilled into
                doc/rados/troubleshooting/troubleshooting-mon.rst
                by accident back to this commit.
(cherry picked from commit 24faea7ce446bbf09cbd4a9d3434dd5444a6c295)

mon/AuthMonitor: make AuthMonitor::IncType public

so ceph-objectstore-tool is able to use it when rebuilding monitor
db.

Fixes: http://tracker.ceph.com/issues/17179
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 19ef4f16b3aba04119ac647cd6261c74a57ce829)

Merge pull request #10885 from dachary/wip-17145-jewel

jewel: PG::choose_acting valgrind error or ./common/hobject.h: 182: FAILED assert(!max || (*this == hobject_t(hobject_t::get_max())))

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10883 from dachary/wip-17141-jewel

jewel: PG::_update_calc_stats wrong for CRUSH_ITEM_NONE up set items

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #11420 from dachary/wip-17556-jewel

jewel: librbd::Operations: update notification failed: (2) No such file or directory

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10857 from dachary/wip-16984-jewel

jewel: Disabling pool mirror mode with registered peers results orphaned mirrored images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10645 from dachary/wip-16458-jewel

jewel: Potential crash during journal::Replay shut down

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10652 from dachary/wip-16951-jewel

jewel: ceph 10.2.2 rbd status on image format 2 returns (2) No such file or directory

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #11337 from SUSE/wip-17060-jewel

jewel: librbd: cannot disable journaling or remove non-mirrored, non-primary image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10650 from dachary/wip-16868-jewel

jewel: Prevent the creation of a clone from a non-primary mirrored image

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11433 from dillaman/wip-17416-jewel

jewel: rbd-mirror: improve resiliency of stress test case

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10796 from dillaman/wip-17059-jewel

jewel: rbd: bench io-size should not be larger than image size

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11414 from dachary/wip-17477-jewel

jewel: Crash in Client::_invalidate_kernel_dcache when reconnecting during unmount

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11412 from dachary/wip-17479-jewel

jewel: Duplicate damage table entries

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11415 from dachary/wip-17476-jewel

jewel: Failure in snaptest-git-ceph.sh

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11416 from dachary/wip-17474-jewel

jewel: Failure in dirfrag.sh

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11418 from dachary/wip-17246-jewel

jewel: Log path as well as ino when detecting metadata damage

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #11419 from dachary/wip-17244-jewel

jewel: Failure in snaptest-git-ceph.sh

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10877 from dachary/wip-16946-jewel

jewel: client: nlink count is not maintained correctly

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #10758 from dachary/wip-17007-jewel

jewel: ceph-disk should timeout when a lock cannot be acquired

Reviewed-by: Loic Dachary <ldachary@redhat.com>

rgw: avoid corruption when running old radosgw-admin on a newer rgw

Fixes:http://tracker.ceph.com/issues/17371
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
(cherry picked from commit 8eab9454b4d45ed8092a1292bd1904ef928c2120)

rgw: Fix missing master zone for default region conversion

Fixes:http://tracker.ceph.com/issues/17371
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
(cherry picked from commit cd3bccdef2ecd1d294ab1192ab3b9e731b5592e0)

rgw: mark configuration as converted to avoid corruption when running older admins

Fixes: http://tracker.ceph.com/issues/17371
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
(cherry picked from commit ceafa2863dd9c02da8a30c5a5324b472ed5d3404)

Merge pull request #10864 from dachary/wip-17131-jewel

jewel: Jewel: segfault in ObjectCacher::FlusherThread

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #11417 from dachary/wip-17347-jewel

jewel: ceph-create-keys: sometimes blocks forever if mds allow is set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11272 from aiicore/wip-17402-jewel

jewel: OSDMonitor: Missing nearfull flag set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11193 from SUSE/wip-17377-jewel

jewel: LIBRADOS modify Pipe::connect() to return the error code

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10861 from dachary/wip-17135-jewel

jewel: ceph mon Segmentation fault after set crush_ruleset ceph 10.2.2

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #10855 from dachary/wip-16657-jewel

jewel: i386 tarball gitbuilder failure on master

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11409 from dachary/wip-17341-jewel

jewel: librados memory leaks from ceph::crypto (WITH_NSS)

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11311 from dzafman/wip-scrub-boundary-jewel

jewel: osd: adjust scrub boundary to object without SnapSet

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #11231 from badone/wip-17376

jewel: common: Log.cc: Assign LOG_INFO priority to syslog calls

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9872 from odivlad/fix-init-el7-jewel

jewel: remove SYSTEMD_RUN from initscript

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

AArch64: Detect crc32 extension support from assembler

The used compiler may or may not be recent enough to recognize the
crc32 extended cpu type. However, it does not really have to know about
them either, since all we do is pass inline assembly instructions to
the assembler.

This patch moves the crc cpu extension detection from compiler based
to assembler based, so that we can build optimized code even when the
compiler does not know about the cpu type yet.

Signed-off-by: Alexander Graf <agraf@suse.de>
(manual backport of e70ab48b7f6d39a281b3ec65098535a55018b681 - manual backport
was undertaken because jewel uses autotools)

Merge pull request #10891 from dachary/wip-16793-jewel

jewel: rgw: upgrade from old multisite to new multisite fails

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10889 from dachary/wip-17143-jewel

jewel: rgw: rgw file uses too much CPU in gc/idle thread

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10868 from dachary/wip-17064-jewel

jewel: rgw: radosgw daemon core when reopen logs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10867 from dachary/wip-17118-jewel

jewel: rgw： period commit return error when the current period has a zonegroup which doesn't have a master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #11367 from linuxbox2/jewel-s3-versioning

jewel: rgw: S3 object versioning fails when applied on a non-master zone

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #11342 from SUSE/wip-17505-jewel

jewel: rgw: doc: description of multipart part entity is wrong

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11330 from cbodley/wip-17073

jewel: rgw: RGWDataSyncCR fails on errors from RGWListBucketIndexesCR

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #11139 from cbodley/wip-radosgw-admin-man-jewel

jewel: add orphan options to radosgw-admin --help and man page

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10832 from ceph/jewel-default-quota

jewel: rgw - default quota fixes

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10107 from xiaoxichen/wip-16313-jewel

jewel: client: FAILED assert(root_ancestor->qtree == __null)

Reviewed-by: John Spray <john.spray@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

tests: populate /dev/disk/by-partuuid for scsi_debug

The scsi_debug SCSI devices do not have a symlink in /dev/disk/by-partuuid
because they are filtered out by 60-persistent-storage.rules. That was
worked around by 60-ceph-partuuid-workaround-rules which has been
removed by 9f76b9ff31525eac01f04450d72559ec99927496.

Add create rules targetting this specific case, only for tests since the
problem does not show in real use cases.

Fixes: http://tracker.ceph.com/issues/17100
Signed-off-by: Loic Dachary <loic@dachary.org>
(cherry picked from commit 7cbf1f0a5eced402e6c7242015550668e3b568f3)

Allow building RGW with LDAP disabled

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>
(cherry picked from commit 38abfcb5d036c050a363533a2c7b658d528d3612)

doc: Modification for "TEST S3 ACCESS" section

Signed-off-by: SirishaGuduru sirishaguduru99@gmail.com
(cherry picked from commit 27d6cf0c6958de9f0533bedb90a74c4020b53675)

rgw:response information is error when geting token of swift account
The header is X-Trans-Id instead of x-amz-request-id in the response header

Fixes:#15195
Signed-off-by: Qiankun Zheng <zheng.qiankun@h3c.com>
(cherry picked from commit 08e909d0b844e8be3cb82d6add3e87d38e19e80e)

doc: Radosgw multitenancy config paramater change

Radosgw multitenancy configuration parameter
"rgw keystone make new tenants" never works even
applied. When gone through the code, itseems this
parameter is not used. But "rgw keystone implicit
tenants" works as the code looks for this.

Modified the configuration parameter in two files
mentioned below from "rgw keystone make new tenants"
to "rgw keystone implicit tenants"

Fixes: http://tracker.ceph.com/issues/17293
Signed-off-by: SirishaGuduru <SGuduru@walmartlabs.com>
(cherry picked from commit bd9695d50cfd472508a8a704f1ea5fc62dc08faf)

rgw: don't fail if lost race when setting acls

Fixes: http://tracker.ceph.com/issues/16930
When concurrently setting acls on object/bucket, we could lose in a race.
Instead of retry, just return success (same effect as if we won and then
other writer overwrote us).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 6e9a915b565923081f609048072b8d75716a74ea)