mon: A PG with PG_STATE_REPAIR doesn't mean damaged data, PG_STATE_INCONSISTENT does

Fixes: http://tracker.ceph.com/issues/38070
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a4b1574532ee4f789b30634f0eca03cc30ddcf8d)

Merge pull request #25385 from ifed01/wip-ifed-osd-mkfs-fix

mimic: osd/OSD: OSD::mkfs asserts when reusing disk with existing superblock.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26201 from smithfarm/wip-37688-mimic

mimic: tests: Command failed on smithi191 with status 1: 'sudo yum -y install ceph-radosgw'

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26206 from neha-ojha/wip-37902-mimic

mimic: osd/mon: fix upgrades for pg log hard limit

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc: pglog_hardlimit flag recommendations

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25348 from ifed01/wip-ifed-fix-expand-mimic

mimic: core: os/bluestore_tool: fix bluefs expand

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25525 from smithfarm/wip-36751-mimic

mimic: cmake: do not pass -B{symbolic,symbolic-functions} to linker on FreeBSD

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #25531 from smithfarm/wip-37642-mimic

mimic: ceph-create-keys: fix octal notation for Python 3 without losing compatibility with Python 2

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25857 from ashishkumsingh/wip-37822-mimic

mimic: mds: log evicted clients to clog/dbg

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25885 from pdvian/wip-37818-mimic

mimic: mds: remove wrong assertion in Locker::snapflush_nudge

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26010 from smithfarm/wip-37819-mimic

mimic: mds: create separate config for heartbeat timeout

Reviewed-by:  Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #26055 from smithfarm/wip-37898-mimic

mimic: mds: purge queue recovery hangs during boot if PQ journal is damaged

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #26214 from pdvian/wip-38101-mimic

mimic: mds: trim cache after journal flush

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #26009 from sebastian-philipp/dashboard-mirror_mode

mimic: mgr/dashboard: Set mirror_mode to None

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by:  Volker Theile <vtheile@suse.com>

Merge pull request #26208 from vumrao/wip-vumrao-38107

mimic: mgr/DaemonServer: log pgmap usage to cluster log

Reviewed-by: Neha Ojha <nojha@redhat.com>

mds: allow boot on read-only

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit c7ce967b778a0b86b335f6801301e484aaf6ebc3)

Conflicts:
src/mds/MDSRank.cc
- no ceph_assert in mimic

mds: setup readonly mode for PurgeQueue

If the PQ faces an error, it should go read-only along with the MDS rank.

Fixes: http://tracker.ceph.com/issues/37543
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 4cccc4dffb0915ef9e7d3b446e9a32f277646562)

Conflicts:
src/mds/PurgeQueue.cc
- Mutex::Locker instead of std::lock_guard
- no ceph_assert in mimic

mds: add missing locks for PurgeQueue methods

These could race with the asynchronous workings of the PQ.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit c7350ac23c73867b52cd9b7bb23b6c618eebe44d)

Conflicts:
src/mds/PurgeQueue.cc
- use Mutex::Locker instead of std::lock_guard
- no "ceph_assert" in mimic

mds: delete on_error context on des

Otherwise it leaks.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 33279822eabb64380f5968cc645735a8f99a3ac1)

mds: trim cache after journal flush

Dropping cache should trim the cache only after journal flush otherwise dirty
inodes will remain pinned in cache.

Fixes: http://tracker.ceph.com/issues/38010
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit d7f6c1583b4162fade434e376176f3068472f53f)

mgr/DaemonServer: log pgmap usage to cluster log

- change default of mon_cluster_log_file_level to debug
- log to clog->debug()

Fixes: https://tracker.ceph.com/issues/37886
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit f64a69e5054fb319e9f717bbcb7ddd8541e01c6f)

 Conflicts:
src/common/options.cc
        dropped master .add_service, .add_see_also

Merge pull request #25306 from trociny/wip-37437-mimic

mimic: crushtool: add --reclassify operation to convert legacy crush maps to use device classes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #25313 from pdvian/wip-36639-mimic

mimic: os/bluestore: rename does not old ref to replacement onode at old name

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #25526 from smithfarm/wip-36753-mimic

mimic: mon: log last command skips latest entry

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25603 from smithfarm/wip-37686-mimic

mimic: core: list-inconsistent-obj output truncated, causing osd-scrub-repair.sh failure

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #25694 from joscollin/wip-37624-mimic

mimic: pybind/mgr/status: fix ceph fs status in py3 environments.

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25708 from joscollin/wip-37755-mimic

mimic: osd/PrimaryLogPG: fix the extent length error of the sync read

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25822 from batrick/i36209

mimic: mds: runs out of file descriptors after several respawns

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25852 from ashishkumsingh/wip-37832-mimic

mimic: objecter: avoid race when reset down osd's session

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #26128 from xiexingguo/wip-mimic-upmap-fixes

mimic: osd: backport recent upmap fixes

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #26147 from ashishkumsingh/wip-38045-mimic

mimic: tests: qa/overrides/short_pg_log.yaml: reduce osd_{min,max}_pg_log_entries

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25413 from tombarron/mimic

mimic: ceph-volume-client: allow setting mode of CephFS volumes

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25515 from smithfarm/wip-36578-mimic

mimic: qa: teuthology may hang on diagnostic commands for fuse mount

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25517 from smithfarm/wip-37424-mimic

mimic: qa: wrong setting for msgr failures

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25519 from smithfarm/wip-37603-mimic

mimic: mds: severe internal fragment when decoding xattr_map from log event

Reviewed-by:  Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #25521 from smithfarm/wip-37607-mimic

mimic: mds: directories pinned keep being replicated back and forth between exporting mds and importing mds

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25523 from smithfarm/wip-37605-mimic

mimic: mds: PurgeQueue write error handler does not handle EBLACKLISTED

Reviewed-by:  Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #25557 from smithfarm/wip-37611-mimic

mimic: qa: pjd test appears to require more than 3h timeout for some configurations

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25559 from smithfarm/wip-37626-mimic

mimic: mds: fix incorrect l_pq_executing_ops statistics when meet an invalid item in purge queue

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25561 from smithfarm/wip-37628-mimic

mimic: mds: do not call Journaler::_trim twice

Reviewed-by:  Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #25642 from vshankar/wip-37609

mimic: mds: disallow dumping huge caches to formatter

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25676 from joscollin/wip-37699-mimic

mimic: mds: fix bug filelock stuck at LOCK_XSYN leading client can't read data

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25678 from joscollin/wip-37695-mimic

mimic: client: fix failure in quota size limitation when using samba

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #25680 from joscollin/wip-37634-mimic

mimic: mds: fix races of updating wanted caps

Reviewed-by:  Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #25681 from joscollin/wip-37632-mimic

mimic: mds: remove duplicated l_mdc_num_strays perfcounter set

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #25683 from joscollin/wip-37630-mimic

mimic: client: do not move f->pos untill success write

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25685 from joscollin/wip-37738-mimic

mimic: MDSMonitor: allow beacons from stopping MDS that was laggy

Reviewed-by: Yan, Zheng <zyan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25803 from ashishkumsingh/wip-37757-mimic

mimic: mds: clean up log messages for standby-replay

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25903 from pdvian/wip-37828-mimic

mimic: client: fix fuse client hang because its pipe to mds is not ok

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25980 from pdvian/wip-37907-mimic

mimic: mds: wait shorter intervals if beacon not sent

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26047 from pdvian/wip-37923-mimic

mimic: qa: test_damage performs truncate test on same object repeatedly

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26051 from ukernel/mimic-37594

mimic: mds: handle state change race

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26066 from pdvian/wip-37986-mimic

mimic: core: Objecter::calc_op_budget: Fix invalid access to extent union member

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26069 from pdvian/wip-37988-mimic

mimic: MDSMonitor: missing osdmon writeable check

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26072 from smithfarm/wip-37952-mimic

mimic: qa: test_damage needs to silence MDS_READ_ONLY

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26180 from pdvian/wip-37894-mimic

mimic: doc: updated Ceph documentation links

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: downgrade librados2,librbd1 for thrash-old-clients tests

librados2 and librbd1 are installed as a dependency of qemu-kvm.
qemu-kvm is installed by ceph-cm-ansible, see [1].

in thrash-old-clients, jewel packages are installed, but yum does
not allow downgrade unless it's required explicitly. in this change,
we downgrade librbd1 and librados2 to address this issue.

currently, the ceph packages shipped by CentOS/RHEL 7 are still an old
version of jewel. so this issue only kicks in when we try to install
hammer.

this change should address failures like

Command failed on smithi136 with status 1: '\n sudo yum -y install
rbd-fuse\n '

found in rados/thrash-old-clients tests.

---
[1]
https://github.com/ceph/ceph-cm-ansible/commit/3db1cbdc226ecdea532bcc7cd6e710b5ad2bb0d6#diff-f2b05d775fedff6c5c6689f564b32f1c

Fixes: http://tracker.ceph.com/issues/37618
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 1d973c1e900d87824298ec82e8e08a1775dca7b1)

Conflicts:
qa/suites/rados/thrash-old-clients/1-install/hammer.yaml

qa/suites/upgrade/luminous-x/stress-split: set flag pglog_hardlimit

We should be able to set this flag after latest luminous is installed.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25511 from smithfarm/wip-37465-mimic

mimic: rgw: master zone deletion without a zonegroup rm would break rgw rados init

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #25578 from pdvian/wip-37556-mimic

mimic: multisite: es sync null versioned object failed because of olh info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25624 from pdvian/wip-37560-mimic

mimic: rgw: rgwgc: process coredump in some special case

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25954 from pdvian/wip-37890-mimic

mimic: rbd: fix delay time calculation for trash move

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #26049 from pdvian/wip-37920-mimic

mimic: test/rbd_mirror: wait for release leader lock fully complete

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #26137 from ashishkumsingh/wip-38032-mimic

mimic: librbd: properly filter out trashed non-user images on purge

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/OSD: OSD::mkfs asserts when reusing disk with existing superblock.

Fixes: http://tracker.ceph.com/issues/37404
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 328ab2d9f0a27aa38d153469f8ed11e21a6c3ae8)

qa/suites/upgrade/mimic-p2p: restructure parallel and add stress-split

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: updated Ceph documentation links

Fixes include:

- Updating Doxygen link for reference (http://docs.ceph.com/docs/master/dev/documenting/)
- Adding :ref: label for User Management page (http://docs.ceph.com/docs/master/rados/operations/user-management/)
- Update User Management link for reference (http://docs.ceph.com/docs/master/rbd/rbd-snapshot/)

Fixes: https://tracker.ceph.com/issues/37793
Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>
(cherry picked from commit 7ff2f215a3d051c4ef226bfdc81184d89520e152)

qa/suites/upgrade/luminous-x: add pg log settings

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 24c3e2d669a293f12be5374e994946d587900e13)

include/rados.h: hide CEPH_OSDMAP_PGLOG_HARDLIMIT from ceph -s

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit dfc292923548fe1e1b5329555dd46949feb96b99)

mon/OSDMonitor.cc: make a note about reusing jewel feature bit

For OSD_PGLOG_HARDLIMIT, we have reused a jewel feature bit that was retired
in luminous. Therefore, we need to check the release version for
>= CEPH_RELEASE_LUMINOUS, before using it.

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 6abcc20dca0ee5a08a3fe7c560750f904fe3fa65)

mon: add and use OSD_PGLOG_HARDLIMIT feature bit

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 6b0a3ded5baabb19809618de16cdf67c925a8e5a)

osd/mon: fix upgrades for pg log hard limit

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 6ceeab6e204286148a69eb147fbc4045beddef49)

Conflicts:
src/mon/MonCommands.h: trivial resolution

osd: bring back old calc_trim_to and rename new method

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 0536aef761fefda9a86d5bc72a12ca8cab931ccb)

Merge pull request #25699 from xiexingguo/wip-37744

mimic: osd/OSDMap: cancel mapping if target osd is out

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25292 from smithfarm/wip-37417-mimic

mimic: mgr: various python3 fixes

Reviewed-by: Noah Watkins <noahwatkins@gmail.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #25299 from smithfarm/wip-37428-mimic

mimic: msg/async: crashes when authenticator provided by verify_authorizer not implemented

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25718 from ashishkumsingh/wip-37552-mimic

mimic: osdc/Objecter: update op_target_t::paused in _calc_target

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25759 from pdvian/wip-37599-mimic

mimic: pybind/mgr: drop unnecessary iterkeys usage to make py-3 compatible

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #25864 from ashishkumsingh/wip-37826-mimic

mimic: mgr: fix crash due to multiple sessions from daemons with same name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25368 from pdvian/wip-37477-mimic

mimic: mgr: race between daemon state and service map in 'service status'

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25513 from smithfarm/wip-37342-mimic

mimic: osd: Prioritize user specified scrubs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #25605 from smithfarm/wip-37698-mimic

mimic: osd: failed assert when osd_memory_target options mismatch

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25809 from pdvian/wip-37742-mimic

mimic: build/ops: rpm: require ceph-base instead of ceph-common

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25844 from pdvian/wip-37810-mimic

mimic: mon/OSDMonitor: do not populate void pg_temp into nextmap

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25846 from pdvian/wip-37812-mimic

mimic: mon: shutdown messenger early to avoid accessing deleted logger

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25850 from ashishkumsingh/wip-37814-mimic

mimic: qa/workunits/rados/test_health_warnings: prevent out osds

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25854 from ashishkumsingh/wip-37824-mimic

mimic: os/bluestore: KernelDevice::read() does the EIO mapping now.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #25958 from xiexingguo/wip-mimic-async-fixes

mimic: msg/async: backport recent messenger fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #26118 from ceph/backport-mimic-26084

mimic: ceph-volume: mark a device not available if it belongs to ceph-disk

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #25386 from pdvian/wip-37483-mimic

mimic: rgw: Don't treat colons specially in resource part of ARN

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25449 from smithfarm/wip-37518-mimic

mimic: rgw: fix max-size in radosgw-admin and REST Admin API

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25459 from pdvian/wip-37536-mimic

mimic: librbd: reset snaps in rbd_snap_list()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25485 from pdvian/wip-37548-mimic

mimic: rgw_file: user info never synced since librgw init

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #25505 from smithfarm/wip-37550-mimic

mimic: rgw: multisite: sync gets stuck retrying deletes that fail with ERR_PRECONDITION_FAILED

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25643 from pdvian/wip-37562-mimic

mimic: rgw: fix version bucket stats

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25740 from pdvian/wip-37649-mimic

mimic: rgw: feature -- log successful bucket resharding events

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #25899 from ashishkumsingh/wip-37558-mimic

mimic:  rgw: handle S3 version 2 pre-signed urls with meta-data

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #26003 from smithfarm/wip-37948-mimic

mimic: rgw: debug logging for v4 auth does not sanitize encryption keys

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25607 from joscollin/wip-37684-mimic

mimic: doc/user-management: Remove obsolete reset caps command

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25501 from joscollin/wip-37601-mimic

mimic: doc: broken link on troubleshooting-mon page

Reviewed-by: Nathan Cutler <ncutler@suse.com>