Merge pull request #30718 from smithfarm/wip-epel-release-snafu-mimic

mimic: build/ops: backport miscellaneous install-deps.sh and ceph.spec.in fixes from master

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #29241 from smithfarm/wip-38206-mimic

mimic: core: osds allows to partially start more than N+2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29225 into mimic

* refs/pull/29225/head:
common/secret.c: fix key parsing when doing a remount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30291 from tchaikov/wip-mimic-41447

mimic: osd/PrimaryLogPG: Avoid accessing destroyed references in finish_degr…

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #30447 from tchaikov/wip-mimic-37866

mimic: mgr/localpool: pg_num is an int arg to 'osd pool create'

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30481 from pdvian/wip-41863-mimic

mimic: mon: ensure prepare_failure() marks no_reply on op

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30485 from sobelek/wip-41922-mimic

mimic: mon: show pool id in pool ls command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30543 from smithfarm/wip-39317-mimic

mimic: mgr/dashboard: RGW proxy can't handle self-signed SSL certificates

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30275 from pdvian/wip-41704-mimic

 mimic: osd/PrimaryLogPG: update oi.size on write op implicitly truncating ob…

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph.spec.in: fix Cython package dependency for Fedora

Fedora distros do not have python3?-Cython packages, but they do have
python3-Cython ones. Fix the BuildRequires so that we only use the
python3_version_nodots based version string for RHEL.

Fixes: https://tracker.ceph.com/issues/42032
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 40c5a18f81c67c96f6a35e592ac3141b87c19701)

ceph.spec.in: s/pkgversion/version_nodots/

`python3_pkgversion` is now defined as 3, while we don't have packages
like python3-Cython yet in EPEL7. but we do have `python36-Cython`. so
let's use `python3_version_nodots` instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6d2f66846d1b84595cfdf8ac13a7ed4e1ae608ce)

rpm: make Python build deps somewhat less confusing

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 0b79741a770acb5cee6de2dd88022cd2c5d15ef5)

Conflicts:
ceph.spec.in

rpm: Install python2-Cython in python2 environment

do_cmake.sh tests for Cython module version in python2

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 6de7111e735683f2c73d4c013d9611fc02b68985)

Conflicts:
ceph.spec.in

install-deps: do not install if rpm already installed

there is chance that the system already have epel-release-latest-7.noarch.rpm
installed, in that case, install-deps.sh should not fail.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 45a4173bcfe15a9bf56f6aeb6e6c35386a00d1c5)

install-deps.sh: add EPEL repo for non-x86_64 archs as well

Fixes: https://tracker.ceph.com/issues/42023
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 49d740118f9a0ee6b6bb82954630c654946f1751)

install-deps.sh: extract munge_debian_control()

this function will uncomment necessary dependencies on demand.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4fd179654e3c37c837899372859959283479cee6)

Conflicts:
install-deps.sh

install-deps.sh: use latest setuptools

this change reverts "install-deps.sh: workaround setuptools' dependency on six"

This reverts commit 30ce5e55460e333e1240704112a16fae68b3325f.

tl;dr: this change addresses the failures of "make check" runs on arm64
builders when they try to build `mgr-dashboard-test-venv` target.

long story: without this change, we will fail to pull in
setuptools >= 36, and as a result pip will fail to import
`setuptools.build_meta` in `pip/_vendor/pep517/_in_process.py`. and will
a `BackendUnavailable` exception thrown by `_call_hook()` in
`pip/_vendor/pep517/wrappers.py`. since the issue addressed by 30ce5e55
has been addressed since setuptools >= 36.0.1, we should be safe to
upgrade to the latest setuptools now.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 8b55b47cc75e43ef683cd230f792cd92f1c46fa7)

install-deps.sh: skip unavailable repos

as centos-sclo-rh-source leads us to 404 at this moment. and we are not
using the source repo for building ceph. so we can just skip any
unavailable repo.

Fixes: http://tracker.ceph.com/issues/37707
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit aa2b9e9a643441d9a2891e17920677a7725754c4)

install-deps.sh: correct gcc version info as major.minor.patch

On ubuntu 18.04, "gcc -dumpversion" output "7", it result in
that below check will return false:
    dpkg --compare-versions 7 ge 7.0
Then, this script will install other gcc.

Actully, the full gcc version "gcc -dumpfullversion" output is
"7.3.0", than below check will turn true:
    dpkg --compare-version 7.3.0 ge 7.0
So, there's no need to install other gcc.

In case of hitting error on ubuntu16.04, use below parameter:
   gcc -dumpfullversion -dumpversion

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>
(cherry picked from commit ab7344e61755b357c558aef3630e1479d5dcc7b3)

install-deps: install setuptools before upgrading virtualenv

this should address the failures when running install-deps.sh, like

Downloading/unpacking virtualenv
  Running setup.py egg_info for package virtualenv
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown
distribution option: 'python_requires'
      warnings.warn(msg)
    error in virtualenv setup command: 'extras_require' must be a
dictionary whose values are strings or lists of strings containing valid
project/version requirement specifiers.
    Complete output from command python setup.py egg_info:
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown
distribution option: 'python_requires'

  warnings.warn(msg)

error in virtualenv setup command: 'extras_require' must be a dictionary
whose values are strings or lists of strings containing valid
project/version requirement specifiers.

this only happens on very old virtualenv shipped with RHEL7.4

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 95386752ab2635f56e1d38b50de7144e2135b323)

install-deps.sh: dashboard frontend needs git

Running `npm install` for the dashboard pulls dependenies with git.
Under some setups, git was not installed when building the frontend.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
(cherry picked from commit d8de543b7e8f815f46e6a967fb5df4620f1cfbc4)

install-deps: do not specify unknown options

per apt.conf(5), and
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=550564,
"Acquire::Translation" is not documented and will be deprecated. it
would be better to use "Acquire::Languages" solely. and it's consistent
with https://github.com/ceph/ceph-build/pull/1137 .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 3911a7c7d068fde7c1ea7b1a2af0063e1249d697)

build/ops: unify command substitution in install-deps.sh

The $() form is preferable to `` because folks (like me) might be using
` as a keyboard shortcut to GNU Screen, causing havoc to ensue whenever
copy-pasting the ` character.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit e0042dd617269934bf51efee5bab7e4485875576)

install-deps.sh: import ubuntu-toolchain-r's key without keyserver

we are running into following error recently

gpg: requesting key BA9EF27F from hkp server keyserver.ubuntu.com
gpgkeys: key 1E9377A2BA9EF27F can't be retrieved
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0
gpg: keyserver communications error: keyserver helper general error
gpg: keyserver communications error: unknown pubkey algorithm
gpg: keyserver receive failed: unknown pubkey algorithm

so use the local copy of the pubkey instead

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b94e3790c88d599821b71bc76b20755c9580c081)

install-deps: check the exit status for the $builddepcmd

in some case, the $builddepcmd will failed without any "error:" output.
so we should check the exit status to handle it.

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>
(cherry picked from commit 46e33d15a4132772a6bd11cd49f4181081bcdd55)

install-deps: fix installing gcc on ubuntu when no old compiler

Signed-off-by: Tomasz Setkowski <tom@ai-traders.com>
(cherry picked from commit 0d8d0cf6f7594f4f97677a2069998331998bdfa9)

install-deps.sh: support install gcc7 in xenial aarch64

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>
(cherry picked from commit f92f256f4503d19f41b7c300a4df46b189e81a9b)

install-deps: nuke wheelhouse if it's stale

otherwise, if we update *requirements.txt, and the wheelhouse fails to
offer the newly added dependency, the setup-virtualenv.sh will fail the
build.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit ed0ca70707439f44779ca7093461182d73db1a77)

install-deps.sh: Fixes for RHEL 7

Signed-off-by: Zack Cerza <zack@redhat.com>
(cherry picked from commit a1a3e7960cf2c792ac8c9c71e68caa0ebada8933)

Conflicts:
install-deps.sh

Merge pull request #29242 from smithfarm/wip-38277-mimic

mimic: core: mon,osd: limit MOSDMap messages by size as well as map count

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29249 from smithfarm/wip-38852-mimic

mimic: core: .mgrstat failed to decode mgrstat state; luminous dev version?

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29247 from smithfarm/wip-38437-mimic

mimic: core: crc cache should be invalidated when posting preallocated rx buffers

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge pull request #29593 from pdvian/wip-40884-mimic

mimic: mon/MgrMonitor: fix null deref when invalid formatter is specified

Merge pull request #29732 from pdvian/wip-40949-mimic

mimic: Better default value for osd_snap_trim_sleep

Merge pull request #29741 from smithfarm/wip-41287-mimic

mimic: core: ceph daemon mon.a config set mon_health_to_clog false cause leader mon assert

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29976 from dzafman/wip-38450

mimic: osd: make project_pg_history handle concurrent osdmap publish

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30081 from pdvian/wip-41595-mimic

mimic: ceph-objectstore-tool can't remove head with bad snapset

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #30180 from smithfarm/wip-40083-mimic

mimic: osd: Better error message when OSD count is less than osd_pool_default_size

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30181 from smithfarm/wip-40732-mimic

mimic: mon/AuthMonitor: fix initial creation of rotating keys

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30182 from smithfarm/wip-41291-mimic

mimic: os/filestore: assure sufficient leaves in pre-split

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30183 from smithfarm/wip-41351-mimic

mimic: cmake,run-make-check.sh: disable SPDK by default

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30214 from smithfarm/wip-41490-mimic

mimic: osd/OSDCap: Check for empty namespace

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30222 from smithfarm/wip-41502-mimic

mimic: osd/PeeringState: do not complain about past_intervals constrained by oldest epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30243 from smithfarm/wip-41442-mimic

mimic: core: osd: clear PG_STATE_CLEAN when repair object

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #30391 from pdvian/wip-41805-mimic

mimic: mgr: do not reset reported if a new metric is not collected

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29253 from smithfarm/wip-40464-mimic

mimic: core: osd beacon sometimes has empty pg list

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29245 from smithfarm/wip-38351-mimic

mimic: tests: Limit loops waiting for force-backfill/force-recovery to happen

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29250 from smithfarm/wip-38856-mimic

mimic: core: should set EPOLLET flag on del_event()

Merge pull request #29251 from smithfarm/wip-40179-mimic

mimic: tests: qa/standalone/scrub/osd-scrub-snaps.sh sometimes fails

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #29765 from smithfarm/wip-40130-mimic

mimic: doc: Document behaviour of fsync-after-close

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30687 from smithfarm/wip-42104-mimic

mimic: build/ops: admin/build-doc: use python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: use python3 (follow-on fix)

Unfortunately, a5127318cf2feb12b5fd61aeed35b48bc31b4e34 (first attempt to
cherry-pick ace8cb15a600daf5261c0ab349b43010a238a1d9) was incomplete.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(partial manual cherry-pick of ace8cb15a600daf5261c0ab349b43010a238a1d9)

doc: fix urls in posix.rst

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit c5c5f031f77f778af73f8659b01695b88f158807)

doc: cephfs: add section on fsync error reporting to posix.rst

Fixes: http://tracker.ceph.com/issues/24641
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit b52d85c2a71950db222e3649d5265e5891368ba1)

Merge pull request #30663 from smithfarm/wip-fix-docs-build-check-mimic

mimic: build/ops: admin/build-doc: use python3

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30131 from smithfarm/wip-40736-mimic

mimic: doc/rgw: document use of 'realm pull' instead of 'period pull'

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30099 from smithfarm/wip-40673-mimic

mimic: docs: fix rgw_ldap_dnattr username token

Reviewed-by: Casey Bodley <cbodley@redhat.com>

admin/build-doc: use python3

to address https://github.com/sphinx-doc/sphinx/issues/3620, we need to
use sphinx with its fix at
https://github.com/sphinx-doc/sphinx/commit/e049f86b2de1cfdf8a74c88dc9593d047c85d5cb
in other words, we need to use sphinx v2.0.0 and up. but sphinx 2.0
requires python >= 3.5, so we have to use python3 for building the
documents.

in this change:

* doc-requirements.txt: install python3 packages on debian derivatives
* build-doc: install python3.6 packages from EPEL7, and use python3
  venv for using sphinx2
* doc-requirements.txt: bump up all python packages to latest
  stable.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit ace8cb15a600daf5261c0ab349b43010a238a1d9)

Conflicts:
admin/doc-requirements.txt

Merge pull request #30070 from smithfarm/wip-40139-mimic

mimic: doc: describe metadata_heap cleanup

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29039 from dcasier/wip-40100-mimic

mimic: ceph-volume: lvm.activate: Return an error if WAL/DB devices absent

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29257 from smithfarm/wip-39602-mimic

mimic: doc: rgw: CreateBucketConfiguration for s3 PUT Bucket request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30555 from alfredodeza/mimic-guits-41392

mimic ceph-volume: do not fail when trying to remove crypt mapper

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30096 from jan--f/wip-41612-mimic

mimic: ceph-volume: minor optimizations related to class Volumes's use

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: do not fail when trying to remove crypt mapper

In a containerized context, at some point, need to run `simple scan` on a device
from a separate container (not the existing and running corresponding container
to that device), but this can't work because when it tries to remove the
mapper which is still in use by the corresponding running osd container,
it fails.
This can be a bit more permissive and simply throw a warning.

Closes: https://tracker.ceph.com/issues/41392
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit 80d3e4ca054b4d1334c2fb99fede46ab54ca4305)

mgr/dashboard: RGW proxy can't handle self-signed certificates

Fixes tracker.ceph.com/issues/24677

Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 670522ee310cd9ac816e4db9e204498901bd7590)

Merge pull request #30303 from alfredodeza/mimic-ceph-volume-zap-fix

mimic ceph-volume lvm.zap fix cleanup for db partitions

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mon: show pool id in pool ls command

Fixes: http://tracker.ceph.com/issues/40287
Signed-off-by: Chang Liu <liuchang0812@gmail.com>
(cherry picked from commit 0b44a2d22bfee66dcdedfb77fffdccf7eb9e39b8)

mon: ensure prepare_failure() marks no_reply on op

Signed-off-by: Joao Eduardo Luis <joao@suse.com>
(cherry picked from commit 75ec26d78dfff1ce3b668e08d6a6a3cf05fe07de)

Merge pull request #30334 from jan--f/wip-disjoint-device-lists-mimic

mimic: ceph-volume: ensure device lists are disjoint

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #30333 from jan--f/wip-41762-mimic

mimic: ceph-volume: implement __format__ in Size to format sizes in py3

Reviewed-by: Alfredo Deza <adeza@redhat.com>

mgr/localpool: pg_num is an int arg to 'osd pool create'

In bd565bca3f42d7946c392b2833bb3d9982a8ac5d we start returning EINVAL if
the pg_num argument is not valid, whereas before it was silently ignored.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e02a5d55472bdd8473604730f969fad2dc5eafac)

mgr: do not reset reported if a new metric is not collected

Fixes: https://tracker.ceph.com/issues/41741
Signed-off-by: Ilsoo Byun <ilsoobyun@linecorp.com>
(cherry picked from commit 02cc60f6935a5005aa461da183c6c4332503be83)

Merge pull request #30295 from alfredodeza/mimic-rm41378-2

mimic ceph-volume: pre-install python-apt and its variants before test runs

Merge pull request #30301 from alfredodeza/mimic-rm41660

mimic ceph-volume: fix stderr failure to decode/encode when redirected

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30306 from alfredodeza/mimic-rm40664

mimic ceph-volume  zap always skips block.db, leaves them around

ceph-volume: don't keep device lists as sets

This was introduced by #27754. The explicit device lists were cast to
sets but other parts of the code where not updated accordingly. To avoid
touching all code places, only cast to sets for disjoint test and keep
lists otherwise.

Fixes: https://tracker.ceph.com/issues/41292
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 0534cf188a671096d5ddb9d48cdae3dccc6c0b18)

ceph-volume: batch ensure device lists are disjoint

Fixes: https://tracker.ceph.com/issues/41018
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 8fcdb4d628f3a55e4783f0a3df7ac7c0a36eaa43)

ceph-volume: implement __format__ in Size to format sizes in py3

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 7a741f201a078dfd9bce34e66ed16bcb37d99f7f)

ceph-volume tests: verify that wipefs tries several times

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 845e62cb4ba2636325304db2c2d65a185ace25cf)

ceph-volume lvm.zap: retry wipefs several times to prevent race condition failures

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b7b8b106fa485d3b8345b583fbc7f74e30cbc03e)

ceph-volume lvm.zap fix cleanup for db partitions

this uses the correct type 'db' for db type partitions, else
a block.db parition does not get cleaned up by ceph-volume zap

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
(cherry picked from commit 140af8a7eee8d230ac9d94ba100320a20c137a12)

ceph-volume tests create a test file for checking unicode output

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 79196354aca3b3d30c7de1e2ac8b5a3c06ee8330)

ceph-volume tests create a shell test for functional unicode

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 595e492783283359b613d199a75f3e7495bfb9ed)

ceph-volume tests verify new logging fallback and encodings in terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ee18ebc078b2d380b88f3726737184f34345e2f3)

ceph-volume terminal remove unicode stream handler

This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fe66e01f6a4b19bb15695537713a629a951b28ce)

ceph-volume: instantiate the new terminal logger in main()

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bfb8422e262dcce49c50610a3977bfe1edc2a435)

ceph-volume create a logger for the terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 95b16b516c0736839fe50f12ecbeb7cd62c67335)

ceph-volume create a new tox.ini for shell-based tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4612ab3ae62a3e8e1cc9e51489feab0c938296d3)

qa/standalone/ceph-helpers: resurrect all OSD before waiting for health

address the regression introduced by e62cfceb
in e62cfceb, we wanted to test the newly introduced TOO_FEW_OSDS
warning, so we increased the number of OSD to the size of pool, so if
the number of OSD is less than pool size, monitor will send a warning
message.

but we need to bring all OSDs back if we are expecting a healthy
cluster. in this change, all OSDs are resurrect before
`wait_for_health_ok`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit cdba0f14201a327e824b6ec48e014233281dee2d)

qa/standalone/ceph-helpers: fix health_ok test

Stopping the osd daemon won't reliably get you HEALTH_WARN or ERR; you have
to make sure it is also marked down.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit dcdca44aa45e7dc5aab56da26caec644e32024a6)

ceph-volume tests pre-instrall python-apt to prevent auto-install failing later

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 40cac7b9510f7904c2f9dfb1294d809e04bdcf7e)

osd/PrimaryLogPG: Avoid accessing destroyed references in finish_degraded_object

As follows:
for (auto i = callbacks_for_degraded_object.begin(); i != callbacks_for_degraded_object.end();) {
    finish_degraded_object((i++)->first);
}

void PrimaryLogPG::finish_degraded_object(const hobject_t oid)
{
  if (callbacks_for_degraded_object.count(oid)) {
    contexts.swap(callbacks_for_degraded_object[oid]);
    callbacks_for_degraded_object.erase(oid);   // Release
  }

  map<hobject_t, snapid_t>::iterator i = objects_blocked_on_degraded_snap.find(
    oid.get_head());  // Access
  ...
}

Fixes: https://tracker.ceph.com/issues/41250
Signed-off-by: Tao Ning <ningtao@sangfor.com.cn>
(cherry picked from commit 86d55c1a0ddb48efc0c1934728d27f22cf49dfa1)

Conflicts: src/osd/PGBackend.h: `PrimaryLogPG` derives from
`PGBackend::Listener` in mimic, and it's `PGBackend::Listener` 's only
derived class. so we need to update `PGBackend::Listener` accordingly.

osd/PrimaryLogPG: update oi.size on write op implicitly truncating object up

See "BlueStore::_do_truncate", bluestore will reset ondisk object size
unconditionally to track truncated size. Hence we must adjust logical
size (and usage) accordingly to match the specified metrics.

Fixes: https://tracker.ceph.com/issues/41601
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 52caa5b5841e49da3806bd1bf5adce221c42fafb)

Merge pull request #30188 from smithfarm/wip-41332-mimic

mimic: build/ops: rpm: always build ceph-test package

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30227 from smithfarm/wip-41530-mimic

mimic: doc: default values for mon_health_to_clog_* were flipped

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30244 from smithfarm/wip-41701-mimic

mimic: build/ops: install-deps.sh: install `python*-devel` for python*rpm-macros

Reviewed-by: Kefu Chai <kchai@redhat.com>

install-deps.sh: install `python*-devel` for python*rpm-macros

in 087ea813, we installed '*rpm-macros' for the macros, so we can have
access to the latest python packaging related macros for preparing the
build dependencies.

but we could run into https://bugs.centos.org/view.php?id=16379, if
we already have an old version of python-devel installed. as the newer
version of python-rpm-macros conflicts with it.

it was a chicken-and-egg problem, as we don't know the exact name of
*rpm-macros packages. that's why we chose to install all of them. but
we have to upgrade the existing python-devel package to resolve the
conflict. but the since there is no python3-devel in RHEL7/CentOS7,
what they have is python36-devel. so we have to hardwire the
`%{python3_pkgversion}` to "36" even before we have access to this
macro, and upgrade the python36-devel package beforehand. but this
renders installing the rpm-macro package less useful -- we intend to
use the macro offered by the package to figure out "36".

as a workaround, we pretend that we know the "main" version of python3
in current RHEL/CentOS. and always install python36-devel for
python-rpm-macros. as the former requires the latter.

once all python3*-devel on all builders are upgraded, we will be safe
to install '*rpm-macros' again without installing python36-devel first.
by then, we could revert this change, or continue installing
python36-devel until the distro bumps up the "main" python version to 3.7

Fixes: https://tracker.ceph.com/issues/41603
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 0b2be2b6710695867197ee84e411bdf0a843a06a)

Conflicts:
install-deps.sh
- VERSION_ID in mimic, MAJOR_VERSION in master

mon/PGMap: enable/disable TOO_FEW_OSDS warning with an option

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3b74fbc93d32d6cc367049d3bcbe83d3837abba5)

Conflicts
        src/mon/PGMap.cc
- cct->_conf->get_val in mimic

qa/standalone: remove osd_pool_default_size in test_wait_for_health_ok

Signed-off-by: zjh <jhzeng93@foxmail.com>
(cherry picked from commit e62cfceb9549e806f774ef3da549efd9f4d90171)

osd: Better error message when OSD count is less than osd_pool_default_size

Fixes: http://tracker.ceph.com/issues/38617
Signed-off-by: zjh <jhzeng93@foxmail.com>
(cherry picked from commit 94237d36932b27961afffbe65d433af26c3fe513)

Conflicts:
doc/rados/operations/health-checks.rst
- trivial
        src/mon/PGMap.cc
- cct->_conf->get_val in mimic