rgw-admin: support for processing all gc objects including unexpired.

Signed-off-by: Yao Zongyou <yaozongyou@vip.qq.com>
(cherry picked from commit ceccb54f191182813cae70841e87b9969c3d4aeb)

Merge pull request #28926 from ceph/backport-luminous-28294

luminous: tests: pass --ssh-config to pytest to resolve hosts when connecting

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #27365 from ifed01/wip-ifed-fix-alloc-overflow-luminous

luminous: os/bluestore: fix length overflow.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jianpeng Ma <majianpeng@gmail.com>

Merge pull request #28640 from tchaikov/luminous-38040

luminous: mon,osd: limit MOSDMap messages by size as well as map count

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27459 from ifed01/wip-ifed-no-verify-fm-luminous

luminous: os/bluestore/BitmapFreelistManager: disable bluestore_debug_freelist

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27529 from ifed01/wip-ifed-fix-21312-luminous

luminous: os/bluestore: call fault_range properly prior to looking for blob to …

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27583 from xiexingguo/wip-38242

luminous: backport recent messenger fixes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27739 from ifed01/wip-ifed-fix-39444

luminous: os/bluestore: fix out-of-bound access in bmap allocator.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #27950 from smithfarm/wip-37612-luminous

luminous: build/ops: python-cephfs should depend on python-rados

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #28030 from ifed01/wip-ifed-dump-before-nospanid-luminous

luminous: os/bluestore: dump before "no spanning blob id" abort

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28111 from smithfarm/wip-38750-luminous

luminous: core: should report EINVAL in ErasureCode::parse() if m<=0

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28929 from ceph/backport-luminous-28866

luminous: ceph-volume broken assertion errors after pytest changes

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #28927 from ceph/backport-luminous-28836

luminous: ceph-volume tests add a sleep in tox for slow OSDs after booting

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume api.lvm catch IndexError when parsing dmmapper output

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bfff70d07a1ccf7fb11a96dd7c63335cfc9b5e26)

ceph-volume tests update to use error.value instead of str(error)

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 484be7ffb1e0865ff840090a73036567a1f4f5d2)

ceph-volume tests add a sleep in tox for slow OSDs after booting

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b398f99f810d97c94af6d943aed4c11a5fd69c19)

tests: pass --ssh-config to pytest to resolve hosts when connecting

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ed708e520999c145c3cf8a0c51e4605f275a1771)

Merge pull request #28536 from ukernel/luminous-40211

luminous: mds: fix corner case of replaying open sessions

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #28543 from batrick/i40041

luminous: mds: avoid trimming too many log segments after mds failover

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #28544 from ukernel/luminous-40221

luminous: mds: reset heartbeat inside big loop

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #28669 from smithfarm/wip-40347-luminous

luminous: tests: use curl in wait_for_radosgw() in util/rgw.py

Merge pull request #27536 from pdvian/wip-38975-luminous

luminous: rbd: krbd: return -ETIMEDOUT in polling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27990 from smithfarm/wip-39589-luminous

luminous: qa/tasks/rbd_fio: fixed missing delimiter between 'cd' and 'configure'

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28124 from dillaman/wip-39460-luminous

luminous: rbd-mirror: clear out bufferlist prior to listing mirror images

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28126 from dillaman/wip-39427-luminous

luminous: librbd: async open/close should free ImageCtx before issuing callback

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28134 from dillaman/wip-38440-luminous

luminous: librbd: ensure compare-and-write doesn't skip compare after copyup

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28137 from dillaman/wip-38674-luminous

luminous: librbd: improve object map performance under high IOPS workloads

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28140 from dillaman/wip-38564-luminous

luminous: librbd: race condition possible when validating RBD pool

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #28745 from dillaman/wip-40551-luminous

luminous: qa/workunits/rbd: use more recent qemu-iotests that support Bionic

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #26554 from pdvian/wip-38397-luminous

luminous: rgw: when exclusive lock fails due existing lock, log add'l info

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #27110 from pdvian/wip-38748-luminous

luminous: rgw_sync: drop ENOENT error logs from mdlog

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27123 from smithfarm/wip-38887-luminous

luminous: rgw: GetBucketCORS API returns Not Found error code when CORS configuration does not exist

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27330 from pdvian/wip-38908-luminous

luminous: rgw: fix read not exists null version return wrong

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27424 from pdvian/wip-38925-luminous

luminous: rgw: Adding tcp_nodelay option to Beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27040 from pdvian/wip-38686-luminous

luminous: qa/cephfs: relax min_caps_per_client check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #27342 from vshankar/wip-38445

luminous: qa: misc cache drop fixes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27679 from rishabh-d-dave/wip-38877-luminous

luminous: mds: dont print subtrees if they are too big or too many

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #27737 from pdvian/wip-39191-luminous

luminous: mds: drop reconnect message from non-existent session

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #27801 from pdvian/wip-39198-luminous

luminous: mds/server: check directory split after rename.

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #27840 from pdvian/wip-39208-luminous

luminous: mds: initialize cap_revoke_eviction_timeout with conf

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #27848 from ashishkumsingh/wip-39468-luminous

luminous: mds: better output of 'ceph health detail'

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #28432 from ukernel/luminous-39221

luminous: mds: remove superfluous error in StrayManager::advance_delayed()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #28436 from batrick/i39231

luminous: cephfs: kclient: nofail option not supported

Merge pull request #28437 from jtlayton/wip-40160-luminous

luminous: client: clean up error checking and return of _lookup_parent

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #28438 from batrick/i39213

luminous: mds: there is an assertion when calling Beacon::shutdown()

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #28445 from jan--f/ceph_volume_client-use-fs-commands

luminous: ceph_volume_client: remove ceph mds calls in favor of ceph fs calls

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/workunits/rbd: use more recent qemu-iotests that support Bionic

Fixes: http://tracker.ceph.com/issues/24668
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit b184acd218b36b85f5a25678d773c35407a2b745)

qa: use curl in wait_for_radosgw() in util/rgw.py

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 9c6afa3fb570629a57612f250a6a4890a65cbc6e)

Conflicts:
qa/tasks/rgw.py
- rgw task was refactored for mimic

Merge pull request #27085 from pdvian/wip-38719-luminous

luminous: crush/CrushWrapper: ensure crush_choose_arg_map.size == max_buckets

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27636 from smithfarm/wip-39343-luminous

luminous: ceph-objectstore-tool: rename dump-import to dump-export

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #27674 from smithfarm/wip-38873-luminous

luminous: pybind: Rados.get_fsid() returning bytes in python3

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27715 from pdvian/wip-39042-luminous

luminous: osd/PGLog: preserve original_crt to check rollbackability

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #27751 from ashishkumsingh/wip-39431-luminous

luminous: osd/PG: discover missing objects when an OSD peers and PG is degraded

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27810 from pdvian/wip-39204-luminous

luminous: osd: shutdown recovery_request_timer earlier

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27878 from pdvian/wip-39218-luminous

luminous: osd: FAILED ceph_assert(attrs || !pg_log.get_missing().is_missing(soid) || (it_objects != pg_log.get_log().objects.end() && it_objects->second->op == pg_log_entry_t::LOST_REVERT)) in PrimaryLogPG::get_object_context()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27908 from smithfarm/wip-39563-luminous

luminous: mon: Error message displayed when mon_osd_max_split_count would be exceeded is not as user-friendly as it could be

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #28185 from dzafman/wip-39719

luminous: osd: Include dups in copy_after() and copy_up_to()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mon/OSDMonitor: limit MOSDMap message size by bytes

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit dcf1cd00cc6e2595c283b4a73a4de9df50e7330a)

Conflicts:
src/mon/OSDMonitor.cc:
1. replace g_conf() with g_conf, g_conf was replaced
with g_conf() to prepare for the crimson project. and that change was
not backported to luminous. so restore all `g_conf()` to `g_conf` in
this backport.
2. replace std::min() and std::max() with MIN() and
MAX() macros, as we modernize it in an unrelated change. and that
change was not backported to luminous. so let's be conservative, and
restore to the old style macros.

common/options: add osd_map_message_max_bytes

Limit MOSDMap message size by bytes as well as map count.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit e15be00560c1492e0807d0779cfd0d6088396b3e)

Conflicts:
src/common/config_values.h: the changes in this file is applied
to src/common/config.h instead. because, src/common/config_values.h
is extracted in 4718b7cb2fac65b2ac7014f42ba1a10181350f0b. which
happened after luminous, and that commit was not backported.
  also, the OPT_SIZE is changed to OPT_U64, as we don't have OPT_SIZE by
then, the size_t option support was added after luminous, and that
change was not backported. so let's use the uint64_t as an alternative,
as other *bytes settings are also using OPT_U64. and OSDMonitor.cc
will be accessing that setting using the legacy way instead of using
conf.get_val<>() interface, so this change will have no impact on the
fix.

Merge pull request #27982 from cbodley/wip-23237

luminous: rgw: update civetweb submodule to match version in mimic

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mds: reset heartbeat inside big loop

Fixes: https://tracker.ceph.com/issues/40171
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit d90403c8df90ed06564c84bb2dd38c16861cf34e)

 Conflicts:
src/mds/Locker.h
src/mds/MDCache.cc

mds: avoid sending too many osd requests at once after mds restarts
Fixes: http://tracker.ceph.com/issues/40028
Signed-off-by: simon gao <simon29rock@gmail.com>
(cherry picked from commit 78484352f553d6b67f7d5c82384a186455f05537)

Merge pull request #27332 from pdvian/wip-38920-luminous

luminous: rgw: don't crash on missing /etc/mime.types

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27935 from cbodley/wip-39572

luminous: rgw:send x-amz-version-id header when upload files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28332 from cbodley/wip-37497

luminous: rgw: get or set realm zonegroup zone need check user's caps

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28069 from smithfarm/wip-39615-luminous

luminous: civetweb frontend: response is buffered in memory if content length is not explicitly specified

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28350 from smithfarm/wip-39409-luminous

luminous: rgw: inefficient unordered bucket listing

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mds: ignore sessionmap version mismatch if mds_wipe_sessions is set

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 2a20605c0393692bae4780ef3bd0cc684995d028)

 Conflicts:
src/mds/SessionMap.h
src/mds/journal.cc

mds: fix corner case of replaying open sessions

Marking a session dirty may flush all existing dirty sessions. MDS
calls Server::finish_force_open_sessions() for log event that opens
multiple sessions. The function marks sessions dirty one by one. So
sessions opened by a log event may get flushed partially.

When replaying a log event that opens multiple sessions, mds need to
check if some of these sessions have already been flushed.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 277327565fae93fad8e03cc5f27b78c98354c0dc)

 Conflicts:
src/mds/SessionMap.cc
src/mds/SessionMap.h
src/mds/journal.cc

Merge pull request #28519 from ceph/backport-luminous-28060

luminous: ceph-volume: use the Device.rotational property instead of sys_api

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume: skip missing interpreters when running tox tests

This will skip tests for versions of python that don't exist on the node
that's running the tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit f1db0a864814d9b81192d89b6eebb6c624ddec3b)

ceph-volume: use the Device.rotational property instead of sys_api

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1666822

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 8ebff475dfa136c88101ef7364820fe3c5cee547)

Merge pull request #27453 from pdvian/wip-38958-luminous

luminous: rgw-admin: fix data sync report for master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27668 from smithfarm/wip-39360-luminous

luminous: rgw: failed to pass test_bucket_create_naming_bad_punctuation in s3test

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27881 from pdvian/wip-39227-luminous

luminous: rgw_file: save etag and acl info in setattr

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28314 from smithfarm/wip-39180-luminous

luminous: rgw: orphans find perf improvments

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa: fix mdsmap lookup

We were returning the file system structure and not the mdsmap.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0e4027efdc4fd84ae3cf7eef0e2374b3429f111d)

ceph-volume-client: fix command argument order

Fixes: http://tracker.ceph.com/issues/22038
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit cd30d28c6e691c974d6a10b4441d03febfec544f)

qa: use fs rm_data_pool

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit b897ef9445e31004595df4d6243ac8333691c116)

 Conflicts:
doc/cephfs/administration.rst
qa/workunits/cephtool/test.sh
qa/workunits/fs/misc/mkpool_layout_vxattrs.sh
    Changes were dropped.

ceph_volume_client: fix usage of get_mds_map()

get_mds_map() is an instance method, and not a global
function. Use it as such.

Fixes: http://tracker.ceph.com/issues/22524
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 181e3ea6ee8890a66e460f66a4e8c2aa74a9c002)

ceph_volume_client: remove use of obsolete mds cmd

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit cbbdd0da7d40e4e5def5cc0b9a9250348e71019f)

Merge pull request #28493 from tchaikov/luminous-39164

luminous: qa: install python36-{cephfs,rados} instead of python34-*

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa: install python3-{cephfs,rados} instead of python34-*

we install the latest python-rpm-macros on all builders since
https://github.com/ceph/ceph-build/pull/1283 . now that we started
building python36-* after that change, for testing the python3 packages on
CentOS/RHEL 7, we need to install python36-* instead of python34-*.

and after the change of 8ae1947, python36-* now "Provides" python3-*, we
can just install python3-* for fulfill the requirement for testing
python3 cephfs bindings.

Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts: this change is not cherry-picked from master, because,
  in master, we don't install python3 packages after 7e5c85b604.
(cherry picked from commit 6790821afc749b14b1ddac68a0889059419eebb3)

install-deps.sh: install '*rpm-macros'

so `yum-builddep` can have access to the latest macros for preparing the
build dependencies

Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 087ea813a061a1d6858aeae57950e90965f4ae15)

Conflicts:
install-deps.sh: trivial resolution

qa/tasks/ceph_deploy: install python3.6 instead of python3.4 for py3 tests

EPEL7 has switched over to python3.6 as the main python3. and we started
packaging python bindings for python3.6 since
https://github.com/ceph/ceph-build/pull/1283

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 42167b87909c20c9bb4d32bac02b1e01ec94af13)

PendingReleaseNotes: note on python3.6 changes

Fixes: http://tracker.ceph.com/issues/39164
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 77c5ee06311d610f9accd94edf14becdf5820542)

Conflicts:
PendingReleaseNotes: the next release of luminous is 12.2.13
so adapt the cherry-picked change accordingly.

rpm: add "Provides: python3-*" for python packages

so user can install python3-rados, instead of python36-rados, without
specifying the minor version of python. also, we should not break our
teuthology tests with this naming scheme change. for instance, our
cephfs qa suite installs `python3-cephfs` for testing the `cephfs-shell`

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 8ae1947728837bf4a4bcc8e9325aea69ff14b706)

rpm: always use 3 as the default python3_{pkgversion,version}

as we have the rpm macros installed in
https://github.com/ceph/ceph-build/pull/1283 .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 11f0b8d5a8951cb50437f5e87055f3b9405f4556)

rpm: use python 3.6 as the default python3

some of our centos7 jenkins builders are failing to build ceph master and
nautilus branches. because EPEL7 recently switched from python3.4 to
python3.6 as the native python3. see
https://lists.fedoraproject.org/archives/list/epel-announce@lists.fedoraproject.org/message/EGUMKAIMPK2UD5VSHXM53BH2MBDGDWMO/

and one of our BuildRequires, cmake3,
was offered by EPEL7. it also followed the python3.6 switch-over to
rebuild against python3.6. as a result, the cmake3-data-3.13.4-2.el7
started to depend on /usr/bin/python3.6, which is in turn offered by
python36 package. after installing python36 as a dependency of the
updated cmake3. but in cmake, we originally checks for the latest
python3 interpreter if WITH_PYTHON3 is enabled, that's why these
builders which happen to install these updated packages started to fail
when detecting the existence of python3.6 related build dependencies.

as a fix, in d1e83082,
python%{python3_pkgversion}-{devel,setuptools,Cython} are listed as
BuildRequires to reflect this change in EPEL7. before d1e83082, we
hardwired them to python34-*.

but as following analysis puts, there are cases where `yum-builddep`
is inconsistent with `rpmbuild`. as `yum-builddep` changes the how
`python3_pkgversion` and `python3_version` macros are expanded:

- none of the packages installed by `yum-builddep` installs the python3
  related rpm macros, so the system stays with whatever python3 it was
  using. in this case, `rpmbuild` won't complain, as the
  `python3_pkgversion` and `python_version` are consistent before and
  after `yum-builddep`.
- system has python3.4 installed before `yum-builddep`. but
  `yum-builddep` installed python3.6 and also the updated
  `python-rpm-macros` packages, which points `python3_version` and
  `python3_pkgversion` to 3.6 and 36 respectively. in this case,
  `rpmbuild` will complain, because when we run `yum-builddep`,
  `python3_version` was still "3.4".
- system does not have python3 installed before `yum-builddep`. so
  it was using python34 for preparing the "BuildRequires". but some
  of the packages installed by `yum-builddep` installs python36, and
  also the updated `python-rpm-macros` packages, which points
  `python3_version` and `python3_pkgversion` to 3.6 and 36 respectively.
  in this case, `rpmbuild` will complain, because the python36 related
  dependencies are missing. what the system has is python34
  dependencies.
- system does not have python3 installed before `yum-builddep`. so
  it was using python34 for preparing the "BuildRequires". but some
  of the packages installed by `yum-builddep` installs python34, and
  also the updated `python-rpm-macros` packages, which points
  `python3_version` and `python3_pkgversion` to 3.4 and 34 respectively.
  in this case, `rpmbuild` won't complain, as the
  `python3_pkgversion` and `python_version` are also consistent before and
  after `yum-builddep`.

as we cannot tell if the system has python3 or what the python3 version
the system has before `yum-builddep`, so what we can do is to ensure
`rpmbuild` has what it needs to build Ceph. so let's just stick with
python3.6.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 658ca2b0d79022edbc6cb1e488b243844e87bfbe)

cmake: should PYTHON3_VERSION_STRING of libpython3

and make sure the version matches exactly the requested one

in future, we should use FindPython.cmake

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7b15b682b1201929e0ba9efc332ebb8652be40a8)

Conflicts:
src/CMakeLists.txt: trivial resolution

rpm: pass %{python3_version} to cmake

to force cmake to use the python3 and python3 modules for building
python3 bindings

on the debian side, it's okay to continue using "-DWITH_PYTHON3=ON", as

- cmake does normalize "ON" to 3
- debian's cmake extension lives on /usr/lib/python3/dist-packages/
  not in a specific /usr/lib/python3.x/dist-packages directory

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7681a38ce51743a19ec5679d49b362f04e271ddb)

Conflicts:
ceph.spec.in: trivial resolution

cmake: check for libpython of the same version of interpreter

actually cython or python3-cython's dependencies should take care of it.
but we should at least get it right on our side if we check it.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit c961e002d22ab7ceec20f1019f8c558ec9fcce6c)

cmake: use specified python3 version if any

use might have multiple python3 installed, some of them has/have all
dependencies installed and is good enough for building Ceph. we should
not always use the latest python installed in the system and complain that
there is missing dependencies, even if user has installed all the
python3 dependencies for the older python3.

put in other words, if user only installs cython module for python3.4, but
she has both python3.6 and python3.4 in her system. we should not force
her to uninstall python3.6 for installing Ceph.

this change also aligns with MGR_PYTHON_VERSION. i am not applying the
same change to WITH_PYTHON2, because python2 is already stablized. and distros
are not likely to release new python2 releases.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d8c13e86048b7aac458f3eda101f9778e3d45dcf)

Conflicts:
src/CMakeLists.txt: in luminous, WITH_PYTHON3 was "CHECK" by
default. as it's complicatd to support this behavior. it is changed to
"ON" in this change to be consistent with mimic and up. since we always
specify -DWITH_PYTHON3=ON explicitly when building rpm and deb packages,
this change is not visible to our CI or package builders.

Merge pull request #27728 from neha-ojha/wip-39420-luminous

luminous: osd/OSDMap: Replace get_out_osds with get_out_existing_osds

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #27858 from ashishkumsingh/wip-39463-luminous

luminous: msg: output peer address when detecting bad CRCs

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

client: clean up error checking and return of _lookup_parent

ll_lookup_inode can end up getting back 0 from _lookup_parent,
without zeroing out the parent pointer, which ends up remaining
uninitialized.

Fix this by moving most of the sanity checks in _lookup_parent
into ll_lookup_inode, and only have it issue the call to the MDS.

This also allows us to do the checks in a more sane order.

Fixes: http://tracker.ceph.com/issues/40085
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 3ade7c046c6c5eaf20517d0713a805a681128831)

Conflicts:
src/client/Client.cc : std::lock_guard() => Mutex::Locker()
       ceph_assert() => assert()
       dentries.empty() => dn_set.empty()

mds: there is an assertion when calling Beacon::shutdown()

If you run MDSDaemon::init(), an exception occurs that causes MDSDaemon::suicide()--->Beacon::shutdown()--->sender.join();
this problem occurs because sender is just Default-constructed is in Beacon.h(std::thread sender;), there is no call
to Beacon::init() to construct the sender, so the sender has no "joinable"

Fixes: http://tracker.ceph.com/issues/38822
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>
(cherry picked from commit 526bbda6f68873a500c78685a7c39a129d8531e5)

mount.ceph.c: do not pass nofail to the kernel

nofail is a userspace level option and shouldn't be passed to the kernel

Fixes: https://tracker.ceph.com/issues/23262
Signed-off-by: Kenneth Waegeman <kenneth.waegeman@ugent.be>
(cherry picked from commit 51676e602969c5643c31ed26d7c611f43470c8e4)

qa: drop rfc2616 checking on civetweb

Since newer versions of civetweb are also strict on rfc2616 checks let's enforce
strict rfc2616 checks in s3tests

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit f8ed817cccff626c0b63f776c0c5203989c348a1)

Conflicts:
qa/tasks/s3tests.py: also lifecycle -> lifecycle_expiration

mds: open import bounding dirfrags in batch

Fixes: http://tracker.ceph.com/issues/38679
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 1f0dcf2b7f43e93212d56739a9d9d2db771b327f)

 Conflicts:
src/mds/Migrator.cc