doc: update bluestore cache settings and clarify data fraction

Fixes: http://tracker.ceph.com/issues/39522
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 9d8336a7f418fe2bb11361dd74a214403b1e5be7)

Merge pull request #30242 from smithfarm/wip-41489-luminous

luminous: cephfs: client: return -EIO when sync file which unsafe reqs have been dropped

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge pull request #31166 from alfredodeza/wip-rm42292

luminous: qa/ceph-disk: use a Python2.7 compatible version of pytest

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #31172 from dzafman/wip-missed-backports

Luminous corrections for tests

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28502 from smithfarm/wip-40166-luminous

luminous: cephfs: client: ceph.dir.rctime xattr value incorrectly prefixes 09 to the nanoseconds component

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge pull request #29097 from batrick/i40807

luminous: mds: msg weren't destroyed before handle_client_reconnect returned, if the reconnect msg was from non-existent session

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29226 from smithfarm/wip-40163-luminous

luminous: cephfs: mount: key parsing fail when doing a remount

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29229 from smithfarm/wip-40218-luminous

luminous: tests: cephfs: TestMisc.test_evict_client fails

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29829 from batrick/i39691

luminous: mds: check dir fragment to split dir if mkdir makes it oversized.

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29830 from ukernel/luminous-41000

luminous: cephfs: client: unlink dentry for inode with llref=0

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #28701 from batrick/i40286

luminous: qa: remove ubuntu 14.04 testing

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30934 from smithfarm/wip-42039-luminous

luminous: client: _readdir_cache_cb() may use the readdir_cache already clear

Reviewed-by: Venky Shankar <vshankar@redhat.com>

qa/ceph-disk: use a Python2.7 compatible version of pytest

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #30230 from dzafman/wip-network-luminous

luminous: core: Health warnings on long network ping times

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30785 from smithfarm/wip-41919-luminous

luminous: core: osd: scrub error on big objects; make bluestore refuse to start on big objects

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30902 from smithfarm/wip-42138-luminous

luminous: osd: Remove unused osdmap flags full, nearfull from output

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #31020 from smithfarm/wip-42393-luminous

luminous: common/ceph_context: avoid unnecessary wait during service thread shutdown

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31036 from smithfarm/wip-40502-luminous

luminous: core: osd: rollforward may need to mark pglog dirty

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #31049 from vshankar/wip-mds-session-ref

luminous: mds: add reference when setting Connection::priv to existing session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #31091 from smithfarm/wip-42415-luminous

luminous: doc/rbd: s/guess/xml/ for codeblock lexer

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: s/guess/xml/ for codeblock lexer

this change silences the warning of

```
doc/rbd/qemu-rbd.rst:174: WARNING: Pygments lexer name 'guess' is not
known
```

see http://pygments.org/docs/lexers/, we should use "xml" for XML .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit df226da996e468d2707b08eb012d54b4e37ffdc6)

Merge pull request #30425 from smithfarm/wip-41544-luminous

luminous: test/librbd: set nbd timeout due to newer kernels defaulting it on

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30147 from smithfarm/wip-41266-luminous

luminous: rgw: don't throw when accept errors are happening on frontend

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30323 from smithfarm/wip-41713-luminous

luminous: rgw: fix refcount tags to match and update object's idtag

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mds: fix MDSRank::send_message_client_counted()

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 28f35b5dd00c773936f97af9dacc0346c3214213)

mds: add reference when setting Connection::priv to existing session

the bug was introduced by commit 72883956c26 "use intrusive_ptr for
holding Connection::priv"

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 4478e353e35d1a6b217e075bbc1c551c1c009e29)

osd: accident of rollforward may need to mark pglog dirty

refers: https://github.com/ceph/ceph/pull/27015/files#r294114392

Fixes: http://tracker.ceph.com/issues/40403
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>
(cherry picked from commit 35cb184becd6562edd240553dfa50f47bb120b7f)

OSD: rollforward may need to mark pglog dirty

if we rollforward at the end of PG::activate(), we may advance the *crt*,
but we did not mart the log dirty, this means we will not update the crt
within the transaction of rollforward, so it is inconsistent.

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>
(cherry picked from commit 10d0990dc69310864b4845ee57b32610a642464f)

Conflicts:
src/osd/PGLog.h
- dfbe5e070cc978253abcb30b86de5faa7e6a1efc is not being backported
- retain !touched_log as part of conditional in is_dirty()

Merge pull request #30981 from tchaikov/wip-luminous-42361

luminous: build/ops: python3-cephfs should provide python36-cephfs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30118 from smithfarm/wip-41439-luminous

luminous: rbd-mirror: ignore errors relating to parsing the cluster config file

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/ceph_context: avoid unnecessary wait during service thread shutdown

Fixes: https://tracker.ceph.com/issues/42332
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e8249d3b28f8789b2d4aca6fb75c75103a5cbea1)

Conflicts:
src/common/ceph_context.cc
- luminous uses Mutex::Locker instead of std::unique_lock

Merge pull request #29875 from trociny/wip-41421-luminous

luminous: qa/workunits/rbd: stress test `rbd mirror pool status --verbose`

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/workunits/rbd: stress test `rbd mirror pool status --verbose`

Fixes: https://tracker.ceph.com/issues/40923
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 0866a2456b773843537681c768d93d1c9e7718a4)

Conflicts:
qa/workunits/rbd/rbd_mirror_helpers.sh:
          - trivial resolution in test_status_in_pool_dir
          - adding missing XMLSTARLET definition (added in master by 917f8a0)

Merge pull request #30298 from smithfarm/wip-40082-luminous

luminous: osd: Better error message when OSD count is less than osd_pool_default_size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30132 from smithfarm/wip-40735-luminous

luminous: doc/rgw: document use of 'realm pull' instead of 'period pull'

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30071 from smithfarm/wip-40138-luminous

luminous: doc: describe metadata_heap cleanup

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30690 from smithfarm/wip-42106-luminous-follow-on

luminous: build/ops: admin/build-doc: use python3 (follow-on fix)

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph.spec.in: provide python2-<modname>

to be consistent with other python2 packages, and their python3
counterparts

the `python_provide` macro is offered by `python-rpm-macros` package,
which is in turn required by python*-devel

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit fb6290b1fab0bd8a73043f2d68210c010e2fb425)

ceph.spec.in: use python_provide macro

our python3 bindings are now named `python3-<modname>` after python3 is
now maintained by RHEL/CentOS instead EPEL. to help the users using
`python36-<modname>`, we should "Provide" `python36-<modname>`.

the `python_provide` macro is offered by `python-rpm-macros` package,
which is in turn required by python*-devel. and we do install
`python36-devel` in install-deps.sh, and install `python3-devel` in
ceph-*build/build/setup_rpm

see also
https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_provides

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 50b19e673d8200306c3e36d1abaec414a3d336b9)

Merge pull request #30608 from vumrao/wip-vumrao-luminous-rados-tools-list-objects-in-a-pg

luminous: rados tools list objects in a pg

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30905 from smithfarm/wip-42153-luminous

luminous: mon/OSDMonitor: trim not-longer-exist failure reporters

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30922 from vumrao/wip-vumrao-42241

luminous: osd/PG: Add PG to large omap log message

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #30926 from smithfarm/wip-42127-luminous

luminous: osd/OSDMap: do not trust partially simplified pg_upmap_item

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30855 from dillaman/wip-39673-luminous

luminous: qa/workunits/rbd: wait for rbd-nbd unmap to complete

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #30856 from dillaman/wip-40499-luminous

luminous: rbd: use the ordered throttle for the export action

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #30857 from dillaman/wip-40880-luminous

luminous: rbd: Reduce log level for cls/journal and cls/rbd expected errors

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #30858 from dillaman/wip-41621-luminous

luminous: tools/rbd-ggate: close log before running postfork

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #29202 from smithfarm/wip-40266-luminous

luminous: common: data race in OutputDataSocket

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28989 from smithfarm/wip-39537-luminous

luminous: osd/ReplicatedBackend.cc: 1321: FAILED assert(get_parent()->get_log().get_log().objects.count(soid) && (get_parent()->get_log().get_log().objects.find(soid)->second->op == pg_log_entry_t::LOST_REVERT) && (get_parent()->get_log().get_log().object

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29564 from k0ste/luminous_backports4

luminous: osd/bluestore: Actually wait until completion in write_sync

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30271 from smithfarm/wip-41733-luminous

luminous: osd: clear PG_STATE_CLEAN when repair object

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #30722 from smithfarm/wip-epel-release-snafu-luminous

luminous: build/ops: backport miscellaneous install-deps.sh and ceph.spec.in fixes from master

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/standalone/ceph-helpers: fix test_wait_for_clean

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 30fc7f5e972bd41537c92eb135c9763b46dd09d2)

Conflicts:
qa/standalone/ceph-helpers.sh
- mimic does not have separate WAIT_FOR_CLEAN_TIMEOUT

qa/standalone: make sure an osd is running before create_rbd_pool

'rbd pool init' now does IO.  Drop the pool, or change the pool size to 1.

Fixes: http://tracker.ceph.com/issues/38585
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit cba0483b090a6370390e8ed88d3f9d7767d56be5)

qa: specify filestore for misc tests

Signed-off-by: Vikhyat Umrao <vumrao@redhat.com>
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 8a694fc2f9e75864e064fe591feda9fab943c15e)

qa/standalone/ceph-helpers: resurrect all OSD before waiting for health

address the regression introduced by e62cfceb
in e62cfceb, we wanted to test the newly introduced TOO_FEW_OSDS
warning, so we increased the number of OSD to the size of pool, so if
the number of OSD is less than pool size, monitor will send a warning
message.

but we need to bring all OSDs back if we are expecting a healthy
cluster. in this change, all OSDs are resurrect before
`wait_for_health_ok`.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit cdba0f14201a327e824b6ec48e014233281dee2d)

qa/standalone/ceph-helpers: fix health_ok test

Stopping the osd daemon won't reliably get you HEALTH_WARN or ERR; you have
to make sure it is also marked down.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit dcdca44aa45e7dc5aab56da26caec644e32024a6)

mon/PGMap: enable/disable TOO_FEW_OSDS warning with an option

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3b74fbc93d32d6cc367049d3bcbe83d3837abba5)

Conflicts
        src/mon/PGMap.cc
- cct->_conf->get_val in luminous
src/common/options.cc
- no mon_max_snap_prune_per_epoch in luminous
- in luminous, osd_pool_default_size is INT_T (int64_t). In master, it is UINT_T
  (uint64_t). So I changed the type to int64_t.

qa/standalone: remove osd_pool_default_size in test_wait_for_health_ok

Signed-off-by: zjh <jhzeng93@foxmail.com>
(cherry picked from commit e62cfceb9549e806f774ef3da549efd9f4d90171)

osd: Better error message when OSD count is less than osd_pool_default_size

Fixes: http://tracker.ceph.com/issues/38617
Signed-off-by: zjh <jhzeng93@foxmail.com>
(cherry picked from commit 94237d36932b27961afffbe65d433af26c3fe513)

Conflicts:
doc/rados/operations/health-checks.rst
- trivial
        src/mon/PGMap.cc
- cct->_conf->get_val in luminous

client: _readdir_cache_cb() may use the readdir_cache already clear

Calling function _readdir_cache_cb() means to get dir information from the cache, but in the while loop,
the contents of readdir_cache are not static.

For example, _getattr() will trigger insert_trace to update readdir_cache, such as:
1. insert_trace()->clear_dir_complete_and_ordered() or
2. insert_trace()->add_update_inode()->add_update_cap()-> check_cap_issue()->clear_dir_complete_and_ordered()
So, there is a possibility that dn is empty after the _getattr() call.

Fixes: http://tracker.ceph.com/issues/41148
Signed-off-by: huanwen ren <ren.huanwen@zte.com.cn>
(cherry picked from commit 404c06784e46016703104db59ea73f36d09b578b)

osd/OSDMap: do not trust partially simplified pg_upmap_item

If we simplified a partially no-op pg_upmap_item, we shall still
continue to verify that the remaining part is valid.
The bug is introduced by 02e5499b350bcd7d9eac98b2072052a9a4a1f535,
before which we always validate the correctness of a pg_upmap_item
before trying to cancel or simplify it.

Fixes: https://tracker.ceph.com/issues/42052
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 4196b13283144de966eeba40e6765f10b254dac6)

Conflicts:
src/osd/OSDMap.cc

mon/OSDMonitor: trim no-longer-exist failure reporters

remove the report from failure_info in OSDMonitor::check_failure(), if
the reporter does not exist in the osdmap any longer. otherwise, we will
run into assert() failure when trying to lookup the osd in osdmap using
`osdmap.get_xinfo()`.

Fixes: https://tracker.ceph.com/issues/41680
Signed-off-by: NancySu05 <su_nan@inspur.com>
(cherry picked from commit 99ccc166ede5f296b86abfe9db0a3d597100edde)

Conflicts:
src/mon/OSDMonitor.cc
- ceph_assert, g_conf()
- account for C++17ism by moving iter initialization out of if statement

osd/PG: Add PG to large omap log message

Fixes: https://tracker.ceph.com/issues/41359
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit a71d51bfa154a278a350e6dbeb842fb3585b74f5)

osd: Remove unused osdmap flags full, nearfull from output

Fixes: https://tracker.ceph.com/issues/42015
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d9c645d7d8c7355c8aaf3ff9121f2b8fc3b1290a)

osd/OSDMap: misleading message in print_oneline_summary()
The output scared people as showing so many osds were full/nearfull

Improve the output to make the message clearer

Fixes: http://tracker.ceph.com/issues/22350
Signed-off-by: Gu Zhongyan <guzhongyan@360.cn>
(cherry picked from commit 7004b35f71a657a0d9479b7cc46acc210de303d4)

Merge pull request #29713 from cbodley/wip-41322

luminous: rgw: datalog/mdlog trim commands loop until done

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

tools/rbd-ggate: close log before running postfork

Otherwise the assert in Log:open() will trigger since
all data is kept during the fork and also the `is_started` flag

Fixes: https://tracker.ceph.com/issues/41592
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit 21bdd4c6ecd24b2a7dbf8691705f228ac11481ad)

Conflicts:
src/tools/rbd_ggate/main.cc : Resolved in do_map

cls/journal: reduce verbosity of debug logs for non-errors

Fixes: http://tracker.ceph.com/issues/40865
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7779860d38725636fb5438e2f52e432994dc3926)

cls/rbd: reduce verbosity of debug logs for non-errors

Fixes: http://tracker.ceph.com/issues/40865
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7a6a2aefd72faf1da6594ca6bc5f8bf14a387edd)

rbd: use the ordered throttle for the export action

This allows exports to STDOUT to use multiple concurrent operations
and also fixes a potential race condition with concurrent callbacks
and file seeking.

Fixes: http://tracker.ceph.com/issues/40435
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 751bf6957020ad02b3a29b005b9792b9c6547f96)

Conflicts:
src/tools/rbd/action/Export.cc
- mimic has g_conf->get_val and int64_t "rbd_concurrent_management_ops"

qa/workunits/rbd: wait for rbd-nbd unmap to complete

The "unmap" request is asynchronous, so wait for a short amount
of time for the "rbd-nbd" daemon process to exit.

Fixes: http://tracker.ceph.com/issues/39598
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 87278548a5a476692511b76b661278501bf72429)

Conflicts:
qa/workunits/rbd/rbd-nbd.sh : Resolved for map/unmap namespace test
(cherry picked from commit b9e4644140ce9538a5162e466615d7e6b53efa7a)

rbd-nbd: output format support for list-mapped command

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit e372e9492bb7f535c0f15a078f1eab2de3fad113)

rbd-nbd: support optionally setting device timeout

Fixes: http://tracker.ceph.com/issues/22333
Signed-off-by: Mykola Golub <to.my.trociny@gmail.com>
(cherry picked from commit bad51ffe7fbfe8c2e33eb0db2ef7acb89a03bd1d)

rbd-nbd: allow to unmap by image or snap spec

(to match the behaviour of krbd unmap)

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 02e211c2822db7d3fb3792ce18cdb217c5de24ee)

qa/workunits/rbd: bench-write is part of exclusive option test

Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 47e38ca1b9f748e2f2ce1c035cd3f982d68eb492)

Merge pull request #29269 from smithfarm/wip-40135-luminous

luminous: rgw: the Multi-Object Delete operation of S3 API wrongly handles the Code response element

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30345 from dillaman/wip-41772-luminous

luminous: common/config: hold lock while accessing mutable container

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #27974 from iain-buclaw-sociomantic/luminous-rgw-cls-bi-list-log-level

luminous: rgw: cls/rgw: raise debug level of bi_log_iterate_entries output

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27992 from smithfarm/wip-39016-luminous

luminous: rgw: unable to cancel reshard operations for buckets with tenants

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29153 from davebiffuk/s3_policy_tracker_39272

luminous: rgw: policy fix for nonexistent objects

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29170 from cbodley/wip-qa-rgw-tempest-luminous

luminous: qa/rgw: remove incomplete rgw/tempest suite from luminous

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #29271 from smithfarm/wip-40143-luminous

luminous: rgw: multisite: 'radosgw-admin bucket sync status' should call syncs_from(source.name) instead of id

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29295 from dillaman/wip-40233-luminous

luminous: rbd/action: fix error getting positional argument

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #29826 from linuxbox2/wip-luminous-apiname

luminous: rgw: fix issue for CreateBucket with BucketLocation param

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29874 from trociny/wip-41285-luminous

luminous: rbd-mirror: don't overwrite status error returned by replay

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30375 from smithfarm/wip-41480-luminous

luminous: rgw: make dns hostnames matching case insensitive

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30666 from smithfarm/wip-41104-luminous

luminous: rgw: mitigate bucket list with max-entries excessively high

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #30677 from sobelek/wip-42049-luminous

luminous: ceph-volume: fix warnings raised by pytest

osd/PG: scrub error when objects are larger than osd_max_object_size

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 5928fe8ca08f9e307c0353bd81753cf263e33d90)

Conflicts:
qa/standalone/scrub/osd-scrub-repair.sh
- luminous does not have "manifest objects"

os/bluestore: refuse to mkfs or mount if osd_max_object_size >= MAX_OBJECT_SIZE

BlueStore has its own object size limit (2^32-1).  Make sure the cluster
limit is below that or refuse to mkfs or mount.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 377fdb1484e9ed34c3d46d2a9b99df926531c675)

ceph.spec.in: s/pkgversion/version_nodots/

`python3_pkgversion` is now defined as 3, while we don't have packages
like python3-Cython yet in EPEL7. but we do have `python36-Cython`. so
let's use `python3_version_nodots` instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts:
ceph.spec.in

install-deps: do not install if rpm already installed

there is chance that the system already have epel-release-latest-7.noarch.rpm
installed, in that case, install-deps.sh should not fail.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 45a4173bcfe15a9bf56f6aeb6e6c35386a00d1c5)

install-deps.sh: add EPEL repo for non-x86_64 archs as well

Fixes: https://tracker.ceph.com/issues/42023
Signed-off-by: Kefu Chai <kchai@redhat.com>
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 49d740118f9a0ee6b6bb82954630c654946f1751)

Conflicts:
install-deps.sh

install-deps.sh: use latest setuptools

this change reverts "install-deps.sh: workaround setuptools' dependency on six"

This reverts commit 30ce5e55460e333e1240704112a16fae68b3325f.

tl;dr: this change addresses the failures of "make check" runs on arm64
builders when they try to build `mgr-dashboard-test-venv` target.

long story: without this change, we will fail to pull in
setuptools >= 36, and as a result pip will fail to import
`setuptools.build_meta` in `pip/_vendor/pep517/_in_process.py`. and will
a `BackendUnavailable` exception thrown by `_call_hook()` in
`pip/_vendor/pep517/wrappers.py`. since the issue addressed by 30ce5e55
has been addressed since setuptools >= 36.0.1, we should be safe to
upgrade to the latest setuptools now.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 8b55b47cc75e43ef683cd230f792cd92f1c46fa7)

install-deps.sh: skip unavailable repos

as centos-sclo-rh-source leads us to 404 at this moment. and we are not
using the source repo for building ceph. so we can just skip any
unavailable repo.

Fixes: http://tracker.ceph.com/issues/37707
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit aa2b9e9a643441d9a2891e17920677a7725754c4)

install-deps: install setuptools before upgrading virtualenv

this should address the failures when running install-deps.sh, like

Downloading/unpacking virtualenv
  Running setup.py egg_info for package virtualenv
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown
distribution option: 'python_requires'
      warnings.warn(msg)
    error in virtualenv setup command: 'extras_require' must be a
dictionary whose values are strings or lists of strings containing valid
project/version requirement specifiers.
    Complete output from command python setup.py egg_info:
    /usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown
distribution option: 'python_requires'

  warnings.warn(msg)

error in virtualenv setup command: 'extras_require' must be a dictionary
whose values are strings or lists of strings containing valid
project/version requirement specifiers.

this only happens on very old virtualenv shipped with RHEL7.4

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 95386752ab2635f56e1d38b50de7144e2135b323)

build/ops: unify command substitution in install-deps.sh

The $() form is preferable to `` because folks (like me) might be using
` as a keyboard shortcut to GNU Screen, causing havoc to ensue whenever
copy-pasting the ` character.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit e0042dd617269934bf51efee5bab7e4485875576)

Conflicts:
install-deps.sh
- include a part of b6d9464806b74c9f40e03258a004921f472d122e

install-deps: check the exit status for the $builddepcmd

in some case, the $builddepcmd will failed without any "error:" output.
so we should check the exit status to handle it.

Signed-off-by: Yunchuan Wen <yunchuan.wen@kylin-cloud.com>
(cherry picked from commit 46e33d15a4132772a6bd11cd49f4181081bcdd55)

install-deps: fix installing gcc on ubuntu when no old compiler

Signed-off-by: Tomasz Setkowski <tom@ai-traders.com>
(cherry picked from commit 0d8d0cf6f7594f4f97677a2069998331998bdfa9)