pybind/mgr/rbd_support: use image ids to detect duplicate tasks

This helps to to avoid the case where new tasks were not being scheduled
when an image name was re-used after having a task created under the
same name.

Fixes: https://tracker.ceph.com/issues/41032
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29408 from neha-ojha/wip-bs-fsck-mnt

common/options.cc: change default value of bluestore_fsck_on_mount_deep to false

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #28637 from linuxbox2/wip-rgw-blorderflat

rgw: s/std::map/boost::container::flat_map/ cls_bucket_list_ordered

rgw: s/std::map/boost::container::flat_map/ cls_bucket_list_ordered
(RGWRados and CLS).

Probably faster, allocating less.  Definitely not slower.

Examples from single-OSD vstart.sh cluster, Ceph built at -O2

BEFORE

[mbenjamin@lemon python]$ time  s3cmd -c s3cfg_userx ls  s3://DOCREQUEST_750/CSV/SUB1/ > /dev/null

real 4m48.991s
user 3m45.260s
sys 0m7.174s

(2nd run)

radosgw
Samples: 81K of event 'cycles:ppp', 4000 Hz, Event count (approx.): 3189324729
Overhead  Shared Object         Symbol
   7.06%  libtcmalloc.so.4.5.1  [.] tcmalloc::CentralFreeList::FetchFromOneSpans
   6.85%  libstdc++.so.6.0.25   [.] std::__ostream_insert<char, std::char_traits<char> >
   6.15%  librados.so.2.0.0     [.] ceph::buffer::v14_2_0::list::iterator_impl<true>::copy
   4.12%  librados.so.2.0.0     [.] ceph::buffer::v14_2_0::ptr::copy_out
   4.11%  libstdc++.so.6.0.25   [.] std::basic_streambuf<char, std::char_traits<char> >::xsputn
   3.49%  libc-2.27.so          [.] __memmove_avx_unaligned_erms
   3.33%  libtcmalloc.so.4.5.1  [.] tc_deletearray_aligned_nothrow
   3.04%  radosgw               [.] std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pa
   2.46%  radosgw               [.] std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare
   2.45%  libtcmalloc.so.4.5.1  [.] operator new[]
   2.39%  libstdc++.so.6.0.25   [.] std::ostream::sentry::sentry
   2.36%  radosgw               [.] std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pa
   2.07%  librados.so.2.0.0     [.] ceph::buffer::v14_2_0::list::iterator_impl<true>::advance
   1.94%  libc-2.27.so          [.] __memcmp_avx2_movbe
   1.93%  radosgw               [.] std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pa
   1.85%  librados.so.2.0.0     [.] ceph::buffer::v14_2_0::list::iterator_impl<true>::copy
   1.79%  radosgw               [.] rgw_bucket_dir::decode
   1.76%  libceph-common.so.0   [.] operator<<
   1.42%  radosgw               [.] ceph::decode
   1.35%  libstdc++.so.6.0.25   [.] std::_Rb_tree_insert_and_rebalance
   1.33%  libtcmalloc.so.4.5.1  [.] tcmalloc::CentralFreeList::ReleaseToSpans
   1.31%  librados.so.2.0.0     [.] std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate
   1.24%  [kernel]              [k] copy_user_enhanced_fast_string
For a higher level overview, try: perf top --sort comm,dso

osd

Samples: 23K of event 'cycles:ppp', 4000 Hz, Event count (approx.): 5059851086
Overhead  Shared Object         Symbol
   8.57%  libc-2.27.so          [.] vfprintf
   4.90%  ceph-osd              [.] ceph::logging::Log::_flush
   4.66%  libc-2.27.so          [.] _IO_default_xsputn
   3.49%  libtcmalloc.so.4.5.1  [.] operator new[]
   2.93%  ceph-osd              [.] StackStringBuf<4096ul>::xsputn
   2.70%  libstdc++.so.6.0.25   [.] std::__ostream_insert<char, std::char_traits<char> >
   2.11%  libpthread-2.27.so    [.] __pthread_mutex_unlock_usercnt
   1.91%  libc-2.27.so          [.] __memmove_avx_unaligned_erms
   1.75%  libstdc++.so.6.0.25   [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<unsigned l
   1.64%  libc-2.27.so          [.] _itoa_word
   1.58%  libtcmalloc.so.4.5.1  [.] tc_deletearray_aligned_nothrow
   1.39%  libstdc++.so.6.0.25   [.] std::ostream::sentry::sentry
   1.33%  [kernel]              [k] stackleak_erase
   1.27%  [kernel]              [k] copy_user_enhanced_fast_string
   1.22%  libc-2.27.so          [.] __strlen_avx2
   1.15%  ceph-osd              [.] ceph::buffer::v14_2_0::list::append
   1.12%  libstdc++.so.6.0.25   [.] std::ostream::_M_insert<unsigned long>
   1.10%  ceph-osd              [.] ceph::buffer::v14_2_0::ptr::append
   1.04%  libpthread-2.27.so    [.] __pthread_mutex_lock
   1.01%  libc-2.27.so          [.] __tz_convert
   0.94%  [kernel]              [k] entry_SYSCALL_64
   0.94%  ceph-osd              [.] ceph::buffer::v14_2_0::list::iterator_impl<true>::copy
   0.93%  ceph-osd              [.] ceph::logging::Log::submit_entry
For a higher level overview, try: perf top --sort comm,dso

AFTER

[mbenjamin@lemon python]$ time  s3cmd -c s3cfg_userx ls  s3://DOCREQUEST_750/CSV/SUB1/ > /dev/null

real 4m51.488s
user 3m36.785s
sys 0m5.689s

(1st run)

radosgw

Samples: 52K of event 'cycles:ppp', 4000 Hz, Event count (approx.): 4426952205
Overhead  Shared Object         Symbol
   6.11%  librados.so.2.0.0     [.] ceph::buffer::v14_2_0::list::iterator_impl<true>::copy
   5.83%  libstdc++.so.6.0.25   [.] std::__ostream_insert<char, std::char_traits<char> >
   3.89%  radosgw               [.] rgw_bucket_dir::decode
   3.73%  radosgw               [.] rgw_bucket_dir_entry::rgw_bucket_dir_entry
   3.68%  librados.so.2.0.0     [.] ceph::buffer::v14_2_0::ptr::copy_out
   3.37%  libstdc++.so.6.0.25   [.] std::basic_streambuf<char, std::char_traits<char> >::xsputn
   3.14%  libtcmalloc.so.4.5.1  [.] tcmalloc::CentralFreeList::FetchFromOneSpans
   2.55%  libc-2.27.so          [.] __memmove_avx_unaligned_erms
   2.22%  libtcmalloc.so.4.5.1  [.] tc_deletearray_aligned_nothrow
   2.02%  libstdc++.so.6.0.25   [.] std::ostream::sentry::sentry
   1.79%  librados.so.2.0.0     [.] ceph::buffer::v14_2_0::list::iterator_impl<true>::advance
   1.77%  [kernel]              [k] n_tty_write
   1.74%  libc-2.27.so          [.] vfprintf
   1.71%  libtcmalloc.so.4.5.1  [.] operator new[]
   1.65%  librados.so.2.0.0     [.] ceph::buffer::v14_2_0::list::iterator_impl<true>::copy
   1.58%  radosgw               [.] ceph::decode
   1.38%  librados.so.2.0.0     [.] std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate
   1.22%  [kernel]              [k] stackleak_erase
   1.12%  libceph-common.so.0   [.] operator<<
   1.09%  libc-2.27.so          [.] _IO_default_xsputn
   1.06%  libc-2.27.so          [.] __memcmp_avx2_movbe
   1.04%  [kernel]              [k] copy_user_enhanced_fast_string
   0.93%  librados.so.2.0.0     [.] std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append
For a higher level overview, try: perf top --sort comm,dso

osd

Samples: 134K of event 'cycles:ppp', 4000 Hz, Event count (approx.): 26819176020
Overhead  Shared Object         Symbol
   8.82%  libc-2.27.so          [.] vfprintf
   6.88%  ceph-osd              [.] ceph::logging::Log::_flush
   4.80%  libc-2.27.so          [.] _IO_default_xsputn
   4.15%  libpthread-2.27.so    [.] __pthread_mutex_unlock_usercnt
   2.69%  ceph-osd              [.] StackStringBuf<4096ul>::xsputn
   2.54%  libstdc++.so.6.0.25   [.] std::__ostream_insert<char, std::char_traits<char> >
   2.40%  libtcmalloc.so.4.5.1  [.] operator new[]
   2.39%  libc-2.27.so          [.] __memmove_avx_unaligned_erms
   2.13%  libcls_rgw.so.1.0.0   [.] boost::container::dtl::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<ch
   1.61%  libtcmalloc.so.4.5.1  [.] tc_deletearray_aligned_nothrow
   1.54%  libc-2.27.so          [.] _itoa_word
   1.49%  libstdc++.so.6.0.25   [.] std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >::_M_insert_int<unsigned l
   1.41%  libpthread-2.27.so    [.] __pthread_mutex_lock
   1.41%  ceph-osd              [.] ceph::logging::Log::submit_entry
   1.23%  libstdc++.so.6.0.25   [.] std::ostream::sentry::sentry
   1.19%  libc-2.27.so          [.] __tz_convert
   1.08%  libc-2.27.so          [.] __strlen_avx2
   1.08%  [kernel]              [k] stackleak_erase
   1.06%  libcls_rgw.so.1.0.0   [.] rgw_bucket_list
   1.02%  ceph-osd              [.] ceph::buffer::v14_2_0::list::append
   0.99%  ceph-osd              [.] ceph::buffer::v14_2_0::ptr::append
   0.96%  [kernel]              [k] copy_user_enhanced_fast_string
   0.93%  [kernel]              [k] entry_SYSCALL_64
For a higher level overview, try: perf top --sort comm,dso

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #25962 from greenx/master

rgw: swift: bugfix: https://tracker.ceph.com/issues/37765

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29401 from jan--f/c-v-inventory-parttype

ceph-volume: detect ceph-disk osd if PARTLABEL is missing

Merge pull request #29046 from p-na/wip-pna-disable-user

mgr/dashboard: Provide user enable/disable capability

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29088 from votdev/issue_24662

mgr/dashboard: Allow disabling redirection on standby Dashboards

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #29379 into master

* refs/pull/29379/head:
os/bluestore: s/align_down/p2align/

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #28881 into master

* refs/pull/28881/head:
os/Transaction: dump alloc hint flags in op

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29349 from smithfarm/wip-40975

rpm: put librgw lttng SOs in the librgw-devel package

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29397 from wjwithagen/wjw-fix-malloc.h

common: use of malloc.h is deprecated

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29396 from wjwithagen/wjw-fix-boost-1.70

cmake: update FindBoost.cmake

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29181 from liewegas/wip-rgw-meta-pool-priority

rgw: add rgw_rados_pool_recovery_priority (default 5)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28587 from zhangsw/rgw-bugfix-versioning

rgw: fix a bug that lifecycle expiraton generates delete marker continuously

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28789 from taodd/rgw-fix-remove-bucket

rgw: fix drain handles error when deleting bucket with bypass-gc option

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #29178 from ivancich/wip-small-rgw-efficiency

rgw: add a small efficiency

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29179 from ivancich/wip-bucket-list-excessive-max-entries

rgw: mitigate bucket list with max-entries excessively high

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29240 from mdw-at-linuxbox/wip-master-rgw-swift-metadata-versioning

rgw: Fix bucket versioning vs. swift metadata bug.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28331 from linuxbox2/wip-rgw-uinfo-by-acc

rgw: permit rgw-admin to populate user info by access-key

Merge pull request #29413 from tchaikov/wip-remove-test/python

test/python: remove stale tests

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29445 from ricardoasmarques/fix-doc-rbd-namespace

doc: Fix rbd namespace documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: Fix rbd namespace documentation

Pool and namespace should be separated by a slash.

Fixes: https://tracker.ceph.com/issues/41046
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #29446 from tchaikov/wip-cmake-tox-test

cmake: add add_tox_test()

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29423 from rhcs-dashboard/wip-41023-master

doc: Show Jenkins commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29297 from iotcg/starlingx

ceph-objectstore-tool: set log date format

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: fall back to PARTTYPE if PARTLABEL is empty

In some cases ceph-disk does not populate PARTLABEL for wal and db
partitions. This commit adds the assumption that the empty string is a
valid label and falls back to identifying those disks by PARTTYPE.

Fixes: https://tracker.ceph.com/issues/40917
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: adjust tests for empty PARTLABEL fields

It seems like ceph-disk sometimes creates OSDs where the wal and db
partitions have no PARTLABEL set. This commit alters tests so that the
empty string is considered a valid label.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

cmake: add add_tox_test()

to consolidate the tox tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29364 from capri1989/doc-pg-num

doc: pg_num should always be a power of two

Reviewed-By: Jan Fajerski <jfajerski@suse.com>
Reviewed-By: Sage Weil <sage@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: removing dead code in upgrade user database

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: tests: fix test_access_control due to the merge of 79b9021

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: frontend: use checkIcon for enabled column in user list

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: frontend: use checkbox for enabled field in user form

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Provide user enable/disable capability in the frontend

Fixes: http://tracker.ceph.com/issues/25229
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Provide user enable/disable capability in the backend

Fixes: http://tracker.ceph.com/issues/25229
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #29063 from sebastian-philipp/orchestrator-get_feature_set

mgr/orchestrator: Split *_stateless_service and add get_feature_set

Merge pull request #29131 from sebastian-philipp/rook-caching

mgr/rook: Add caching for the Dashboard

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #29334 into master

* refs/pull/29334/head:
qa/tasks/mgr/dashboard/test_mgr_module: adjust expected schema
mgr/telemetry: separate out cluster config vs running daemons
mgr/telemetry: include any config options that are customized

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge PR #29340 into master

* refs/pull/29340/head:
mgr/telemetry: specify license when opting in

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #29419 from vshankar/wip-fix-rbd-support

pybind/rbd: fix call to unregister_osd_perf_queries

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

common/options.cc: common/options.cc: change default value of bluestore_fsck_on_umount_deep to false

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29406 from yuriw/wip-yuriw-upgrade-rwg-master_2

qa/tests: added rgw into upgrade sequence to improve coverage

Merge pull request #29428 from yuriw/wip-yuriw-crontab-master

qa/tests: added mimic-x on master run

Merge PR #29104 into master

* refs/pull/29104/head:
qa/cephfs: test case for timeout config of individual session
mds: add command that config individual client session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29430 from uweigand/fix-no-boost-context-build

rgw: tests: Fix building with -DWITH_BOOST_CONTEXT=OFF

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29412 into master

* refs/pull/29412/head:
doc: mds-config-ref: update 'mds_log_max_segments' value.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29355 into master

* refs/pull/29355/head:
mgr/volumes: set uid/gid of FS client's mount as 0/0

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29133 from ricardoasmarques/prevent-iscsi-target-delete-with-sessions

mgr/dashboard: Prevent deletion of iSCSI IQNs with open sessions

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #29388 into master

* refs/pull/29388/head:
qa: sleep briefly after resetting kclient

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge PR #29377 into master

* refs/pull/29377/head:
mds: add perf counter for finisher of MDSRank

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: mitigate bucket list with max-entries excessively high

When listing a bucket with radosgw-admin, the user can specify the
maximum number of entries. That number can be unreasonably large, and
can affect the performance and memory availability. For example:

    radosgw-admin bucket list --bucket mybucket1 --max-entries=10000000

This has the potential for creating large data structures at multiple
levels in the the call stack of the radosgw(-admin) process,
potentially causing the process to run out of memory. This change
limits the maximum number of entries requested in all but the high
level code to help mitigate this issue.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

qa/tests: added rgw into upgrade sequence to improve coverage

added rgw into upgrade sequence to improve coverage - splits
See https://github.com/ceph/ceph/pull/29234 https://github.com/ceph/ceph/pull/29282

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw: tests: Fix building with -DWITH_BOOST_CONTEXT=OFF

Attempting to build with -DWITH_BOOST_CONTEXT=OFF will result in
a "conflicting declaration" error when building
test/rgw/test_rgw_dmclock_scheduler.cc

Fixed by avoiding the boost/asio/spawn.hpp include in that case.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

qa/tests: added mimic-x on master run

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc: Address further comments on choosing pg_num

Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>

doc: adjust examples to use 2^n pg_num

The examples used pg_num set to 300 or 1000. This misled some users.

Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>

common: use of malloc.h is depricated

And the file no longer exests on FreeBSD.
Usage is only advised for the non-posix parts that are declared
in the Linux variant

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #29405 from wjwithagen/wjw-fix-env-vs-args.t

test/cli: Make the ceph-conf test more liberal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29414 from tchaikov/wip-mgr-api-test

pybind/mgr: install setuptools >= 12

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #27754 from jan--f/c-v-validate-device-lists

ceph-volume: batch ensure device lists are disjoint

doc: Show Jenkins commands

Fixes: https://tracker.ceph.com/issues/41023
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

test/cli: Make the ceph-conf test more liberal

The tests look for the exact error text.
But they are OS dependant, and thus on FreeBSD the output is
slightly different:
2019-07-30T19:04:16.581+0000 80bc48000 -1 parse_file: filesystem error: in file_size: No such file or directory [from-env]

This stems from the fact that different OSes use different
standard libraries: libc++ <> libstd++

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

pybind/rbd: fix call to unregister_osd_perf_queries

.. should be invoked as self.unregister_osd_perf_queries()
rather than self.module.unregister_osd_perf_queries().

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29318 from tspmelo/wip-backport-component

script/ceph-backport.sh: Allow to set component label

pybind/mgr: install setuptools >= 12

otherwise we have
```
  Running setup.py egg_info for package apache-libcloud

    libcloud/utils/py3.py:76: UserWarning: Missing backports.ssl_match_hostname package
      warnings.warn("Missing backports.ssl_match_hostname package")
    Traceback (most recent call last):
...
      File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66, in _warn_if_setuptools_outdated

...
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
...
Command python setup.py egg_info failed with error code 1 in /tmp/tmp.i95Fg82Ea4/venv/build/apache-libcloud
```

and more important thing is, setuptools >= 12 allows us to install the
python modules in venv, without uninstalling existing modules if they
are also installed in by system packages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rook: monkey-patch for CoreV1Api.list_node

`kubernetes.client.apis.core_v1_api.CoreV1Api#list_node` is
broken and needs a monkey patch. Might be related to
an incompatible k8s cluster version.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: better format `service ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Implement fallback for CentOS 7

`kubernetes-client` requires `urllib3` > 1.15, but CentOS 7 ships 1.10

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Move KubernetesResource to rook_cluser.py

Use `KubernetesResource` as cache for pods, inventory maps and nodes.
This is required for the dashboard.

Also:
  properly return K8s nodes
  fix minor issues

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Add feature to gather k8s storageclass information

Provide ability to fetch or watch k8s resources,
and hold the resulting objects in a dict. A method
has been added to the rook module to allow other
mgr modules to fetch the state of specific k8s
resources.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/orchestrator_clli: Always return str

Also: Don't return `"None"`
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ansible: Moved RGW defaults to ansible

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Fix RGW creation

* Moved RGWSpec defaults to a new method

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Split *_stateless_service and add get_feature_set

Split *_stateless_service, because every orchestrator
did some method dispatching internally anyway.

Also: Added test

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-volume: batch ensure device lists are disjoint

Fixes: https://tracker.ceph.com/issues/41018
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

test/python: remove test for ceph cli

we have abundant test for exercising `ceph` cli, like

qa/standalone/mon/misc.sh.

and `test_ceph.py` does not test any functionality of this python script
other than importing it, which basically runs it through python's syntax
check.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/python: remove test for brag/client

ceph-brag is now take replaced by telemetry. and it was removed in
cf97b5c1 .

so no need to keep its test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #29363 into master

* refs/pull/29363/head:
qa/suites/multimds/basic/tasks/ceph_test_snapshots: disable RECENT_CRASH
qa/suites/kcephfs/recovery/failover.yaml: disable RECENT_CRASH
qa/suites/fs/multifs/tasks/failover.yaml: disable RECENT_CRASH

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/mgr/dashboard/test_mgr_module: adjust expected schema

Signed-off-by: Sage Weil <sage@redhat.com>

doc: mds-config-ref: update 'mds_log_max_segments' value.

Also drop options that not exists on code:

* mds_log_max_expiring
* mds_log_eopen_size

Signed-off-by: Konstantin Shalygin <k0ste@k0ste.ru>

Merge PR #29185 into master

* refs/pull/29185/head:
os/bluestore: add tail length when caculating reserved buffer size
os/bluestore: record next_offs when offset is not sequential.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29356 into master

* refs/pull/29356/head:
pybind/mgr/restful: use dict.items() for py3 compatibility

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #29098 from liewegas/wip-unhexdump-c

src/script/unhexdump-C: script to reverse a 'hexdump -C' style hexdump

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common/options.cc: change default value of bluestore_fsck_on_mount_deep to false

Fixes: https://tracker.ceph.com/issues/41017
Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29394 from mkogan1/wip-rgw-bi-list-shard-id

rgw: add option to specify shard-id for "bi list" admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #29362 into master

* refs/pull/29362/head:
Update nautilus release for 14.2.2
Updated schedule with latest Nautilus release

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28239 into master

* refs/pull/28239/head:
cephfs-shell: Add tox for testing with flake8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29386 from alfredodeza/wip-rm40987

ceph-volume  simple: better detection when 'type' file is not present

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

cephfs-shell: Add tox for testing with flake8

Fixes: https://tracker.ceph.com/issues/39947
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge PR #29331 into master

* refs/pull/29331/head:
common: add comment about pod memory requests/limits

Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #29161 from valentinbajrami/improve-grep-regex

src/script/check_commands.sh: fix grep regex class range

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: update FindBoost.cmake

FreeBSD is moving to Boost 1.70
we should allow user to use boost 1.70.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

ceph-volume: refactor ceph-disk_member unittests

This adds a parameterized fixture as input to the tests. Should make
writing new tests easier since only fixture parameters need to be added.
In the future we should be able to reuse the fixtures to build other
fixtures and reuse the parametrization.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

rgw: add option to specify shard-id for "bi list" admin command

related to: https://tracker.ceph.com/issues/40700
Signed-off-by: Mark Kogan <mkogan@redhat.com>

src/script/check_commands.sh: fix grep regex class range

Correct the faulty regex class.

Signed-off-by: Valentin Bajrami <valentin.bajrami@gmail.com>

Merge pull request #28915 from sebastian-philipp/python-common-package-2

python-common:  Python common package

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #27336 from SUSE/wip-orchestrator-deepsea-iscsi-ganesha

mgr/deepsea: return ganesha and iscsi endpoint URLs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: pg_num should always be a power of two

Signed-off-by: Kai Wagner <kwagner@suse.com>

Merge pull request #29387 from ceph/wip-ceph-volume-tests-no-dashboard

ceph-volume: when testing disable the dashboard