rgw: change the format of the cast

Signed-off-by: jiahuizeng <jhzeng93@foxmail.com>
(cherry picked from commit c44a335aeae3f830ff70fe9b4ed04b53f32c07d6)

Conflicts:
src/rgw/rgw_lc.cc
- make_timespan() call in s3_expiration_header is different in nautilus

rgw: avoid expiration early triggering caused by overflow

Fixes: https://tracker.ceph.com/issues/42634
Signed-off-by: jiahuizeng <jhzeng93@foxmail.com>
(cherry picked from commit 260df7dfd819a4117ae76aaf699370afda653baf)

Conflicts:
src/rgw/rgw_lc.cc
- make_timespan() call in s3_expiration_header is different in nautilus

Merge pull request #38589 from smithfarm/wip-48128-nautilus

nautilus: rgw: keep syncstopped flag when copying bucket shard headers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38556 from smithfarm/wip-48576-nautilus

nautilus: rgw: Disable prefetch of entire head object when GET request with range header

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #38527 from s0nea/wip-48537-nautilus

nautilus: mgr/dashboard: test_standby* (tasks.mgr.test_dashboard.TestDashboard) failed locally

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #38504 from callithea/wip-48512-nautilus

nautilus: mgr/dashboard: remove pyOpenSSL version pinning

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #38493 from ivancich/nautilus-better-ordered-listing-skipping

nautilus: rgw: in ordered bucket listing skip namespaced entries internally when possible

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38332 from votdev/wip-48395-nautilus

nautilus: mgr/dashboard: Disable TLS 1.0 and 1.1

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38772 from batrick/pr38188-fix

nautilus: qa: restore file name

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/dashboard: Disable TLS 1.0 and 1.1

Disable these TLS versions because of security issues.

Fixes: https://tracker.ceph.com/issues/48360
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 5cf222b6d2a5cdc7ff23357b12878d34a4e68801)

Conflicts:
- src/pybind/mgr/dashboard/module.py
Limiting TLS version seems to work only with CherryPy >= 8.9.1 (Ubuntu 18.04 or SUSE Leap 15.1) based on tests in Teuthology. It failed on Ubuntu 16.04 (CherryPy 3.5.0) and RHEL 7.9 (CherryPy 3.2.2).
- qa/tasks/mgr/test_dashboard.py
Tests must be removed because they will not work in Teuthology because of outdated cherrypy versions on most platforms. Only Ubuntu 18.04 (and downstream SUSE Leap 15.1) will work, but it's not possible to limit QA runs to that specific platforms.

Merge pull request #38467 from Vicente-Cheng/wip-48457-nautilus

nautilus: client: do not use g_conf().get_val<>() in libcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37991 from Vicente-Cheng/wip-48110-nautilus

nautilus: cephfs: client: increment file position on _read_sync near eof

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37988 from Vicente-Cheng/wip-48097-nautilus

nautilus: cephfs: osdc: restart read on truncate/discard

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37986 from Vicente-Cheng/wip-48095-nautilus

nautilus: mds : move start_files_to_recover() to recovery_done

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa: restore file name

Changing it caused symbolic links to not resolve which broke the kcephfs
suite (and probably others).

Fixes: 0e3d45d3486d47ea50045fb2449199f7158effe3
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38614 from neha-ojha/wip-48614-nautilus

nautilus: Do not add sensitive information in Ceph log files

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge remote-tracking branch 'upstream/heads/nautilus-saved' into HEAD

14.2.16

Merge branch 'CVE-2020-27781-nautilus' into nautilus

* CVE-2020-27781-nautilus:
  tasks/cephfs/test_volume_client: Add tests for authorize/deauthorize
  pybind/ceph_volume_client: Optionally authorize existing auth-ids
  pybind/ceph_volume_client: Preserve existing caps while authorize/deauthorize auth-id
  pybind/ceph_volume_client: Disallow authorize auth_id
  pybind/ceph_volume_client: Fix PEP-8 SyntaxWarning

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Kotresh Hiremath Ravishankar <khiremat@redhat.com>

tasks/cephfs/test_volume_client: Add tests for authorize/deauthorize

1. Add testcase for authorizing auth_id which is not added by
   ceph_volume_client
2. Add testcase to test 'allow_existing_id' option
3. Add testcase for deauthorizing auth_id which has got it's caps
   updated out of band

Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit aa4beb3d993649a696af95cf27150cc460baaf70)

Conflicts:
qa/tasks/cephfs/test_volume_client.py

pybind/ceph_volume_client: Optionally authorize existing auth-ids

Optionally allow authorizing auth-ids not created by ceph_volume_client
via the option 'allow_existing_id'. This can help existing deployers
of manila to disallow/allow authorization of pre-created auth IDs
via a manila driver config that sets 'allow_existing_id' to False/True.

Fixes: https://tracker.ceph.com/issues/48555
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 77b42496e25cbd4af2e80a064ddf26221b53733f)

pybind/ceph_volume_client: Preserve existing caps while authorize/deauthorize auth-id

Authorize/Deauthorize used to overwrite the caps of auth-id which would
end up deleting existing caps. This patch fixes the same by retaining
the existing caps by appending or deleting the new caps as needed.

Fixes: https://tracker.ceph.com/issues/48555
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 47100e528ef77e7e82dc9877424243dc6a7e7533)

pybind/ceph_volume_client: Disallow authorize auth_id

This patch disallow the ceph_volume_client to authorize the auth_id
which is not created by ceph_volume_client. Those auth_ids could be
created by other means for other use cases which should not be modified
by ceph_volume_client.

Fixes: https://tracker.ceph.com/issues/48555
Signed-off-by: Ramana Raja <rraja@redhat.com>
Signed-off-by: Kotresh HR <khiremat@redhat.com>
(cherry picked from commit 3a85d2d04028a323952a31d18cdbefb710be2e2b)

pybind/ceph_volume_client: Fix PEP-8 SyntaxWarning

Signed-off-by: Đặng Minh Dũng <dungdm93@live.com>
(cherry picked from commit 3ce9a89a5a1a2d7fa3d57c597b781a6aece7cbb5)

mgr/ActivePyModules.cc: don't log config value in get_typed_config

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit f4b3d6b7692332c329b567d91460b8ada96741e2)

mon/ConfigMonitor.cc: don't log config values in refresh_config

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 77932ae49bffd03f905f1039ae4add303738eec6)

 Conflicts:
src/mon/ConfigMonitor.cc - trivial resolution

mgr: don't add config values to mgr log in config_callback and module_config

The original code has been commented out and left for future debugging
purposes.

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 19000fad573138b2b0576c093237c4a708b76020)

 Conflicts:
src/mgr/PyModuleRegistry.cc - maintain dout level

mon/ConfigMonitor.cc: don't dump all keys and values in config map

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit a68b4597785f92a0cb98440030be61f8004e3151)

mon: don't log "config set" and "config-key set" dispatch and finished messages

Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 4b83dfb1f74e8a59c802ff3c0eb4595f7e763762)

messages/MMonCommand, MMonCommandAck: don't log values for "config set" and "config-key set"

This acts like a big hammer to avoid adding sensitive information, like passwords
into mon/mgr/cluster logs when using "config set" and "config-key set" to set keys
whose values should be secure.

Fixes: https://tracker.ceph.com/issues/37503
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 3d54660ca1a9a7ae54e884c3181fca17a40d8cd3)

 Conflicts:
src/messages/MMonCommand.h - trivial resolution
src/messages/MMonCommandAck.h - trivial resolution

rgw: keep syncstopped flag when copying bucket shard headers

Fixes: https://tracker.ceph.com/issues/48037
Signed-off-by: Ilsoo Byun <ilsoobyun@linecorp.com>
(cherry picked from commit 5a94be030b67085ef08c7b3a4450d3d19420a9c4)

Merge pull request #38310 from ifed01/wip-ifed-finer-mempool-granularity-nau

nautilus: os/bluestore: mempool's finer granularity + adding missed structs

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #37589 from kamoltat/wip-ksirivad-nautilus-backports

nautilus: mgr/progress: make it so progress bar does not get stuck forever

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38404 from callithea/wip-48344-nautilus

nautilus: mgr/dashboard: Disable sso without python3-saml

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

rgw: Disable prefetch of entire head object when GET request with range header

Disable prefetch of entire head object when GET request with range header.
The current behavior for the RGW is getting the whole object although the client asked only for a small bytes offset.
For example: If the client asked for bytes=0-1, The RGW will anyway fetch 0-4194304

Fixes: https://tracker.ceph.com/issues/44508
Signed-off-by: Or Friedmann <ofriedma@redhat.com>
(cherry picked from commit 2be5af0006169cb54547034aa98b7eacb8751d59)

Merge pull request #38354 from ifed01/wip-ifed-fix-statfs-out-nau

nautilus: mgr: don't update osd stat which is already out

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38334 from b-ranto/wip-prom-fixes-nautilus

nautilus: mgr/prometheus: Make module more stable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38085 from orztt/wip-rgw-versioning-nautilus

nautilus: rgw: cls/rgw/cls_rgw.cc: fix multiple lastest version problem

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37895 from smithfarm/wip-48040-nautilus

nautilus: rbd: librbd: ensure that thread pool lock is held when processing throttled IOs

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

qa/mgr/test_progress: fix bug 48217

Fixes a failing test case regarding osd coming back
after being marked out. The old test case wasn't accounting
for a specific event, therefore this resulted in the failure.
The fix basically accounts for a specific event of osd being
marked in/out.

Fixes: https://tracker.ceph.com/issues/48217
Signed-off-by: Kamoltat <ksirivad@redhat.com>
(cherry picked from commit 52fe9dbdae87b65a99a6c3bac3519c9528f22ea2)

Conflicts:
qa/tasks/mgr/test_progress.py - trivial fix

mgr/dashboard: test_standby* (tasks.mgr.test_dashboard.TestDashboard) failed locally

Fixes: https://tracker.ceph.com/issues/48449
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 9d4282e53af6c4a4c48641deeaa5f018781d5006)

mgr/dashboard: remove pyOpenSSL version pinning

Tests require a newer version of pyOpenSSL.

Fixes: https://tracker.ceph.com/issues/48506
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>
(cherry picked from commit 2695075c4b2fc69735a834c49f6f8e768de4f85c)

Conflicts:
src/pybind/mgr/dashboard/requirements.txt
 - This manual change was required in order to get the
   backport working in nautilus

rgw: in ordered bucket listing skip namespaced entries when possible

When listing non-namespaced entries in the bucket index, the code
would march through the namespaced entries in blocks, requesting all
of them from the CLS layer. When there were many namespaced entries,
it would significantly affect the performance of ordered listing.

This commit adds code to advance the marker passed to lower layers to
skip past namespaced entries. This is challenging in that
non-namespaced entries can appear in the middle of the namespaced
entries. We'll ignore the issue instance tags in names to simplify the
following discussion. Non-namespaced entries are indexed by
"name". Namespaced entries are indexed by _namespace_name, using
underscores to surround the namespace. The challenge comes with
entries such as "_name", where the name begins with an underscore. In
that case we index them by "__name", quoting the underscore with
another.

Now the extra challenge comes due to the lexic ordering of the
following:

    ASP
    _BAT_cat
    __DOG
    _eel_FOX
    goat

Note that the namespaced entries are in positions 2 and 4, and the
non-namespaced entries are in positions 1, 3, and 5. So when skipping
past the namespaced entries, we have to be careful not to skip past
the non-namespaced entries that begin with underscore.

Additional code clean-ups done as well.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
Resolves: rhbz#1883283
(cherry picked from commit acfa6409597768b3c1dc8dcab3668d23761a4cbc)

client: do not use g_conf().get_val<>() in libcephfs

The g_conf() will try to dereference the "g_ceph_context" to get
the "_conf", but the "g_ceph_context" won't be set in none fuse
use case in libcephfs, so it will crash in when doing
g_conf().get_val<>.

Switch to use cct->_conf.get_val<> instead.

Fixes: https://tracker.ceph.com/issues/48206
Signed-off-by: Xiubo Li <xiubli@redhat.com>
(cherry picked from commit a467274f1c4dcf357de48fb736d40f7488bc4d7c)

mgr/prometheus: don't store exception as e

Python's logging module's exception() method will log the full exception
and stack trace for us, so we do not need to store the exception in the
"e" variable here.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit a17c603effd3367dc64c87a1d6c53d6d3d794fc7)

Merge pull request #38416 from kamoltat/wip-fix-bug-48434

nautilus: mgr/progress: delete all events over the wire

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #38411 from dzafman/wip-48444

nautilus: osd: Check for nosrub/nodeep-scrub in between chunks, to avoid races

Reviewed-by: Neha Ojha <nojha@redhat.com>

librbd: ensure that thread pool lock is held when processing throttled IOs

There previously was a potential race for throttled IOs to complete prior
to the main worker thread finishing the processing of the blocked IO.

Fixes: https://tracker.ceph.com/issues/47371
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 2d86e0935aa6f0c392df428676d9ab0a338fccae)

Conflicts:
    src/test/librbd/io/test_mock_ImageRequestWQ.cc
- in Octopus, commit 792d6c53fedc695199cc18916347c1b545fe42c2 did a global
  replace of Mutex to ceph::mutex, so to fix this for Nautilus, we just need to
  do that in test_mock_ImageRequestWQ.cc since the get_pool_lock() method is
  returning a Mutex instead of a ceph::mutex

Merge pull request #37959 from callithea/wip-47995-nautilus

nautilus: monitoring: Use null yaxes min for OSD read latency

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/progress: 'progress clear' command should clear events in 'ceph -s'

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6f60d33115d2f583331d31b95a0a33b96a614f09)

osd: Check for nosrub/nodeep-scrub in between chunks, to avoid races

Fixes: https://tracker.ceph.com/issues/47767
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 9b7f911d9a139cc347f2d3ac3068fc1d212058c7)

Conflicts:
src/osd/PG.cc (manual merge due to code rearrangement)

mgr/dashboard: Disable sso without python3-saml

Removed the requirement for the python3-saml package when wanting to disable SSO for the dashboard, this is currently relevant since the official container that runs Ceph mgr does not have this package installed.
So when upgrading from an older, non-containerized version, you would be stuck using a non-functional dashboard.

This pull requests changes that and allows the ceph dashboard sso disable command without the requirement of the library so that we SSO can always be disabled again.

Fixes: https://tracker.ceph.com/issues/48237 Signed-off-by: Kevin Meijer <admin@kevinmeijer.nl>
(cherry picked from commit 0c18437d2c786ef1ade8b89e42dbf4b0e163aafe)

Merge pull request #38362 from badone/wip-nautilus-mon-scrub-testing

nautilus: mon scrub testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #38372 into nautilus

* refs/pull/38372/head:
ceph-volume: implement the --log-level flag

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38371 into nautilus

* refs/pull/38371/head:
lvm/create.py: fix a typo in the help message

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #38382 from badone/wip-nautilus-run-tox-mgr-insights-six-missing

nautilus: mgr/insights: Test environment requires 'six'

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/insights: Test environment requires 'six'

Not a backport because python2 support was dropped in master and only
nautilus seems to be affected at this time.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

cls/rgw/cls_rgw.cc: fix multiple lastest version problem

Fixes: https://tracker.ceph.com/issues/47919
Signed-off-by: Ruan Zitao <ruanzitao@kuaishou.com>
Signed-off-by: Yang Honggang <yanghonggang@kuaishou.com>
(cherry picked from commit f60f9ace1a4bceeda256373cf4603058e1947fa8)

Conflicts:
      src/cls/rgw/cls_rgw.cc
- nautilus does not have "rgw_bucket_dir_entry::FLAG_VER"; use "RGW_BUCKET_DIRENT_FLAG_VER" instead

qa/suites/rados/monthrash: Exercise mon scrub error injectors

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit a5bcca7f415790521a76213620ae079318e7bee1)

 Conflicts:
qa/suites/rados/monthrash/ceph.yaml - whitelist vs. ignorelist

ceph-volume: implement the --log-level flag

The --log-level flag was being ignored and
the file log level was always set to DEBUG.

Fixes: https://tracker.ceph.com/issues/48045
Resolves: rhbz#1867717

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit ecbd6c13f116b390c782c9ae14b5becd0bdecc8e)

lvm/create.py: fix a typo in the help message
ceph_volume/devices/lvm/create.py:corrected typo of the word when using ceph-volume lvm create -h

Fixes: https://tracker.ceph.com/issues/48273
Signed-off-by: ZhenLiu94 <zhenliu94@163.com>
(cherry picked from commit e3c7d6ff4cec80ee0135abb50d795411c5dc2283)

Merge PR #38279 into nautilus

* refs/pull/38279/head:
ceph-volume batch: reject partitions in argparser

Reviewed-by: Rishabh Dave <ridave@redhat.com>

qa/config/rados.yaml: Test mon scrub

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit f85001e5d5fb11718ab2fd8b708402cd2db951d4)

Merge pull request #37840 from smithfarm/wip-47990-nautilus

nautilus: qa/cephfs: add session_timeout option support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37838 from smithfarm/wip-47988-nautilus

nautilus: cephfs: client: fix inode ll_ref reference count leak

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37836 from smithfarm/wip-47953-nautilus

nautilus: vstart.sh: fix fs set max_mds bug

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37822 from smithfarm/wip-47957-nautilus

nautilus: mon/MDSMonitor do not ignore mds's down:dne request

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37821 from smithfarm/wip-47939-nautilus

nautilus: mon/MDSMonitor: divide mds identifier and mds real name with dot

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37820 from smithfarm/wip-47935-nautilus

nautilus: mds: account for closing sessions in hit_session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37725 from rishabh-d-dave/wip-46611-nautilus

nautilus: pybind/cephfs: add special values for not reading conffile

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38118 from neha-ojha/wip-48227-nautilus

nautilus: mon: Log "ceph health detail" periodically in cluster log

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr: don't update osd stat which is already out

Ceph status still reports slow requests on the OSD which is already out.
When orignal PG monitor handled PGSTATS msg, it wouldn't update osd stat
if this OSD is not in OSD map, but current MGR had no checks on that.

Fixes: https://tracker.ceph.com/issues/46440
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
(cherry picked from commit 493ec9d3acd3f57eed3e4b96ad7c6739c2089ff1)

mgr/prometheus: use threading.Event instead of sleep

This allows us to avoid waiting for the sleep to finish when waiting for
the thread to finish.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit dd5886c3c006e388283df50cc87addeffb3b2b52)

mgr/prometheus: Log collection issues

Log any issues encountered during the data collection and continue to
collect the data anyway (after a sleep).

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 28a5c13bf993679e3098d73df27ded249f34dc99)

mgr/prometheus: Use mgr.release_name for always on modules

The host_version is not populated properly in the early stages of ceph
mgr start up process. We can use mgr.release_name instead. It is more
stable and it provides the data even if mgr_map does not contain the
versions, yet.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit aa0650092da3cbf1a73151999874001352cfb9ef)

mgr/prometheus: Clean up collection thread

We need to clean up the metrics collection thread.

Signed-off-by: Boris Ranto <branto@redhat.com>
(cherry picked from commit 03fcaccafc877d10a894b1c39af5547f172c1ed3)

Conflicts:
prometheus/module.py: Pass _global_instance as an argument to
        MetricCollectionThread, collect can't be a static function
        anymore

Merge pull request #38295 from badone/wip-nautilus-dont-run-tests-if-build-fails

nautilus: run-make-check.sh: Don't run tests if build fails

Reviewed-by: Kefu Chai <kchai@redhat.com>

run-make-check.sh: Make sure a build failure will exit

We 'set -e' but that is ignored because 'build tests' is executed in a
'&&' list (see 'man set') so move the echo to the following line.

Follow-up to 03ff2146f95

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit e70483133db87a3f04bc1fff31d8472465c305b3)

 Conflicts:
run-make-check.sh: cmake call differences and trivial logging
        output change.

run-make-check.sh: Don't run tests if build fails

When run-make was taken out we lost the 'set -e' call and therefore
continue after an error.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 03ff2146f95c7e03a84df1f8c3b38bbbb315b708)

os/bluestore: bind attrs to bluestore_cache_meta mempool.

This seems to be a better choice over bluestore_cache_data

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f4e09fc103454fa8a41b5ffb822639368a89522d)

os/bluestore: add missing structs to bluestore_cache_other mempool

Fixes: https://tracker.ceph.com/issues/47551
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 4592c5df554190065fa9a994f7b6dc3fed7ea214)

os/Bluefs: fix file reader/writer buffer mempool

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
(cherry picked from commit 6a50c85067c03884473922318267769a9e77ef2e)

bluestore: _get_used_bytes should reflect new mempool fields

with bluestore_cache_other split further, the _get_used_bytes should
take into account the new mempool splits as well.

see also: 0ce8ee4b123eacfc2bfb14f554487e928512415a
  68ee931fd4b6bf748f95b14dae31d11738fbaf1e

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
(cherry picked from commit d9f84b41eff99357585f27c89449e379a28a9212)

common/mempool: More detailed bluestore stats

Splits bluestore bluestore_cache_other, exposes accounting of some internals.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit bb488df5e4ffa1ec2b5d25d9af0ce7f5093d7470)

Introduce separate mempool for FileReaderBuffer and FileWriter

FileReaderBuffer and FileWriter uses Bufferlists which needs to be
tracked by mempool to get the proper memory usage.

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
(cherry picked from commit 9bb741c12cf1ffe3e9fec8e0d2fbaae107947215)

 Conflicts:
src/os/bluestore/BlueFS.cc
  trivial

split mempool allocation for bluestore_cache_other

While doing root cause analysis it bluestore_cache_other gives a bit of
a crude estimate, something more helpful would be to have it split into
the following fields:

- bluestore_cache_onode
- bluestore_cache_meta
- bluestore_onode_attrs

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
(cherry picked from commit 1c6bc3893bf4cadae344ef19290f5c7e55dd5a62)

 Conflicts:
src/os/bluestore/BlueStore.cc
   Legacy vs. New lock used
src/os/bluestore/bluestore_types.h
   Lacked explicit std:: namespace reference in headers which pacific has

Merge pull request #38268 from idryomov/wip-relax-preauth-asserts-again-nautilus

nautilus: msg/async/ProtocolV2: allow rxbuf/txbuf get bigger in testing, again

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38024 from rhcs-dashboard/wip-48180-nautilus

nautilus: mgr/dashboard: Display users current bucket quota usage

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37995 from callithea/wip-48133-nautilus

nautilus: mgr/dashboard: disable cluster selection in NFS export editing form

Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37756 from tspmelo/wip-47198-nautilus

nautilus: mgr/dashboard: Datatable catches select events from other datatables

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38188 from rakeshgm/rhel_nautilus_repos

nautilus: qa/distros: add rhel 7.9

qa/mgr/progress: Update the test suite for progress module

Update the test suite to reflect a feature
change that has been merged to master in
progress module where you create an event
when an osd is marked in.

Fixes: http://tracker.ceph.com/issues/40618
with success QA run in sepia:
http://pulpito.ceph.com/kchai-2019-07-16_10:10:01-rados-master-distro-basic-mira/

Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
(cherry picked from commit 5604ba4ec1e0dac17f12e287a831cec9a4fad7ce)

Conflicts:
qa/tasks/mgr/test_progress.py - change `ev` to `new_event`

qa/distros: add rhel 7.9

Signed-off-by: rakeshgm <rakeshgm@redhat.com>

Merge pull request #38296 from badone/wip-nautilus-pin-importlib_metadata

nautilus: mgr: Pin importlib_metadata version 2.1.0

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/progress: Bug fix complete event when OSD marked in

Swap the order of checking for complete event
when OSD is marked in and creating a
PG Recovery event so that the event
doesn't cancel itself.

Signed-off-by: Kamoltat (Junior) Sirivadhna <ksirivad@redhat.com>
(cherry picked from commit 93d4d9d7044e991a7bbdb70b0aef02284e6eda22)

Conflicts:
src/pybind/mgr/progress/module.py - change self._events.item()
    to list(self._events)
  - add self.get("pg_ready")

pybind/mgr: Pin importlib_metadata version 2.1.0

Latest release of importlib_metadata breaks the Nautilus build.

Master does not appear to be affected, probably because it uses
python3.6 or greater which is compatible with the latest
importlib_metadata version.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #37605 from smithfarm/wip-47803-nautilus

nautilus: test/librados: fix endian bugs in checksum test cases

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38015 from dsavineau/wip-48087-nautilus

nautilus: ceph-volume: consume mount opt in simple activate

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38048 into nautilus

* refs/pull/38048/head:
ceph-volume: fix lvm help test
ceph-volume: remove mention of dmcache from docs and help text

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #37723 into nautilus

* refs/pull/37723/head:
ceph-volume: add no-systemd argument to zap

Reviewed-by: Jan Fajerski <jfajerski@suse.com>