qa/suites/upgrade: disable min pg per osd warning

disable the TOO_FEW_PGS warning, as
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4 is not backported to mimic, we
will have TOO_FEW_PGS warnings when a healthy cluster is expected when
upgrading from mimic.

this change disables this warning by setting "mon_pg_warn_min_per_osd" to
"0".

this change is not cherry-picked from master. as
1ac34a5ea3d1aca299b02e574b295dd4bf6167f4 is already included by master,
and we don't perform upgrade from mimic on master branch.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34833 from smithfarm/wip-44653-nautilus

nautilus: rgw: increase log level for same or older period pull msg

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34413 from votdev/wip-42151-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34862 from tchaikov/nautilus-el-7.8

nautilus: qa/distros: point {centos,rhel}_latest.yaml to 7.8

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge pull request #34832 from smithfarm/wip-45330-nautilus

nautilus: bluestore/bdev: initialize size when creating object

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34796 from smithfarm/wip-45045-nautilus

nautilus: os/bluestore: Don't pollute old journal when add new device

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #34677 from Vicente-Cheng/wip-44468-nautilus

nautilus: mon: Get session_map_lock before remove_session

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #34611 from ifed01/wip-ifed-beter-bluefs-expand-nautilus

nautilus: os/bluestore: open DB in read-only when expanding DB/WAL

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34774 from ajarr/nautilus-fix-45229-45225

nautilus: mon/FSCommands: Fix 'add_data_pool' command and 'fs new' command

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #34579 from tchaikov/nautilus-mimic-x

nautilus: qa/tasks/ceph_manager: do not cancel pending pg num changes on mimic

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/distros: point {centos,rhel}_latest.yaml to 7.8

this change is not cherry-picked from master, as master is using el8
already

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: do not cancel pending pg num changes on mimic

mimic does not support auto split/merge, but we do test mimic-x on
nautilus, which ends up with failures like:

ceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_git_teuthology_py2/teuthology/contextutil.py", line 34, in nested
    yield vars
  File "/home/teuthworker/src/git.ceph.com_ceph_nautilus/qa/tasks/ceph.py", line 1928, in task
    ctx.managers[config['cluster']].stop_pg_num_changes()
  File "/home/teuthworker/src/git.ceph.com_ceph_nautilus/qa/tasks/ceph_manager.py", line 1806, in stop_pg_num_changes
    if pool['pg_num'] != pool['pg_num_target']:
KeyError: 'pg_num_target'

so we need to skip this if 'pg_num_target' is not in pg_pool_t::dump().

this change is not cherry-picked from master, as we don't test
mimic-x on master.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: increase log level for same or older period pull msg

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 90cce10fc1364df16ab12632b2dca403894cbe44)

bluestore/bdev: initialize size when creating object.

The other partss of the struct are initialized by their
ctors. Only for unint_64 there is no ctor.

Otherwise ceph-dencoder tests will fail in comparing
the exported output.
[~/master36] wjw@cephdev.digiware.nl> build/bin/ceph-dencoder type bluestore_bdev_label_t select_test 1 encode export /tmp/typ-yFISvjvgj
[~/master36] wjw@cephdev.digiware.nl> hexdump -C !$
hexdump -C /tmp/typ-yFISvjvgj
00000000  62 6c 75 65 73 74 6f 72  65 20 62 6c 6f 63 6b 20 |bluestore block |
00000010  64 65 76 69 63 65 0a 30  30 30 30 30 30 30 30 2d |device.00000000-|
00000020  30 30 30 30 2d 30 30 30  30 2d 30 30 30 30 2d 30 |0000-0000-0000-0|
00000030  30 30 30 30 30 30 30 30  30 30 30 0a 02 01 28 00 |00000000000...(.|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
00000050  00 00 70 74 03 04 00 00  00 00 00 00 00 00 00 00 |..pt............|
00000060  00 00 00 00 00 00 00 00  00 00 |..........|
0000006a

[~/master36] wjw@cephdev.digiware.nl> build/bin/ceph-dencoder type bluestore_bdev_label_t select_test 1 encode decode encode export /tmp/typ-MjWXdCpzJ
[~/master36] wjw@cephdev.digiware.nl> hexdump -C !$
hexdump -C /tmp/typ-MjWXdCpzJ
00000000  62 6c 75 65 73 74 6f 72  65 20 62 6c 6f 63 6b 20 |bluestore block |
00000010  64 65 76 69 63 65 0a 30  30 30 30 30 30 30 30 2d |device.00000000-|
00000020  30 30 30 30 2d 30 30 30  30 2d 30 30 30 30 2d 30 |0000-0000-0000-0|
00000030  30 30 30 30 30 30 30 30  30 30 30 0a 02 01 28 00 |00000000000...(.|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00 |................|
00000050  00 00 73 64 00 00 00 00  00 00 00 00 00 00 00 00 |..sd............|
00000060  00 00 00 00 00 00 00 00  00 00 |..........|
0000006a

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>
(cherry picked from commit d411ee26fb2f6cbe610f8bbc81b777cf28d839c2)

Merge pull request #34518 from tchaikov/nautilus-27422

nautilus: doc: fix parameter to set pg autoscale mode

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: fix parameter to set pg autoscale mode

osd_pool_default_pg_autoscale_mode is the right parameter to
set placement-group autoscale mode.

Signed-off-by: Changcheng Liu <changcheng.liu@intel.com>
(cherry picked from commit c0df98fc7e78bbd366333d810f78ddbeed0e6729)

Merge pull request #34736 from smithfarm/wip-44648-nautilus

nautilus: qa/workunits/rbd: wait for nbd map to close after unmap

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

os/bluestore: Don't pollute old journal when add new device

Fixes: https://tracker.ceph.com/issues/44774
Signed-off-by: Yang Honggang <yanghonggang@kuaishou.com>
(cherry picked from commit ade4d46981e660c8d57cec64180b5afa4561b945)

Merge pull request #34735 from smithfarm/wip-45260-nautilus

nautilus: rgw: reshard: skip stale bucket id entries from reshard queue

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34628 from shyukri/wip-45070-nautilus

nautilus: mgr/telegraf: catch FileNotFoundError exception

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34512 from Vicente-Cheng/wip-44324-nautilus

nautilus: osd/PeeringState.h: ignore RemoteBackfillReserved in WaitLocalBackfillReserved

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #34500 from Vicente-Cheng/wip-44289-nautilus

nautilus: mon: stash newer map on bootstrap when addr doesn't match

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34419 from votdev/wip-44954-nautilus

nautilus: monitoring: root volume full alert fires false positives

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mon/FSCommands: Fix 'fs new' command

After creating a filesystem using the 'fs new' command, the value
of the 'data' and 'metadata' key of the datapool and metadatapool's
application tag 'cephfs' should be the filesystem's name. This
didn't happen when the data or metadata pool's application metadata
'cephfs' was enabled before the pool was used in the 'fs new' command.
Fix this during the handling of the 'fs new' command by setting the
value of the key of the pool's application metadata 'cephfs' to the
filesystem's name even when the application metadata 'cephfs' is
already enabled or set.

Fixes: https://tracker.ceph.com/issues/43761
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 2f45558eb84782e334d997e2c545fd99ab455783)

mon/FSCommands: Fix 'add_data_pool' command

After making a RADOS pool a filesystem's data pool using the
'add_data_pool' command, the value of the 'data' key of the pool's
application metadata 'cephfs' should be the filesystem's name. This
didn't happen when the pool's application metadata 'cephfs' was
enabled before the pool was made the data pool. Fix this during the
handling of the 'add_data_pool' command by setting the value of
the 'data' key of the pool's application metadata 'cephfs' to the
filesystem's name even when the application metadata 'cephfs' is
already enabled or set.

Fixes: https://tracker.ceph.com/issues/43061
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 3c49092322720b7f1d612f45e73fb26820ff7885)

Merge pull request #34662 from votdev/wip-45157-nautilus

nautilus: mgr/dashboard: Refactor Python unittests and controller

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #34627 from shyukri/wip-45043-nautilus

nautilus: mgr/PyModule: fix missing tracebacks in handle_pyerror()

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34542 from smithfarm/wip-44486-nautilus

nautilus: common/ceph_time: tolerate mono time going backwards

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34539 from smithfarm/wip-45073-nautilus

nautilus: build/ops: address SElinux denials observed in rgw/multisite test run

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34527 from smithfarm/wip-45035-nautilus

nautilus: spec: address some warnings raised by RPM 4.15.1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34256 from smithfarm/wip-44793-nautilus

nautilus: tests: qa/tasks/ceph.py: fail if any osd role doesn't get a device

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits/rbd: wait for nbd map to close after unmap

The unmap action only sends a signal to the kernel to notify the
rbd-nbd daemon to disconnect. Therefore, it's possible that an
unmap followed by an immediate re-map to the same device might
fail since the unmap is still in-progress.

Fixes: https://tracker.ceph.com/issues/44567
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 80a3f18cafb4add1624cc690bba436a1284dc634)

rgw: reshard: skip stale bucket id entries from reshard queue

If we encounter a reshard queue entry that has an older ID compared to the
bucket's current ID, it'd mean that some other process or a manual reshard has
already processed this entry, skip processing the entry this time. An
alternative is to verify the num_shards that we have in queue >= the current
shards, but this would mean that we may reshard a recently manual resharded
bucket again which might not be intended

Fixes: https://tracker.ceph.com/issues/45134
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 02664fc091674e28233559cd1c42f954d5776d86)

Conflicts:
src/rgw/rgw_reshard.cc
- immediately preceding "get_bucket_info" call looks different in
  nautilus

Merge pull request #34551 from tspmelo/wip-45085-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #34550 from tspmelo/wip-45082-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34497 from votdev/wip-45019-nautilus

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34399 from votdev/wip-44920-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mon: Get session_map_lock before remove_session

Get session_map_lock to prevent race.

Fixes: https://tracker.ceph.com/issues/44407
Signed-off-by: Xiaofei Cui <cuixiaofei@sangfor.com.cn>
(cherry picked from commit c916b8a8156f9855fb74c62508b54308ec5e7b32)

Merge pull request #34597 from smithfarm/wip-43134-nautilus

nautilus: rgw: find oldest period and update RGWMetadataLogHistory()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34585 from smithfarm/wip-45040-nautilus

nautilus: mon: calculate min_size on osd pool set size

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34489 from rhcs-dashboard/nautilus-fix-coverage-path

nautilus: mgr/dashboard: fix COVERAGE_PATH in run-backend-api-tests.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34394 from votdev/wip-44917-nautilus

nautilus: monitoring: alert for prediction of disk and pool fill up broken

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #34392 from tchaikov/nautilus-cbt

nautilus: qa/suites: use "nautilus" branch for cbt based testing

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34609 from smithfarm/wip-44267-nautilus

nautilus: rgw: fix bug with (un)ordered bucket listing and marker w/ namespace

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34599 from smithfarm/wip-44232-nautilus

nautilus: rgw: ReplaceKeyPrefixWith and ReplaceKeyWith can not set at the same …

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34598 from smithfarm/wip-44136-nautilus

nautilus: rgw: set bucket attr twice when delete lifecycle config.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34278 from jan--f/wip-44688-nautilus

nautilus: ceph-volume: prepare: use *-slots arguments for implicit sizing

Merge pull request #34478 from jan--f/wip-44998-nautilus

nautilus: ceph-volume/batch: return success when all devices are filtered

Merge pull request #34481 from jan--f/wip-45002-nautilus

nautilus: ceph-volume/batch: check lvs list before access

Merge pull request #34445 from jan--f/wip-44974-nautilus

nautilus: devices/simple/scan: Fix string in log statement

Merge pull request #33616 from jan--f/wip-44325-nautilus

nautilus: ceph-volume: lvm: get_device_vgs() filter by provided prefix

ceph-volume: pass *-slots arguments to LV creation

Fixes: https://tracker.ceph.com/issues/44494
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 61f0c3af0b0f36a03748733b16de1b11048342e6)

use extent count for slots conversion instead of free count

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 4a81b4a2fbb4cafe45a8fe8cceeece636e51adb6)

ceph-volume: available_lvm: vg space takes precedence

This changes available_lvm to check for generic reasons only if no VGs
were found. A VG can contain a (mounted) lv, which triggers the
ro/locked test, despite the VG having space available.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c6d71eb0022935731bb9380098c975d9c5f27a86)

mgr/dashboard: Refactor Python unittests

* Make use of the KVStoreMockMixin class to get rid off duplicate code.
* Fake the index.html file to be able to run tests/test_home.py locally without building the frontend in production mode.
* Encapsulate helper functions in controllers/home.py, otherwise tests/test_feature_toggles.py need to fake the filesystem because load_controllers() will load the home.py controller and fail due missing files in the filesystem.

Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit e542255b04457df269587bc5639b14a935a125d3)

# Conflicts:
src/pybind/mgr/dashboard/requirements.txt
src/pybind/mgr/dashboard/tests/test_home.py
src/pybind/mgr/dashboard/tests/test_rgw_client.py

ceph-volume: don't create Volume/VolumeGroup objects with empty name

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 235b19eb66b9ba9beab99508f2fdf7b6afe26f31)

ceph-volume: get_device_lvs filter by provided prefix

Fixes: https://tracker.ceph.com/issues/44124
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ac43d37f581510d1854e76afb634a142ad9a0fc0)

ceph-volume: lvm.get_device_vgs: don't create VolumeGroup with empty name

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 3c4c8c5e4a242985389c8d1c85b45cac89b2dc89)

ceph-volume: lvm: get_device_vgs() filter by provided prefix

Fixes: https://tracker.ceph.com/issues/44246
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit b48caf4873c11690d36a88760504f98b46101995)

Merge pull request #34282 from votdev/issue_44809_select_tab

nautilus: mgr/dashboard: Tabs does not handle click events

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34424 from smithfarm/wip-44838-nautilus

nautilus: pybind/rbd: ensure image is open before permitting operations

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mgr/dashboard: Fix iSCSI's username and password validation

When using regex, instead of strings, angular doesn't assume the pattern must
validate the entire string.

Add validation to the backend.

Fixes: https://tracker.ceph.com/issues/44624
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit 532b271572f7b6410c099362dc046148f5a686a1)

Merge pull request #34618 from neha-ojha/wip-45135-nautilus

nautilus: mon: disable min pg per osd warning

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

mgr/telegraf: catch FileNotFoundError exception

in tasks/module_selftest.yaml, `TestModuleSelftest.test_telegraf()` is
called. but we fail to prepare a unix domain socket to which the telegraf
module can send stats. and telegraf module does not catch
FileNotFoundError exception, so the exception is populated to ceph-mgr
and is found by the test, hence the test is marked a failure whenever
telegraf is tested.

in this change,

* catch this exception, so it won't be caught by ceph-mgr
* whitelist the error message, so the test can pass

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit f6d028dfa0c24d43c33670f1bb4547a6310963ea)

mgr/PyModule: fix missing tracebacks in handle_pyerror()

In certain cases, errors raised in mgr modules don't actually result in a
proper traceback in the mgr log; all you see is a message like "'Hello'
object has no a ttribute 'dneasdfasdf'", but you have no idea where that
came from, which is a complete PITA to debug.

Here's what's going on: handle_pyerror() calls PyErr_Fetch() to get
information about the error that occurred, then passes that information
back to python's traceback.format_exception() function to get the traceback.
If we write code in an mgr module that explicitly raises an exception
(e.g.: 'raise RuntimeError("that didn't work")'), the error value returned
by PyErr_Fetch() is of type RuntimeError, and traceback.format_exception()
does the right thing.  If however we accidentally write code that's just
broken (e.g.: 'self.dneasdfasdf += 1'), the error value returned is not
an actual exception, it's just a string.  So traceback.format_exception()
freaks out with something like "'str' object has no attribute '__cause__'"
(which we don't actually ever see in the logs), which in turn dumps us in a
"catch (error_already_set const &)" block, which just prints out the
single line error string.

https://docs.python.org/3/c-api/exceptions.html#c.PyErr_NormalizeException
tells us that "Under certain circumstances, the values returned by
PyErr_Fetch() below can be “unnormalized”, meaning that *exc is a class
object but *val is not an instance of the same class.".  And that's exactly
the problem we're having here.  We're getting a 'str', not an Exception.
Adding a call to PyErr_NormalizeException() turns the value back into a
proper Exception type and traceback.format_exception() now always does the
right thing.

I've also added calls to peek_pyerror() in the catch blocks, so if anything
else ever somehow causes traceback.format_exception to fail, we'll at least
have an idea of what it is in the log.

Fixes: https://tracker.ceph.com/issues/44799
Signed-off-by: Tim Serong <tserong@suse.com>
(cherry picked from commit dee598087a37623238c35d7595348a4c674c43f3)

Merge pull request #34520 from smithfarm/wip-45060-nautilus

nautilus: qa/suites/rados/rest: don't pass empty dict as data arg

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34326 from rzarzynski/wip-bug-24995-nautilus

nautilus: mgr: synchronize ClusterState's health and mon_status.

Reviewed-by: Neha Ojha <nojha@redhat.com>

mon: disable min pg per osd warning

Now that the pg_autoscaler is on by default, it is "normal" (and okay) to
have a small number of PGs in the cluster if the overall cluster usage is
also low.  This setting just results in a health warning out of the box
when you create a pool and haven't written any data yet.

Fixes: https://tracker.ceph.com/issues/41735
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1ac34a5ea3d1aca299b02e574b295dd4bf6167f4)

rgw: fix bug with (un)ordered bucket listing and marker w/ namespace

When listing without specifying a namespace, the returned entries
could be in one or more namespaces. The marker used to continue the
listing may therefore contain a namespace, and that needs to be
preserved. This fixes a bug in both ordered and unordered listings
where it was not preserved.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 9457abb729754458aafc07a9ee514407906a1011)

Conflicts:
src/rgw/rgw_rados.cc
- trivial (nautilus is missing some whitespace-finessing that happened in master)

os/bluestore: avoid bluefs log compaction when doing cold_close

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit db6f3f66fcc28c3db2eba01611664676a820cb90)

 Conflicts:
src/os/bluestore/BlueFS.cc
src/os/bluestore/BlueFS.h
 Caused by lack of https://github.com/ceph/ceph/pull/30593

os/bluestore/BlueFS: replace flush_log with sync_metadata.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>
(cherry picked from commit cdd0e87a73637c25e042caa4bd67497dc72c0386)

os/bluestore: modify return type for BlueFS::_get_slow_device_id()

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit de79da6a065315eb92b203308598d4c452adcc75)

os/bluestore: open db in read-only when expanding standalone DB/WAL

Replace mount() with cold_open() to force opening db in read-only mode.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 976079216cec202d47b7193d19b16f5a8578f269)

 Conflicts:
src/os/bluestore/BlueStore.cc
 Cause by the lack of https://github.com/ceph/ceph/pull/30109

kv/rocksdb: remove obsolete func declarations.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 58e12372e8306e453f44688ac0bc15f5b08bdf09)

os/bluestore: be more informative about main device's avail space.

Log it on OSD startup. Previous output lacked bluefs occupation.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit d2ff682ebcbc9c394df7e17fde854efc52e4351c)

os/bluestore: show space available from bluestore when dumping bluefs devices.

This effectively allows ceph-bluestore-tool to report space available from bluestore
for bluefs-bdev-sizes command, i.e. estimate how utilized main device is.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 7662fce0439456cc3aa4da23248321a4ff90745e)

rgw: when a period lookup for oldest_realm_epoch returns an ENOENT,
find the oldest one and update RGWMetadataLogHistory. This is to avoid an
empty cursor being passed in to ceph_assert() in PurgePeriodLogsCR::operate()
in case of incomplete period history.

Fixes: https://tracker.ceph.com/issues/40341
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit 0ea65d81d4d5431ca22340149dbd714e395ecdc3)

Conflicts:
src/rgw/services/svc_mdlog.cc
- file does not exist in nautilus; made the changes manually to
  src/rgw/rgw_metadata.cc

rgw: ReplaceKeyPrefixWith and ReplaceKeyWith can not set at the same time.
and support some HttpErrorCodeReturnedEquals and HttpRedirectCode limit.

fix https://tracker.ceph.com/issues/43563

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>
(cherry picked from commit 0b4689865db3ab8efc39a73070a5d60bd9992fbd)

rgw: set bucket attr twice when delete lifecycle config.

Fixes: https://tracker.ceph.com/issues/44093
Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>
(cherry picked from commit c480b029f205c417935df2478a6aa66211f13658)

Conflicts:
src/rgw/rgw_op.cc
- set_bucket_instance_attrs call looks different in nautilus, presumably
  due to post-nautilus refactoring

Merge pull request #34343 from Vicente-Cheng/wip-44480-nautilus

nautilus: mds: just delete MDSIOContextBase during shutdown

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #34308 from Vicente-Cheng/wip-44328-nautilus

nautilus: client: fix bad error handling in lseek SEEK_HOLE / SEEK_DATA

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34307 from Vicente-Cheng/wip-44337-nautilus

nautilus: mds: handle bad purge queue item encoding

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/workunits/mon: add check for min_size

verify whether min_size is recalculated when osd
pool size is changed.

fixes: https://tracker.ceph.com/issues/44862
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
(cherry picked from commit 0a7325a2b2ee82752b4191c240e7c7a6f14a2b3e)

Conflicts:
qa/workunits/mon/pool_ops.sh
- in nautilus, "ceph osd pool set foo size 1" does not take
  a "--yes-i-really-mean-it" option

mon: calculate min_size on osd pool set size

currently `osd pool set size` only modifies min_size when it is above new size,
while it should be recalculated unconditionally.

fixes: https://tracker.ceph.com/issues/44862
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
(cherry picked from commit b69dcaa18cc30d3091630606b981290c91091ce5)

mgr/dashboard: Add more debug information to Dashboard RGW backend

Fixes: https://tracker.ceph.com/issues/44914
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 4febdd6ddaddebd2a8d07f44c8d0f79524305766)

mgr/dashboard: Improve workaround to redraw datatables

The current approach to workaround this issue is to fire a 'resize' event to force the datatable to redraw the grid with correct column widths. This approach is like using a sledgehammer to crack a nut. The new approach simply uses Angular's change-detection mechanism. The datatable is marked as changed and Angular will do the rest for us. This should also finally solve the error that variables have been changed after Angular's change-detection has been done.

Fixes: https://tracker.ceph.com/issues/41787
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 08a5ab12ce17de2b4e54d6990e930fa4973d6ee2)

# Conflicts:
# src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.ts

monitoring: root volume full alert fires false positives

Fixes: https://tracker.ceph.com/issues/44780
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit f8e347f771cdd1df512507685efb5300e1dc87d0)

mgr/dashboard: use FQDN for failover redirection

Fixes: https://tracker.ceph.com/issues/44923
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit 2e790c1fbcbb67dfb1c433b80ee3261b5453d72c)

mgr/dashboard: Update bootstrap

Fixes: http://tracker.ceph.com/issues/39326
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit 07f28050dba61b851f149a2ed3b63cf5349c04cd)

nautilus: mgr/dashboard: Tabs does not handle click events
Fixes: https://tracker.ceph.com/issues/44809
Conflicts:
src/pybind/mgr/dashboard/frontend/angular.json
src/pybind/mgr/dashboard/frontend/package-lock.json
src/pybind/mgr/dashboard/frontend/package.json
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-details/iscsi-target-details.component.spec.ts
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-details/iscsi-target-details.component.ts
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-form/iscsi-target-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-image-settings-modal/iscsi-target-image-settings-modal.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-iqn-settings-modal/iscsi-target-iqn-settings-modal.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-list/iscsi-target-list.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-form/rbd-snapshot-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-list/rbd-trash-list.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/crushmap/crushmap.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/crushmap/crushmap.component.spec.ts
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/logs/logs.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-list/osd-list.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form-client/nfs-form-client.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form/nfs-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-form/pool-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-details/rgw-user-details.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-form/rgw-user-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/auth/sso/sso-not-found/sso-not-found.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/administration/administration.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/dashboard-help/dashboard-help.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/identity/identity.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/notifications/notifications.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/task-manager/task-manager.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/not-found/not-found.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.scss
src/pybind/mgr/dashboard/frontend/src/app/shared/components/grafana/grafana.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/components/loading-panel/loading-panel.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/components/select-badges/select-badges.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/components/submit-button/submit-button.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-actions/table-actions.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-actions/table-actions.component.ts
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-key-value/table-key-value.component.spec.ts
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.spec.ts
src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-notification.ts
src/pybind/mgr/dashboard/frontend/src/styles.scss
- all those files are not needed to fix the tab select issue.

monitoring: alert for prediction of disk and pool fill up broken

Fixes: https://tracker.ceph.com/issues/44776
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 6935dc55921ef533e939719bad7dc12324450bbb)

Conflicts:
  monitoring/prometheus/alerts/ceph_default_alerts.yml

mgr/dashboard: fix COVERAGE_PATH in run-backend-api-tests.sh

As we cannot backport directly https://github.com/ceph/ceph/pull/33407
to nautilus and COVERAGE_PATH is invalid for CentOS7 + py2,
this fix is applied directly to nautilus.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #34388 from smithfarm/wip-44896-nautilus

nautilus: pybind/rbd: fix no lockers are obtained, ImageNotFound exception will be output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mds: handle bad purge queue item encoding

The bad encoding was introduced by commit a88f8d5eb4

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit ba06fcbe3e8345dc2c4c4c3dc3bcc18acf5ab076)
Conflicts:
        src/mds/PurgeQueue.cc: advance changed to +=
Fixes: https://tracker.ceph.com/issues/36635
Note: This commit from v13.2.3 fixes a bad backport in v13.2.2. It is
      also required in Octopus/Nautilus to handle upgrades.
(cherry picked from commit b73d1989bcbea227017607f8dd6e79633ec11f8f)
Conflicts:
            src/mds/PurgeQueue.cc: += changed to advance

client: fix error handling in lseek SEEK_HOLE/SEEK_DATA cases

The error handling here looks quite wrong. If the offset represents
an area outside the file, then we want to just return -ENXIO
immediately.

Fixes: https://tracker.ceph.com/issues/44021
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 2ca8db71364698e05c3a6ac9a5aa4da5421a35bc)

Conflicts:
src/client/Client.cc
  - minor modification for nautilus code flow

test: add new ceph_lseek test

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 339b09dc5921f81621f76faa55c56c31c16d4937)

mds: don't assert empty io context list when shutting down

Fixes: https://tracker.ceph.com/issues/44680
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 7f26bb7bb768a90177601915050b4f5020902943)

Revert "mds: make MDSIOContextBase delete itself when shutting down"

This reverts commit 73436961512bd87981244fa48212085faf7028c4.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 18118974294ee1286b4cbae2f5e6578a5a68f6d6)

Revert "mds: just delete MDSIOContextBase during shutdown"

This reverts commit 79f5052a1ddb61043de4e1cbec19ede2a6b4f53b.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 02658de9eeb700364af02f2c7710c3819e776fb2)

mds: just delete MDSIOContextBase during shutdown

We don't want the finisher called during shutdown.

Fixes: https://tracker.ceph.com/issues/44295
Fixes: 73436961512bd87981244fa48212085faf7028c4
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 79f5052a1ddb61043de4e1cbec19ede2a6b4f53b)

mds: add comment explaining unusual locking for MDSIOContextBase

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit c2a079084c7653b6ed7e0912d4e737fe10a36836)