rgw: reshard: skip stale bucket id entries from reshard queue

If we encounter a reshard queue entry that has an older ID compared to the
bucket's current ID, it'd mean that some other process or a manual reshard has
already processed this entry, skip processing the entry this time. An
alternative is to verify the num_shards that we have in queue >= the current
shards, but this would mean that we may reshard a recently manual resharded
bucket again which might not be intended

Fixes: https://tracker.ceph.com/issues/45134
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 02664fc091674e28233559cd1c42f954d5776d86)

Conflicts:
src/rgw/rgw_reshard.cc
- immediately preceding "get_bucket_info" call looks different in
  nautilus

Merge pull request #34551 from tspmelo/wip-45085-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #34550 from tspmelo/wip-45082-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34497 from votdev/wip-45019-nautilus

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34399 from votdev/wip-44920-nautilus

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34597 from smithfarm/wip-43134-nautilus

nautilus: rgw: find oldest period and update RGWMetadataLogHistory()

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34585 from smithfarm/wip-45040-nautilus

nautilus: mon: calculate min_size on osd pool set size

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34489 from rhcs-dashboard/nautilus-fix-coverage-path

nautilus: mgr/dashboard: fix COVERAGE_PATH in run-backend-api-tests.sh

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34394 from votdev/wip-44917-nautilus

nautilus: monitoring: alert for prediction of disk and pool fill up broken

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

Merge pull request #34392 from tchaikov/nautilus-cbt

nautilus: qa/suites: use "nautilus" branch for cbt based testing

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34609 from smithfarm/wip-44267-nautilus

nautilus: rgw: fix bug with (un)ordered bucket listing and marker w/ namespace

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34599 from smithfarm/wip-44232-nautilus

nautilus: rgw: ReplaceKeyPrefixWith and ReplaceKeyWith can not set at the same …

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34598 from smithfarm/wip-44136-nautilus

nautilus: rgw: set bucket attr twice when delete lifecycle config.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34278 from jan--f/wip-44688-nautilus

nautilus: ceph-volume: prepare: use *-slots arguments for implicit sizing

Merge pull request #34478 from jan--f/wip-44998-nautilus

nautilus: ceph-volume/batch: return success when all devices are filtered

Merge pull request #34481 from jan--f/wip-45002-nautilus

nautilus: ceph-volume/batch: check lvs list before access

Merge pull request #34445 from jan--f/wip-44974-nautilus

nautilus: devices/simple/scan: Fix string in log statement

Merge pull request #33616 from jan--f/wip-44325-nautilus

nautilus: ceph-volume: lvm: get_device_vgs() filter by provided prefix

ceph-volume: pass *-slots arguments to LV creation

Fixes: https://tracker.ceph.com/issues/44494
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 61f0c3af0b0f36a03748733b16de1b11048342e6)

use extent count for slots conversion instead of free count

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 4a81b4a2fbb4cafe45a8fe8cceeece636e51adb6)

ceph-volume: available_lvm: vg space takes precedence

This changes available_lvm to check for generic reasons only if no VGs
were found. A VG can contain a (mounted) lv, which triggers the
ro/locked test, despite the VG having space available.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c6d71eb0022935731bb9380098c975d9c5f27a86)

ceph-volume: don't create Volume/VolumeGroup objects with empty name

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 235b19eb66b9ba9beab99508f2fdf7b6afe26f31)

ceph-volume: get_device_lvs filter by provided prefix

Fixes: https://tracker.ceph.com/issues/44124
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit ac43d37f581510d1854e76afb634a142ad9a0fc0)

ceph-volume: lvm.get_device_vgs: don't create VolumeGroup with empty name

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 3c4c8c5e4a242985389c8d1c85b45cac89b2dc89)

ceph-volume: lvm: get_device_vgs() filter by provided prefix

Fixes: https://tracker.ceph.com/issues/44246
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit b48caf4873c11690d36a88760504f98b46101995)

Merge pull request #34282 from votdev/issue_44809_select_tab

nautilus: mgr/dashboard: Tabs does not handle click events

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34424 from smithfarm/wip-44838-nautilus

nautilus: pybind/rbd: ensure image is open before permitting operations

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

mgr/dashboard: Fix iSCSI's username and password validation

When using regex, instead of strings, angular doesn't assume the pattern must
validate the entire string.

Add validation to the backend.

Fixes: https://tracker.ceph.com/issues/44624
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit 532b271572f7b6410c099362dc046148f5a686a1)

Merge pull request #34618 from neha-ojha/wip-45135-nautilus

nautilus: mon: disable min pg per osd warning

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34520 from smithfarm/wip-45060-nautilus

nautilus: qa/suites/rados/rest: don't pass empty dict as data arg

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #34326 from rzarzynski/wip-bug-24995-nautilus

nautilus: mgr: synchronize ClusterState's health and mon_status.

Reviewed-by: Neha Ojha <nojha@redhat.com>

mon: disable min pg per osd warning

Now that the pg_autoscaler is on by default, it is "normal" (and okay) to
have a small number of PGs in the cluster if the overall cluster usage is
also low.  This setting just results in a health warning out of the box
when you create a pool and haven't written any data yet.

Fixes: https://tracker.ceph.com/issues/41735
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 1ac34a5ea3d1aca299b02e574b295dd4bf6167f4)

rgw: fix bug with (un)ordered bucket listing and marker w/ namespace

When listing without specifying a namespace, the returned entries
could be in one or more namespaces. The marker used to continue the
listing may therefore contain a namespace, and that needs to be
preserved. This fixes a bug in both ordered and unordered listings
where it was not preserved.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 9457abb729754458aafc07a9ee514407906a1011)

Conflicts:
src/rgw/rgw_rados.cc
- trivial (nautilus is missing some whitespace-finessing that happened in master)

rgw: when a period lookup for oldest_realm_epoch returns an ENOENT,
find the oldest one and update RGWMetadataLogHistory. This is to avoid an
empty cursor being passed in to ceph_assert() in PurgePeriodLogsCR::operate()
in case of incomplete period history.

Fixes: https://tracker.ceph.com/issues/40341
Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>
(cherry picked from commit 0ea65d81d4d5431ca22340149dbd714e395ecdc3)

Conflicts:
src/rgw/services/svc_mdlog.cc
- file does not exist in nautilus; made the changes manually to
  src/rgw/rgw_metadata.cc

rgw: ReplaceKeyPrefixWith and ReplaceKeyWith can not set at the same time.
and support some HttpErrorCodeReturnedEquals and HttpRedirectCode limit.

fix https://tracker.ceph.com/issues/43563

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>
(cherry picked from commit 0b4689865db3ab8efc39a73070a5d60bd9992fbd)

rgw: set bucket attr twice when delete lifecycle config.

Fixes: https://tracker.ceph.com/issues/44093
Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>
(cherry picked from commit c480b029f205c417935df2478a6aa66211f13658)

Conflicts:
src/rgw/rgw_op.cc
- set_bucket_instance_attrs call looks different in nautilus, presumably
  due to post-nautilus refactoring

Merge pull request #34343 from Vicente-Cheng/wip-44480-nautilus

nautilus: mds: just delete MDSIOContextBase during shutdown

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #34308 from Vicente-Cheng/wip-44328-nautilus

nautilus: client: fix bad error handling in lseek SEEK_HOLE / SEEK_DATA

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #34307 from Vicente-Cheng/wip-44337-nautilus

nautilus: mds: handle bad purge queue item encoding

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

qa/workunits/mon: add check for min_size

verify whether min_size is recalculated when osd
pool size is changed.

fixes: https://tracker.ceph.com/issues/44862
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
(cherry picked from commit 0a7325a2b2ee82752b4191c240e7c7a6f14a2b3e)

Conflicts:
qa/workunits/mon/pool_ops.sh
- in nautilus, "ceph osd pool set foo size 1" does not take
  a "--yes-i-really-mean-it" option

mon: calculate min_size on osd pool set size

currently `osd pool set size` only modifies min_size when it is above new size,
while it should be recalculated unconditionally.

fixes: https://tracker.ceph.com/issues/44862
Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>
(cherry picked from commit b69dcaa18cc30d3091630606b981290c91091ce5)

mgr/dashboard: Add more debug information to Dashboard RGW backend

Fixes: https://tracker.ceph.com/issues/44914
Signed-off-by: Volker Theile <vtheile@suse.com>
(cherry picked from commit 4febdd6ddaddebd2a8d07f44c8d0f79524305766)

mgr/dashboard: use FQDN for failover redirection

Fixes: https://tracker.ceph.com/issues/44923
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>
(cherry picked from commit 2e790c1fbcbb67dfb1c433b80ee3261b5453d72c)

mgr/dashboard: Update bootstrap

Fixes: http://tracker.ceph.com/issues/39326
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit 07f28050dba61b851f149a2ed3b63cf5349c04cd)

nautilus: mgr/dashboard: Tabs does not handle click events
Fixes: https://tracker.ceph.com/issues/44809
Conflicts:
src/pybind/mgr/dashboard/frontend/angular.json
src/pybind/mgr/dashboard/frontend/package-lock.json
src/pybind/mgr/dashboard/frontend/package.json
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-details/iscsi-target-details.component.spec.ts
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-details/iscsi-target-details.component.ts
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-form/iscsi-target-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-image-settings-modal/iscsi-target-image-settings-modal.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-iqn-settings-modal/iscsi-target-iqn-settings-modal.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/iscsi-target-list/iscsi-target-list.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-snapshot-form/rbd-snapshot-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/block/rbd-trash-list/rbd-trash-list.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration-form/configuration-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/configuration/configuration.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/crushmap/crushmap.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/crushmap/crushmap.component.spec.ts
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/logs/logs.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/cluster/osd/osd-list/osd-list.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form-client/nfs-form-client.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/nfs/nfs-form/nfs-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/pool/pool-form/pool-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-details/rgw-user-details.component.html
src/pybind/mgr/dashboard/frontend/src/app/ceph/rgw/rgw-user-form/rgw-user-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/auth/sso/sso-not-found/sso-not-found.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/auth/user-form/user-form.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/administration/administration.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/dashboard-help/dashboard-help.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/identity/identity.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/navigation/navigation.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/notifications/notifications.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/navigation/task-manager/task-manager.component.html
src/pybind/mgr/dashboard/frontend/src/app/core/not-found/not-found.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/components/critical-confirmation-modal/critical-confirmation-modal.component.scss
src/pybind/mgr/dashboard/frontend/src/app/shared/components/grafana/grafana.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/components/loading-panel/loading-panel.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/components/select-badges/select-badges.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/components/submit-button/submit-button.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-actions/table-actions.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-actions/table-actions.component.ts
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table-key-value/table-key-value.component.spec.ts
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.html
src/pybind/mgr/dashboard/frontend/src/app/shared/datatable/table/table.component.spec.ts
src/pybind/mgr/dashboard/frontend/src/app/shared/models/cd-notification.ts
src/pybind/mgr/dashboard/frontend/src/styles.scss
- all those files are not needed to fix the tab select issue.

monitoring: alert for prediction of disk and pool fill up broken

Fixes: https://tracker.ceph.com/issues/44776
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>
(cherry picked from commit 6935dc55921ef533e939719bad7dc12324450bbb)

Conflicts:
  monitoring/prometheus/alerts/ceph_default_alerts.yml

mgr/dashboard: fix COVERAGE_PATH in run-backend-api-tests.sh

As we cannot backport directly https://github.com/ceph/ceph/pull/33407
to nautilus and COVERAGE_PATH is invalid for CentOS7 + py2,
this fix is applied directly to nautilus.

Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #34388 from smithfarm/wip-44896-nautilus

nautilus: pybind/rbd: fix no lockers are obtained, ImageNotFound exception will be output

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mds: handle bad purge queue item encoding

The bad encoding was introduced by commit a88f8d5eb4

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit ba06fcbe3e8345dc2c4c4c3dc3bcc18acf5ab076)
Conflicts:
        src/mds/PurgeQueue.cc: advance changed to +=
Fixes: https://tracker.ceph.com/issues/36635
Note: This commit from v13.2.3 fixes a bad backport in v13.2.2. It is
      also required in Octopus/Nautilus to handle upgrades.
(cherry picked from commit b73d1989bcbea227017607f8dd6e79633ec11f8f)
Conflicts:
            src/mds/PurgeQueue.cc: += changed to advance

client: fix error handling in lseek SEEK_HOLE/SEEK_DATA cases

The error handling here looks quite wrong. If the offset represents
an area outside the file, then we want to just return -ENXIO
immediately.

Fixes: https://tracker.ceph.com/issues/44021
Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 2ca8db71364698e05c3a6ac9a5aa4da5421a35bc)

Conflicts:
src/client/Client.cc
  - minor modification for nautilus code flow

test: add new ceph_lseek test

Signed-off-by: Jeff Layton <jlayton@redhat.com>
(cherry picked from commit 339b09dc5921f81621f76faa55c56c31c16d4937)

mds: don't assert empty io context list when shutting down

Fixes: https://tracker.ceph.com/issues/44680
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 7f26bb7bb768a90177601915050b4f5020902943)

Revert "mds: make MDSIOContextBase delete itself when shutting down"

This reverts commit 73436961512bd87981244fa48212085faf7028c4.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 18118974294ee1286b4cbae2f5e6578a5a68f6d6)

Revert "mds: just delete MDSIOContextBase during shutdown"

This reverts commit 79f5052a1ddb61043de4e1cbec19ede2a6b4f53b.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 02658de9eeb700364af02f2c7710c3819e776fb2)

mds: just delete MDSIOContextBase during shutdown

We don't want the finisher called during shutdown.

Fixes: https://tracker.ceph.com/issues/44295
Fixes: 73436961512bd87981244fa48212085faf7028c4
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 79f5052a1ddb61043de4e1cbec19ede2a6b4f53b)

mds: add comment explaining unusual locking for MDSIOContextBase

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit c2a079084c7653b6ed7e0912d4e737fe10a36836)

mgr/dashboard: Prevent iSCSI target recreation when editing controls

Also prevents updating target auth, when this was not modified.

Fixes: https://tracker.ceph.com/issues/44659
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit 7d370ce22259a5b6a6c25995a1857d960dc5d27d)

mgr/dashboard: Remove duplicate code in iscsi.py

Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit 7291e82ffc878394debbbb00e81dd565c6f6a128)

Merge pull request #34524 from smithfarm/wip-45054-nautilus

nautilus: doc/releases/nautilus: restart OSDs to make them bind to v2 addr

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/releases/nautilus: restart OSDs to make them bind to v2 addr

Fixes: https://tracker.ceph.com/issues/43896
Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 4facd9daa524f1e2d77511ee6b1a18e108c07d31)

Conflicts:
doc/releases/nautilus.rst
- nautilus does not have 98ae20c2c874

qa/suites/rados/rest: Don't pass empty dict as data arg

Passing an empty 'args' dict as a data argument when calling
requests.get somehow confuses the transaction, causing it to fail. Pass
'None' instead.

Fixes: https://tracker.ceph.com/issues/43720
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit e20e6fc3295da836fb5f098d46d6190ab28195d5)

pybind/rbd: ensure image is open before permitting operations

Fixes: https://tracker.ceph.com/issues/44610
Signed-off-by: Mykola Golub <mgolub@suse.com>
(cherry picked from commit 99287a1e552387f8eec505bffdffae7f4674cc8b)

Conflicts:
src/pybind/rbd/rbd.pyx
- no "snap_exists", "snap_get_name", "snap_get_id",
  "mirror_image_create_snapshot", "mirror_image_get_mode", "config_set",
  "config_get", "config_remove", "snap_get_mirror_namespace",  in nautilus
- nautilus "mirror_image_enable" does not take any argument

pybind/rbd: fix no lockers are obtained, ImageNotFound exception will be output

No lockers are obtained, ImageNotFound exception will be output,
but tht image is always exist.when lockers number is zero,
Should not output any exceptions。

Fixes: https://tracker.ceph.com/issues/44613
Signed-off-by: zhangdaolong <zhangdaolong@fiberhome.com>
(cherry picked from commit a183aac978dac69f996250324975073a78cb476b)

Merge remote-tracking branch 'origin/nautilus' into nautilus

Conflicts:
src/msg/async/ProtocolV2.cc - rename of outcoming_bl to outgoing_bl

14.2.9

Merge pull request #34473 from Vicente-Cheng/wip-44995-nautilus

nautilus: mgr/dashboard: Fix error in unit test caused by timezone

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34055 from callithea/wip-44668-nautilus

nautilus: qa: use small default pg count for CephFS pools

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34354 from smithfarm/wip-44711-nautilus

nautilus: mgr/DaemonServer: fix pg merge checks

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Xiaoxi Chen <xiaoxi.chen@intel.com>

Merge pull request #34297 from vumrao/wip-vumrao-44818

nautilus: common/options: Disable bluefs_buffered_io by default again.

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #33889 from ifed01/wip-ifed-flexible-bluefs-nautilus

nautilus: os/bluestore: more flexible DB volume space usage

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: fix tests in order to match pg num conventions

Update the tests test_ganesha and test_rbd_mirroring in order
to match the PG num conventions. It prevents the health warning
'POOL_PG_NUM_NOT_POWER_OF_TWO' from being shown.

Signed-off-by: Tatjana Dehler <tdehler@suse.com>
(cherry picked from commit ee46e58e5dc779418492ef6b8c5e687f36ecb3f6)

test/store_test: test coverage for anti-spillover framework.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 57c6a79136a390c74a800d03bc595f3363718c2f)

 Conflicts:
src/os/bluestore/BlueFS.h
Future stuff tried to get into.

os/bluestore: introduce bluestore_volume_selection_reserved cfg option.

This is an alternative (and straightforward) way to specify barred space not allowed for 'use
some extra' BlueFS volume selector policy.
Disabled by default since it should depend on RocksDB settings and actual volume size.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 185e6a1adb2add91209d5f937c61c309528518f9)

os/bluestore: beautify RocksDBBlueFSVolumeSelector::dump output

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit fc84588536b4757bbe75065951a7215ffbcc0f60)

os/bluestore: introduce 'bluestore bluefs stats' admin socket command

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit f5568eb7209e43e6b916ea8450b2bd0c9484919a)

 Conflicts:
src/os/bluestore/BlueFS.cc
 Future stuff, legacy admin command iface
        src/os/bluestore/BlueStore.cc
 Need to use legacy bluefs_shared_bdev

os/bluestore: implement more intelligent DB volume space managment for BlueFS.

It allows excessive space usage for higher DB levels.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 5f72c376deb64562e5e88be2f22339135ac7372b)

 Conflicts:
src/common/legacy_config_opts.h
src/common/options.cc
Future options tried to get into.

ceph-volume/batch: check lvs list before access

Fixes: https://tracker.ceph.com/issues/44989
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit c6dcf3161b2d69b88acdda473b85c9dd2c8433e3)

rgw: reject control characters in response-header actions

S3 GetObject permits overriding response header values, but those inputs
need to be validated to insure only characters that are valid in an HTTP
header value are present.

Credit: Initial vulnerability discovery by William Bowling (@wcbowling)
Credit: Further vulnerability discovery by Robin H. Johnson <rjohnson@digitalocean.com>
Signed-off-by: Robin H. Johnson <rjohnson@digitalocean.com>

rgw: EPERM to ERR_INVALID_REQUEST

As per Robin's comments and S3 spec

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw: reject unauthenticated response-header actions

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d8dd5e513c0c62bbd7d3044d7e2eddcd897bd400)

msg/async/crypto_onwire: fix endianness of nonce_t

As a AES-GCM IV, nonce_t is implicitly shared between server and
client.  Currently, if their endianness doesn't match, they are unable
to communicate in secure mode because each gets its own idea of what
the next nonce should be after the counter is incremented.

Several RFCs state that the nonce counter should be BE, but since we
use LE for everything on-disk and on-wire, make it LE.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

msg/async/ProtocolV2: avoid AES-GCM nonce reuse vulnerabilities

The secure mode uses AES-128-GCM with 96-bit nonces consisting of a
32-bit counter followed by a 64-bit salt.  The counter is incremented
after processing each frame, the salt is fixed for the duration of
the session.  Both are initialized from the session key generated
during session negotiation, so the counter starts with essentially
a random value.  It is allowed to wrap, and, after 2**32 frames, it
repeats, resulting in nonce reuse (the actual sequence numbers that
the messenger works with are 64-bit, so the session continues on).

Because of how GCM works, this completely breaks both confidentiality
and integrity aspects of the secure mode.  A single nonce reuse reveals
the XOR of two plaintexts and almost completely reveals the subkey
used for producing authentication tags.  After a few nonces get used
twice, all confidentiality and integrity goes out the window and the
attacker can potentially encrypt-authenticate plaintext of their
choice.

We can't easily change the nonce format to extend the counter to
64 bits (and possibly XOR it with a longer salt).  Instead, just
remember the initial nonce and cut the session before it repeats,
forcing renegotiation.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Conflicts:
src/msg/async/ProtocolV2.cc [ context: commit 697aafa2aad2
  ("msg/async/ProtocolV2: remove unused parameter") not in
  nautilus ]
src/msg/async/ProtocolV2.h [ context: commit ed3ec4c01d17
  ("msg: Build target 'common' without using namespace in
  headers") not in nautilus ]

msg/async: rename outcoming_bl -> outgoing_bl in AsyncConnection.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
(cherry picked from commit 7997a3ea193344ea9782d2594cca295ac5bdf59d)

ceph-volume/batch: return success when all devices are filtered

batch should only return an error if some (but not all) devices are
filtered. When only some devices are filtered the resulting osd layout
could look very different from what a user expects. If all devies are
filtered just return success.

Fixes: https://tracker.ceph.com/issues/44994
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit cb3eade3d00fdcdb34c4156f3c488bbb8747cd6f)

mgr/dashboard: Fix error in unit test caused by timezone

Unit test was only accepting positive timezones

Fixes: https://tracker.ceph.com/issues/42812
Signed-off-by: Tiago Melo <tmelo@suse.com>
(cherry picked from commit f5e88e1d113d5fbeeac2e52a2b2f9b03ace174fa)

Merge pull request #34373 from votdev/wip-44690-nautilus

nautilus: mgr/dashboard: list configured prometheus alerts

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34433 from badone/wip-44260-nautilus

nautilus: selinux: Allow ceph to setsched

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #34434 from badone/wip-nautilus-ceph-mgr-httpd-selinux-denial

nautilus: selinux: Allow ceph-mgr access to httpd dir

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #31842 from sseshasa/wip-42662-nautilus

nautilus: osd/OSDMap: Show health warning if a pool is configured with size 1

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34120 from tspmelo/wip-rbd-error

nautilus: mgr/dashboard: Fix error when listing RBD while deleting or moving

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #34099 from shyukri/wip-44472-nautilus

nautilus: rgw: clear ent_list for each loop of bucket list

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33641 from tchaikov/nautilus-42477

nautilus: tools/rados: Unmask '-o' to restore original behaviour

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #33160 from shyukri/wip-43995-nautilus

nautilus: ceph_argparse: increment matchcnt on kwargs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34417 from smithfarm/wip-44952-nautilus

nautilus: mgr/dashboard: Repair broken grafana panels

Reviewed-by: Laura Paduano <lpaduano@suse.com>

devices/simple/scan: Fix string in log statement

A multiline string enclosed by () gained a comma, making it into a
tuple, which has no format() method.

Fixes: https://tracker.ceph.com/issues/44949
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 292ae0305db3e48a87db45bcc7ca8cb837a08ead)

selinux: Allow ceph to setsched

In several places, such as common/numa.cc we call sched_setaffinity
which requires this permission.

Fixes: https://tracker.ceph.com/issues/44196
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit 43103e0207bfacf02f7f9533b36443d65f95d718)

selinux: Allow ceph-mgr access to httpd dir

ceph-mgr loads modules which require read access and this causes a
denial on el7.

Fixes: https://tracker.ceph.com/issues/44216
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

dashboard: Convert FQDN to hostname in grafana panels

The $ceph_hosts variable contained the FQDN for hosts
while the instance label created by ceph only has
the hostname.

Fixes: https://tracker.ceph.com/issues/44784
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
(cherry picked from commit b7abaab5bd2b72f48f41b3d2a9732743afdcecfc)

dashboard: Resolve FQDN / hostname mismatch in hosts overview panel

In the AVG Disk Utilization panel, the result is calculated
by combining the output of node_disk_io_time_seconds_total
with the output of ceph_disk_occupation. However, the
first vector encodes the instance label with the full FQDN
while the ceph label only contains the hostname:port. In
order for these to match correctly, the domain name and port
has to be stripped from the labels.

Fixes: https://tracker.ceph.com/issues/44784
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
(cherry picked from commit 136d21e21dc3c05ec8c586a47eed1904ffbda578)

dashboard: Use exported_instance to identify OSDs

When moving to LVM-based ceph-volume setups, several
grafana dashboards stopped working. The problem is that
(device, instance) no longer results in unique labels
which causes errors like:

"many-to-many matching not allowed: matching labels must be unique on one side"

Fixes: https://tracker.ceph.com/issues/44784
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
(cherry picked from commit 8b61b8d3d781b9d762098cf45b41b596a32509ab)

dashboard: AVG RAM Utilization panel always showed "N/A"

The references to `$osd_hosts` etc. were encoded as
`[[osd_hosts]]` in the PromQL expression divisor, and
the panel always displayed N/A as the result of the
query.

Replacing the `[[...]]` with `$...` makes the expression
work again.

Fixes: https://tracker.ceph.com/issues/44784
Signed-off-by: Kristoffer Grönlund <kgronlund@suse.com>
(cherry picked from commit 4444333243aaa81de1d7347bf0c57e039b657d43)