mgr/dashboard: Fix random failure in images e2e test

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove duplicated method

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove helper.po.ts

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove E2E warning

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Extract delete method into page-helper

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Use suites in protractor

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Split user-mgmt e2e files

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Move some e2e into new "ui" folder

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Improve clickCheckbox with some waits

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Improve images e2e test (#30212)

mgr/dashboard: Improve images e2e test

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

mgr/dashboard: Wait for breadcrumb text is present in e2e tests (#30267)

mgr/dashboard: Wait for breadcrumb text is present in e2e tests

mgr/dashboard: Rendering datatable causes Angular runtime error (#30209)

mgr/dashboard: Rendering datatable causes Angular runtime error

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #30318 from tchaikov/wip-run-make.sh

script/run-make.sh: do not pass cmake options twice

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

script/run-make.sh: do not pass cmake options twice

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30292 from Reperator/wip-doc-dashboard-fix-typo

doc: Fixes typo for ceph dashboard command.

Merge pull request #30281 from sebastian-philipp/doc-orchestrator-feature-update

doc/orchestrator: Sync status with reality

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #30231 from xiexingguo/wip-blue-balance

os/bluestore: add bluestore_bluefs_max_free; smooth space balancing a bit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30315 from tchaikov/wip-crimson-perf-test

test/crimson: add acceptable section to tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #29809 into master

* refs/pull/29809/head:
common/config_proxy: hold lock while accessing mutable container

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: add acceptable section to tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30124 from rzarzynski/wip-crimson-conn-in-msg

crimson/osd: Message has non-null ref to SocketConnection now.

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30309 from neha-ojha/wip-41661

qa/suites/rados/perf/objectstore: do not symlink to qa/objectstore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30255 from xiexingguo/wip-bluestore-gift-ratio

os/bluestore: fix space balancing overflow

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #30308 into master

* refs/pull/30308/head:
doc/dev: add teuthology priority recommendations

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

qa/suites/rados/perf/objectstore: do not symlink to qa/objectstore

Removing the symlink since we do not want to run performance tests
on filestore-xfs.yaml.

Fixes: https://tracker.ceph.com/issues/41661
Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/dev: add teuthology priority recommendations

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30269 from dabukalam/patch-1

doc: Update ceph-deploy docs from dumpling to nautilus

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: Update ceph-deploy docs from dumpling to nautilus

Signed-off-by: Danny Abukalam <danny@softiron.co.uk>

Merge pull request #30274 from alfredodeza/wip-rm41660

ceph-volume: fix stderr failure to decode/encode when redirected

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #30287 from tchaikov/wip-run-cbt

script/run-cbt.sh: check option correctly

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30290 from tchaikov/wip-doc-cbt

doc/dev/crimson.rst: document CBT testing

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30289 from wjwithagen/wjw-fix-pylint-dashboard-progress.py

dashboard/services: fix lint error

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

dashboard/services: fix lint error

```
lint run-test: commands[0] | flake8
 many blank lines (3)
1     E303 too many blank lines (3)
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc: Fixes typo for ceph dashboard command.

Signed-off-by: Fabian Bonk <fabian.bonk@croit.io>

os/bluestore: smooth space balancing a bit

For workloads that need to handle tons of small objects,
the space of "slow" could become extremely fragmented,
which makes it is hard to transmit free space between
bluestore and bluefs.

Hard cap alloc size at 1GB at per request, so there is
1024 small allocations at most. Hence we don't block
write ops for long.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph-volume tests create a test file for checking unicode output

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create a shell test for functional unicode

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify new logging fallback and encodings in terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume terminal remove unicode stream handler

This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: instantiate the new terminal logger in main()

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume create a logger for the terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #30155 into master

* refs/pull/30155/head:
mgr/MgrClient: send MMgrCommand for octopus+ mgrs
mon/MgrMap: include mgr_features in mgrmap
mgr: accept MMgrCommand[Reply] messages (in addition to MCommand[Reply])

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30162 from cyx1231st/wip-seastar-msgr-test-failover-crimson

crimson/test: v2 failover tests with crimson FailoverTestPeer

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson.rst: document CBT testing

Signed-off-by: Kefu Chai <kchai@redhat.com>

script/run-cbt.sh: check option correctly

* s/source_dir/source-dir/: as the option is named "--source-dir", not
  "--source_dir"
* cd $build_dir, before launching/stopping vstart. as vstart.sh is
  supposed to run from the build directory

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30111 from tspmelo/wip-nfs-read

mgr/dashboard: Set RO as the default access_type for RGW NFS exports

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: progress: support rbd_support module async tasks (#29424)

mgr/dashboard: progress: support rbd_support module async tasks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #30095 from rjfd/wip-fix-progress-qa

qa/mgr/progress: fix timeout error when waiting for osd in event

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

mgr/dashboard: Wait for breadcrumb text is present in e2e tests

When the cluster is busy it might take some time until the tested component has loaded it's data and is rendered in the browser. Waiting for the breadcrumb text to be present might help to fix errors in the e2e tests.

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #30078 from rhcs-dashboard/fix-41602-master

mgr/dashboard: disable pylint's --py3k flag

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #30126 from dvanders/dvanders_spec

ceph.spec.in: reserve more memory per build job

Reviewed-By: Nathan Cutler <nculter@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30258 from tchaikov/wip-41721

test/cls_rbd/test_cls_rbd: update TestClsRbd.sparsify

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc/orchestrator: Sync status with reality

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

os/bluestore: fix space balancing overflow

When gifting, bluestore should not gift more than it actually has.
same for bluefs when reclaming.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30273 from neha-ojha/wip-documenting-ceph

doc/start/documenting-ceph.rst: make better doc recommendations

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc/start/documenting-ceph.rst: make better doc recommendations

I do not think "Editors" is relevant anymore.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #30187 into master

* refs/pull/30187/head:
script/backport-resolve-issue: better error message

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume create a new tox.ini for shell-based tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #30226 from neha-ojha/wip-41385-2

osd/MissingLoc.cc: do not rely on missing_loc_sources only

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed By: Xie Xingguo <xie.xingguo@zte.com.cn>

Merge PR #30196 into master

* refs/pull/30196/head:
qa/tasks/ceph: restart: stop osd, mark down, then start

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30268 from cbodley/wip-41719

cls/rgw: fix bilog trim tests in ceph_test_cls_rgw

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #30220 into master

* refs/pull/30220/head:
test: Allow fractional milliseconds to make test possible

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cls/rgw: fix bilog trim tests in ceph_test_cls_rgw

the bucket index prepare step no longer writes a bilog entry, so the
tests were expecting more entries

Fixes: https://tracker.ceph.com/issues/41719
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29127 from bk201/wip-40337

mgr/dashboard: orchestrator integration initial works

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

test/cls_rbd/test_cls_rbd: update TestClsRbd.sparsify

after #30061, extents returned by sparse read will always be
an empty map as long as the extents to be read is empty or the extent(s)
in it are empty, even of the objectstorage does not support sparse read.

Fixes: https://tracker.ceph.com/issues/41721
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30115 from alfredodeza/wip-rm41378-2

ceph-volume: pre-install python-apt and its variants before test runs

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30142 from votdev/issue_41648

mgr/dashboard: Handle always-on Ceph Manager modules correctly

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30079 from uweigand/issue-41605-fixes

Fix big-endian handling

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30087 from egggHang/ec_benchmark

test/erasure-code: add exception handling to k & m

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29892 from Yan-waller/wip-yanj-fixreed_sol_r6_op

osd: fix the missing default value m=2 of reed_sol_r6_op in profile

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30125 from majianpeng/msg-protocolv1-reduece-swap

msg/async/ProtocolV1: avoid unnecessary bufferlist::swap.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

PendingReleaseNotes: note about changes in `ceph osd erasure-code-profile set`

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

osd: add sanity check for invalid m/w of liber8tion

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

ec/jerasure: save the default m=2 of reed_sol_r6_op in profile

save the default value of m to profile so that it could be dumped
in command 'ceph osd erasure-code-profile get xxx', which is more
useful and friendly to user.

Signed-off-by: Yan Jun <yan.jun8@zte.com.cn>

Merge pull request #30109 from rzarzynski/wip-bug-25098-bluefs_layout_t

os/bluestore: verify disk layout of BlueFS

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30197 from bk201/wip-41675

mgr/orchestrator: fix refs property of progresses

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Handle always-on Ceph Manager modules correctly

Currently always-on modules are not marked as enabled in the WebUI and can be disabled. This PR will fix that.
Note, this PR will NOT implement code that will prevent a developer from trying to disable an always-on module through the REST API. The Mgr Python extension will throw an adequate exception.

This PR will also do:
* Remove old code fragments from a previous Mgr Module management UI that is obsolete now.
* Cleanup code in BaseMgrModule code.

Fixes: https://tracker.ceph.com/issues/41648
Signed-off-by: Volker Theile <vtheile@suse.com>

os/bluestore: add bluestore_bluefs_max_free option

This prevents bluefs from trying to acquire more space than
it actually needs, e.g., due to batch deletion of user data.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #30199 from IlsooByun/cmake_ptyhon_var

build/ops: fix build fail related to PYTHON_EXECUTABLE variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29986 into master

* refs/pull/29986/head:
mgr/pg_autoscaler: use 'stored' for pool_logical_used

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30139 into master

* refs/pull/30139/head:
mgr/DaemonServer: skip adjusting pgp_num when merging is in-progress

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30061 from xiexingguo/wip-sparse-read

os, osd: readv

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30059 from xiexingguo/wip-clean-vs-repair

osd/PG: fix _finish_recovery vs repair race

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30190 from tchaikov/wip-python-rpm-macro

install-deps.sh: install `python*-devel` for python*rpm-macros

Reviewed-by: Boris Ranto <branto@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

osd/MissingLoc.cc: do not rely on missing_loc_sources only

In 624ade487ea4aeaf988cc1767e0b293f76addd5b, we relied on missing_loc_sources
to check for strays and remove an OSD from missing_loc. However, it is
possible that missing_loc_sources is empty while there are still OSDs
present in missing_loc. Since the aim is to just remove a stray OSD from
missing_loc, we do not need to rely on missing_loc_sources. We still
clean missing_loc_sources if any stray is present in it.

Signed-off-by: Neha Ojha <nojha@redhat.com>

test: Allow fractional milliseconds to make test possible

Fixes: https://tracker.ceph.com/issues/41689
Signed-off-by: David Zafman <dzafman@redhat.com>

install-deps.sh: install `python*-devel` for python*rpm-macros

in 087ea813, we installed '*rpm-macros' for the macros, so we can have
access to the latest python packaging related macros for preparing the
build dependencies.

but we could run into https://bugs.centos.org/view.php?id=16379, if
we already have an old version of python-devel installed. as the newer
version of python-rpm-macros conflicts with it.

it was a chicken-and-egg problem, as we don't know the exact name of
*rpm-macros packages. that's why we chose to install all of them. but
we have to upgrade the existing python-devel package to resolve the
conflict. but the since there is no python3-devel in RHEL7/CentOS7,
what they have is python36-devel. so we have to hardwire the
`%{python3_pkgversion}` to "36" even before we have access to this
macro, and upgrade the python36-devel package beforehand. but this
renders installing the rpm-macro package less useful -- we intend to
use the macro offered by the package to figure out "36".

as a workaround, we pretend that we know the "main" version of python3
in current RHEL/CentOS. and always install python36-devel for
python-rpm-macros. as the former requires the latter.

once all python3*-devel on all builders are upgraded, we will be safe
to install '*rpm-macros' again without installing python36-devel first.
by then, we could revert this change, or continue installing
python36-devel until the distro bumps up the "main" python version to 3.7

Fixes: https://tracker.ceph.com/issues/41603
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22574 from aclamk/bs-allocator-ageing-test

bluestore/allocator: Ageing test for bluestore allocators.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30028 from tchaikov/wip-rpath

cmake: selectively rewrite install rpath

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #30015 from neha-ojha/wip-41429

mon/Monitor.cc: fix condition that checks for unrecognized auth mode

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Joao Eduardo Luis <joao@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30023 from tchaikov/wip-get-req

mon,osd: use get_req<> instead of static_cast<>(get_req())

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #30210 from ctrlaltdel/doc-typo

doc: Fix a typo in balancer documentation

Reviewed-by: Lenz Grimmer <LGrimmer@suse.com>

Merge pull request #29520 from pcuzner/add-events-mgr-module

mgr/k8sevents: Add mgr module for kubernetes event integration

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

mgr/dashboard: Improve images e2e test

The 'should move the image to the trash' e2e fails because the datatable content has not been updated at the time the test checks for the existence of the deleted image.

To fix that the helper function waits until the presence of the datatable row (the same approach is done by other helper functions, too).

Signed-off-by: Volker Theile <vtheile@suse.com>

doc: Fix a typo in balancer documentation

Signed-off-by: Francois Deppierraz <francois@ctrlaltdel.ch>

mgr/dashboard: Rendering datatable causes Angular runtime error

Due some timing effects it might happen that the re-rendering of the datatable causes an Angular runtime error caused by changes outside Angulars' change-detection.

Fixes: https://tracker.ceph.com/issues/41683
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge PR #29786 into master

* refs/pull/29786/head:
allow for multiple "build" dirs to exist

Reviewed-by: Sage Weil <sage@redhat.com>

build/ops: fix build fail related to PYTHON_EXECUTABLE variable

Fixes: https://tracker.ceph.com/issues/41676
Signed-off-by: Ilsoo Byun <ilsoobyun@linecorp.com>

ceph.spec.in: reserve 2500MB per build job

On our 14GB build machines, 8 parallel jobs results in the compiler
going OOM. Reserving 2500MB per job allows it to succeed with 5 jobs.

Fixes: https://tracker.ceph.com/issues/41659
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

allow for multiple "build" dirs to exist

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>