mgr/dashboard: Fix e2e host test

There is a random problem with mds service listing
that is causing a e2e test to fail.

This will temporarily fix this problem.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Increase the global value of jasmine timeout (#30359)

mgr/dashboard: Increase the global value of jasmine timeout

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #29761 from wjwithagen/wjw-fix-venv-python-version

test/venv: do not hardwire to py2.7 for tox tests

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix calculation of PG Status percentage (#30343)

mgr/dashboard: Fix calculation of PG Status percentage

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #30285 from IlsooByun/slow_op_report

mgr: do not reset reported if a new metric is not collected

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30112 from liewegas/wip-autoscaler-on

common: default pg_autoscale_mode=on for new pools

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29574 into master

* refs/pull/29574/head:
mds: Reorganize class members in FSMapUser header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29883 into master

* refs/pull/29883/head:
mds: Reorganize class members in InoTable header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30090 into master

* refs/pull/30090/head:
mds: Reorganize class members in Anchor header

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #30143 into master

* refs/pull/30143/head:
mds: Reorganize class members in LocalLock header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30205 into master

* refs/pull/30205/head:
mds: Reorganize class members in LogEvent header

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #30191 into master

* refs/pull/30191/head:
cls/hello: disable write returning data clobbering demo
messages/MOSDOpReply: assert no write result payloads
osd/PrimaryLogPG: assert write result == 0
osd/PrimaryLogPG: pass TMAPGET errors up
osd/PrimaryLogPG: set result=0 for successful writes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29792 from sebastian-philipp/orchestrator-api-replace

mgr/orchestrator: Document OSD replacement

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Tim Serong <tserong@suse.com>

common: default pg_autoscale_mode=on for new pools

Signed-off-by: Sage Weil <sage@redhat.com>

cls/hello: disable write returning data clobbering demo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29820 into master

* refs/pull/29820/head:
osd/PeeringState: log_weirdness during peering
osd/PeeringState: send new message types
osd/PeeringState: give require_osd_release to BufferedRecoveryMessages
osd: add new notify, query, and info messages
osd/PeeringState: use send_info() for replica activation ack
osd/PeeringState: remove old info_map member
osd/PeeringState: send infos via message_map (not info_map)
osd/PeeringState: remove old query_map member
osd/PeeringState: send queries via message_map (not query_map)
osd/PeeringState: use return value from discover_all_missing
osd/PeeringState: give PeeringCtxWrapper a BufferedRecoveryMessages ref
osd/PeeringState: send notifies via message_map (not notify_list)
osd/PeeringState: add message_map to PeeringCtx/BufferedRecoveryMessages
osd: dispatch_context inside PG lock

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #30146 from ivancich/wip-fix-min-unordered-bkt-list

rgw: fix minimum of unordered bucket listing

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

mgr/dashboard: Display the duration of each e2e test

Now we are able to see which tests are taking
too much time and may need improvements.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Increase the global value of jasmine timeout

Some tests are taking longer than the default jasmine timeout to finish
and are causing a failure.

This commit will increase the timeout to 5 minutes,
removing the need to increase it for each failing test.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #30106 from smithfarm/wip-more-cbs

script/ceph-backport.sh: make the script idempotent

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: key-value-table doesn't render booleans (#30203)

 mgr/dashboard: key-value-table doesn't render booleans

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fix error message not displayed in pool form (#30137)

mgr/dashboard: Fix error message not displayed in pool form

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28936 from tchaikov/wip-unaligned-access

cmake: check for unaligned access

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28939 from trociny/wip-37529

rbd-mirror: add namespace support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30192 from dzafman/wip-41669

Make dumping of reservation info congruent between scrub and recovery

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30347 from neha-ojha/wip-add-backend

qa/suites/rados/perf: test min recommended osd_memory_target

Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #30319 from zhangsw/rgw-fix-zone-rm

rgw: fix incorrect "radosgw-admin zonegroup rm" info.

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

qa/suites/rados/perf: test min recommended osd_memory_target

Add bluestore-basic-min-osd-mem-target.yaml to objectstore.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge PR #30346 into master

* refs/pull/30346/head:
script: raw_input was renamed to input in py3

Reviewed-by: Sage Weil <sage@redhat.com>

script: raw_input was renamed to input in py3

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30014 into master

* refs/pull/30014/head:
doc: update CephFS overview in introductory page
doc: filesystem to file system

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30221 from cbodley/wip-rgw-dns-name-iends-with

rgw: fix dns name comparison for virtual hosting

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #30310 from rzarzynski/wip-crimson-op_effects

crimson/osd: extend OpsExecuter to carry about op effects.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix calculation of PG Status percentage

We were reading a wrong value for the total of PGs.

Fixes: https://tracker.ceph.com/issues/41536
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: E2E enhancements (#30017)

mgr/dashboard: E2E enhancements

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29606 from sseshasa/bluestoreCfgObserver

os/bluestore: Add config observer for osd memory specific options.

Merge pull request #29793 from sebastian-philipp/ansible-default-realm-zonegroup

mgr/ansible: Change default realm and zonegroup

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #30321 from tchaikov/wip-run-cbt.sh

script/run-cbt.sh: always use python3

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Update protractor dependencies

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Enable waitforAngular in E2E

Convert waitForTableData into a method,
using it as a decorator was causing problems.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add protractor-fast-fail plugin

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add click call inside waitClickable

All waitClickable calls were being followed by a click.
This will remove the need for the click call.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix error in pool list component

Sometimes the pools list result comes without the tiers field.
This was causing an error in the code.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix random failure in images e2e test

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove duplicated method

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove helper.po.ts

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove E2E warning

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Extract delete method into page-helper

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Use suites in protractor

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Split user-mgmt e2e files

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Move some e2e into new "ui" folder

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Improve clickCheckbox with some waits

Signed-off-by: Tiago Melo <tmelo@suse.com>

script/ceph-backport.sh: help the user set up the script

This commit adds several options:

    --setup
    --setup-advice
    --usage-advice
    --troubleshooting-advice

It replaces the "comment block at the beginning of the script" which was lamely
masquerading as documentation.

It also implements heuristic deduction of GitHub remotes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

script/ceph-backport.sh: check Backport ticket assignee

Don't work on tickets assigned to someone else.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Improve images e2e test (#30212)

mgr/dashboard: Improve images e2e test

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pnawracay@suse.com>

mgr/dashboard: Wait for breadcrumb text is present in e2e tests (#30267)

mgr/dashboard: Wait for breadcrumb text is present in e2e tests

mgr/dashboard: Rendering datatable causes Angular runtime error (#30209)

mgr/dashboard: Rendering datatable causes Angular runtime error

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

script/run-cbt.sh: always use python3

run-cbt.sh will be used by the "ceph-perf-pull-requests" jenkins job,
which will prepare the runtime dependencies by installing python3
modules.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30318 from tchaikov/wip-run-make.sh

script/run-make.sh: do not pass cmake options twice

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: fix incorrect "radosgw-admin zonegroup rm" info.

Signed-off-by: zhang Shaowen <zhangshaowen@cmss.chinamobile.com>

script/run-make.sh: do not pass cmake options twice

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30292 from Reperator/wip-doc-dashboard-fix-typo

doc: Fixes typo for ceph dashboard command.

Merge pull request #30281 from sebastian-philipp/doc-orchestrator-feature-update

doc/orchestrator: Sync status with reality

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #30231 from xiexingguo/wip-blue-balance

os/bluestore: add bluestore_bluefs_max_free; smooth space balancing a bit

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #30315 from tchaikov/wip-crimson-perf-test

test/crimson: add acceptable section to tests

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: check for unaligned access

C and C++ allow us to cast a pointer with an alignment requirement to
one with a different alignment requirement. but if the destination
type's alignment is greater/stricter than the source type, the behavior
is undefined. on amd64, it's allowed. and on aarch64, GCC offers an
option named `-munaligned-access` to enable this behavior. so we cannot
take this as guaranteed.

in denc.h and messenger implementations, we just cast the pointers to
the type we want to access without memcpy(), which is, strictly
speaking, not quite standard compliant.

so, to be on the safe side, we detect the support of unaligned access at
compile time.

please note, this would fail the cross-compile. as the target platform
might not support unaligned access, and the produced executable does not
run on the build platform. so this test is skipped if the building
host's architecture is different than that of target host.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: extract checks into CephChecks.cmake

also use case chosen in the macro definition as per suggested by cmake
document. see
https://cmake.org/cmake/help/latest/command/macro.html#invocation

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #29809 into master

* refs/pull/29809/head:
common/config_proxy: hold lock while accessing mutable container

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: add acceptable section to tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30124 from rzarzynski/wip-crimson-conn-in-msg

crimson/osd: Message has non-null ref to SocketConnection now.

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30309 from neha-ojha/wip-41661

qa/suites/rados/perf/objectstore: do not symlink to qa/objectstore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30255 from xiexingguo/wip-bluestore-gift-ratio

os/bluestore: fix space balancing overflow

Reviewed-by: Sage Weil <sage@redhat.com>

crimson/osd: extend OpsExecuter to carry about op effects.

This commit brings a new infrastructural piece that will be
needed by e.g. the watch/notify mechanism.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #30308 into master

* refs/pull/30308/head:
doc/dev: add teuthology priority recommendations

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

osd: Replace active/pending scrub tracking for local/remote

This is similar to how recovery reservations are split between
local and remote.

It was the case that scrubs_pending was used for reservations at
the replicas as well as at the primary while requesting reservations
from the replicas.  There was no need for scrubs_pending to turn
into scrubs_active at the primary as nothing treated that value
as special.  scrubber.active = true when scrubbing is
actually going.

Now scurbber.local_reserved indicates scrubs_local incremented
Now scrubber.remote_reserved indicates scrubs_remote incremented

Fixes: https://tracker.ceph.com/issues/41669
Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add dump_scrub_reservations to show scrub reserve tracking

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Rename dump_reservations to dump_recovery_reservations

Signed-off-by: David Zafman <dzafman@redhat.com>

qa/suites/rados/perf/objectstore: do not symlink to qa/objectstore

Removing the symlink since we do not want to run performance tests
on filestore-xfs.yaml.

Fixes: https://tracker.ceph.com/issues/41661
Signed-off-by: Neha Ojha <nojha@redhat.com>

doc/dev: add teuthology priority recommendations

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30269 from dabukalam/patch-1

doc: Update ceph-deploy docs from dumpling to nautilus

Reviewed-by: Alfredo Deza <adeza@redhat.com>

doc: Update ceph-deploy docs from dumpling to nautilus

Signed-off-by: Danny Abukalam <danny@softiron.co.uk>

Merge pull request #30274 from alfredodeza/wip-rm41660

ceph-volume: fix stderr failure to decode/encode when redirected

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Andrew Schoen <aschoen@redhat.com>

mgr: do not reset reported if a new metric is not collected

Fixes: https://tracker.ceph.com/issues/41741
Signed-off-by: Ilsoo Byun <ilsoobyun@linecorp.com>

doc: update CephFS overview in introductory page

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30287 from tchaikov/wip-run-cbt

script/run-cbt.sh: check option correctly

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30290 from tchaikov/wip-doc-cbt

doc/dev/crimson.rst: document CBT testing

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc: filesystem to file system

"Filesystem" is not a word (although fairly common in use).

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd/PeeringState: log_weirdness during peering

Log this in a few more places so that we can catch issues sooner.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30289 from wjwithagen/wjw-fix-pylint-dashboard-progress.py

dashboard/services: fix lint error

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

dashboard/services: fix lint error

```
lint run-test: commands[0] | flake8
 many blank lines (3)
1     E303 too many blank lines (3)
```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

doc: Fixes typo for ceph dashboard command.

Signed-off-by: Fabian Bonk <fabian.bonk@croit.io>

os/bluestore: smooth space balancing a bit

For workloads that need to handle tons of small objects,
the space of "slow" could become extremely fragmented,
which makes it is hard to transmit free space between
bluestore and bluefs.

Hard cap alloc size at 1GB at per request, so there is
1024 small allocations at most. Hence we don't block
write ops for long.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

ceph-volume tests create a test file for checking unicode output

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests create a shell test for functional unicode

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests verify new logging fallback and encodings in terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume terminal remove unicode stream handler

This caused problems in environments where stderr was redirected, since
stderr sets the encoding to None. Getting it back again allows
everything to work correctly, and keeps all the current unit tests
passing

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: instantiate the new terminal logger in main()

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume create a logger for the terminal

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge PR #30155 into master

* refs/pull/30155/head:
mgr/MgrClient: send MMgrCommand for octopus+ mgrs
mon/MgrMap: include mgr_features in mgrmap
mgr: accept MMgrCommand[Reply] messages (in addition to MCommand[Reply])

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30162 from cyx1231st/wip-seastar-msgr-test-failover-crimson

crimson/test: v2 failover tests with crimson FailoverTestPeer

Reviewed-by: Kefu Chai <kchai@redhat.com>