mgr/dashboard: RGW user E2E Tests

Tests creating, editing and deleting RGW users.
Tests providing invalid input to user creation and editing forms.

Fixes: https://tracker.ceph.com/issues/40364
Fixes: https://tracker.ceph.com/issues/40908
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

Merge pull request #29152 from tchaikov/wip-cmake-crimson-test

test/crimson/: use 256M mem and 1 cpu core for each test

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #29114 into master

* refs/pull/29114/head:
mgr/dashboard: qa: whitelist client eviction warning

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #29164 into master

* refs/pull/29164/head:
mgr: cleanup idle debug log at level 4.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29172 into master

* refs/pull/29172/head:
mgr/dashboard: Fix dashboard health test failure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29169 into master

* refs/pull/29169/head:
qa/suites/rados/mgr/tasks/module_selftest: whitelist mgr client getting backlisted

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26690 from jecluis/dev/wip-vstart-fix-rgw

vstart.sh: run multiple rgws with different ids

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29195 from ricardoasmarques/fix-ceph-iscsi-required-version

doc: Update 'ceph-iscsi' min version

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

vstart.sh: run multiple rgws with different ids

We should expect being able to run multiple rgws with different ids
using vstart, and yet that was not the case as they were sharing the
same name (and same key).

We now mimic each rgw running on a different host, with ids being their
ports; as such, we'll have 'client.rgw.X', 'client.rgw.X+1', and so
forth. These will match the ports the daemons are running on.

Signed-off-by: Joao Eduardo Luis <joao@suse.com>

Merge pull request #29132 from votdev/issue_40829

mgr/dashboard: RGW User quota validation is not working correctly

Reviewed-by: Ricardo Marques <rimarques@suse.com>

doc: Update 'ceph-iscsi' min version

Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #28986 from sebastian-philipp/orchestrator-verify-certificate

pybind/mgr: Remove code duplication

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29184 from neha-ojha/wip-1731894

mon/OSDMonitor.cc: better error message about min_size

Merge pull request #29070 from adk3798/reformat

mgr/dashboard: Update formatting of e2e test files

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29042 from sebastian-philipp/orchestrator-doc-disable

mgr/orchestrator: Fix disabling the orchestrator

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #29058 from tchaikov/wip-ceph-conf

tool/ceph-conf: s/global_pre_init()/global_init()/

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29151 from tchaikov/wip-rgw-kill-gcc-warnings

rgw/rgw_crypt.cc: silence -Wsign-compare GCC warning

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29155 from tchaikov/wip-msg-cleanup

msg/Message: remove unused local variables

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mon/OSDMonitor.cc: better error message about min_size

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29183 from gabriellasroman/patch-2

doc/dev/developer_guide: remove web address

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #29136 into master

* refs/pull/29136/head:
Client: bump ll_ref from int32 to uint64_t

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29119 into master

* refs/pull/29119/head:
common: add ref header

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28686 into master

* refs/pull/28686/head:
mds: cleanup truncating inodes when standby replay mds trim log segments

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28628 into master

* refs/pull/28628/head:
pybind: Fixes print of path as byte object in error message
pybind: Print standard error messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28557 into master

* refs/pull/28557/head:
ceph_volume_client: don't convert None to str object
ceph_volume_client: convert string to bytes object
ceph_volume_client: make UTF-8 encoding explicit

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #28447 into master

* refs/pull/28447/head:
cephfs-shell: Fix print of error messages to stdout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22645 into master

* refs/pull/22645/head:
qa/tests: test if unresponsive MDS client with no caps is evicted directly
qa/tests: add a method to signal a MDS client
qa/cephfs: memoize FUSE client pid

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/dev/developer_guide: remove web address

Removed http://pulpito.ovh.sepia.ceph.com:8081/
in "The nightlies" because it was unreachable

Signed-off-by: gabriellasroman <gsroman@bu.edu>

Merge PR #29121 into master

* refs/pull/29121/head:
qa/suites/rados/multimon: whitelist SLOW_OPS while thrashing mons

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #29052 into master

* refs/pull/29052/head:
common/ceph_time: make operator<< for timespan less useless
include/encoding: fix encode/decode for signed durations
osd/osd_types: make pg_history_t operator<< more concise
osd: do not rebind heartbeat messengers on wrongly marked down
ceph_test_osd_stale_read: add StaleRead test
osd/PeeringState: restructure init_primary_up_acting a bit
msg/async: add ms_blackhole_$type options

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #29089 into master

* refs/pull/29089/head:
mon/MgrMonitor: fix null deref when invalid formatter is specified

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge PR #29093 into master

* refs/pull/29093/head:
osd: set collection pool opts on collection create, pg load

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29096 into master

* refs/pull/29096/head:
pybind/rados: put lens array in outer scope
pybind/rados: fix set_omap() crash on py3

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29109 into master

* refs/pull/29109/head:
qa/tasks/ceph_manager: wait for clean before asserting clean on minsize test

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29144 into master

* refs/pull/29144/head:
qa/tasks/ceph_manager: fix thrash_pg_upmap_items when no pools

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29171 from smithfarm/wip-release-notes-14.2.2-take-two

doc: nautilus 14.2.2 release notes, take three

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29165 from dillaman/wip-40865

cls: reduce log level for non-fatal errors

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #28988 from zhangsw/rgw-fix-bug-listobjectv2

rgw: continuation token doesn't work in list object v2 request.

doc/releases/nautilus.rst: fix typo in name

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/nautilus: add missing PRs to 14.2.2

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc: release/nautilus.rst: drop stray "b'" strings

The ceph-release-notes script renders strings as bytes and these get exported
as b'<str>'. . . the script needs some more work to play nicely with Python 3.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

mgr/dashboard: Fix dashboard health test failure

Fixes: https://tracker.ceph.com/issues/40869
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Revert "doc/release/nautilus: 14.2.2 changes redone"

This reverts commit 7962a36849b91a3a11fa8a2fe90238b101868c7d.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

qa/suites/rados/mgr/tasks/module_selftest: whitelist mgr client getting backlisted

The mgr's libcephfs client gets evicted after the mgr fails over.
Whitelist the message.

Fixes: http://tracker.ceph.com/issues/40867
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #27420 from pooja-gautam/wip-dashboard-progress-bar

mgr/dashboard Displays progress bar in notification tray for background tasks

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29135 from dillaman/wip-40486

doc/rbd: re-organize top-level and add live-migration docs

Reviewed-by: Mykola Golub <mgolub@suse.com>

cls/journal: reduce verbosity of debug logs for non-errors

Fixes: http://tracker.ceph.com/issues/40865
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: reduce verbosity of debug logs for non-errors

Fixes: http://tracker.ceph.com/issues/40865
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Update formatting for dashboard e2e test files

Fixes: https://tracker.ceph.com/issues/40789
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

mgr/dashboard Displays progress bar in notification tray

Signed-off-by: Pooja <pooja.gautam@ts.fujitsu.com>

mgr: cleanup idle debug log at level 4.

Changed all messages to 10 that are printed repeatedly in an idle cluster.

This makes debugging Python modules easier as the log
is no longer pullued by mgr core messages.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/rbd: initial live-migration documentation

Fixes: http://tracker.ceph.com/issues/40486
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: RGW User quota validation is not working correctly

Fixes: https://tracker.ceph.com/issues/40829
Signed-off-by: Volker Theile <vtheile@suse.com>

msg/Message: remove unused local variables

and avoid creating a temporary instance of `ceph_msg_footer`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28764 from rhcs-dashboard/rgw-bucket-placement-target

mgr/dashboard: select placement target on RGW bucket creation

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test/crimson/: use 256M mem and 1 cpu core for each test

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw/rgw_crypt.cc: silence -Wsign-compare GCC warning

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29100 from tchaikov/wip-cmake-python-osx

cmake: use latest FindPython*.cmake

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29012 into master

* refs/pull/29012/head:
os/bluestore: proper locking for BlueFS prefetching

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29068 into master

* refs/pull/29068/head:
os/bluestore: cleanup around allocator calls

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29072 into master

* refs/pull/29072/head:
qa/suites/upgrade: set pg_autoscale_mode=off on existing pools
mon/MgrMonitor: make pg_autoscaler always on
qa/tasks/ceph.conf.template: osd_pool_default_pg_autoscale_mode = off

Reviewed-by: Sébastien Han <seb@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29085 into master

* refs/pull/29085/head:
os/bluestore: add slow op detector for collection listing
os/bluestore: parametrize latency threshold for log_latency funcs..
os/bluestore: cleanup around slow op logging.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29145 into master

* refs/pull/29145/head:
doc/release/nautilus: 14.2.2 changes redone

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

doc/dev/macos.rst: update to be in sync with master

* we've moved to cmake 3.5, so no need to hack for newer cmake
* nss dependency is dropped, no need to install it anymore
* pass clang and clang++ using cmake variable to be more consistent
* pass CMAKE_EXE_LINKER_FLAGS as `/usr/local/opt/llvm/lib` is
  not in default library paths.
* mention https://github.com/boostorg/atomic/issues/15

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass PYTHON_VERSION explicitly

allow `distutils_*()` functions to choose the python according to this
parameter, instead of implicitly using the variable of ${PYTHON_VERSION}
inherited from the caller.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: update FindPython* modules

use the ones shiped from the latest cmake. which

* enables us to find the recent python intepreter and development files,
* find intepreter and development in a single `find_pacakge()` command,
  simpler this way and less error prone.

and to accomodate this change:

* all `PYTHON${PYTHON_VERSION}_*` variables are renamed to
  `Python${PYTHON_VERSION}_*` if we use `find_package(Python2...)` or
  `find_package(Python3...)` to find python2 or python3 instead of using
  `find_package(Python...)`.
* use "2" explicitly when using python2, as `Python_*` variables are not
  defined anymore
* when compiling python support of ceph-mgr, continue using `Python_*`
  variables. because we find the python interpreter and development
  files using `find_pacakge(Python...)` for ceph-mgr.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tests: test if unresponsive MDS client with no caps is evicted directly

Test if an unresponsive MDS client session holding no caps is evicted
directly at session_autoclose without being marked as stale at
session_timeout.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #29080 from tchaikov/wip-pybind

pybind: check "CEPH_LIBDIR" not "MAKEFLAGS"

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28763 from tchaikov/wip-dpdk-fix-ftbfs

msg/async/dpdk: fix FTBFS

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29120 from neha-ojha/wip-add-write-omap

qa: add basic omap testing capability

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/release/nautilus: 14.2.2 changes redone

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/perf: add op_size to radosbench_omap_write.yaml

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/tasks/ceph_manager: fix thrash_pg_upmap_items when no pools

Follow-on to e7ca5a92d4b666f9933779c708614b3575bade5a

Fixes: https://tracker.ceph.com/issues/40635
Signed-off-by: Sage Weil <sage@redhat.com>

common: add ref header

This enables the use of these handy ref templates outside of Message.h.
This is necessary for a future rework of #26348.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29123 into master

* refs/pull/29123/head:
test_sessionmap: use sudo_write_file() from teuthology.misc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28946 from cbodley/wip-rgw-list-v1-close-section

rgw: add missing close_section in send_versioned_response

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Mark Kogan <mkogan@redhat.com>

Client: bump ll_ref from int32 to uint64_t

ll_ref can surge up if application repeatly lookup same dentry
and finally cause overflow.

https://tracker.ceph.com/issues/40775 is an example.

in kernel fuse driver and libfuse, uint64_t is used, update
ceph-fuse side to match.

Fixes: https://tracker.ceph.com/issues/40775
Signed-off-by: Xiaoxi CHEN <xiaoxchen@ebay.com>

Merge pull request #29101 from valentinbajrami/master

mgr/dashboard: Improve e2e script

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Rafael Quintero <rquinter@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/suites/upgrade: set pg_autoscale_mode=off on existing pools

Signed-off-by: Sage Weil <sage@redhat.com>

doc/rbd: tweak config reference

Clean up the caching sections to match the current implementation and
clarify which sections apply only to librbd.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: select placement target on RGW bucket creation

* Select a placement target from the zone that the RGW daemon is running on.

Fixes: https://tracker.ceph.com/issues/40567
Signed-off-by: alfonsomthd <almartin@redhat.com>

doc/man: note that rbd-fuse is not intended for production workloads

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

doc/rbd: restructure into operations and integrations

This avoids the long table of contents list of intermingled subjects.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29129 from tchaikov/wip-remove-sepia-bt.sh

script/sepia_bt.sh: remove stale script

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

script/sepia_bt.sh: remove stale script

this script is not actively maintained anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29057 from cyx1231st/wip-seastar-msgr-loggingv2

crimson/net: clean-up and fixes of messenger

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29105 from tspmelo/wip-prevent-propagation

mgr/dashboard: Disable event propagation in the helper icon

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29107 from ricardoasmarques/label-col-size-mgr-module-form

mgr/dashboard: Increase column size on mgr module form

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29112 from ricardoasmarques/iscsi-gateway-status-health-page

mgr/dashboard: Show iSCSI gateways status in the health page

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #28942 from ricardoasmarques/validate-iscsi-controls

mgr/dashboard: Validate iSCSI controls min/max value

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28898 from ricardoasmarques/cephfs-client-evict

mgr/dashboard: Evict a CephFS client

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Improve e2e script

Passing -r flag to jq to remove quotes. sed not needed
Using $() command substitution instead of the legacy and unsafe backticks `...`

Signed-off-by: Valentin Bajrami <valentin.bajrami@gmail.com>

Merge pull request #28081 from sebastian-philipp/kubejacker-doc-update

doc/dev/kubernetes: Update

Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #29125 from bk201/wip-40827

mgr/dashboard: fix tox test failure

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Evict a CephFS client

Fixes: https://tracker.ceph.com/issues/24892
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Show iSCSI gateways status in the health page

Fixes: https://tracker.ceph.com/issues/39028
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: fix tox test failure

`api/task` now also returns tasks in progress module.
Mock related function for tests.

Fixes: https://tracker.ceph.com/issues/40827
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

test_sessionmap: use sudo_write_file() from teuthology.misc

Instead of defining a method to write files within the testsuite, use
sudo_write_file() from teuthology.misc.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #29067 into master

* refs/pull/29067/head:
qa/tasks: add exception in do_thrash()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks: add exception in do_thrash()

Fixes the bug during DaemonWatchdog testing:

Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-daemonwatchdog-testing13/qa/tasks/daemonwatchdog.py", line 38, in _run
    self.watch()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-daemonwatchdog-testing13/qa/tasks/daemonwatchdog.py", line 112, in watch
    if thrasher.e is not None:
AttributeError: MonitorThrasher instance has no attribute 'e'

Signed-off-by: Jos Collin <jcollin@redhat.com>