qa/suites/rgw: disable valgrind for tests that require py2/ubuntu

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw: disable remaining ragweed test

Needs to be py3 since it relies on the python bindings.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw: pin swift tests to py2/ubuntu

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rgw: ragweed on ubuntu

ragweed uses nosetests; stick to py2 for now.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites: run s3tests on ubuntu

Note that this does not capture the rgw/website test on valgrind, since
valgrind is centos only.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32198 into master

* refs/pull/32198/head:
cephadm: Fix Python2.7 isssue
cephadm: fix how uid/gid is determined
cephadm: add sample configuration for prometheus
cephadm: support deployment of prometheus

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32211 from rhcs-dashboard/43284-fix-frontend-e2e-script

mgr/dashboard: run-frontend-e2e-tests.sh: allow user defined BASE_URL…

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31819 from ricardoasmarques/wip-42687

mgr/dashboard: iSCSI targets not available if any gateway is down

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #32246 into master

* refs/pull/32246/head:
mgr/cephadm: remove redundant /dev when blinking device light

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #32257 from dillaman/wip-41556

rbd-mirror: extract journal replaying logic from image replayer

Reviewed-by: Mykola Golub <mgolub@suse.com>

cephadm: Fix Python2.7 isssue

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: CephFS quota management (#31811)

mgr/dashboard: CephFS quota management

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: add password expiration date (#30939)

mgr/dashboard: add password expiration date

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Display OSD device class in OSD list (#32092)

mgr/dashboard: Display OSD device class in OSD list

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31815 from tchaikov/wip-mgr-dashboard-gevent

mgr/dashboard: install teuthology using pip

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Ernesto Puertat <epuertat@redhat.com>

rbd-mirror: switch image replayer to new standalone journal replayer

Remove all the original journal replaying code embedded in the image
replayer and instead rely on the new journal replayer class.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: extract journal replay logic to its own class

This will help to greatly reduce the journal-specific code in the
current image replayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved image asok/perf counter name helper to utils

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved event pre-processor to sub-namespace

The current implementation is tied to journal-based replaying so
move it down into the journal sub-namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: moved replay status formatter to sub-namespace

The current implementation is tied to journal-based replaying so
move it down into a new journal sub-namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: drop 'using namespace' in image replayer state machine

This is going to cause conflicts w/ the '::journal' namespace and
future 'rbd::mirror::image_replayer::journal' namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32142 from changchengx/doc_libvirt

doc/rbd: simplify libvirt usage

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32243 from trociny/wip-43289

rbd-mirror: fix race on namespace replayer initialization failure

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32184 from trociny/wip-43004

rbd-mirror: rename per-image replication perf counters

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32113 from lixiaoy1/master

librbd: finish write request early

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #32251 into master

* refs/pull/32251/head:
mds/MDSDaemon: fix asok 'exit' and 'respawn' commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32260 into master

* refs/pull/32260/head:
.organizationmap: Wido 42on -> 42on

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephadm: fix how uid/gid is determined

The uid/gid is determined by looking at file ownership. It
was looking at /etc/ceph, but on current builds this is owned by root so switched to using /var/lib/ceph which
is the homedir of the ceph account.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: add sample configuration for prometheus

Provides an example definition that sets up an
initial prometheus server with a scrape of itself.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

cephadm: support deployment of prometheus

Initial support to enable cephadm to deploy
monitoring containers like prometheus. This
patch adds support for prometheus. It is the callers
responsibility to provide a valid prometheus.yml
file passed within the --config-json parameter.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

.organizationmap: Wido 42on -> 42on

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32158 into master

* refs/pull/32158/head:
cephadm: fix exception when no time sync is running
cephadm: properly extract osd fsid during adoption
cephadm: do ceph-volume activate+deactivate as part of systemd unit
ceph-daemon: behave with no /etc/ceph/ceph.conf

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

cephadm: fix exception when no time sync is running

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: properly extract osd fsid during adoption

This needs to be passed to the deploy_daemon_units method so that we can
set up the pre and post c-v calls properly.

Signed-off-by: Sage Weil <sage@redhat.com>

cephadm: do ceph-volume activate+deactivate as part of systemd unit

1- Do the activate as part of the unit on start, since it may include
setup that disappears on a reboot, like starting up the dmcrypted volume.

2- Do the matching deactivate step when the container stops, like shutting
down the dmcrypt volume.

Right now the OSD is the only thing that needs this special behavior.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: behave with no /etc/ceph/ceph.conf

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32244 into master

* refs/pull/32244/head:
qa/workunits/cephadm/test_cephadm.sh: re-enable `adopt` tests

Reviewed-by: Sage Weil <sage@redhat.com>

mds/MDSDaemon: fix asok 'exit' and 'respawn' commands

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32253 into master

* refs/pull/32253/head:
mgr/orchestrator: make hosts and label args consistent

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32209 into master

* refs/pull/32209/head:
mgr/cephadm: Add unittest for service_action

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: make hosts and label args consistent

- hosts is always n=N; labels is not
- hosts always before labels

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #32192 into master

* refs/pull/32192/head:
common: handle return value from read(2)

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32195 into master

* refs/pull/32195/head:
osd/PeeringState.cc: skip peer_purged when discovering all missing

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #32203 into master

* refs/pull/32203/head:
Revert "mgr/DaemonServer: safe-to-destroy - do not consider irrelevant pgs"

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32219 into master

* refs/pull/32219/head:
common: move xattr -> os/filestore/os_xattr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #32250 into master

* refs/pull/32250/head:
mgr/cephadm: fix temp file prefixes
qa: fix lingering ceph-mgr-ssh -> ceph-mgr-cephadm refs

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32144 into master

* refs/pull/32144/head:
doc: mention multimds suite in dev guide
doc: clarify difference between fs and kcephfs suite

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #32024 into master

* refs/pull/32024/head:
stop: Unmount cephFS volumes

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/cephadm: fix temp file prefixes

Signed-off-by: Sage Weil <sage@redhat.com>

qa: fix lingering ceph-mgr-ssh -> ceph-mgr-cephadm refs

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32226 from neha-ojha/wip-four-percent

doc/rados: Better block.db size recommendations for bluestore

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

doc/rados: Better block.db size recommendations for bluestore

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32040 from smithfarm/wip-rn-13-2-8-mimic

doc/releases: add release notes for mimic v13.2.8

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/cephadm: remove redundant /dev when blinking device light

Fixes: https://tracker.ceph.com/issues/43223
Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases: add Mimic v13.2.8 to releases timeline

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge PR #32236 into master

* refs/pull/32236/head:
pybind/mgr: add_tox_test: Add mypy to TOX_ENVS

Reviewed-by: Sage Weil <sage@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: re-enable `adopt` tests

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: Avoid escaped quotes using TSLint

The problem was that the TSLint linting hasn't matched the
linting of prettier regarding the quotes of a string.

Explanation:
'\'' will be converted to "'" with prettier, but TSLint thrown an
error that it wanted the string to look like '\''.

Now TSLint will not error if it sees an single quote inside a double
quote to match the behavior of prettier.

Fixes: https://tracker.ceph.com/issues/38287
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Allow string descriptions in confirmation modal

Fixes: https://tracker.ceph.com/issues/38287
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add binary min and max validators

Adds simple validators that resemble the min and max validators for
numbers.

As a binary size (the string) has first be converted into bytes
(the number) to validate against some other number and in case of an
error the other number want's to be converted into a size to display,
the whole process needs a bunch of conversion, but these validators take
care of all that.

Fixes: https://tracker.ceph.com/issues/38287
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Make form modal more flexible

Now any input type is supported and the special binary type which will
use the cd-binary input and automatically format the binary size into
bytes when submitting.

Now any field can have custom validators and error messages.
The form will fallback on predefined error messages if not configured.
It will use the error messages provided by the binary min and max
validators. It still provides the error message for a required field.

Fixes: https://tracker.ceph.com/issues/38287
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #32005 from s0nea/wip-dashboard-reactivate-suites

mgr/dashboard: reactivate dashboard test suites

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

pybind/mgr: add_tox_test: Add mypy to TOX_ENVS

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: CephFS quota management

Now both CephFS quotas can be changed with a validation against the next
tree maximum in the file tree, that prevents setting the quotas in a way
that would not be usable.

Fixes: https://tracker.ceph.com/issues/38287
Signed-off-by: Stephan Müller <smueller@suse.com>

rbd-mirror: fix race on namespace replayer initialization failure

Fixes: https://tracker.ceph.com/issues/43289
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #31851 from votdev/issue_42609_device_lights

mgr/dashboard: Add support for blinking enclosure LEDs

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

stop: Unmount cephFS volumes

When stop.sh is run while the cephFS is mounted,
mount processes are retained and can't be killed.
And also the mount and df commands hang after this.
This patch unmounts the mounted volumes cleanly.

Signed-off-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #32186 from rhcs-dashboard/43263-fix-rgw-subuser-key

mgr/dashboard: fix RGW subuser auto-generate key.

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31753 from smithfarm/wip-bci-resolve-fix

backport-create-issue: resolve parent only if parent has backport issues

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Simplify PrometheusService::getRules implementa… (#32130)

mgr/dashboard: Simplify PrometheusService::getRules implementation

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: iSCSI targets not available if any gateway is down

Fixes: https://tracker.ceph.com/issues/42687
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/cephadm: Add unittest for service_action

(That was missing previously)

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32177 from Takios/ceph_volume_fix

ceph-volume: use correct extents if using db-devices and >1 osds_per_device

mgr/dashboard: reactivate dashboard test suites

Reactivate the dashboard test suites that were commented
out in https://github.com/ceph/ceph/pull/30864 because
https://tracker.ceph.com/issues/41538 has been resolved
in the meanwhile.

Fixes: https://tracker.ceph.com/issues/42652
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: add password expiration warning banner

It adds a new component to notify the user about the expiration
of the password. The banner is shown on top of the page and can
be dismissed by the user.

Fixes: https://tracker.ceph.com/issues/40329
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: add password expiration date

Add a 'pwd_expiration_date' field to the User object to be able
to set password expiration date per user.

There are two options to set a password expiration date for a
user:

1. Set the 'USER_PWD_EXPIRATION_SPAN' setting. If defined
   the expiration date of the password will be calculated
   automatically based on the given value. It will also be
   updated automatically when the user changes the password.
2. Set an expiration date by setting the 'pwdExpirationDate'
   field when creating a user.

Add two more settings in addition: USER_PWD_EXPIRATION_WARNING_1
and USER_PWD_EXPIRATION_WARNING_2. These two settings are
defining the amount of days to notify the user that his password
will expiration soon.

It's not possible to set the 'pwd_expiration_date' field to a
date in the past.

If the password of a user is already expired he is no longer
allowed to log into the dashboard.

Fixes: https://tracker.ceph.com/issues/40329
Signed-off-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #32224 from tspmelo/wip-e2e-insecure

mgr/dashboard: Fix e2e chromedriver problem

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #31255 into master

* refs/pull/31255/head:
mds/MDSDaemon: make con disposable before passing message to asok
mds/SessionMap: interpret bare id as a valid filter
mds: get rid of unused send_command_reply helpers
mds: remove old handle_command machinery
mds/MDSRank: remove C_ExecAndReply helpers
mds/MDSRank: too many words about op_tracker
mds: convert 'cache drop'
mds: convert 'damage ls' and 'damage rm'
mds: convert '{session,client} config'
mds: convert 'scrub {start,abort,pause,resume,status}'
mds: convert 'session ls' and 'client ls'
mds: convert 'session evict' and 'session kill'
mds: convert 'heap' and 'cpu_profiler'
mds: convert 'exit', 'respawn'
mds/MDSDaemon: remove old duplicated tell commands
mds: send tell commands to asok infrastructure
common/admin_socket: switch tell queue to cref_t (from ref_t)
mds: switch asok calls over to call_async interface

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #32230 from shellwedance/ceph-volume_typo_fix

doc: fix a typo in a command

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #32170 from rosinL/wip-rss

msg/async/DPDK: refactor set_rss_table to support DPDK 19.05

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc: fix a typo in a command

Co-authored-by: donggyu_park <donggyu_park@tmax.co.kr>
Signed-off-by: taeuk_kim <taeuk_kim@tmax.co.kr>

doc/rbd: align with other part in the context

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge PR #32179 into master

* refs/pull/32179/head:
lvm: add deactivate subcommand
util/system: add unmount_tmpfs helper
api/lvm: add get_lv_by_osd_id method
api/lvm: add deactivate method to Volume class

Reviewed-by: Sage Weil <sage@redhat.com>

doc/rbd: correct targe dev name and existence check

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge PR #32225 into master

* refs/pull/32225/head:
qa: add force-branch to suites running s3readwrite & s3roundtrip tasks

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #32003 into master

* refs/pull/32003/head:
mgr/ssh fix tests for update_mds/mgr/mds/rgw/rbd-mirror
Allow List[HostSpec] in PlacementSpec
mgr/rook: adapt to arg passing change in orchestrator_cli.py
mgr/ssh: Add SimpleScheduler and streamline arg passing
orch_cli: adapt to changes in PlacementSpec and ServiceSpec
orch: extend PlacementSpec and add StatefulServiceSpec

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #32218 into master

* refs/pull/32218/head:
cephadm: use raw string during `re.sub`
cephadm: fix ConfigParser DeprecationWarning in py3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #32187 into master

* refs/pull/32187/head:
ceph.in: (py3) use binary stdout
pybind/mgr: index=4, sort_keys=True for json.dumps throughput

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #32013 from liewegas/doc-foundation

doc/foundation: add cloudbase and vexxhost

Merge pull request #32197 from neha-ojha/wip-report-doc-bug

doc/_templates/page.html: redirect to etherpad

Reviewed-by: Zac Dover zac.dover@gmail.com
Reviewed-by: Sage Weil sage@redhat.com

qa: add force-branch to suites running s3readwrite & s3roundtrip tasks

Signed-off-by: Ali Maredia <amaredia@redhat.com>

Merge PR #32220 into master

* refs/pull/32220/head:
qa/workunits/cephadm/test_cephadm.sh: need --fsid always

Reviewed-by: Michael Fritch <mfritch@suse.com>

mds/MDSDaemon: make con disposable before passing message to asok

Signed-off-by: Sage Weil <sage@redhat.com>

mds/SessionMap: interpret bare id as a valid filter

This works around the fact that the old asok 'session evict' took a bare
id and the tell version took a filter.

Signed-off-by: Sage Weil <sage@redhat.com>

mds: get rid of unused send_command_reply helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mds: remove old handle_command machinery

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSRank: remove C_ExecAndReply helpers

Signed-off-by: Sage Weil <sage@redhat.com>

mds/MDSRank: too many words about op_tracker

Signed-off-by: Sage Weil <sage@redhat.com>

mds: convert 'cache drop'

Signed-off-by: Sage Weil <sage@redhat.com>

mds: convert 'damage ls' and 'damage rm'

Signed-off-by: Sage Weil <sage@redhat.com>