qa/tasks/ceph2: pull image using sha1

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: docker needs quay.io/ prefix for image name

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rados/test_python: make sure rbd pool exists

The ceph2 task doesn't create it for you.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/rados/ssh: new tests!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: pull ceph-ci/ceph:$branch

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: register_daemons after pods start

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: fix conf

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: add restart

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: pass ceph-daemon path to DaemonState

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: tolerate no mdss or 1 mgr

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: replace wait_for_osds_up with manager.wait_for_all_osds_up

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: wait-until-healthy

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: set up managers

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: use seed ceph.conf

Based on ceph.conf.template, but edited down a bit.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph: healthy: use manager helpers (instead of teuthology/misc ones)

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: name mds daemons

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: fix osd ordering

Sort by the numeric key

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: start up mdss

TODO: name them properly

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: set up daemon handles and use them to stop

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: make it multicluster-aware

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: can bring up mon, mgr, osds!

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/ceph2: basic task to bring up cluster with ceph-daemon and ssh

This is pretty minimal at this point, but it works so far (or will, once a few open PRs merge).

Main problematic area right now is that the multi-cluster stuff isn't right.  We should fix that before continuing, since ceph-daemon for once can do multiple clusters well.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Cross sign button not working for some modals (#31783)

mgr/dashboard: Cross sign button not working for some modals

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: sort monitors by open sessions correctly. (#31752)

mgr/dashboard: sort monitors by open sessions correctly.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31738 from ricardoasmarques/wip-41776

mgr/dashboard: Unable to set boolean values to false when default is true

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #31773 from idryomov/wip-krbd-snapshot-removal-test

qa: kernel.sh: update for read-only changes

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Don't use any xlf file when building the defaul… (#31727)

mgr/dashboard: Don't use any xlf file when building the default language

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #31748 into master

* refs/pull/31748/head:
qa/standalone/test_ceph_daemon.sh: remove old vg before creating
qa/standalone/test_ceph_daemon.sh: sudo for untar
qa/standalone/test_ceph_daemon.sh: sudo for losetup etc
qa/standalone/test_ceph_daemon.sh: fix overwrites of temp files

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: Update dev documentation URL (#31724)

mgr/dashboard: Update dev documentation URL

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Use new ImageSpec class (#31622)

mgr/dashboard: Use new ImageSpec class

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Dashboard can't handle self-signed cert on Graf… (#31129)

mgr/dashboard: Dashboard can't handle self-signed cert on Grafana API

Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: open files with UTF-8 encoding in Grafana check… (#31750)

mgr/dashboard: open files with UTF-8 encoding in Grafana checking script

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Cross sign button not working for some modals

Fixes: https://tracker.ceph.com/issues/42179
Signed-off-by: Ricardo Marques <rimarques@suse.com>

qa/standalone/test_ceph_daemon.sh: remove old vg before creating

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_daemon.sh: sudo for untar

The deepsea.tgz tar contains actual device nodes for the OSD block devices
(not symlinks or files).  Must be root to untar.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_daemon.sh: sudo for losetup etc

Signed-off-by: Sage Weil <sage@redhat.com>

qa/standalone/test_ceph_daemon.sh: fix overwrites of temp files

mktemp creates these files, so we have to pass --allow-overwrite (or
delete them after we get the unique name but before we write to them--this
is easier).

Broken by c7fe27a72a61d1345a66b8830fd17e7b922abd44

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31418 into master

* refs/pull/31418/head:
test: use distinct subvolume/group/snapshot names

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #30553 into master

* refs/pull/30553/head:
qa/tasks: check if fs mounted in umount_wait

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31534 into master

* refs/pull/31534/head:
mds: fix invalid access of mdr->dn[0].back()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31548 into master

* refs/pull/31548/head:
mds: properly evaluate unstable locks when evicting client

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31531 into master

* refs/pull/31531/head:
cmake: link ceph-fuse against librt

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa: kernel.sh: update for read-only changes

Snapshot existence validation code was removed from krbd.  It was racy
and relied on having watch established for snapshots.

Fixes: https://tracker.ceph.com/issues/42916
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge PR #31745 into master

* refs/pull/31745/head:
qa/suites/rados/thrash-old-clients: skip TestClsRbd.mirror

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31392 from dillaman/wip-42598

journal: fix flush by age and in-flight byte tracking

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Use new ImageSpec class

This class should be used when dealing with RBD image specs.
It allows the creation of an ImageSpec given the names of the pool, image and
namespace. Alternatively you can also create one with an already existing
image spec string.

With it you keep the access to each individual component and can also convert it
to a well formated string.

Fixes: https://tracker.ceph.com/issues/42787
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: fix restored RBD image naming issue (#31590)

mgr/dashboard: fix restored RBD image naming issue

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #31758 from votdev/fix_typo

python-common: Fix typo in device type

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Don't use any xlf file when building the default language

The build-i18n script no longer uses a xlf file when building the default
language. This means that we don't need to keep the messages.en-US.xlf in the repository anymore.

Fixes: https://tracker.ceph.com/issues/42693
Signed-off-by: Sebastian Krah <skrah@suse.com>

Merge pull request #31410 from jan--f/c-v-create-vgs-on-demand

ceph-volume: allow raw block devices everywhere

mgr/dashboard: Fixes removal of custom tags during pool edit (#31373)

mgr/dashboard: Fixes removal of custom tags during pool edit

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

python-common: Fix typo in device type

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #31751 from tchaikov/wip-run-tox.sh-python3

script/run_tox.sh: do not use python2 if we have python3

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31607 from tspmelo/wip-pool-pattern

mgr/dashboard: Add invalid pattern message for Pool name

Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: show "Rename" in header & button when renaming… (#31551)

mgr/dashboard: show "Rename" in header & button when renaming RBD snapshot

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: CephFS snapshot management UI (#30996)

mgr/dashboard: CephFS snapshot management UI

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31563 from tspmelo/wip-default-i18n

mgr/dashboard: Use default language when running "npm run build"

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

doc: update ceph-volume lvm prepare

Add option to pass raw physical devices everywhere, restructure a little
(bluestore section before filestore) and reword a few things.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: make lvm report fields into constants

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: api/lvm create or reuse a vg

This changes create_lv so one can pass the desired device and either a
VG with a name starting with ceph is re-used or a new one is created.
This commit also adds two new lvm primitives, making use of lvm's select
feature. The goal is to eventually avoid keeping a full list of lv's (or
vg's) around and query the lvm system as needed.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31755 from tchaikov/wip-mgr-orch

mgr/orch: pass unicode string to ipaddress.ip_network()

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/orch: pass unicode string to ipaddress.ip_network()

otherwise it complains like

>               raise e
E               AddressValueError: '10.1.1.10' does not appear to be an
IPv4 or IPv6 address. Did you pass in a bytes (str in Python 2) instead
of a unicode object?

orchestrator.py:94: AddressValueError

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: add ipaddress to requirements.txt

ipaddress dependency was introduced in 16f281d2. because the tox testenv
is isolated from the host python env, the test env has no access to the
ipaddress python module installed by install-deps.sh. we have to install
ipaddress for python<3.3 for testing which uses `orchestrator.py`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Update dev documentation URL

Update the developer documentation URL because the
old one is only returning "403 Forbidden".

Signed-off-by: Tatjana Dehler <tdehler@suse.com>

mgr/dashboard: sort monitors by open sessions correctly.

Fixes: https://tracker.ceph.com/issues/42893
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

script/run_tox.sh: do not use python2 if we have python3

always use python3 for bootstrapping tox environment unless the tox-env
contains "py2" but not "py3"

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31574 from jan--f/c-v-refactor-get_devices

ceph-volume: make get_devices fs location independent

mgr/dashboard: open files with UTF-8 encoding in Grafana checking script

Fixes: https://tracker.ceph.com/issues/42892
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #30698 into master

* refs/pull/30698/head:
common/options: Set bluestore min_alloc size to 4K

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31638 into master

* refs/pull/31638/head:
mgr/pg_autoscaler: only generate target_* health warnings if targets set

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #31572 into master

* refs/pull/31572/head:
mgr/diskprediction_local: some debug output during predict (and self-test)

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/thrash-old-clients: skip TestClsRbd.mirror

Older versions have this test and fail it.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31721 into master

* refs/pull/31721/head:
mgr/ssh: set UserKnownHostsFile

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31660 from ivancich/wip-add-some-const

rgw: add const correctness to some rest functions

Merge pull request #16715 from adamemerson/wip-I-Object!

osdc/Objecter: Boost.Asio (I object!)

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #31672 into master

* refs/pull/31672/head:
mgr/orch: adapt update_mgr/mon funcs to new parsing
mgr/orch: improve commandline parsing

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #31679 into master

* refs/pull/31679/head:
ceph-daemon: add osd create test(s)

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: unable to set bool values to false when default is true

Fixes: https://tracker.ceph.com/issues/41776
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #31731 from varshar16/wip-doc-update-orchestrator_cli-url

mgr/orchestrator_cli: Update doc link in README

mgr/ssh: set UserKnownHostsFile

Set this to /dev/null, since we have StrictHostKeyChecking
off anyway.  This gets rid of messages to stderr every time we connect
to a new host.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-volume: refactor tests for refactored get_devices

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

ceph-volume: refactor get_devices, don't use os.path.realpath

Fixes: https://tracker.ceph.com/issues/42777
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

journal: properly handle immediate overflow upon advancing object set

While the old set is being closed, additional IO can be queued up within
the old, closed objects while the in-flight IO settles. It's therefore
possible that the queued IO that is transferred from the old set to the
new set causes an immediate overflow of the new set.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: fix flush by age and in-flight byte tracking

The flush by age was always causing an immediate flush due to a
backwards comparison. Additionally, the in-flight byte tracker was
never decremented which caused premature closure of the journal
object.

Finally, there was a potential race condition between closing the
object and in-flight notification callbacks executing. Now we keep
the lock held for both closed and overflow callbacks to prevent
the small chance of a race.

Fixes: https://tracker.ceph.com/issues/42598
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: simplify object recorder locker aggregation

Use move semantics and RAII to control the locking of the
per-object recorder lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29399 from ofriedma/master

rgw: Fix expiration header does not return the earliest rule

Merge pull request #29937 from ofriedma/wip-ofriedma-days0

Add days0 to rgw lc

Merge pull request #31642 from trociny/wip-rbd-mirror-service-daemon-ns-support

rbd-mirror: add namespace support to service daemon

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31627 from trociny/wip-pybind-rbd-snap_remove

pybind/rbd: provide snap remove flags

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31608 from trociny/wip-mock-warnings

test/rbd_mirror: fix mock warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31606 from trociny/wip-test_rbd.py-tabs

test/pybind: inconsistent use of tabs and spaces in indentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: CephFS snapshot management UI

CephFS snapshots can now be created on a directory basis. Multiple
snapshots can be deleted at once.

Fixes: https://tracker.ceph.com/issues/41791
Signed-off-by: Volker Theile <vtheile@suse.com>
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/orchestrator_cli: Update doc link in README

Update the orchestrator cli doc link to fix page not found error. Also remove
some unnecessary blank lines.

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #31594 from smithfarm/wip-bri-regex-fix

backport-resolve-issue: narrow regular expression and read key/token from files

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #30336 from joscollin/wip-F40959-subvolume-uid-gid

mgr/volumes: uid, gid for subvolume create and subvolumegroup create commands

mgr/dashboard: show Rename in modal header & button when renaming RBD snapshot

Fixes: https://tracker.ceph.com/issues/42762
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #31546 from zjcmszh/execute

pybind/rados: add WriteOp::execute()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30968 from majianpeng/osd-optimze-send_message_osd_cluster

osd: optimize send_message to peers

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31550 from wido/issue-42761

systemd: Wait 5 seconds before attempting a restart of an OSD

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #31623 from badone/wip-tracker-42387-extend-admin-socket-timeout

common/admin_socket: Increase socket timeouts

Reviewed-by: Sage Weil <sage@redhat.com>