ceph-volume: add proper size attribute to partitions

Fixes: https://tracker.ceph.com/issues/42710
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #31467 into master

* refs/pull/31467/head:
ceph-daemon: Add basic unittest infrastructure and cmake integration
ceph-daemon: Move ceph-daemon executable to own directory

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

ceph-daemon: Add basic unittest infrastructure and cmake integration

To be able to write unittests, add a tox.ini (for testing in venvs)
and a first single test case as example.
Tests can be executed with eg:

$ tox

Or by running the usual:

$ ./run-make-check.sh

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: Move ceph-daemon executable to own directory

Moving ceph-daemon into src/ceph-daemon/ makes it simpler to add extra
code (eg. tox.ini, README, unittests, ...) specific to ceph-daemon.
That way related files are in a single directory.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

Merge PR #31477 into master

* refs/pull/31477/head:
ceph-daemon: unmount osd data dir during `adopt`

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31478 into master

* refs/pull/31478/head:
ceph-daemon: add --skip-pull
ceph-daemon: add explicit pull at bootstrap start

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31177 into master

* refs/pull/31177/head:
doc: update with zone bucket and straw2 addition

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31476 into master

* refs/pull/31476/head:
ceph-daemon: make mon container privileged

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

ceph-daemon: add --skip-pull

It occurs to me there might be cases where the user *doesn't* want to pull
the latest image (e.g., because it is a partially disconnected enviroment,
and they know the image is already in the local registry).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31464 into master

* refs/pull/31464/head:
ceph-daemon: help users find the shell/CLI too
ceph-daemon: enable the dashboard during bootstrap
ceph-daemon: add CLI helper to bootstrap

Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Kai Wagner <kwagner@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31479 from rishabh-d-dave/cv-dd-sync

ceph-volume: use fsync for dd command

Merge pull request #29041 from jan--f/c-v-too-small-disks-unavailable

ceph-volume: reject disks smaller then 5GB in inventory

Merge pull request #31260 from trociny/wip-42405

rbd-mirror: removing non-primary trash snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31188 from zy751713126/rbd_features

librbd: features converting bitmask and string API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31457 from jschmid1/ssh_orch_dev_env

mgr/ssh: update ssh-orch bootstrap guide (Vagrantfile & docs)

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: do not show non-pool data in pool details (#31456)

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Fix grafana dashboards (#31219)

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31189 from bk201/wip-42075

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31021 from sebastian-philipp/python-common-make-drive-selection

python-common: Add drive selection

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

ceph-volume: use fsync for dd command

fsync makes sure that data and metadata is actually written to device
being zapped before finishing.

Fixes: https://tracker.ceph.com/issues/39156
Signed-off-by: Rishabh Dave <ridave@redhat.com>

ceph-daemon: add explicit pull at bootstrap start

This is less confusing for users since the pull can be slow and the
uid/gid check is a weird thing to hang on for a long time.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: unmount osd data dir during `adopt`

Signed-off-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: make mon container privileged

libudev needs to be privileged in order to query the underlying hardware
devices, as reported by the 'ceph device ...' command set, and to scrape
smart metrics, etc.

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: help users find the shell/CLI too

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: enable the dashboard during bootstrap

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #31221 from weiyingze-git/patch

test: add test_remove_snap_ImageNotFound test case in remove snap part

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #31414 from cbodley/wip-qa-rgw-more-crypto-backend

qa/rgw: use 'testing' kms backend for other rgw subsuites

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #31398 into master

* refs/pull/31398/head:
pybind/mgr/pg_autoscaler: implement shutdown method

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>

Merge PR #31064 into master

* refs/pull/31064/head:
test: Test balancer module commands
mgr: Improve balancer module status
mgr: Release GIL before calling OSDMap::calc_pg_upmaps()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #30976 into master

* refs/pull/30976/head:
cmake: bump libceph-common SO version for compliance

Reviewed-by: Tim Serong <tserong@suse.com>

ceph-volume: reject disks smaller then 5GB in inventory

Volumes smaller than 5GB won't be created anyway. This also guards
against drives with size 0, as can happen in mdraid setups.

Fixes: http://tracker.ceph.com/issues/40776
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #31438 into master

* refs/pull/31438/head:
mgr/ssh: implement blink_device_light

Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31462 into master

* refs/pull/31462/head:
mgr/ssh: remove superfluous parameters

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31445 into master

* refs/pull/31445/head:
ceph-daemon: set container_image during bootstrap

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: add CLI helper to bootstrap

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: implement blink_device_light

On my test box I get as far as

  NO_SUPPORT(153): SCSI VPD page 0x83 is not supported

because my hardware doesn't support this.  The same command works
on another box that does.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31443 into master

* refs/pull/31443/head:
ceph-daemon: fix 'version' field for legacy `ls`

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31458 into master

* refs/pull/31458/head:
ceph-daemon: Only run in the __main__ scope

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: set container_image during bootstrap

Set the new cluster's container_image to match the image we bootstrapped
with.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31463 into master

* refs/pull/31463/head:
ceph-daemon: switch default image

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31448 into master

* refs/pull/31448/head:
ceph-daemon: make 'ls' log less noisy

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: switch default image

Use this until octopus is ready.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: remove superfluous parameters

in _set_ssh_config and _clear_ssh_config

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/{dashboard,prometheus}: return FQDN instead of '0.0.0.0' (#31430)

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/ssh: move and adapt documentation for ssh-orchestrator

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/dashboard: Use htmllint (#30998)

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Set iSCSI disk WWN and LUN number from the UI (#31056)

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

ceph-daemon: Only run in the __main__ scope

That makes unit testing easier to setup because the code is not loaded
when ceph-daemon gets imported. Instead it is only loaded when
executed.
For that, the parser also moved to a function instead of being on
module level.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

mgr/ssh: adapt vagrantfile for ssh-orchestration bootstrap

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #31070 from sebastian-philipp/dashbaord-run-backend-zsh

mgr/dashboard: Fix zsh support in run-backend-api-tests.sh

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: do not show non-pool data in pool details

Fixes: https://tracker.ceph.com/issues/42674
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: fix e2e test failure for OSDs (#31451)

mgr/dashboard: fix e2e test failure for OSDs

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test/pybind: add features test case

add features_to_string and features_from_string test case

Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>

librbd: features converting bitmask and string API

Signed-off-by: Zheng Yin <zhengyin@cmss.chinamobile.com>

mgr/dashboard: fix e2e test failure for OSDs

When clicking a selected row, the row becomes unselected and the detail
component is hidden. Rearrange related tests by selecting the row only
once.

Fixes: https://tracker.ceph.com/issues/42671
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #31350 from chenpan1012/rbd_testcase

cls/rbd: add snapshot limit UINT64_MAX test case

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-daemon: make 'ls' log less noisy

Signed-off-by: Sage Weil <sage@redhat.com>

Merge branch 'master' into patch

Merge pull request #31208 from weiyingze-git/master

test:add test_remove_snap2 interface to remove  snap when it's protected

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

ceph-daemon: fix 'version' field for legacy `ls`

The `ls` command was reporting the git commit id instead of the ceph
version.

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #31441 into master

* refs/pull/31441/head:
github/codeowners: Add orchestrator team

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31437 into master

* refs/pull/31437/head:
ceph-daemon: Remove data dir during adopt

Reviewed-by: Sage Weil <sage@redhat.com>

github/codeowners: Add orchestrator team

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge PR #31440 into master

* refs/pull/31440/head:
qa/standalone/test_ceph_daemon: Make container images configurable
qa/standalone/test_ceph_daemon: Allow running from root dir
qa/standalone/test_ceph_daemon: Fix hang when CEPH_DAEMON is not set

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31439 into master

* refs/pull/31439/head:
ceph-daemon: fix traceback during `ls` command

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31385 into master

* refs/pull/31385/head:
mgr/ssh: add 'osd rm'
mgr/ssh: keep inventory and service cache keys in sync
mgr/orchestrator_cli: fix 'service ls' table format
mgr/ssh: invalidate service state on create, remove, and service action
mgr/orchestrator: add invalidate() to OutdatableDictMixin
mgr/ssh: cache services
mgr/ssh: drop specialized _remove_{mds,mgr,rgw}
mgr/ssh: change inventory_cache_timeout to seconds (not minutes)
mgr/orchestrator: make wait poll every 1s (not 5s)

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/standalone/test_ceph_daemon: Make container images configurable

Instead of hardcoding the images, make them configureable via
environment variables.
That way, downstream can use the script with custom images.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: fix traceback during `ls` command

fixup for 3a1ac3ff6ac2ef6e3e8c361c0c55562b732b9aac

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/standalone/test_ceph_daemon: Allow running from root dir

Allow running the script from root directory via:

./qa/standalone/test_ceph_daemon.sh

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

qa/standalone/test_ceph_daemon: Fix hang when CEPH_DAEMON is not set

When running test_ceph_daemon.sh from the root dir and not setting
$CEPH_DAEMON manually, the call hangs at:

$ ./qa/standalone/test_ceph_daemon.sh
[...]
+ for p in $PYTHONS
+ echo '=== re-running with python3 ==='
=== re-running with python3 ===
++ which python3
+ ln -s /usr/bin/python3 /tmp/tmp.6hneCsNMio/python
+ echo '#!/tmp/tmp.6hneCsNMio/python'
+ cat

Check that there is a ceph-daemon found before continue.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

ceph-daemon: Remove data dir during adopt

Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: add missing test_orchestrator suite (#31198)

mgr/dashboard: add missing test_orchestrator suite

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31422 into master

* refs/pull/31422/head:
mgr/MgrClient: fix open condition fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/ssh: add 'osd rm'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: keep inventory and service cache keys in sync

Our host list is implicitly defined as the set of keys for the inventory
(and service) caches.  Ensure they are in sync by unioning and inserting
any missing items.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: fix 'service ls' table format

Make this match the rest of Ceph.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: invalidate service state on create, remove, and service action

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add invalidate() to OutdatableDictMixin

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: cache services

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: drop specialized _remove_{mds,mgr,rgw}

_remove_daemon is sufficient

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: change inventory_cache_timeout to seconds (not minutes)

The descriptio didn't match the implementation (which took minutes).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: make wait poll every 1s (not 5s)

More responsive!

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/prometheus: return FQDN for default server_addr

Instead of returning '0.0.0.0' the prometheus module returns the FQDN of
the host, as it is the case for '::'.

Fixes: https://tracker.ceph.com/issues/42664
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: return FQDN for default server_addr

Instead of returning '0.0.0.0' the dashboard returns the FQDN of the
host, as it is the case for '::'.

Fixes: https://tracker.ceph.com/issues/42664
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: Use htmllint

This package will replace htmlhint.

Some rules from the previous installed packages were not working that well.
This new package has more rules and seems to work better.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add support for device management (#30759)

mgr/dashboard: Add support for device management

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Searchable objects for table (#31199)

mgr/dashboard: Searchable objects for table

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #26768 from sebastian-philipp/upstream-pull-26684

mgr/orchestrator: device lights

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #31409 into master

* refs/pull/31409/head:
mds: Fix MDCache.h reorder compiler warnings

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31401 into master

* refs/pull/31401/head:
mds: clarify comment

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #31420 into master

* refs/pull/31420/head:
ceph-daemon: do not screw up line splitting
mgr/ssh: raise RuntimeError when ceph-daemon invocation fails

Reviewed-by: Michael Fritch <mfritch@suse.com>

ceph-daemon: do not screw up line splitting

When we log stdout and stderr, we were artificially adding line splits
due to reading 1024 bytes at a time and assuming it was aligned with a
newline.

Fix by keeping partial lines buffered and only log them once they are
complete (or the call finishes).

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/MgrClient: fix open condition fix

Fix fc60989bf7a72c35b8f6b8fec2407b3080ad9bbd, which reversed this
condition.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: raise RuntimeError when ceph-daemon invocation fails

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: disable pg_autoscale for new pools in e2e tests (#31396)

mgr/dashboard: disable pg_autoscale for new pools in e2e tests

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge PR #31365 into master

* refs/pull/31365/head:
ceph-daemon: make rm-daemon reset-failed
ceph-daemon: make rm-cluster handle failed unit cleanup

Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Fix error on ceph-iscsi version pre controls_limits

Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/dashboard: Set iSCSI disk WWN and LUN number from the UI

Fixes: https://tracker.ceph.com/issues/41749
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge PR #31303 into master

* refs/pull/31303/head:
ceph-daemon: make systemd unit descriptions Capitalized
ceph-daemon: handle daemon names with multiple .'s
mgr/ssh: rgw: ensure map rgws are mapped to the zone
mgr/ssh: use . as delimiter between service group and id
mgr/ssh: implement 'rgw update'
mgr/orchestrator_cli: add 'rgw update' CLI command
mgr/ssh: add 'rgw rm'
mgr/ssh: implement 'rgw add'
ceph-daemon: handle 'rgw' type daemons
mgr/orchestrator_cli: rgw add: optionally take count and node names

Reviewed-by: Sebastian Wagner <swagner@suse.com>

ceph-daemon: make systemd unit descriptions Capitalized

Same style as other units.

Signed-off-by: Sage Weil <sage@redhat.com>