mgr/ssh: Port raising exceptions from completion handlers to Py2

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Eye candy updates to datatable pages (#31764)

mgr/dashboard: Eye candy updates to datatable pages

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31903 from majianpeng/os-bluestore-ceph-assert

os/bluestore: don't round_up_to in apply_for_bitset_range.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #31774 from dzafman/wip-balancer3

osd,tools: Balancer fixes without all of the calc_pg_upmaps() rewrites

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31317 from wjwithagen/wjw-update-FindBoost.cmake

cmake: update FindBoost.cmake for 1.71

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31841 from rosinL/wip-support-dpdk-numa

cmake: Add dpdk numa support

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31915 from majianpeng/osd-remove-useless-ceph-assert

osd: remove useless ceph_assert.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31723 from rosinL/wip-add-bench-log-usage

test/bench_log: add usage function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31840 from rosinL/wip-fix-dpdk

msg/async/dpdk: fix compilation errors when WITH_DPDK=on

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30631 from xxhdx1985126/wip-bluefs-sync-leak-fix

BlueFS: prevent BlueFS::dirty_files from being leaked when syncing metadata

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #31900 into master

* refs/pull/31900/head:
mgr: fix debug typo

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31442 into master

* refs/pull/31442/head:
kv/RocksDBStore: Add minimum key limit before invoking DeleteRange.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

msg/async/dpdk: Porting seastar's init_port_start to fix to resolve compilation errors

Ceph currently uses dpdk19.05, seastar dpdk_device::init_port_start has
been upgraded to DPDK19.05, so porting changes to fix compiler error.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

Merge pull request #31165 from votdev/improve_py_unittests

mgr/dashboard: Refactor Python unittests and controller

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31856 from rzarzynski/wip-crimson-errorator-handle_error

crimson/common/errorator.h: add handle_error() method.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31853 from liewegas/fix-rn-typo

PendingReleaseNotes: fix typo

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31918 from rzarzynski/wip-crimson-errorator-simplify_safe_then

crimson/common/errorator.h: simplify the compound safe_then() variant.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31838 into master

* refs/pull/31838/head:
ceph-daemon: chown uid/gid files during `adopt`
ceph-daemon: use 'iglob' instead of 'glob'
ceph-daemon: Fix handling for symlinks on python2

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

mgr/dashboard: Eye candy updates to datatable pages

- Introduce boolean and array pipes
- Introduce configurable 'badge' datatable column
- Refactor the Inventory, OSD, RBD, iSCSI, ... datatable pages

Signed-off-by: Volker Theile <vtheile@suse.com>

crimson/common/errorator.h: simplify the compound safe_then() variant.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common/errorator.h: add handle_error() method.

In contrast to `safe_then` it deals only with errors
and, on return, leaves the value type `future<ValueT>`
embodies unchanged.
While being basically a specialization of `safe_then`,
it's also a convenience method to squeeze boilerplate.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #31914 into master

* refs/pull/31914/head:
mgr/orchestrator: set node labels to empty list if none specified

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: support creating OSDs on spare devices (#30921)

mgr/dashboard: support creating OSDs on spare devices

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

cmake: Add shared library as dpdk dependency

librte_eal.a refers to libpthread.so and libdl.so, and libceph-common.so
refers to librte_eal.a. Many test application link libceph-common.so
without linking pthread, it will cause link error.
Such as unittest_interval_set, target_link_libraries (unittest_interval_set
ceph-common)

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

cmake: Add dpdk numa support

Find and link the numa library for DPDK.

Fixes: https://tracker.ceph.com/issues/42275
Signed-off-by: Hu Ye <yehu5@huawei.com>
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

Merge pull request #31785 from votdev/cleanup_remove_useless_var

mgr/dashboard: Remove unused variable

Reviewed-by: Kiefer Chang kiefer.chang@suse.com
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #31720 from rhcs-dashboard/fix-42708-master

mgr/dashboard: remove traceback/version assertions

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31843 from votdev/rename_list_pipe

mgr/dashboard: Rename pipe list -> join

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

os/bluestore: don't round_up_to in apply_for_bitset_range.

I met the following error messages:
>sceph7: /home/Jianpeng/ceph/src/os/bluestore/bluestore_common.h: In function 'void apply_for_bitset_range(uint64_t, uint64_t, uint64_t, Bitset&, Func) [with Bitset = boost::dynamic_bitset<long unsigned int>; Func = BlueFS::_replay(bool, bool)::<lambda(uint64_t, boost::dynamic_bitset<long unsigned int>&)>; uint64_t = long unsigned int]' thread 7f7d55a3dd40 time 2019-11-27T23:26:05.949827+0800
>sceph7: /home/Jianpeng/ceph/src/os/bluestore/bluestore_common.h: 28: FAILED ceph_assert(end <= bitset.size())
>sceph7:  ceph version 15.0.0-7747-g22a9a28e28 (22a9a28e2859718c4168f5b3cf6e20a9186e18bc) octopus (dev)
>sceph7:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x56322c4440ef]
>sceph7:  2: (()+0x50d2d9) [0x56322c4442d9]
>sceph7:  3: (BlueFS::_replay(bool, bool)+0x5639) [0x56322cb51e49]
>sceph7:  4: (BlueFS::mount()+0x269) [0x56322cb53289]
>sceph7:  5: (BlueStore::_open_bluefs(bool)+0x29e) [0x56322ca6e38e]
>sceph7:  6: (BlueStore::_open_db(bool, bool, bool)+0x59b) [0x56322ca6e9cb]
>sceph7:  7: (BlueStore::mkfs()+0x749) [0x56322caaa259]
>sceph7:  8: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1c8) [0x56322c5099e8]
>sceph7:  9: (main()+0x1787) [0x56322c4ab117]
>sceph7:  10: (__libc_start_main()+0xeb) [0x7f7d55b5fb6b]
>sceph7:  11: (_start()+0x2a) [0x56322c4de71a]
>sceph7: *** Caught signal (Aborted) **
>sceph7:  in thread 7f7d55a3dd40 thread_name:ceph-osd

If bdev[i]->get_size() / alloc_size[i] != 0,  round_up_to(off+lengh,alloc_size[i])/alloc_size is larger than(one)
ownend_blocks[i].size().

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

mgr/orchestrator: set node labels to empty list if none specified

f06366836ff tries to ' '.join(node.labels), which raises an exception
if labels is None, so we need to default it to an empty list.

Signed-off-by: Tim Serong <tserong@suse.com>

osd: remove useless ceph_assert.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #31588 from usefulalgorithm/patch-1

doc/rados/operations: OSD_OUT_OF_ORDER_FULL fullness order is wrong

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

test: Add test case based on Xie script in commit comment

Other test fixes to reflect changes

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: ceph_pg_upmaps() use any overfull when there are still underfull

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Create more_underfull with below target that aren't in underfull

Use more_underfull for finding remaps for overfull OSDs

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: calc_pg_upmaps() pick most overfull remap from try_pg_upmap()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: ceph_pg_upmaps() use max_deviation to determine perfect distribution

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: osdmaptool sync with balancer module behavior

This can be backported only to nautilus

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: osdmaptool: Perform upmap calculation as ceph-mgr does

This is the backportable commit that works with older balancer module.py

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: For balancer crush needs the rule passed to get_parent_of_type()

Fixes: https://tracker.ceph.com/issues/42718
Signed-off-by: David Zafman <dzafman@redhat.com>

osdmaptool: Match default max value of 10

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: odsmaptool truncate target upmap file

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Fix balancer print

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #31870 into master

* refs/pull/31870/head:
ceph-daemon: fix `systemctl is-enabled` bool

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31886 into master

* refs/pull/31886/head:
ceph-daemon: error out if not root
ceph-daemon: user-friendly exception/error messages
ceph-daemon: user-friendly exception/error messages

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31883 from jan--f/ceph-volume-code-owner

github/codeowners: add ceph-volume

Merge PR #31892 into master

* refs/pull/31892/head:
ceph-daemon: seek relative to the start of file

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31899 into master

* refs/pull/31899/head:
mgr: drop session with Ceph daemon when not ready
mgr: improve debug message information

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: error out if not root

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: user-friendly exception/error messages

- Rename `CDError` to `Error`
- Display traceback on console when `-v` is specified
- Other misc clean-up

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #30262 from sebastian-philipp/orchestrator-pomise-future

mgr/orchestrator: Make Completions composable

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/ssh: Disable host labels.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31896 from rsevero/missing_virtualenv

doc: add missing virtualenv for build-doc

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: use finalize to set TrivialReadCompletion result

This is needed to correctly set the state of the completion, so that
Completion.has_result returns true.  Without this, _orchestrator_wait()
will spin forever if given a TrivialReadCompletion.

Signed-off-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add pretty_print()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Fix Promises

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

import typing optionally

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Fix testing error

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/deepsea: Fixed DeepSeaReadCompletion arguments

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: fix tests

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

make check fixes

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchstrator: Removed all type anotations incompatible to Python 2

* Reduce mypy's type checking capability
* Remove useful code documentation

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/orchestrator: Adapted to new Completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add cancel

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/orchestrator: Adapt teuthology test to new completions

Also: Adapt mgr/test_orchestrator to new completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/deepsea: Adapt to new orchestrator completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ansible: Adapt to new orchestrator completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Adapt ssh orch to new Completions interface

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Adapt to new completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Introduce composable completions

Also:

* unify Read and Write completions
* Distinguish should_wait and is_finished
* Removed `all_hosts` parameter to `osd_create`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: rename wait() -> process()

Because wait() should actually not wait!

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31904 from tchaikov/wip-crimson-stop-msgr-after-stopping-osd

crimson: stop osd before stopping messengers

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Remove unused variable

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Rename pipe list -> join

Signed-off-by: Volker Theile <vtheile@suse.com>

crimson: stop osd before stopping messengers

so we can drain all pending tasks referencing messenger before
destroying it, otherwise they will be using a already-stopped
messenger when they are still alive.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31756 into master

* refs/pull/31756/head:
pybind / cephfs: remove static typing in LibCephFS.chown

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: fix debug typo

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #31828 into master

* refs/pull/31828/head:
mgr/orchestrator: include container_image in 'service ls' output
ceph-daemon: include container_image_name and container_image_id in 'ls' output
common/options: default to ceph/ceph:latest-master-devel (for now)

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31854 into master

* refs/pull/31854/head:
mgr/orchestrator_cli: print table for 'host ls'
mgr/ssh: populate labels
mgr/orchestrator: add 'labels' property to InventoryNode
mgr/ssh: add/remove labels
mgr/orchestrator: orchestrator host label {add,rm}
mgr/ssh: add explicit host list

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31864 into master

* refs/pull/31864/head:
osd/PeeringState: clear LAGGY and WAIT states on exiting Started

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #31865 into master

* refs/pull/31865/head:
cls/rbd: improve efficiency of mirror image status queries

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #31866 into master

* refs/pull/31866/head:
osd/SnapMirror: no need to record purged_snaps every epoch

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr: drop session with Ceph daemon when not ready

If the mgr is waiting on daemon metadata from the mons, it has no
DaemonState associated with the daemon yet. If we try to process this
MgrOpen, the metadata sent by the daemon (like its config) will not be
recorded.

Fixes: https://tracker.ceph.com/issues/43037
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mgr: improve debug message information

Connection pointer is not helpful.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: missing virtualenv for build-doc

The install deps command recommended by admin/build-doc doesn't install virtualenv which is required by admin/build-doc.

After manually installing virtualenv, admin/build-doc runned without issues.

Signed-off-by: Rodrigo Severo <rodrigo@fabricadeideias.com>

Merge PR #27467 into master

* refs/pull/27467/head:
cephfs-shell: add snapshot management

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #31871 from smithfarm/wip-43021

doc: mention --namespace option in rados manpage

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

ceph-daemon: seek relative to the start of file

python3 only allows seeks relative to the start of a text file

Signed-off-by: Michael Fritch <mfritch@suse.com>

doc: mention --namespace option in rados manpage

Fixes: https://tracker.ceph.com/issues/43021
Signed-off-by: Nathan Cutler <ncutler@suse.com>

ceph-daemon: user-friendly exception/error messages

We don't want tracebacks for most error conditions in ceph-daemon.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator_cli: print table for 'host ls'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: populate labels

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: add 'labels' property to InventoryNode

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add/remove labels

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orchestrator: orchestrator host label {add,rm}

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/ssh: add explicit host list

Keep the host list in an explicit 'inventory' key, and provide a dict for
each host so that we can keep other metadata (like labels).

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #31837 into master

* refs/pull/31837/head:
ceph-daemon: add additional debug logging

Reviewed-by: Sage Weil <sage@redhat.com>

github/codeowners: add ceph-volume

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #31874 from tchaikov/wip-crimson-cyanstore-restructure

crimson: move os/cyan_* down to os/cyanstore/*

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Refactor Python unittests

* Make use of the KVStoreMockMixin class to get rid off duplicate code.
* Fake the index.html file to be able to run tests/test_home.py locally without building the frontend in production mode.
* Encapsulate helper functions in controllers/home.py, otherwise tests/test_feature_toggles.py need to fake the filesystem because load_controllers() will load the home.py controller and fail due missing files in the filesystem.

Signed-off-by: Volker Theile <vtheile@suse.com>