crimson/osd/pg_backend: factor out load_object_state/snapset

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson: add ObjectContext/Registry

ObjectContext will hold the ephemeral state and lock status
for an object as in classic osd.

Signed-off-by: Samuel Just <sjust@redhat.com>

src/osd: factor out RWState, move SnapSetContext, ObjectState, RWState

We'll want to reuse this logic in crimson, but the actual operation
wake/wait mechanics will be different.

Signed-off-by: Samuel Just <sjust@redhat.com>

common/: introduce intrusive_lru

Signed-off-by: Samuel Just <sjust@redhat.com>

osd/: factor OSD::init_op_flags into seperate class

We'll want to reuse this in crimson, extract the logic
for setting flags from MOSDOp into osd_types.h.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator.h: add finally method for resource release

This way we can add things like resource handlers which are otherwise
transparent to passing values and errors.

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart: add --debug for crimson if debug is enabled

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #31947 from zdover23/wip-doc-ubuntu-1804-python-minimal

doc: add Python 2 to Ubuntu 18.04 installations

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31958 from tchaikov/wip-cmake-new-policy

cmake: do not use CMP0074 unless it is supported

Reviewed-by: Samuel Just <sjust@redhat.com>

cmake: do not use CMP0074 unless it is supported

CMake CMP0074 was introduced by CMake 3.12, but we only require CMake
v3.10.2. so default to check `Boost_ROOT` unless specified otherwise.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #31913 into master

* refs/pull/31913/head:
ceph-daemon: Allow env var for setting the used image

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #31919 from idryomov/wip-krbd-udev-improvements

krbd: do away with explicit memory management and other cleanups

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #31933 into master

* refs/pull/31933/head:
mgr/ssh: `test_mon_update` needs to set a mon name

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31274 into master

* refs/pull/31274/head:
osd/OSD: enhance osd numa affinity compatibility

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #31348 into master

* refs/pull/31348/head:
tools/rados: update advisory lock break usage with --lock-cookie required

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31669 into master

* refs/pull/31669/head:
bluestore: tiny tracepoints improvement

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #31943 from tchaikov/wip-31224

python-common: move pytest integration from setup.py to tox.ini

Reviewed-by: Sebastian Wagner <swagner@suse.com>

doc: add Python 2 to Ubuntu 18.04 installations

This commit updates the "Create a Cluster" section of the
"Storage Cluster Quick Start" document. Ubuntu 18.04 does
not have Python 2 installed by default. This commit instructs
users of Python 18.04 to install the package "python-minimal"
to provide Python 2 prior to running "ceph-deploy".

This commit fixes Bug number 9 in the list here:
https://pad.ceph.com/p/Report_Documentation_Bugs

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #31361 from hairesis/kms_transit_backend

rgw: extend SSE-KMS with Vault using transit secrets engine

Reviewed-by: Casey Bodley <cbodley@redhat.com>

python-common: move pytest integration from setup.py to tox.ini

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31279 from lixiaoy1/rwl-image-writeback-cache-seq1

rbd/cache: Replicated Write Log core codes part 1

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Display the aggregated number of request (#31369)

mgr/dashboard: Display the aggregated number of request

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Resolve some npm audit issues (#31845)

mgr/dashboard: Resolve some npm audit issues

Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Trim IQN on iSCSI target form (#31888)

mgr/dashboard: Trim IQN on iSCSI target form

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/ssh: `test_mon_update` needs to set a mon name

... in order to exercise a code path to make use of `_get_services()`

Fixes: https://tracker.ceph.com/issues/43075
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Eye candy updates to datatable pages (#31764)

mgr/dashboard: Eye candy updates to datatable pages

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #31903 from majianpeng/os-bluestore-ceph-assert

os/bluestore: don't round_up_to in apply_for_bitset_range.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #31774 from dzafman/wip-balancer3

osd,tools: Balancer fixes without all of the calc_pg_upmaps() rewrites

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #31317 from wjwithagen/wjw-update-FindBoost.cmake

cmake: update FindBoost.cmake for 1.71

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31841 from rosinL/wip-support-dpdk-numa

cmake: Add dpdk numa support

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31915 from majianpeng/osd-remove-useless-ceph-assert

osd: remove useless ceph_assert.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31723 from rosinL/wip-add-bench-log-usage

test/bench_log: add usage function

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31840 from rosinL/wip-fix-dpdk

msg/async/dpdk: fix compilation errors when WITH_DPDK=on

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30631 from xxhdx1985126/wip-bluefs-sync-leak-fix

BlueFS: prevent BlueFS::dirty_files from being leaked when syncing metadata

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #31900 into master

* refs/pull/31900/head:
mgr: fix debug typo

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #31442 into master

* refs/pull/31442/head:
kv/RocksDBStore: Add minimum key limit before invoking DeleteRange.

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

msg/async/dpdk: Porting seastar's init_port_start to fix to resolve compilation errors

Ceph currently uses dpdk19.05, seastar dpdk_device::init_port_start has
been upgraded to DPDK19.05, so porting changes to fix compiler error.

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

Merge pull request #31165 from votdev/improve_py_unittests

mgr/dashboard: Refactor Python unittests and controller

Reviewed-by: Ernesto Puertat <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #31856 from rzarzynski/wip-crimson-errorator-handle_error

crimson/common/errorator.h: add handle_error() method.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31853 from liewegas/fix-rn-typo

PendingReleaseNotes: fix typo

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #31918 from rzarzynski/wip-crimson-errorator-simplify_safe_then

crimson/common/errorator.h: simplify the compound safe_then() variant.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #31838 into master

* refs/pull/31838/head:
ceph-daemon: chown uid/gid files during `adopt`
ceph-daemon: use 'iglob' instead of 'glob'
ceph-daemon: Fix handling for symlinks on python2

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

krbd: misc cleanups

Introduce get_devnode() and append_unmap_options(); make some functions
static.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: make wait_for_udev_add() clearer and a bit more efficient

Collect only /dev/rbd* block events and dispose of them as soon as
possible; match on devnode and assert on major/minor instead of the
other way around.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

krbd: do away with explicit memory management

Wrap udev_monitor, udev_enumerate and udev_device with std::unique_ptr.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: Eye candy updates to datatable pages

- Introduce boolean and array pipes
- Introduce configurable 'badge' datatable column
- Refactor the Inventory, OSD, RBD, iSCSI, ... datatable pages

Signed-off-by: Volker Theile <vtheile@suse.com>

crimson/common/errorator.h: simplify the compound safe_then() variant.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common/errorator.h: add handle_error() method.

In contrast to `safe_then` it deals only with errors
and, on return, leaves the value type `future<ValueT>`
embodies unchanged.
While being basically a specialization of `safe_then`,
it's also a convenience method to squeeze boilerplate.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #31914 into master

* refs/pull/31914/head:
mgr/orchestrator: set node labels to empty list if none specified

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/dashboard: support creating OSDs on spare devices (#30921)

mgr/dashboard: support creating OSDs on spare devices

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

cmake: Add shared library as dpdk dependency

librte_eal.a refers to libpthread.so and libdl.so, and libceph-common.so
refers to librte_eal.a. Many test application link libceph-common.so
without linking pthread, it will cause link error.
Such as unittest_interval_set, target_link_libraries (unittest_interval_set
ceph-common)

Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

cmake: Add dpdk numa support

Find and link the numa library for DPDK.

Fixes: https://tracker.ceph.com/issues/42275
Signed-off-by: Hu Ye <yehu5@huawei.com>
Signed-off-by: Chunsong Feng <fengchunsong@huawei.com>

Merge pull request #31785 from votdev/cleanup_remove_useless_var

mgr/dashboard: Remove unused variable

Reviewed-by: Kiefer Chang kiefer.chang@suse.com
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #31720 from rhcs-dashboard/fix-42708-master

mgr/dashboard: remove traceback/version assertions

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #31843 from votdev/rename_list_pipe

mgr/dashboard: Rename pipe list -> join

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

rgw: improvements to SSE-KMS with Vault

* Minor improvements to Vault documentation
* Add teuthology tests for Transit secrets engine
* Add unit tests for KV secrets engine, minor improvements to Transit
  secrets engine
* use string_view::npos instead of string::npos

Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>

rgw: Vault's Transit Secrets updates and docs

  * Drop polymorfism for KMS class
  * Fix issue in kms-key selection
  * Update documentation for Vault section

Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>

rgw: Add support to Vault's Transit Secrets Engine in SSE-KMS

* refactor rgw_kms.cc to support extension to multiple secret engines.
* introduced support to Vault Namesapces
* added support for Vault Agent

Signed-off-by: Andrea Baglioni <andrea.baglioni@workday.com>
Signed-off-by: Sergio de Carvalho <sergio.carvalho@workday.com>

os/bluestore: don't round_up_to in apply_for_bitset_range.

I met the following error messages:
>sceph7: /home/Jianpeng/ceph/src/os/bluestore/bluestore_common.h: In function 'void apply_for_bitset_range(uint64_t, uint64_t, uint64_t, Bitset&, Func) [with Bitset = boost::dynamic_bitset<long unsigned int>; Func = BlueFS::_replay(bool, bool)::<lambda(uint64_t, boost::dynamic_bitset<long unsigned int>&)>; uint64_t = long unsigned int]' thread 7f7d55a3dd40 time 2019-11-27T23:26:05.949827+0800
>sceph7: /home/Jianpeng/ceph/src/os/bluestore/bluestore_common.h: 28: FAILED ceph_assert(end <= bitset.size())
>sceph7:  ceph version 15.0.0-7747-g22a9a28e28 (22a9a28e2859718c4168f5b3cf6e20a9186e18bc) octopus (dev)
>sceph7:  1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x14f) [0x56322c4440ef]
>sceph7:  2: (()+0x50d2d9) [0x56322c4442d9]
>sceph7:  3: (BlueFS::_replay(bool, bool)+0x5639) [0x56322cb51e49]
>sceph7:  4: (BlueFS::mount()+0x269) [0x56322cb53289]
>sceph7:  5: (BlueStore::_open_bluefs(bool)+0x29e) [0x56322ca6e38e]
>sceph7:  6: (BlueStore::_open_db(bool, bool, bool)+0x59b) [0x56322ca6e9cb]
>sceph7:  7: (BlueStore::mkfs()+0x749) [0x56322caaa259]
>sceph7:  8: (OSD::mkfs(CephContext*, ObjectStore*, uuid_d, int)+0x1c8) [0x56322c5099e8]
>sceph7:  9: (main()+0x1787) [0x56322c4ab117]
>sceph7:  10: (__libc_start_main()+0xeb) [0x7f7d55b5fb6b]
>sceph7:  11: (_start()+0x2a) [0x56322c4de71a]
>sceph7: *** Caught signal (Aborted) **
>sceph7:  in thread 7f7d55a3dd40 thread_name:ceph-osd

If bdev[i]->get_size() / alloc_size[i] != 0,  round_up_to(off+lengh,alloc_size[i])/alloc_size is larger than(one)
ownend_blocks[i].size().

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

ceph-daemon: Allow env var for setting the used image

Instead of always adding "--image my-custom-image" when calling
ceph-daemon with a non-standard image, allow to set the environment
variable called CEPH_DAEMON_IMAGE which will adjust the --image
default.
That way, the command line arguments when using ceph-daemon with a
custom image are a bit shorter.

Signed-off-by: Thomas Bechtold <tbechtold@suse.com>

mgr/orchestrator: set node labels to empty list if none specified

f06366836ff tries to ' '.join(node.labels), which raises an exception
if labels is None, so we need to default it to an empty list.

Signed-off-by: Tim Serong <tserong@suse.com>

osd: remove useless ceph_assert.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #31588 from usefulalgorithm/patch-1

doc/rados/operations: OSD_OUT_OF_ORDER_FULL fullness order is wrong

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

test: Add test case based on Xie script in commit comment

Other test fixes to reflect changes

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: ceph_pg_upmaps() use any overfull when there are still underfull

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Create more_underfull with below target that aren't in underfull

Use more_underfull for finding remaps for overfull OSDs

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: calc_pg_upmaps() pick most overfull remap from try_pg_upmap()

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: ceph_pg_upmaps() use max_deviation to determine perfect distribution

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: osdmaptool sync with balancer module behavior

This can be backported only to nautilus

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: osdmaptool: Perform upmap calculation as ceph-mgr does

This is the backportable commit that works with older balancer module.py

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: For balancer crush needs the rule passed to get_parent_of_type()

Fixes: https://tracker.ceph.com/issues/42718
Signed-off-by: David Zafman <dzafman@redhat.com>

osdmaptool: Match default max value of 10

Signed-off-by: David Zafman <dzafman@redhat.com>

tools: odsmaptool truncate target upmap file

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr: Fix balancer print

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge PR #31870 into master

* refs/pull/31870/head:
ceph-daemon: fix `systemctl is-enabled` bool

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31886 into master

* refs/pull/31886/head:
ceph-daemon: error out if not root
ceph-daemon: user-friendly exception/error messages
ceph-daemon: user-friendly exception/error messages

Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #31883 from jan--f/ceph-volume-code-owner

github/codeowners: add ceph-volume

Merge PR #31892 into master

* refs/pull/31892/head:
ceph-daemon: seek relative to the start of file

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #31899 into master

* refs/pull/31899/head:
mgr: drop session with Ceph daemon when not ready
mgr: improve debug message information

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-daemon: error out if not root

Signed-off-by: Sage Weil <sage@redhat.com>

ceph-daemon: user-friendly exception/error messages

- Rename `CDError` to `Error`
- Display traceback on console when `-v` is specified
- Other misc clean-up

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #30262 from sebastian-philipp/orchestrator-pomise-future

mgr/orchestrator: Make Completions composable

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/dashboard: Trim IQN on iSCSI target form

Fixes: https://tracker.ceph.com/issues/43027
Signed-off-by: Ricardo Marques <rimarques@suse.com>

mgr/ssh: Disable host labels.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #31896 from rsevero/missing_virtualenv

doc: add missing virtualenv for build-doc

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/orchestrator: use finalize to set TrivialReadCompletion result

This is needed to correctly set the state of the completion, so that
Completion.has_result returns true.  Without this, _orchestrator_wait()
will spin forever if given a TrivialReadCompletion.

Signed-off-by: Tim Serong <tserong@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add pretty_print()

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Fix Promises

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

import typing optionally

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Fix testing error

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/deepsea: Fixed DeepSeaReadCompletion arguments

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: fix tests

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

make check fixes

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchstrator: Removed all type anotations incompatible to Python 2

* Reduce mypy's type checking capability
* Remove useful code documentation

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/orchestrator: Adapted to new Completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: add cancel

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

qa/orchestrator: Adapt teuthology test to new completions

Also: Adapt mgr/test_orchestrator to new completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/deepsea: Adapt to new orchestrator completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ansible: Adapt to new orchestrator completions

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ssh: Adapt ssh orch to new Completions interface

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>