qa/tasks/rook: add OSD creation to Rook QA

This commit adds OSD creation to the Rook QA tasks. The Rook task will
explicitly wait for the mgr to start and the CLI to work (instead of
implicitly doing so while waiting for 'ceph osd dump' to work).
Then it will do `ceph orch apply osd --all-available-devices` to create
OSDs on the rest of the PVs.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

Merge pull request #42852 from sebastian-philipp/centos_8.3_container_tools_3.0.yaml

qa/cephadm: centos_8.3_container_tools_3.0.yaml

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41950 from zenomri/rgw-crypt-dpp

src/rgw : ldout and lderr calls were replaced by ldpp_dout calls

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #42851 from dang/wip-dang-zipper-raw_obj

RGW - Zipper remove unused files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42834 from josephsawaya/rook-orch-qa

qa/tasks/rook: add `ceph orch device ls` command to the rook qa task

Merge pull request #42757 from josephsawaya/wip-mgr-rook-osd-creation

mgr/rook: Rook orchestrator OSD creation using `ceph orch apply osd`

qa/cephadm: centos_8.3_container_tools_3.0.yaml

Let's avoid latest stable

Fixes: https://tracker.ceph.com/issues/52279
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

RGW - Zipper remove unused files

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42796 from dang/wip-dang-zipper-raw_obj

RGW - Remove last rgw_raw_obj from zipper

Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42793 from sebastian-philipp/dashboard-fix-standby-redirect

pybind/mgr: Fix IPv6 url generation

Merge pull request #42302 from myoungwon/wip-51000

osd: fix to allow inc manifest leaked

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #42820 from ronen-fr/wip-ronenf-mgr-argvvec

common: create a faster & cleaner alternative to argv_to_vec()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42823 from mgfritch/ceph-volume-py3-exec

cmake: s/Python_EXECUTABLE/Python3_EXECUTABLE/

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42799 from varshar16/wip-update-email-id

.githubmap: update mail address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42835 from ceph/wip-ctest-fail

run-make-check.sh: Increase failure output log size

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42828 from tchaikov/wip-doc-dev-config

doc/dev/config: update to in sync with latest .yaml.in changes

Reviewed-by: Neha Ojha <nojha@redhat.com>

run-make-check.sh: Increase failure output log size

Fixes: https://tracker.ceph.com/issues/43566
Signed-off-by: David Galloway <dgallowa@redhat.com>

qa/tasks/rook: add device ls command to rook task

This commit adds the device ls command to the rook qa task
since that command should be working from now on.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

RGW - Zipper - remove last rgw_raw_obj from API

Remove the remaining rgw_raw_obj from the API.  This consists of:

- move get/delete raw_obj from Store to RadosStore; it's only needed by
  RadosStore code
- Add a layout formatter to the API
- remove head_obj and get_manifest() from ReadOp
- Remove GCChain (no longer needed)

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Fix uninitialized warnings

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42827 from ronen-fr/wip-ronenf-std-fix2

key_value_store: fix missing std

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42825 from hualongfeng/fix_blk_pmem

blk/pmem: Add namespace std for map,string

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: fix generation of wrong IPv6 urls

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/restful: Fix url generation for IPv6 hosts

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/prometheus: Fix url generation for IPv6 hosts

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/dashboard: Fix redirect to active MGR

Fixes: https://tracker.ceph.com/issues/52117
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

pybind/mgr: move build_url from dashboard to mgr_util

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42690 from adk3798/offline2

mgr/cephadm: make scheduler able to accomodate offline/maintenance hosts

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Cory Snyder <csnyder@iland.com>

doc/dev/config: update to in sync with latest .yaml.in changes

we migrated to a build process which translate the options from .yaml.in
files. so update the document to reflect this change.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42592 from vfreex/cephadm-better-port-stripping

cephadm: better port stripping

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42590 from vfreex/cephadm-propagate-env-vars

cephadm: propagate environment variables to subprocesses

Merge pull request #42589 from vfreex/fix-cephadm-ipv6

cephadm: Fix bootstrap error with IPv6 mon-ip

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42816 from sebastian-philipp/cephadm-upgrade-status-message

mgr/cephadm/upgrade: Upgrade status: We also upgrade non-ceph daemons

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>

key_value_store: fix missing std

following the removal of 'using std' in PR #42742, and missed
in PR #42790.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #41968 from CongMinYin/fix-ssd-bug

librbd/cache/pwl/ssd: fix m_bytes_allocated exceeding m_bytes_allocated_cap

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42768 from s0nea/wip-dashboard-suppressed-alerts

mgr/dashboard: don't notify for suppressed alerts

blk/pmem: Add namespace std for map,string

Due to not use namespace std for map,string, it leads to
`map,string` not be declared.

Signed-off-by: Feng Hualong <hualong.feng@intel.com>

Merge pull request #42758 from liewegas/gh-template

.github/pull_request_template: fix dashboard e2e test command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42801 from ivancich/wip-update-rados-rm-help

doc: clarify use of `rados rm` command

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42822 from tchaikov/wip-drop-gdbm

ceph.spec.in: drop gdbm from build deps

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

tools/setup-virtualenv: verify the `--python` param

when given an invalid or empty (PYTHON_BINARY)
fixup for 494fea50ced7

Fixes: https://tracker.ceph.com/issues/52304
Signed-off-by: Michael Fritch <mfritch@suse.com>

cmake: s/Python_EXECUTABLE/Python3_EXECUTABLE/

pass the python3 exec when creating the ceph-volume build venv
fixup for 5fc657b40dc7

Fixes: https://tracker.ceph.com/issues/52304
Signed-off-by: Michael Fritch <mfritch@suse.com>

mgr/rook: make use of SizeMatcher in Creator classes

This commit uses the SizeMatcher Class in the Creator functions
to parse and filter devices according to the size specified in a
drive_group.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

common: create a faster & cleaner alternative to argv_to_vec()

New function signature follows current C++ guidance regarding
returning values from functions. It is also faster
- 1.3X for a sample test of 10 arguments.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>
Co-authored-by: Kefu Chai <tchaikov@gmail.com>

ceph.spec.in: drop gdbm from build deps

neither gdbm nor libgdbm is used when building ceph, so let's drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rook: fix convert_size method in DefaultFetcher

This commit fixes the convert_size method by getting it
to use the re python module to split the digits and letters
to support all units a PV could be expressed in.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: create get_storage_class method on RookCluster

this commit creates the get_storage_class method on the RookCluster
class used to get the storage class matching the name provided
in the ceph config.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: apply_drivegroups fix and coding style fixes

This commit fixes the apply_drivegroups method in RookOrchestrator
to process the entire list of drive groups passed.

This commit also fixes some coding style errors in RookCluster.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: keep drive groups updated and clean up cluster CR

This commit creates a new threaded methods on the RookCluster class
that keeps the cluster updated by re-applying drive groups in a
loop.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: OSD creation using Creator classes

This commit implements the apply_drivegroups method in the
RookOrchestrator class and creates the DefaultCreator and
LSOCreator classes that handle creating OSDs. The add_osds
method in RookCluster will use each creator based on what
storage class the user provided in the ceph config.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

mgr/rook: add supplemental information to device object in LSOFetcher

This commit adds the PV name, node name, vendor and model to the Device object
created in the LSOFetcher device method, this information is useful for adding
OSDs.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

Merge pull request #42738 from sebastian-philipp/mypy-3.7

pybind/mgr: tox: also test mypy with python 3.7

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: allowing >= the real refcount to avoid false alarm

Current dedup allow to contain multiple same sources using
multiset, which results in inconsistent situation as follow
(during set_chunk, but not confined in set_chunk).

1. User issues set_chunk
2. OSD receives the set_chunk, and sends increment message
to an object in the low tier (INPROGRESS).
3. OSD map is changed (841 → 843)
3.5. on_change() is called
4. the set_chunk op is reenqueued by requeue_op()
5. OSD handles the duplicated set_chunk, but it is not able to
 know the set_chunk is duplicated because it does not log on the disk yet.
6. OSD issues increment message again to the object
in the low tier. (increment operation is executed twice)

To fix this, this commit allows >= the real refcount in test cases

fixes: https://tracker.ceph.com/issues/51000

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #42811 from tchaikov/wip-dashboard-test

mgr/dashboard/api: set a UTF-8 locale when running pip

Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #42732 from rhcs-dashboard/bug-1950644

mgr/dashboard: disable create snapshot with subvolumes

Merge pull request #42717 from rhcs-dashboard/iscsi-target-refresh-btn

mgr/dashboard: Refresh button on the iscsi targets page

mgr/cephadm/upgrade: Upgrade status: We also upgrade non-ceph daemons

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

librbd/cache/pwl/ssd: ensure bdev and pool sizes match

m_log_pool_size should be multiple of 1M but, just in case, prevent
is_valid_io() assert in KernelDevice::aio_write().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42786 from aaryanporwal/visual-tests

mgr/dashboard: visual tests: Add more ignore regions for dashboard component

mgr/dashboard: disable create snapshot with subvolumes

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #42808 from tchaikov/wip-crimson-test-cleanup

test/crimson: do not discard returned future<>

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard/api: set a UTF-8 locale when running pip

ansible-core started to include files whose filenames are encoded in
non-ascii characters, so we have to use a more capable encoding for the
locale in order to install this package. otherwise we'd have following
error:

Collecting ansible-core<2.12,>=2.11.3
  Using cached ansible-core-2.11.4.tar.gz (6.8 MB)
ERROR: Exception:

Traceback (most recent call last):
  File "/tmp/tmp.fX76ASIrch/venv/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 173, in _main
    status = self.run(options, args)
...
  File "/tmp/tmp.fX76ASIrch/venv/lib/python3.8/site-packages/pip/_internal/utils/unpacking.py", line 226, in untar_file
    with open(path, "wb") as destfp:
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 137-140: ordinal not in range(256)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42807 from tchaikov/wip-crimson-test-more-chatty

test/crimson: re-enable logging output

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #42791 from ideepika/wip-50947-blk-cmake

src/blk: fix block_device_t return if no aio libs present

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42800 from tchaikov/wip-cmake-debug-build

cmake: set CMAKE_BUILD_TYPE to Debug when .git exists

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

test/crimson: do not discard returned future<>

to silence following warning:

../src/test/crimson/seastore/test_randomblock_manager.cc:122:7: warning: ignoring return value of function declared with 'nodiscard' attribute [-Wunused-result]
      rbm_manager->add_free_extent(t.allocated_blocks, r.first * DEFAULT_BLOCK_SIZE,
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: use structured binding

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42798 from xxhdx1985126/wip-52275

crimson/common: allow the tls interrupt_cond to exist when a continuation starts to run

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: re-enable logging output

this change reverts 57877075c0b6885c7e35dc4c4a881b140c30cfb0

per Yingxin's request, we should enable it for better debugging
experience when the test fails.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42787 from rzarzynski/wip-crimson-fix-warn-storenbd

crimson/tools: silence [[maybe_unused]] warning in store-nbd.cc.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

.githubmap: update mail address

Signed-off-by: Varsha Rao <rvarsha016@gmail.com>

Revert "crimson/onode-staged-tree: force test work with invalidated transactions"

This reverts commit 8b6428c4ac08a7d3beabf64f549bfe11e5245978.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: avoid creating interrupt condition within an already interruptible continuation chain

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: assert the existence of interrupt_cond even in release builds

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: allow the tls interrupt_cond to exist when a continuation starts to run

Release build of seastar invokes continuations synchronously if their predecessor futures
are resolved at the time when those continuations are constructed, so we have to allow
these circumstances to happen

Fixes: https://tracker.ceph.com/issues/52275
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #42658 from tchaikov/wip-doc-build

admin/build-doc: rebuild venv if md5 does not match

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge PR #42749 into master

* refs/pull/42749/head:
doc: document that NEARFULL conditions can result in slow writes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #42797 from rzarzynski/wip-crimson-no-prometheus-by-default

crimson/osd: disable Premetheous by default.

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: set CMAKE_BUILD_TYPE to Debug when .git exists

if CMAKE_BUILD_TYPE is an empty string, we should also set it to Debug.

Signed-off-by: Kefu Chai <kchai@redhat.com>

erasure-code/clay: do not set unreferenced variable

plane_count was introduced by 2a3e56b22c5b52e4834a9fd5047b3cdff7e8859f,
and presumably it is only used for debugging. let's drop it as it is not
referenced in production.

this change silences following warning:

../src/erasure-code/clay/ErasureCodeClay.cc:502:7: error: variable 'plane_count' set but not used [-Werror,-Wunused-but-set-variable]
  int plane_count = 0;
      ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

erasure-code/lrc: early return if to_string() fails

so far to_string() always return 0, but it is designed to return
an error code if it fails to parse a string.

this change silences following warning:

../src/erasure-code/lrc/ErasureCodeLrc.cc:399:7: error: variable 'err' set but not used [-Werror,-Wunused-but-set-variable]
  int err = 0;
      ^

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds/CDir: do not set referenced variable

off is set by never get referenced, let's just drop it.

this variable was introduced by
0ec5f3a84aa29ec422e920fb5dc989ed90897e6b.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42731 from liu-chunmei/seastore-alloc-extent-hint

crimson/seastore alloc extent hint

Replace L_ADDR_MIN by obj hash in onode, omap, object_data when alloc_extent.

Reviewed-by: Sam Just <sjust@redhat.com> and Cheng, Yingxin <yingxin.cheng@intel.com>

rgw: disable -Wimplicit-const-int-float-conversion conditionally

to silence warning like:

 unknown warning group '-Wimplicit-const-int-float-conversion', ignored [-Werror,-Wunknown-warning-option]
 #pragma clang diagnostic ignored "-Wimplicit-const-int-float-conversion"

clang-10 does not support this option yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: clarify use of `rados rm` command

The man page did not make it clear that multiple objects could be
specified, nor did it describe use of "--force-full".

Info displayed about "rm" with `rados --help` was poorly formatted and
the wording was adjusted.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

librbd/cache/pwl: make pool size a multiple of 1M

In ssd mode, we need it to be a multiple of bdev block size.
Instead of munging it after opening the bdev in ssd/WriteLog.cc, let's
impose a common restriction and round rbd_persistent_cache_size down to
a 1M boundary.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crimson/seastore: change unit test using hint

Signed-off-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/seastore: replace L_ADDR_MIN by hint in omap

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/seastore: use get_hint in object_data_handler

Signed-off-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/seastore: replace L_ADDR_MIN by obj hash in Onode

Signed-off-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #42340 from mkogan1/wip-d3n-docs

doc/radosgw/d3n: add user docs

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

crimson/osd: disable Prometheous by default.

As we lack the machinery for distributed port allocation, spawning
multiple instances of crimson on the same node is impossible. This
makes almost all Teuthology jobs red very early.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/radosgw/d3n: add user docs

(high-level feature description, setup instructions, config options)
Fixes: https://tracker.ceph.com/issues/51409
Signed-off-by: Mark Kogan <mkogan@redhat.com>

Merge pull request #42790 from ronen-fr/wip-ronenf-std-fix

key_value_store: fix missing std

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42795 from tchaikov/wip-cmake-quiet

cmake: silence build output when building external deps

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

src/blk: fix block_device_t return if no aio libs present

In cases, when no libaio are present in the system, compilation fails,
return block_device_t as unknown

fixes: https://tracker.ceph.com/issues/50947#note-1

Signed-off-by: Deepika Upadhyay <dupadhya@redhat.com>

cmake: do not quote string

otherwise we have

CMake Error at /home/jenkins-build/build/workspace/ceph-pull-requests/build/spdk-ext-prefix/src/spdk-ext-stamp/spdk-ext-build-.cmake:37 (message):
  Command failed: 2

   'env' '-i' 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' 'CC=/usr/bin/clang-10' '/usr/bin/make' 'EXTRA_CFLAGS=' '-fPIC' '-Wno-address-of-packed-member""'

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: silence build output when building external deps

when download/building grafonnet-lib, dpdk, spdk, liburing and fio,
they dump lots of output during configuration and building phrases,
all of which is irrelevant to us. so let's just silence it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42784 from tchaikov/wip-rgw_dencoder

rgw/rgw_dencoder: set identity_type when creating samples

Reviewed-by: Casey Bodley <cbodley@redhat.com>

key_value_store: fix missing std

following PR #42742

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>