test/crimson: mark final class "final"

silences warning from Clang like:

../src/test/crimson/seastore/test_object_data_handler.cc:33:16: warning: class with destructor marked 'final' cannot be inherited from [-Wfinal-dtor-non-final-class]
  ~TestOnode() final = default;
               ^
../src/test/crimson/seastore/test_object_data_handler.cc:20:7: note: mark 'TestOnode' as 'final' to silence this warning
class TestOnode : public Onode {
      ^
1 warning generated.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: do not capture unused variable

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42602 from b-ranto/wip-lto

rados+rpm: Update symver defs and re-enable LTO

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42564 from myoungwon/wip-fix-45423

test: fix wrong alarm (HitSetWrite)

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42277 from tchaikov/wip-vstart-runner-cleanups

qa/tasks/vstart_runner: do not send SIGTERM if no matched pid

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #42663 from cyx1231st/wip-fix-lba-empty-delta

crimson/os/seastore/lba_manager: remove assert non-empty delta

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #42669 from tchaikov/wip-doc-ext-confval

doc/_ext: print out the module which fails the build

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42681 from tchaikov/wip-fmt-a

cmake: link bundled fmt statically

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #42593 from ceph/wip-yuriw-release-15.2.14-master

doc: 15.2.14 Release Notes

doc/releases/octopus.rst: add notable changes for mgr and core

Signed-off-by: Neha Ojha <nojha@redhat.com>

doc: add note about mon trimming of old MDSMaps

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: 15.2.14 Release Notes

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>
Signed-off-by: David Galloway <dgallowa@redhat.com>

Merge pull request #42505 from wzbxqt327/put_obj_rentention

rgw: add the condition of lock mode conversion to PutObjRentention

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42531 from ivancich/wip-user-utilized-stats

rgw: user stats showing 0 value for "size_utilized" and "size_kb_utilized" fields

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42597 from ivancich/wip-rename-f

rgw: clean up some code

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #42125 from ivancich/wip-cls-empty-listing

rgw: resolve empty ordered bucket listing results w/ CLS filtering

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42182 from tengfWang/rgw-swift-check-container-name

rgw: remove the three-character limit when creating a container with swift interface

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #42397 from mkogan1/rgw-d3n-rem-ops-log

rgw: d3n: remove legacy requirement to enable rgw_enable_ops_log

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #42673 from guits/update_cephadm_doc

doc/cephadm: update documentation

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42664 from sebastian-philipp/cephadm-FakePipeWrapper

cephadm: fix unit tests

Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #42665 from sebastian-philipp/cephadm-doc-upgrade-offline

doc/cephadm: Offline hosts will pause upgrades

Reviewed-by: Sage Weil <sage@newdream.net>

cmake: link bundled fmt statically

instead of building the bundled fmt library as shared library, built
static library. otherwise we'd have to link against libfmt.so.

Signed-off-by: Kefu Chai <kchai@redhat.com>

librados/librados_c: check for both symvers

We should check if either asm symver or attribute symver is defined and
not assume that attribute symver implies asm symver.

Fixes: https://tracker.ceph.com/issues/40060
Signed-off-by: Boris Ranto <branto@redhat.com>

cmake: Check -flto flags

We should check if -flto-partition=none is defined when the compiler
does not support symver attribute and fail the build if it is not.

Fixes: https://tracker.ceph.com/issues/40060
Co-authored-by: Kefu Chai <tchaikov@gmail.com>
Signed-off-by: Boris Ranto <branto@redhat.com>

librados/librados_c: Use symver attribute if available

Fixes: https://tracker.ceph.com/issues/40060
Signed-off-by: Boris Ranto <branto@redhat.com>

rpm: Re-enable LTO on supported systems

We can now use LTO when building ceph. The symver issue was fixed by
using the gcc __symver__ attribute. The systems that support it can now
re-enable LTO.

Fixes: https://tracker.ceph.com/issues/40060
Signed-off-by: Boris Ranto <branto@redhat.com>

doc/cephadm: update documentation

This commit fixes a couple of typos in the cephadm documentation.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

doc/_ext: print out the module which fails the build

Signed-off-by: Kefu Chai <kchai@redhat.com>

rados: Update symver defs

The gcc compiler now supports symver attribute. We should update the
symvers to be able to support LTO.

Fixes: https://tracker.ceph.com/issues/40060
Signed-off-by: Boris Ranto <branto@redhat.com>

rados: Fix function ordering

The build fails because the function definitions are out of order. This
also reuses the LIBRADOS_C_API_DEFAULT_F macro for function definitions.

Fixes: https://tracker.ceph.com/issues/40060
Signed-off-by: Boris Ranto <branto@redhat.com>

doc/cephadm: Offline hosts will pause upgrades

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

cephadm: fix unit tests.

Fixes AttributeError: 'FakePipeWrapper' object has no attribute 'readable'

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42242 from sebastian-philipp/cephadm-container-name-dashes

cephadm: use dashes for container names

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

crimson/os/seastore/lba_manager: remove assert non-empty delta

Delta can be empty when measure the effort of invalidated transactions.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42462 from sebastian-philipp/mon-ip-change

doc/cephadm: Move MONs to a different network

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #42619 from jdurgin/wip-release-notes-script

script/ceph-release-notes: improve output for markdown and limit verbosity

Reviewed-by: David Galloway <dgallowa@redhat.com>

script/ceph-release-notes: don't include PR body by default, just the title

This avoids overly long output for merges involving lots of PRs,
e.g. cephadm batch backports.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

script/ceph-release-notes: infer actual title from ptl-tool merges

This helps the notes stay accurate when the commit title has changed,
but the PR title is out of date.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #42636 from zdover23/wip-doc-rados-config-storage-devices-bluestore-2021-08-03

doc/rados: rewrite BlueStore material

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41897 from ivancich/wip-versioned-deletes

rgw: when deleted obj removed in versioned bucket, extra del-marker added

Reviewed-by: Casey Bodley <cbodley@redhat.com>

cephadm: Introduce unit.stop

Reason is we now have to stop two containers named differently. This is possible
with `bash -c ... echo %i | tr . -`. But we're not gainig anything readability
compared to putting it into a unit.stop script.

As not all daemons have this stop script, we still have to call podman for old daemons.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

cephadm: don't log on exit.

At this point, the garbage collector already colleceted the `open` function

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

cephadm: `stats` might return `--` for containers

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

cephadm: exec: also search for old cname

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

cephadm: dashes: use both cnames for `inspect`

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

cephadm: Add CephContainer.for_daemon()

To avoid calling get_container()

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

cephadm: use dashes for container names

podman adds the current container name to the /etc/hosts
file. Turns out, python's `socket.getfqdn()` differs from
`hostname -f`, when we have the container names containing
dots in it.:

[root@sebastians-laptop /]# cat /etc/hosts
127.0.0.1   localhost
::1         localhost
127.0.1.1   sebastians-laptop foo.bar.baz.com
[root@sebastians-laptop /]# hostname -f
sebastians-laptop
[root@sebastians-laptop /]# python3 -c 'import socket; print(socket.getfqdn())'
foo.bar.baz.com

Fascinatingly, this doesn't happen when using dashes.

Fixes: https://tracker.ceph.com/issues/51590
Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42395 from sebastian-philipp/cephadm-improve-no-enigne-msg

cephadm: Clarify no container engine message

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: MON IP change

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42587 from liewegas/rgw-mon-secure

rgw: default ms_mon_client_mode = secure

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #42610 from rhcs-dashboard/52022-fix-rgw-perf-counters

mgr/dashboard: show perf. counters for rgw svc. on Cluster > Hosts

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42554 from rhcs-dashboard/fix-rgw-ssl-cert-validation

mgr/dashboard: fix ssl cert validation for rgw service creation

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #42575 from tchaikov/wip-venv

*: s/virtualenv/python -m venv/

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42594 from sebastian-philipp/mypy-ini-disallow-untyped-root

mypy.ini: Globally enable `disallow_untyped_defs = True`

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42571 from tchaikov/wip-disable-kvs

cmake: disable kvs rados cls by default

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42614 from soumyakoduri/wip-skoduri-sqlite3

cmake: Remove duplicate SQLITE module

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: show perf. counters for rgw svc. on Cluster > Hosts

- Use service_map_id to retrieve perf. counters for rgw services.
- Move logic from controller to CephService.

Fixes: https://tracker.ceph.com/issues/52022
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

rgw: allow ordered bucket listing to work when many filtered out entries

A previous PR moved the much of the filtering that's part of bucket
listing to the CLS layer. One unanticipated result was that it is now
possible for a call to return 0 entries. In such a case we want to
retry the call with the marker moved forward (i.e., advanced),
repeatedly if necessary, in order to either retrieve some entries or
to hit the end of the entries. This PR adds that functionality.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: allow CLSRGWConcurrentIO to handle "advancing" retries

When doing an asynchronous/concurrent bucket index operation against
multiple bucket index shards, a special error code is set aside to
indicate that an "advancing" retry of a/some shard(s) is necessary. In
that case another asynchronous call is made on the indicated shard(s)
from the client (i.e., CLSRGWConcurrentIO).  It is up to the subclass
of CLSRGWConcurrentIO to handle the retry such that it "advances" and
simply doesn't get stuck, looping forever.

The retry functionality only works when the "need_multiple_rounds"
functionality is not in use.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: de-conflate shard_id and request_id in CLSRGWConcurrentIO

When using asynchronous (concurrent) IO for bucket index requests,
there are two int ids that are used that need to be kept separate --
shard id and request id. In many cases they're the same -- shard 0
gets request 0, and so forth.

But in preparation for re-requests, those ids can diverge, where
request 13 maps to shard 2. The existing code maintained the OIDs that
went with each request. This PR also maintains the shard id as
well. Documentation has been beefed up to help future developers
navigate this.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

cmake: disable kvs rados cls by default

libcls_kvs was introduced back in
73d016fdb304ad19bba8aed3f2877b4bdb6ed32e, but we don't have an internal
user so far. to reduce the build time. let's disable the build of it by
default.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42567 from dsavineau/issue_51973

cephadm: set global default container for ingress

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #42559 from tchaikov/wip-rpm-dwz

rpm: enable dwz

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

script/ceph-release-notes: sort PRs case-insensitively

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/dashboard: fix ssl cert validation for rgw service creation

Fixes: https://tracker.ceph.com/issues/51965
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

script/ceph-release-notes: adapt markdown output for new website

* escape underscores
* avoid automatic links from PR titles containing .

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

rgw: clean up some code

In rgw_admin.cc a variable `f` is used frequently. However due to the
sheer size of the file and variable scopes involved, finding the
declaration of `f`, which is a common character, can be
challenging. So it has been renamed `stream_flusher`, which better
indicates its purpose and makes it easier to find the corresponding
declaration.

Additionally some unneeded variables in RGWBucket::check_object_index
were removed. rgw::sal::Bucket::ListParams initializes parameters as
one would hope/expect.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

cmake: Remove duplicate SQLite module

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

doc/rados: rewrite BlueStore material

This PR improves the English text
explaining the BlueStore filesystem
backend.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42605 from aaryanporwal/visual-tests

mgr/dashboard: Jenkins Applitools plugin: add batchId variable

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge PR #42463 into master

* refs/pull/42463/head:
src/pybind/mgr/test_orchestrator: remove nfs pool and namespace
src/vstart: update nfs-ganesha pool name

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #42518 into master

* refs/pull/42518/head:
doc/cephadm: update about disabling logging to journald for quincy
doc/cephadm: remove repeated section on disabling logging to stderr

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #42588 from liewegas/doc-foundation-didi

doc/foundation: drop didi

cephadm: Clarify no container engine message

Multiple people didn't understand the message. Let's try to impove it.

(Also make mypy happy)

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/dashboard: Cluster > Hosts > Performance Counters: do not show context bar.

Do not show the context bar (Object Gateway selection).

Fixes: https://tracker.ceph.com/issues/52022
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

cephadm: Add missing type annotations

Turnd out `command_maintenance` missing an if-branch that returned
None instead of a str

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: update about disabling logging to journald for quincy

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/dashboard: Jenkins Applitools plugin: add batchId variable

batchId variable is mandatory for the applitools test runner to show up in jenkins build reports, for more details, check out: https://plugins.jenkins.io/applitools-eyes/

Signed-off-by: Aaryan Porwal <aaryanporwal2233@gmail.com>

Merge pull request #42601 from idryomov/wip-rbd-qemu-iotests-8stream

qa/workunits/rbd: use xenial version of qemu-iotests for centos stream 8

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #42446 from zdover23/wip-doc-cephadm-troubleshooting-logs-2021-07-20

doc/cephadm: linking to log material

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/cephadm: remove repeated section on disabling logging to stderr

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #42328 from aaryanporwal/visual-tests

mgr/dashboard: Visual regression tests for ceph dashboard

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: aaryanporwal <NOT@FOUND>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #42514 from liu-chunmei/seastore-onode-interruptible-future

crimson/onode-staged-tree: integrate interruptible future

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

rgw: add the condition of lock mode conversion to PutObjRentention

Signed-off-by: wangzhong <wangzhong@cmss.chinamobile.com>

qa/workunits/rbd: use xenial version of qemu-iotests for centos stream 8

It is already used for centos 8(.3) and rhel 8(.4).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

crimson/onode-staged-tree: misc fixes to integrate interruptive-future

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/cephadm: linking to log material

This PR rewrites a section in the Troubleshooting
chapter of the Cephadm Guide. The material that this
section discusses has been covered already in the
Cephadm Guide in the Cephadm Operations chapter.
There's no reason to repeat this information twice,
unless adding technical debt to the documentation
is our goal (which of course it is not, and the
opposite of adding technical debt to the documentation
has been the aim that has guided my work these past
six months).

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42471 from Dorthu/fix-pybind-rbd-mirror-image-get-status

pybind/rbd: fix mirror_image_get_status

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42570 from tchaikov/wip-libcryptsetup

librbd/crypto/luks: require libcryptsetup v2.0.5

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

pybind/mgr: enable `disallow_untyped_defs` for mgr_module

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mypy.ini: Globally enable `disallow_untyped_defs = True`

Plus introduce an ignorelist of all modules that are not compatible yet.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

rgw: default ms_mon_client_mode = secure

Require a secure connection to the monitor.  This is important because
we may store SSL certs on the mon.

Note that secure mode was introduce before Nautilus, so this can be
backported at least to Pacific.

Fixes: https://tracker.ceph.com/issues/52000
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42551 from adk3798/maint-message

mgr/cephadm: make return message for entering maintenance mode more explicit

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

doc/foundation: drop didi

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #42382 from xrmeng8756/master

rgw: avoid occuring radosgw daemon crash when access a conditionally …

Merge pull request #42387 from wzbxqt327/master

rgw:add lock to copy object

Merge pull request #42404 from ivancich/wip-broken-list-plain-entries

 rgw: bucket index list can produce I/O errors

Merge pull request #42582 from tchaikov/wip-doc-health-report

doc/dev: add health-reports.rst

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #42577 from tchaikov/wip-rtd

.readthedocs.yml: use python3.8 and native ditaa

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

doc/dev: add health-reports.rst

to explain the data flow of health metrics.

Signed-off-by: Kefu Chai <kchai@redhat.com>