crimson/os/seastore/cache: replay_delta handle obsolete extents properly

If we see a delta at verison v (v != 0) and we haven't seen version v-1,
we know that the extent is obsolete and can be ignored (presuming that
journal trimming is behaving).

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: pass journal_seq with each replay delta

We'll need this to track where extents became dirty during replay.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: maintain dirty_from value on dirty CachedExtents

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: return addr and seq from submit_record

Reworks journal_seq_t to be a monotonically increasing, but non-dense
value from the current segment_seq_t and segment offset.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add interface for trimming the journal

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: rework pending extents

Previously, we simply added the old extent to the retired set and
treated the new instance seperately.  Instead, let the new pending
instance reference the old instance via a prior_instance reference.
This has a few advantages:
- During commit, we swap cache hooks from the prior instance to
  avoid traversal
- Extents in the retired set are only those extents that are no
  longer mapped/valid.
- btree range pins can be swapped without a traversal
- position in the dirty list can be maintained

That last will be particularly important as the dirty list will
therefore be ordered from the extent with the oldest initial
delta forward.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: factor out LogicalCachedExtent::on_delta_write

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: rename journal_replay_lb to journal_tail

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: factor out common LBANode helpers

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: remove CachedExtent move constructor

I'm not sure why I added this in the first place, there aren't really
any scenarios where invoking it wouldn't be a bug.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../btree_lba_node_impl: mutate_mapping adjust addr for extent state

Addresses must always be adjusted depending on whether the extent will
be written as a block or as a delta.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: improve misc debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore_types: permit paddr_t::operator- to operate on block relative addrs

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: use smaller segments for tests

This way, we'll roll segments more frequently.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree/btree_lba_manager: manage pin in init

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: drop from cache directly during replay if dead

Dead extents can't go through the normal retire_extent path as they don't
have valid pins or other elements of a real extent.  Instead, simply drop
them on the floor directly.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: fix find_hole

find_hole shouldn't check entries prior to min.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: initialize crc *after* read

Previously, all extents ended up initialized to the same crc value
upon read -- the one for an empty buffer.  This bug wasn't detected
because an extent was always evicted and reread following the first
write and subsequently kept in cache once initially written.

Also, add crc to operator<< for extents.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: only add pin if not already pending

If it's already pending, cache.duplicate_for_write is a noop and the extent
will already have a pin.  Also, add some debugging.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: let root exist in the dirty list

Most handling remains special, but this way the dirty list we
work on will include root directly.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: clarify crc initial state

Otherwise, different users may use different values complicating
debugging.  We can re-add this in the future if it becomes
useful.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: let get_extent signal retired extents

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: fix lower_bound

This went undetected because all existing users ensure that
the key does not already exist.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #37186 from guits/guits-cephadm-filter-dangling-images

cephadm: filter dangling images in get_last_local_ceph_image()

Merge pull request #37133 from jschmid1/ptrace_priv

cephadm: capadd and privileged are mutex with podman >= 2.0

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge PR #37310 into master

* refs/pull/37310/head:
client: refresh rstat for quota check of rename

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37041 from bk201/wip-47356

mgr/dashboard: fix security scopes of some NFS-Ganesha endpoints

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37244 from ceph/47543-bucket-list-perf

mgr/dashboard: fix perf. issue when listing large amounts of buckets

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37242 from dang/wip-dang-47520

Fix missing req_state for librgw

Merge pull request #37330 from idryomov/wip-backport-create-issue-key

doc: SubmittingPatches-backports.rst: backport scripts read key from file

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #37136 from p-se/cd-add-per-rbd-graphs

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

SubmittingPatches-backports.rst: backport scripts read key from file

The --key option was removed in commit 1b5f37bbcd4a
("backport-create-issue: read redmine key from file").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37192 from rhcs-dashboard/fix-46898-master

mgr/dashboard/api: move/create OSD histogram in separate endpoint

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37303 from tchaikov/wip-crimson-osd

crimson: do not set_value() twice and fix Message leak

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/mon: move handle_auth_done() to "v1 and v2" section

crimson::mon::Connection::handle_auth_done() is shared by both v1 and v2
authentication implementation, so move it to "v1 and v2" section

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37326 from tchaikov/wip-47324

test/crimson: better error message when bind fails

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #37323 from tchaikov/wip-47593

crimson/osd: add_recovery(oid) before recover_object(oid)

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

test/crimson: better error message when bind fails

See also: https://tracker.ceph.com/issues/47324

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: more constness

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add_recovery(oid) before recover_object(oid)

in 7736bddc534e5df2daf704c4bbc92a0927b0db4d, we assumed that the object
to be recovered did not exist in `recovering` before
`recover_object(oid)` was called. but this turns out not true. so, in
this change, `add_object(oid)` is called before `recover_object(oid)`
gets called.

Fixes: https://tracker.ceph.com/issues/47593
Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use recoverying.at() when appropriate

this helps to avoid the confusion of "where the recovery is added" and
"are we adding a new instance of recovery here".

should call add_recovery() explicitly when we need to add a new recovery
instance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add RecoveryBackend:add_recovering()

before this change, get_recovery() can also be used for adding a
recovery instance to `recovering`. this behavior is error-prone and
confusing.

after this change, add_recovery() is used in the place where we
want to add a new instance of recovery instance.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37290 from tchaikov/wip-run-make-check

run-make-check.sh: fix syntax error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37106 from sebastian-philipp/cephadm-overwritable-templates

mgr/cephadm: make j2 teamplates overwritable

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #37121 from Songweibin/wip-rbd-metadata

librbd: async version of metadata_set/remove

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37083 from orozery/librbd-data-cryptor

librbd: add data encryptor/decryptor using openssl

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37110 from CongMinYin/qa_immutable_object_cache_check

qa: check immutable object cache content

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37237 from MahatiC/wip-fix-rwl-path

librbd/cache: Fix pwl_path read issue

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37206 from bk201/wip-47387

mgr/rook: fix listing daemons

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Varsha Rao <varao@redhat.com>

client: refresh rstat for quota check of rename

Fixes: https://tracker.ceph.com/issues/47224
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #37043 from s0nea/wip-dashboard-47331

mgr/dashboard: log in non-admin users successfully if the telemetry notification is shown

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #37293 from changchengx/osd_comment

osd: fix wrong comment about setting auth_client for messenger

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not add ref when constructing MessageRef

crimson::osd::PG::send_cluster_message() accepts a `Message*`
pointer, and then hand it over to `shard_services.send_to_osd()`,
which expects a `Ref<Message>`. so the raw pointer is used to
construct an `intrusive_ptr<Message>`, which increment the
refcount of that Message instance by one. but that Message
was owned by nobody before that, so we end up with an
`intrusive_ptr<Message>` of 2 refcount, and only a single
owner. hence the memory leak.

in this change, instructs the constructor to not add the refcount.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: do not set_value() again

once the continuation consuming the stored value of the associated
future, we cannot set_value() again. otherwise, ASan complains that we
are accessing the memory on heap after it is freed.

in this change, std::optional<> is used for holding
promise<auth_result_t>, once the promise is fulfilled, `auth_done` is
reset to prevent another call of `set_value()` or `set_exception()`.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: only finish auth when auth is completed

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: s/AuthResult/auth_result_t/

auth_result_t is merely an enum, so make it like so.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/mon: extract mon::Client::_finish_auth()

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd: correct comment about setting auth_client for messenger

client_messenger's auth_client has not been set until call
monc->init();

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37152 from rhcs-dashboard/bottom-links-component

mgr/dashboard: Creating a new Login Page Legal Links Component

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37291 from badone/wip-docker-test-helper-fixes

test/docker-test: helper fixes and support more distro releases

Reviewed-by: Kefu Chai <kchai@redhat.com>

test: Add support for fedora 32, 33 and Ubuntu 20.04

Enable creation and use of these OS images in docker-test.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

tests: Make sure install-deps is run noninteractively

This gets past things like tzconfig stopping for user input.
Remove redundant install of python-virtualenv.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #37200 from tchaikov/wip-crimson-with-lock

crimson/common: add helpers for seastar::with_lock()

Reviewed-by: Samuel Just <sjust@redhat.com>

run-make-check.sh: fix syntax error

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37211 from tchaikov/wip-zbd

rpm,install-deps,run-make-check: enable WITH_ZBD build on demand

Reviewed-by: Neha Ojha <nojha@redhat.com>

test/centos-8: Install git before running install-deps

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

test/docker-test: Fix permissions issue when using podman

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge PR #36776 into master

* refs/pull/36776/head:
systemd: Support Graceful Reboot for AIO Node

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge PR #37227 into master

* refs/pull/37227/head:
qa/cephfs: don't pass args to destroy() in recreate()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37233 into master

* refs/pull/37233/head:
qa/mgr: revert a patch from commit 04ed58f

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37271 from cyx1231st/wip-seastore-fix-non-repeatable-read

crimson/seastore: fix potential non-repeatable-read from RootBlock

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #36927 from idryomov/wip-krbd-noudev

krbd: optionally skip waiting for udev events

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #37213 into master

* refs/pull/37213/head:
mgr/rook: Pass pod namespace to list_namespaced_pod()

Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #37101 from LenzGr/master-documentation

doc: Updated `HACKING.rst` and `README.rst`

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37137 from tspmelo/wip-fix-iscsi-tests

mgr/dashboard: Fix iSCSI backend unit-test

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #37262 from trociny/wip-rbd-nbd-quiesce-hook

rbd-nbd: fix typo in mini help

Reviewed-by: Jason Dillaman <dillaman@rehdat.com>

Merge pull request #35785 from rhcs-dashboard/wip-45957-consolidate_Osd_Endpoints

mgr/dashboard: Consolidate Osd mark endpoints

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

qa: add test for mapping and unmapping from a network namespace

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37234 from guits/guits-quick-fix

ceph-volume: fix wrong type passed in terminal.warning()

Merge pull request #37222 from dillaman/wip-librbd-image-dispatch

librbd: bug fixes and cleanup for IO dispatch path

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37215 from uweigand/fix-librados-test-endian

test/librados: fix endian bugs in checksum test cases

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #36900 from wjwithagen/wjw-enhance-mgr_module.py

mgr/dashboard: Report the missing path in error message

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37087 from tspmelo/wip-iscsi-logged-in

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #37261 from tchaikov/wip-47552

common/BackTrace: do not use len for length of demangled symbol

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #37185 from david-z/wip-fix-osdmaptool

tools/osdmaptool.cc: fix inaccurate pg map result when simulating osd out

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37210 from changchengx/no_tune_message

messenger: remove unused variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37183 from rhcs-dashboard/fix-47434-master

mgr/dashboard: table detail rows overflow

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

ceph-volume: fix wrong type passed in terminal.warning()

`terminal.warning()` excepts a `str`.
Passing `e` means we pass a type `exceptions.RuntimeError`

Changing to `terminal.warning(e.message)` fixes the issue.

Closes: https://bugzilla.redhat.com/show_bug.cgi?id=1877672
Resolves: rhbz#1877672

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #37268 from anthonyeleven/anthonyeleven/doc-fixes

doc/man: Add optional reweight-by-utilization args

Reviewed-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/man: Add optional reweight-by-utilization args
doc/mgr: Grammar and wording for Prometheus labels
doc/rados: Spelling and clarity

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

crimson/seastore: fix potential non-repeatable-read from RootBlock

Load root block into the transaction when read it.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

librbd: async version of metadata_set/remove

In iSCSI scenario (via tcmu-runner), after a client has acquired exclusive
lock on an image, users will no longer be able to set metadata on the image.
This commit try to fix it by forward the request to the lock owner.

Steps to reproduce:

1. Client A:
```
>>> import rbd, rados;client=rados.Rados(conffile='');client.connect();ioctx=client.open_ioctx('rbd');rbd_inst=rbd.RBD();image=rbd.Image(ioctx, 'img1')
>>> from rbd import RBD_LOCK_MODE_EXCLUSIVE
>>> image.lock_acquire(RBD_LOCK_MODE_EXCLUSIVE)
>>>
```

2. Client B:
```
$ rbd image-meta set img1 conf_rbd_qos_iops_limit 10000
2020-09-12T15:19:58.325+0800 7f161affd700 -1 librbd::ManagedLock: 0x7f15f4001d48 handle_acquire_lock: failed to acquire exclusive lock:(30) Read-only file system
failed to set metadata conf_rbd_qos_iops_limit of image : (30) Read-only file system
rbd: setting metadata failed: (30) Read-only file system
$
```

Signed-off-by: songweibin <song.weibin@zte.com.cn>

Merge pull request #37176 from badone/wip-enable-mgr-client-debug

qa: Enable debug_client for mgr tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

messenger: remove unused variable

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

librbd: add data encryptor/decryptor using openssl

This commit adds the first CryptoInterface implementation
which translates bufferlist encryption to the standard api of crypto libraries,
i.e. using block ciphers operating on C-style arrays.

It defines an abstract DataCryptor interface to allow different crypto libraries implementations.
With this commit, we add an OpenSSL based implementation.
Lastly, we implement a context-pool wrapper implementation to optimize performance.

Signed-off-by: Or Ozeri <oro@il.ibm.com>

qa: check immutable object cache content

fix check immutable cache does not work

Signed-off-by: Yin Congmin <congmin.yin@intel.com>

rbd-nbd: fix typo in mini help

Signed-off-by: Mykola Golub <mgolub@suse.com>

common/BackTrace: do not use len for length of demangled symbol

it turns out `len` is longer than the length of demangled symbol,
let's rely on the `\0` sentry in the returned char* string instead.

in this change, use `status` to tell if the demangle is successful or
not.

Fixes: https://tracker.ceph.com/issues/47552
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37072 from idryomov/wip-kcephfs-blacklisted-string

mds: add " (blacklisted)" to session reject error string

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/osd: extract PG::load_head_obc() out

so it can be reused in later refactory

Signed-off-by: Kefu Chai <kchai@redhat.com>