blk: use choose_fd for all filehandle references

This is a leading part of the changes to implement bluestore on
FreeBSD. Without this invalid indexing in the descriptor arrays
will occur.

Creates a new enum: `blk_access_mode_t`
to describe `BUFFERED` and `DIRECT `mode access
with `choose_fd()` to get the correct file for the typed access
and adds
a pretty-printer
boolean convertor `blk_access_mode_t::buffermode(bool)`

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #41977 from rzarzynski/wip-crimson-common-print-more-on-crash

crimson/common: dump more on faults

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41838 from p-se/grafana-clean-up

monitoring: Clean up Grafana dashboards

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: jan--f <NOT@FOUND>
Reviewed-by: p-se <NOT@FOUND>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>

Merge pull request #41721 from aaryanporwal/telemetry-ident-fix

mgr/dashboard: telemetry activate: show ident fields when checked

Reviewed-by: aaryanporwal <NOT@FOUND>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #41991 from dang/wip-dang-bucket-delete

RGW - Bucket Remove Op: Pass in user

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41993 from ronen-fr/wip-ronenf-50346

osd/scrub: replace a ceph_assert() with a test

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42024 from rzarzynski/wip-crimson-load_obc_nocpy

crimson/osd: don't extra copy hobject in PG::load_head_obc().

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: don't extra copy hobject in PG::load_head_obc().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41615 from tchaikov/wip-avl-alloc-ff

os/bluestore/AvlAllocator: introduce bluestore_avl_alloc_ff_max_* options

Reviewed-by: Igor Fedotov <ifedotov@suse,com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38939 from ronen-fr/wip-ronenf-scrub-blocked

osd: issue a warning if the scrubber blocks for too long on an object

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #40850 from varshar16/wip-vstart-support-cephadm-rgw

src/vstart: deploy rgw service with cephadm and create rgw user with system flag

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42020 from athanatos/sjust/wip-cache-assert

crimson/os/seastore: transaction conflict handling improvements

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42003 from cyx1231st/wip-seastore-fix-onode-tree

crimson/onode-staged-tree: fix ref-counter assert failures

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: dump entire siginfo on segmentation fault.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: FatalSignal::signaled() takes siginfo by a reference.

There is no point in having the distincted `nullptr` value.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/common: dump /proc/self/maps on crash.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41889 from ChenFanTony/mkfs_wait_complete

osd/OSD: mkfs need wait for transcation completely finish

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: reset root node after lookup

Otherwise there could be unexpected references that will break the
asserts when remove nodes during insert/delete.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add missing mutable keyword

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #42004 from tchaikov/wip-crimson-osd-fsm

crimson/osd: shutdown if osdmap forces us to do so

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

seastore/.../staged_fltree/node: check for conflict in Node::load

This will be unnecessary once converted to interruptible_future.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree/lba_btree_node_impl: add debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

seastore/.../node_extent_manager/seastore: detect transaction conflicts in read_extent

This won't be necessary once converted to interruptible_future.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: mark conflict in get_extent

After wait_io, the extent may have been mutated again, so it may be
invalid.  Check in the caller and mark the transaction conflicted as
needed.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transasction: expose is_conflicted

Useful for components not yet converted to use interruptible_future.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #41963 from athanatos/sjust/wip-interruptible-tm

crimson/os/seastore: refactor transaction_manager and below to use interruptible_future

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson/test_interruptible_future: disable handle_error

Seems to cause a linker hang with gcc-9 in bionic.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: pass t by ref to submit_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39934 from Jeegn-Chen/wip-tracker-49128

rgw: write meta of a MP part to a correct pool

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41739 from liewegas/rgw-realm-metadata

radosgw: include realm_{id,name} in service map

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

RGW - Bucket Remove Op: Pass in user

When a bucket remove op is called on the non-master zone, the op is
forwarded to the master zone, but this needs a user, so pass the user
in.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #41994 from anthonyeleven/anthonyeleven/adjust-rados-operations-pools

doc/rados/operations: Update pools.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #42005 from trociny/wip-51342

test/librbd: use really invalid domain

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

test/librbd: use really invalid domain

in TestMockMigrationHttpClient.OpenResolveFail

Fixes: https://tracker.ceph.com/issues/51342
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #41828 from tchaikov/wip-btree-alloc

os/bluestore: add BtreeAllocator

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

crimson/osd: document fsm of crimson osd

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark more OSD methods private

they are internal helpers, not part of the public interface of the OSD
class.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: shutdown if osdmap forces us to do so

mirror the change introduced by 5dbae13ce0f5b0104ab43e0ccfe94f832d0e1268
in classic osd.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use discard_result() in stop()

we don't care about the result of shutdown() of messengers, when
shutting down the daemon actually, and we don't handle the failures.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #41935 into master

* refs/pull/41935/head:
mds: avoid journaling overhead for ceph.dir.subvolume for no-op case

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore: convert transaction_manager internally to use interruptible_future

Consumers of TransactionManager use wrapper classes InterruptedTransactionManager
and InterruptedTMRef for now until we convert them.

Also converts users of InterruptedCache etc and removes.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_seastore_cache: use cache directly

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: convert to use interruptible_future

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: convert to use interruptible future

Introduces InterruptedCache wrapper for now for components not yet
converted.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: introduce TransactionConflictCondition interruptor

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache.h: remove unused get_extents

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: invalidate transaction referencing invalid extents

Modify read_set to retain a reverse mapping from extents back to
transactions and use it to update Transaction::conflicted upon
invalidation.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/test_interruptible_future: add tests for errorated behavior

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: add interruptor::base_ertr

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: add safe_then_interruptible for multiple error handlers

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: refactor handle_interruption

handle_interruption can't really be validly used outside of
with_interruption_cond.  Make private, and adjust is_interruption
to not require an instance.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: remove enable/disable_interruption

with_interruption_cond needs to check the condition on the way in.
call_with_interruption_impl already has the required machinery, so
let's just use it and dispense with the other helpers.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: remove unnecessary make_ready_future template

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: add ready|exception_future_marker constructors

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: introduce future<> helper to interruptor

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: introduce si_then as shorthand for safe_then_interruptible

safe_then_interruptible is too long for common use within seastore.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: introduce with_interruption_to_error

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: add handle_interruption

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: add futurize::invoke

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/interruptible_future: add common errorator forwards

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/errorator: add futurize::apply

Signed-off-by: Samuel Just <sjust@redhat.com>

common/interruptible_future: use errorated future as core_type, fix constructor

No reason really to remember the underlying seastar::future type, we should
only be interacting with the errorated future wrapped type.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/test_interruptible_future: using namespace crimson

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add reference type for do_for_each implementations

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: fix typo in comment

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: rename retire_extent_addr for addr overload

Makes InterruptibleCache bit in the later patch simpler, and is somewhat
clearer.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: make complete_transaction void

This really can't result in mutations (the transaction already committed!)
and presently doesn't require any IO at all.  Just make it void.

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/rados/operations: Updates pools.rst

Add clarity, change example PG counts to a power of two.

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

osd/scrub: replace a ceph_assert() with a test

We are using two distinct conditions to decide whether a candidate PG is already being scrubbed. The OSD checks pgs_scrub_active(), while the PG asserts on the value of PG_STATE_FLAG.
There is a time window when PG_STATE_FLAG is set but is_scrub_active() wasn't yet set. is_reserving() covers most of that period, but the ceph_assert is just before the is_reserving check.

fixes: https://tracker.ceph.com/issues/50346
Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #41990 from liewegas/fix-51292

qa/suites/rados/dashboard: fix e2e test

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/suites/rados/dashboard: fix e2e test

Move roles into task yaml.  Rename e2e.

Fixes: https://tracker.ceph.com/issues/51292
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #39358 from Jeegn-Chen/wip-tracker-49206

rgw: avoid infinite loop when deleting a bucket

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #40031 from pritha-srivastava/wip-sts-get-session-ldap

rgw/sts: fixes getsessiontoken authenticated with LDAP,

Merge pull request #41984 from tchaikov/wip-crimson-repeat

crimson: s/crimson::do_until/crimson::repeat/

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41974 from rhcs-dashboard/fix-POST-alerts

mgr/dashboard: Alertmanager fails to POST alerts

crimson: s/crimson::do_until/crimson::repeat/

seastar::do_until() takes a predicate functor and a continuation. while
seastar::repeat() takes a single continuation which returns
stop_iteration::yes or stop_iteration::no. in general, we want to mirror
and extend the facilities offered by seastar instead of changing it in
an unexpected way. while crimson::do_until() only take a single
continuation which returns a bool. in hope to be more consistent, in
this change, all occurances of crimson::do_until are replaced with
crimson::repeat.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41986 from tchaikov/wip-wip-crimson-alienstore-lockless-spin-off

crimson/os/alienstore: cleanups of ThreadPool

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: replace private: section at end of class

to comply to the style guide at https://google.github.io/styleguide/cppguide.html#Declaration_Order

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: replace tab with spaces

to be consistent with the editor variables defined in the header.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: reduce the scope of lock

we just need to synchronize accesses to `stopping`, so only need to
guard the mutation of it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41956 from yuvalif/wip-yuval-fix-51293

rgw/notification: make notifications agnostic of bucket reshard

Merge pull request #41955 from yuvalif/wip-yuval-fix-51305

rgw/notification: send correct size in COPY events

crimson/common/errorator: consider return value as ready future in maybe_handle_error_t

this behavior mirrors seasetar::futurize::apply(), where non-future and
non-void return values are converted to future<>, and returned instead.
this change could simplify some use cases where we always return an
immediately available future.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41831 from yuvalif/wip-yuval-fix-51114

rgw/amqp: remove the explicit "disconnect()" interface

Merge pull request #41902 from idryomov/wip-rbd-thick-conf

rbd: don't disable rbd_discard_on_zeroed_write_same

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #41940 from liu-chunmei/crimson-fix-reactor-stall

crimson: fix reactor stalled and rbd_open failed.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41964 into master

* refs/pull/41964/head:
doc/cephfs/nfs: update recommendation for versions

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41908 into master

* refs/pull/41908/head:
qa: always format the pgid in hex

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41875 into master

* refs/pull/41875/head:
client: check if a mds rank is `up` before fetching connection addr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41385 into master

* refs/pull/41385/head:
mon/FSCommands: add command to rename a file system
qa/cephfs: split test_admin.TestAdminCommands
mds: remove 'fs_name' from MDSRank

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: avoid journaling overhead for ceph.dir.subvolume for no-op case

In preparation for acquiring the xlock on the directory inode, the MDS
must journal a few events before continuing on with the setvxattr. This
can cause significant delays in the volumes ceph-mgr module which needs
to regularly enable this vxattr from multiple code paths. We could cache
in that module whether the vxattr is set but it's also pretty easy to
adjust the MDS to acquire a rdlock on the directory to check if the
subvolume flag is already set. That is much lighter weight and the lock
is generally readily available.

Fixes: https://tracker.ceph.com/issues/51276
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/seastore: fix reactor stalled and rbd_open failed

omap manger use keys' reference, so the keys' string set should
be kept alive during omap operation. use value capture instead of
reference capture.

otherwise, during omap operation, the keys contents will be changed
and cause logger insert_iterator very long time to stall reactor.
omap_get_value also failed, rbd_open failed.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #41967 from sandrobonazzola/patch-1

doc/install/get-packages: point to current stable release

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/install/get-packages: point to current stable release

Point to pacific for downloading the cephadm script

Co-authored-by: Kefu Chai <tchaikov@gmail.com>
Signed-off-by: Sandro Bonazzola <sbonazzo@redhat.com>

Merge pull request #41960 from rzarzynski/wip-crimson-alienstore-sync-shardedwq

crimson/os: synchronize producers with consumers in AlienStore's queues.

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Alertmanager fails to POST alerts

Fixes: https://tracker.ceph.com/issues/51312
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #41957 from liewegas/fix-vstart-registry-url

vstart.sh: fix docker url

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41961 from rzarzynski/wip-crimson-intcltreq-fix-assert

crimson/osd: fix construction of InternalClientRequest in DEBUG builds.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41962 from rzarzynski/wip-crimson-intcltreq-more-asserts

crimson/osd: introduce more asserts to the Watch timeout handling.

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephfs/nfs: update recommendation for versions

... of Ceph and NFS-Ganesha packages.

Signed-off-by: Ramana Raja <rraja@redhat.com>