doc/releases/octopus: use fixed-width font for "ceph fs dump"

We are using fixed-width font to render commands elsewhere, so
let's do so in this release note, too - just for the sake of uniformity.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

PendingReleaseNotes: drop mention of CVE fix

CVEs fixed before the first stable Pacific release need not be mentioned
explicitly in the release notes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

PendingReleaseNotes: add missing ">= 16.0.0" heading

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/octopus: add "default bucket index shards" release note

The RGW "default number of bucket index shards for new buckets was raised from
1 to 11" release note was added to PendingReleaseNotes by
1ec77ec7ba13da68d7919176b3675513ed7d7c21 but it did not find its way into the
official v15.2.0 release notes until now.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/octopus: add "BlockPublicAccess" release note

The RGW BlockPublicAccess release note was added to PendingReleaseNotes
by df384ea95f88196a9723877a5aa9041ef25b5f29 but it did not find its way
into the official v15.2.0 release notes until now.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/octopus: add "rgw bucket granularity" release note

The RGW bucket granularity release note was added to PendingReleaseNotes
by 50f13fd54b709bd0b8ad25e818fc61e2c55828fd but it never found its way
into the official v15.2.0 release notes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/octopus: enforce mds_join_fs cluster affinity

The "enforce mds_join_fs cluster affinity" release note was
added to PendingReleaseNotes by 3fac3b1236c4918e9640e38fe7f5f59efc0a23b9
but it never made its way into the official v15.2.0 release notes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/octopus: osd_calc_pg_upmaps_max_stddev removal

The release note about osd_calc_pg_upmaps_max_stddev removal
was added to PendingReleaseNotes by
65d03bae8b4f50cc3cbaa50640eaeab4cabd711f but never found its
way into the official v15.2.0 release notes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/octopus: add bucket notification JSON array

The release note about bucket notifications being encapsulated in a JSON
array was added to PendingReleaseNotes by
19832a0daeb7b23006d1c38059ad14f0c239d32a but it never found its way into
the actual v15.2.0 release notes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/octopus.rst: add nvme self-classify release note

The release note about OSDs self-classifying themselves as "nvme" in
certain rare cases was added to PendingReleaseNotes by
1d0fd5603e9fd74d54d41b7c4895f9e41c510c29 but it never found its way into
the official v15.2.0 release notes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/octopus.rst: add "-o" behavior release note

The release note about restoring the original behavior of the rados
tool's "-o" option was added to PendingReleaseNotes by
d2b806070de5f56474283035a7ffc91fe96afbab but it never found its way into
the official v15.2.0 release notes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/octopus.rst: include MDS cache memory limit note

This note was added to PendingReleaseNotes by
0efa73f0296276aadc9d20e58120c05a3f47ae03 but never found its way into
the official v15.2.0 release notes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

doc/releases/octopus.rst: add missing target_size_ratio note

This note was added to PendingReleaseNotes by 9118065c4c3803f8a2c26c28c69ffc9f14ff260a
but it apparently never found its way into doc/releases/octopus.rst.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #34980 from sebastian-philipp/cephadm-workunits-rm-ubuntu_2020

qa/cephadm: rm ubuntu_latest for now

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35101 from sebastian-philipp/cephadm-ls-systemd-unit

cephadm: `ls`: also return systemd_unit

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35074 from dillaman/wip-45544

librbd: avoid completing mirror:DisableRequest while holding its lock

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #35062 into master

* refs/pull/35062/head:
qa/tasks/cephfs/fuse_mount.py: retry when the admin socket is not ready

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #34571 into master

* refs/pull/34571/head:
mds: don't shallow copy when decoding xattr map

Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #34144 into master

* refs/pull/34144/head:
Filer: Add error handler in _do_purge_range()
mds: Handle blacklisted error in purge queue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #34596 into master

* refs/pull/34596/head:
test: add a new program for testing ino_release_cb
client: add a new inode release request callback
client: expose ceph_ll_register_callbacks via libcephfs
client: move callback typedefs and arg struct into ceph_ll_client.h
client: rename ceph_statx.h to ceph_ll_client.h
client: make client_dentry_callback_t more friendly for C
client: only override umask_cb with non-NULL values

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #34890 into master

* refs/pull/34890/head:
cephfs-shell: Remove get_error_code() call on OSError type exception objects

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #34951 into master

* refs/pull/34951/head:
qa/cephfs: run() cleanup whether FS was mounted or not

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #35058 into master

* refs/pull/35058/head:
qa/tasks: logger.warn() -> logger.warning()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35108 from trociny/wip-45589

mgr/rbd_support: rename "rbd_trash_trash_purge_schedule" oid

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34900 from gekios/wip-cephadm-bootstrapped

qa/tasks/cephadm.py: Support already bootstrapped clusters

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35071 from sebastian-philipp/cephadm-upgrade-check-mon-command

mgr/cephadm: upgrade: use check_mon_command

Merge pull request #35068 from bk201/wip-45163

mgr/cephadm: config iSCSI gateways in Dashboard

Merge pull request #35084 from jschmid1/caching_osdspec_previews

mgr/cephadm: implement cached osdspec previews

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rbd_support: rename "rbd_trash_trash_purge_schedule" oid

to "rbd_trash_purge_schedule"

Fixes: https://tracker.ceph.com/issues/45589
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge branch 'wip-joshd-fix'

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/orchestrator: adapting orch interface to preview changes

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #35015 from smithfarm/wip-45514

doc: drop obsolete cache tier options

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35083 from p-se/wip-pse-cephadm-prom-scrape-interval

mgr/cephadm: increase global scrape interval

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35092 from varshar16/wip-cephadm-mdcache-ganesha

 mgr/cephadm/nfs: Instead of CACHEINODE use MDCACHE

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: avoid completing mirror:DisableRequest while holding its lock

Ensure that the lock is released before another thread has the chance
to complete the state machine and attempt to destruct the in-use lock.

Fixes: https://tracker.ceph.com/issues/45544
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: upgrade: use check_mon_command

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34964 from dillaman/wip-45137

librbd: replace ImageRequestWQ with pluggable ImageDispatcher

Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/cephadm: Synopsis: add `cephadm unit`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephadm: `ls`: also return systemd_unit

Let users directly know the systemd unit name.

Enable things like

```
cephadm ls | jq -r '.[].systemd_unit' | xargs -n 1 systemctl status | cat
```

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: increase global scrape interval

Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

mgr/dashboard: support setting iSCSI gateways from orchestrator

- Dashboard doesn't actively query iSCSI gateways from orchestrator anymore.
  Instead, Orchestrator backend invokes iscsi-gateway-add CLI after any
  iSCSI daemons are deployed.
- Add an optional argument `name` to `iscsi-gateway-add` command. If
  `name` is provided, Dashboard won't issue API request to iSCSI gateway
  to get the hostname. This is required because when orchestrator sets
  the gateway via CLI, the gateway is just spawned and might be not
  ready yet.

Fixes: https://tracker.ceph.com/issues/45163
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/cephadm: config iSCSI gateways in Dashboard

- Add `daemon_check_post` fuction to allow services implement actions
  needed to be done after daemons are checked.
- After iSCSI daemons are spawned, call dashboard CLI to
  set the API URL of the daemons.

Fixes: https://tracker.ceph.com/issues/45163
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #32682 from lorenzen-b1/patch-1

doc/architecture.rst: clarify scrubbing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35091 from tchaikov/wip-crimson-cleanup

crimson/osd: avoid using variadic future

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #35081 into master

* refs/pull/35081/head:
qa: do not save/restore ip_forward sysctl

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34764 from ianwatsonrh/master

Docs: vault kv and transit engine updates

Reviewed-by: Casey Bodley <cbodley@redhat.com>

 mgr/cephadm/nfs: Instead of CACHEINODE use MDCACHE

As CACHEINODE will be deprecated soon.

Signed-off-by: Varsha Rao <varao@redhat.com>

crimson/osd: avoid using variadic future

it is deprecated by seastar. let's use future<tuple<...>> instead

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35025 from dillaman/wip-45518

librbd: copy API should not inherit v1 image format by default

Merge pull request #34789 from trociny/wip-rbd-nbd-quiesce

rbd-nbd: add quiesce/unquiesce hooks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35023 from yuvalif/fix_issue_45513

cls/2pc_queue: fix timing issue in test

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34959 from yison/rwl-image-writeback-cache-seq10

rbd/cache: Replicated Write Log core codes - load existing cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34930 from trociny/wip-45409

rbd-mirror: wait for events to replay before shut down journal replay

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35042 from tchaikov/wip-cmake-remove-iwyu-cppcheck

cmake: remove cppcheck and iwyu targets

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #35070 from Devp00l/wip-45545-2

mgr/dashboard: Updates jest to fix cache failure

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34913 from athanatos/sjust/wip-seastore-journal

crimson/os/seastore: initial PR

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: move iSCSI service class to its own file

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/cephadm: adapt cephadm to osdspec preview caching

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: add preview caching to inventory.py

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #35052 from tspmelo/wip-unittest-warnings

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35013 from matthewoliver/cephadm-iscsi-log-to-stderr

cephadm: enable logging to stderr in ceph-iscsi

Reviewed-by: Michael Fritch <mfritch@suse.com>

dec/dev/seastore.rst: add more detailed design information and status

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add initial cache support with tests

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: introduce initial journal implementation and tests

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/gtest_seastar: add helpers for building seastar gtest unit tests

Also adds unsafe_get methods for errorator for use within unit tests.

Signed-off-by: Samuel Just <sjust@redhat.com>

qa: do not save/restore ip_forward sysctl

We don't really care if this is left on during the course of the mount.
These settings don't persist across tests anyway.

Fixes: https://tracker.ceph.com/issues/45525
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/cephfs/fuse_mount.py: retry when the admin socket is not ready

Fixes: https://tracker.ceph.com/issues/45552
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #35077 from yuriw/wip-yuriw-crontab-master

qa/tests: reduced powercycle jobs amount

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: reduced powercycle jobs amount

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #32829 from Devp00l/wip-43765

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35076 from yuriw/wip-yuriw-distro-master

qa/tests: run `nautilus-x` on `pacific` on `ubuntu 18.04`

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: run `nautilus-x` on `pacific` on `ubuntu 18.04`

...as we don't build `nauiltus` on `20.04`

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

librbd: QoS delayed dispatch needs to use op work queue

The QoS throttle will invoke its throttle complete callback within
the scope of the ImageCtx timer lock singleton. This would cause
the potential for deadlock if a future dispatch layer attempted
to acquire the same timer lock.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: corrected naming of singleton timer locker

It incorrectly included "Journal" in the name.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: correct misnamed parameters in io::ImageDispatcher

It was incorrectly using to object dispatch names instead of
image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: api::Io now properly passing the parent trace information

A copy-and-paste error from moving the original implentation from
ImageRequestWQ resulted in the parent trace being incorrectly
uninitialized.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32829 from Devp00l/wip-43765

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

librbd: ensure in-flight read-ahead + copy-on-read ops are finalized

Now that the image dispatcher isn't flushing AsyncOperations with each
flush request, we need to ensure that we flush all background ops prior
to fully closing the image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: remove io::ImageRequestWQ

This has been fully deprecated in favor of the new plugable image
dispatcher system.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch to new api::Io dispatch helper methods

This is another step in the removal of ImageRequestWQ in favor
of the new ImageDispatcher.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: created new api::Io helper methods

These will be invoked by the user API and wrap various user API
specific logic.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: exclusive-lock dispatch flush pseudo-handling

Ensure that the exclusive-lock layer tracks all in-flight IOs so that
it can flush the IOs from lower layers when the lock becomes required.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: qos dispatch flush pseudo-handling

Flush requests will be properly queued behind write operations to
ensure data consistency.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: generic helper for tracking in-flight IO and managing flush requests

Layers that potentially queue IOs but not all IOs will need to track all
in-flight IOs to properly ensure that a flush cannot complete while
older IO is still stuck in a queue.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: temporarily redirect ImageRequestWQ IO through ImageDispatcher

The next series of commits will work to remove ImageRequestWQ from librbd.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: complete ImageDispatchSpec upon AioCompletion finalization

This will allow ImageDispatchSpec to invoke any necessary finalizers
for the different dispatch layers and allows in-flight IO to be tracked
since the image dispatch layer finalizers won't be invoked until after
the AioCompletion fires.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switched IO (un)block_write calls to new queue image dispatcher

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: queue image IO dispatch layer

This layer replaces the queue implemented by ImageRequestWQ and handles
blocking all write requests through the dispatch pipeline.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: exclusive lock image IO dispatch layer

This layer will handle acquiring the exclusive lock if required upon
incoming IO requests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: refresh image IO dispatch layer

This layer will handle refreshing the image if required upon incoming
IO requests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: QoS image io dispatch layer

Re-implement the existing QoS throttling behavior from ImageRequestWQ
as a new image IO dispatch layer. The existing QoS hooks in
ImageRequestWQ are temporarily hooked to the new layer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support a finish handler for image IO dispatch layers

This will be faster than allocating per-IO Context callbacks for
layers that need to track the in-flight state of particular IOs.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: init and shut down image IO dispatch layer

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: initial implementation for image IO dispatcher

This is similar to the object IO dispatcher but it expects image extents
for IO operations.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: add image dispatch layer to image dispatch spec

This will allow image extent-based IOs to skip layers. Also added helpful
enums for the start of user-based API requests and internal requests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: added io::ImageDispatchSpec::C_Dispatcher

This will assist with dispatching image extent-based IO requests
through the image IO dispatcher system (to be added in a later commit).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: move generic IO dispatcher logic to base class

This will allow re-use between the existing ObjectDispatcher and
the ImageDispatcher that will be added in a future commit.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: TokenBucketThrottle should use perfect forwarding for item

This will allow more than just a pointer to an item to be passed to
the callback.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Updates jest to fix cache failure

Fixes: https://tracker.ceph.com/issues/45545
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #31889 from ofriedma/wip-ofriedma-reshard-debug

rgw: Fix reshard log so we can know why the reshard is returning error