Merge pull request #37030 from Amrita42/wip-doc-Amy-radosgw

doc: Bucket notifications in the radosgateway-the usage of 'if not'

Merge pull request #37312 from votdev/issue_47578_clipboard_firefox

mgr/dashboard: Copy to clipboard does not work in Firefox

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37411 from p-se/fix-cephadm-custom-template-escaping

mgr/cephadm: do not escape HTML for custom templates

Merge pull request #36622 from dmick/wip-46665

mgr/cephadm: continue trying all hosts for service deployment

Merge pull request #37112 from sebastian-philipp/cephadm-services-types-mandatory

mgr/cephadm: Make type annotations for cephadm.services mandatory

Merge pull request #37397 from adk3798/mirror

mgr/cephadm: get rbd-mirror daemon-id when checking for strays

Merge pull request #37432 from sebastian-philipp/cephadm-upgrade-state-none

mgr/cephadm: upgrade_state can be literally "null"

Merge pull request #36392 from Devp00l/wip-46375

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/cephadm: upgrade_state can be literally "null"

Fixes: https://tracker.ceph.com/issues/47580
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Speed improvements of pool details

Now all variables that are shown in a listing that trigger a
more complex render cycle will only be updated if pool properties have
changed and not only the time series data, which isn't shown anymore,
as it can be seen graphically enhanced in the pool listing.

The pool detail component now uses the `onPush` change detection
strategy, to only call `ngOnChanges` if one of the input variables
have changed.

The function that only updates variables if they have changed is
now available through a helper class in order to provide this
useful functionality through out the dashboard.

Fixes: https://tracker.ceph.com/issues/46375
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #37286 from dillaman/wip-46822

osdc/ObjectCacher: overwrite might cause stray read request callbacks

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37074 from dillaman/wip-librbd-list-snaps-api

librbd: generic internal API for computing snapshot deltas

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37281 from tchaikov/wip-fmt

rpm: add fmt-devel as build deps

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37417 from Songweibin/wip-rbd-metadata-update

librbd: should not permit update image metadata via snapshot

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37429 from trociny/wip-47665

rbd-nbd: don't ignore namespace when unmapping by image spec

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37399 from smithfarm/wip-relax-backporting-rule

doc/SubmittingPatches-backports.rst: clarify where to explain

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #37359 into master

* refs/pull/37359/head:
mon: test before adding/removing required_client_features

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36912 into master

* refs/pull/36912/head:
mds: defer encoding and storing the inode backtrace
mds: defer encoding and storing the CDir dentries
mds: add error handler with lock support

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #36730 into master

* refs/pull/36730/head:
client: switch to unique_lock to avoid use the client_lock directly
client: unlock client_lock when copying data to bufferlist
client: do more check for the client_lock
client: simplify the iterating related code to make it more readable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37424 from ronen-fr/rf_neo_comp

neorados: removing an unused captured variable

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36016 from rhcs-dashboard/wip-45863-master

mgr/dashboard/api: generate static API documentation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Courtney Caldwell <ccaldwel@redhat.com>
Reviewed-by: Karen Norteman <knortema@redhat.com>

Merge pull request #37387 from rhcs-dashboard/expose-always-on

mgr/dashboard: cluster > manager modules

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard/api: generate offline API docs

Generate Sphinx documentation from OpenAPI YAML spec:
- Fixed Docs controller doc generation
- Now dashboard Python doesn't fails if dashboard/frontend/dist doesn't exist
- OpenAPI added to @ceph/api CODEOWNERS
- Renamed Ceph-Dashboard API to Ceph REST or RESTful API.
- OpenAPI Docs: fixed decorators and docstrings.
- Sphinx Docs: updated dashboard and dev ones.

Co-authored-by: Ernesto Puerta <epuertat@redhat.com>
Fixes: https://tracker.ceph.com/issues/45863
Signed-off-by: Courtney Caldwell <ccaldwel@redhat.com>
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37328 from badone/wip-npm-cache-config-changes

cmake: Don't try to access users' home directory

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Don't try to access users' home directory

Don't store .npmrc or cache directory in a user's home directory.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

rbd-nbd: don't ignore namespace when unmapping by image spec

Fixes: https://tracker.ceph.com/issues/47665
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37282 from neha-ojha/wip-46323

qa/*/thrash_cache_writeback_proxy_none.yaml: remove cache-mode proxy

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #37077 from wjwithagen/wjw-fix-cls_fifo.cc-__lea64

cls: replace __le{32,64} types by their Ceph counterparts

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37149 from ifed01/wip-ifed-fix-alloc-cmd-multi-reg

os/bluestore: provide a different name for fallback allocator

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #37150 from pdvian/wip-crush-failure-domain

mon: validate crush-failure-domain

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37263 from ifed01/wip-ifed-more-structs-to-mempool

os/bluestore: add missing structs to bluestore_cache_other mempool

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #37153 from tchaikov/wip-46124

blk/kernel: use open file description lock if available

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Niklas Hambüchen <mail@nh2.me>

Merge pull request #37329 from changchengx/add_conn

mon: set MonClient::_add_conn return type to void

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37321 from changchengx/mon_make

cmake: remove undefined source file lists

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: remove undefined source file lists

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

librbd: should not permit update image metadata via snapshot

Fixes: https://github.com/ceph/ceph/pull/37121
Signed-off-by: songweibin <song.weibin@zte.com.cn>

neorados: removing an unused captured variable

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #37388 from johscheuer/add-missing-python-dependency

debian: Add missing Python dependency for ceph-mgr

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37380 from changchengx/ceph_port

msgr: remove the macro not referenced anywhere

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #36779 from athanatos/sjust/wip-dirty-writeout

crimson/seastore: add dirty extent writeout

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson/seastore/test_transaction_manager: add random overwrite test

Also improves replay() by recycling all structures except for the
segment_manager.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add segment_cleaner

Adds SegmentCleaner component to manage rewriting
dirty segments.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add rewrite_extent

Adds support for writing out an extent to a new location and updating
lba mappings.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add more debugging for replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: initialize current_journal_segment in find_replay_segments

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: return next write location from open_for_write

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: replay_delta handle obsolete extents properly

If we see a delta at verison v (v != 0) and we haven't seen version v-1,
we know that the extent is obsolete and can be ignored (presuming that
journal trimming is behaving).

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: pass journal_seq with each replay delta

We'll need this to track where extents became dirty during replay.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: maintain dirty_from value on dirty CachedExtents

Signed-off-by: Samuel Just <sjust@redhat.com>

librbd: list-snaps needs to writeout fancy-striped extents

If the delta is being generated for a fancy-stiped extent such that
the provided object extents are not within the actual snapshot deltas,
we still need to indicate that the object exists (as a whiteout) to
prevent the possibility of an incorrect read request to the parent.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: get rbd-mirror daemon-id when checking for strays

Currently, list_servers() gets the rbd-mirror service-id instead
of the daemon-id so the daemon is marked as stray. This PR uses
that service-id to find the daemon-id and uses that to check if
the daemon is stray.

Fixes: https://tracker.ceph.com/issues/47639
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #37400 from cbodley/wip-rgw-DatalogTrimImplCR-logging

rgw: raise debug level of DatalogTrimImplCR message

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #36482 from cbodley/wip-rgw-bucket-sync-checkpoint

rgw: add 'bucket sync checkpoint' command to radosgw-admin

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

mgr/cephadm: do not escape HTML for custom templates

Fixes: https://tracker.ceph.com/issues/47648
Signed-off-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #37311 from mgariepy/fix-47502

[ceph-volume]: remove unneeded call to get_devices()

client: switch to unique_lock to avoid use the client_lock directly

Fixes: https://tracker.ceph.com/issues/47039
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: unlock client_lock when copying data to bufferlist

It's no need to hold the lock when copying the data, which may
take a long time.

Fixes: https://tracker.ceph.com/issues/47047
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: do more check for the client_lock

Fixes: https://tracker.ceph.com/issues/47039
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: simplify the iterating related code to make it more readable

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge PR #36685 into master

* refs/pull/36685/head:
mds: allow quota for root directory

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #37009 into master

* refs/pull/37009/head:
client: remove the unecessary client_lock for objector->read/write()
client: fix the comment for RWRef

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37266 into master

* refs/pull/37266/head:
qa: ignore logrotate state rename error

Reviewed-by: Rishabh Dave <ridave@redhat.com>

ceph-volume: remove unneeded call to get_devices()

there is no need to probe the device to generate the argparse help
message.

also removing the test on the function as it's not there anymore.

Signed-off-by: Marc Gariepy <gariepy.marc@gmail.com>
Fixes: https://tracker.ceph.com/issues/47502

Merge pull request #37374 from wjwithagen/wjw-fix-crypto-alloca

librbd: Use stdlib.h instead of alloca.h

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

SubmittingPatches-backports.rst: clarify where to explain

The prevailing consensus is that rationales of direct fixes need
not be included in the commit messages. The matter must be
made clear to the reviewers, but it's not important how this is
accomplished.

(Perusers of the git history interested in finding out this information
are free to search for it by linking the commit in question to the PR it
originated from, and possibly from there to the related tracker issues.)

Signed-off-by: Nathan Cutler <ncutler@suse.com>

rgw: raise debug level of DatalogTrimImplCR message

the message was printed at level 0 even on success

Signed-off-by: Casey Bodley <cbodley@redhat.com>

librbd: Use stdlib.h instead of alloca.h

src/librbd/crypto/BlockCrypto.cc:5:10: fatal error: 'alloca.h' file not found
         ^~~~~~~~~~
1 error generated.

Including <stdlib.h> is enough

fixes: https://tracker.ceph.com/issues/47614
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #37385 from idryomov/wip-krbd-read-only-override

rbd: make common options override krbd-specific options

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37384 from tchaikov/wip-crimson-doc

doc/dev/crimson: add instructions on how to build crimson

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

librbd: skip cache initialization if data IoCtx is invalid

This resolves the potential for an assertion failure in the cache
constructors that occurs when the data IoCtx is not valid. Errors
are deferred until actual IOs are issued against the invalid data
pool.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

debian: Add missing Python dependency for ceph-mgr

The file mgr/volumes/fs/operations/pin_util.py imports distutils.util for using
strtobool and thus the python package is required.

Signed-off-by: Johannes M. Scheuermann <joh.scheuer@gmail.com>

mgr/dashboard: cluster > manager modules

Fixes: https://tracker.ceph.com/issues/47484
Signed-off-by: Avan Thakkar <athakkar@redhat.com>

mds: defer encoding and storing the inode backtrace

When encoding the inode backtrace, it may take a bit longer time.
If we hold the mds_lock or other locks, it may cause other threads
to idle wait.

This will queue the encoding and storing work in the mds finisher
to get rid of the locks holded.

Fixes: https://tracker.ceph.com/issues/47148
Signed-off-by: Xiubo Li <xiubli@redhat.com>

mds: defer encoding and storing the CDir dentries

When encoding the dentries infomation to remove or update, it may
take a bit longer time. If we hold the mds_lock or other locks, it
may cause other threads to idle wait.

This will queue the encoding and storing work in the mds finisher
to get rid of the locks holded.

Fixes: https://tracker.ceph.com/issues/47148
Signed-off-by: Xiubo Li <xiubli@redhat.com>

rbd: make common options override krbd-specific options

ceph-csi has added support for passing custom map and unmap options via
mapOptions and unmapOptions storage class parameters.  However, it also
uses --read-only for implementing ROX (ReadOnlyMany) PVs.  If the user
supplies "mapOptions: rw", they will get around the intended read-only
restriction (at least on the block device).

ceph-csi could be patched to use "-o ro", but it actually makes sense
for common options to win over device type-specific equivalents.

Fixes: https://tracker.ceph.com/issues/47625
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #37236 from jschmid1/osd_rm_doc_fix

docs: remove docs should say osd_id instead of svc_id

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/dev/crimson: add instructions on how to build crimson

and add a link to Scylla's document on debugging with GDB

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: add error handler with lock support

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #37275 from bk201/wip-47494

mgr/dashboard: display devices' health information within a tabset

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #37305 from zdover23/wip-doc-3-dev-guide-basic-workflow-date-update-sep-2020

doc/dev: updated date from 2016

Reviewed-by: Neha Ojha <nojha@redhat.com>

msgr: remove the macro not referenced anywhere

The default port range should be between:
  [ms_bind_port_min, ms_bind_port_max]

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

mon: test before adding/removing required_client_features

Fixes: https://tracker.ceph.com/issues/47006
Signed-off-by: Jos Collin <jcollin@redhat.com>

doc/dev: added the other four tests

This PR adds a list of tests that are performed
when a PR hits Github.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

crimson/os/seastore/journal: return addr and seq from submit_record

Reworks journal_seq_t to be a monotonically increasing, but non-dense
value from the current segment_seq_t and segment offset.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add interface for trimming the journal

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: rework pending extents

Previously, we simply added the old extent to the retired set and
treated the new instance seperately.  Instead, let the new pending
instance reference the old instance via a prior_instance reference.
This has a few advantages:
- During commit, we swap cache hooks from the prior instance to
  avoid traversal
- Extents in the retired set are only those extents that are no
  longer mapped/valid.
- btree range pins can be swapped without a traversal
- position in the dirty list can be maintained

That last will be particularly important as the dirty list will
therefore be ordered from the extent with the oldest initial
delta forward.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: factor out LogicalCachedExtent::on_delta_write

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: rename journal_replay_lb to journal_tail

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: factor out common LBANode helpers

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: remove CachedExtent move constructor

I'm not sure why I added this in the first place, there aren't really
any scenarios where invoking it wouldn't be a bug.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../btree_lba_node_impl: mutate_mapping adjust addr for extent state

Addresses must always be adjusted depending on whether the extent will
be written as a block or as a delta.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: improve misc debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore_types: permit paddr_t::operator- to operate on block relative addrs

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: use smaller segments for tests

This way, we'll roll segments more frequently.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree/btree_lba_manager: manage pin in init

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: drop from cache directly during replay if dead

Dead extents can't go through the normal retire_extent path as they don't
have valid pins or other elements of a real extent.  Instead, simply drop
them on the floor directly.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: fix find_hole

find_hole shouldn't check entries prior to min.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: initialize crc *after* read

Previously, all extents ended up initialized to the same crc value
upon read -- the one for an empty buffer.  This bug wasn't detected
because an extent was always evicted and reread following the first
write and subsequently kept in cache once initially written.

Also, add crc to operator<< for extents.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: only add pin if not already pending

If it's already pending, cache.duplicate_for_write is a noop and the extent
will already have a pin.  Also, add some debugging.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: let root exist in the dirty list

Most handling remains special, but this way the dirty list we
work on will include root directly.

Signed-off-by: Samuel Just <sjust@redhat.com>