crimson/os/seastore/lba_manager: add get_physical_extent_if_live

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add public, cache aware get_extent_by_type variants

Signed-off-by: Samuel Just <sjust@redhat.com>

os/seastore/journal: replay_segment: skip replay on deltas to released segments

See comment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix get_journal_seq

current_journal_segment_seq was misleading -- it's actually the value
for the next one.  Fix naming and update get_journal_seq.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add journal_replay check and debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add segment_header_t operator<<

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add duplicate_for_write debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/journal: check offset against segment length in replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: no need for p2roundup, to_write must already be aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: simplify new extent paddr mapping during complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: improve encode_record debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use scan_segment to add extent scan

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: generalize replay scan

We'll need to do at least two forms of scan:
- deltas for replay
- extents for gc

Most of the mechanics are common, however, so this patch
hoists the common machinery into scan_segemnt.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record extents types and laddr in record metadata

We'll need this to scan segments for live blocks.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use metadata size for block base rather than assuming one block

Uncaught because it's rare for the metadata prefix to exceed 4k.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add space accounting to segment_cleaner and wire in

Adds support for space accounting to SegmentCleaner and wires into
Journal, Cache, and tests.

SegmentCleaner has two tracking implementations, SpaceTrackerSimple
and SpaceTrackerDetailed.  SpaceTrackerSimple simply keeps a count
of live bytes and is intended to be the normal implementation.
SpaceTrackerDetailed maintains a bitmap and is simply useful
for debugging unit tests.  It may be removed in the future.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/.../test_transaction_manager: reset ephemeral segment_manager state on replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: introduce weak transactions

This way, we can do a bulk scan of the store without building up
an unbounded amount of state in Transaction::read_set.  Note that
such transactions will not be snapshot isolated.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager/btree: hold a reference to parent until added to cache

Currently, we need to rely on the Transaction::read_set to ensure cache
residence of a leaf node until TransactionManager adds the logical
extent to the cache.  The next patch, however, will introduce a lazy
flag for Transaction's to enable doing snapshot inconsistent scans
without populating the read_set, so we'll want this to work without
it.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: rearrange to put private members at end

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add scan_mappings

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: expose segment_state_t

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager/ephemeral: fix segment bounds checks

Also adds some debugging.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #37644 from mgfritch/orch-device-lsm_data

mgr/orchestrator: set `lsm_data` to default of Dict type

Merge pull request #37175 from changchengx/log_start

common: remove log_early configuration option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37617 from rhcs-dashboard/46869-sass-modules

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37700 from tchaikov/wip-build-integration-branch

build-integration-branch: send token in the header

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37564 from s0nea/wip-dashboard-nfs-daemons-param

mgr/dashboard/qa: consider `daemons` parameter

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37658 from neha-ojha/wip-crimson-suite

qa/crontab/teuthology-cronjobs: add crimson-rados

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common: remove log_early configuration option

After deciding to always enable tracking log in early phase, there's no
need to keep "log_early" option here and remove it directly.

Suggested-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: remove useless log_early check condition here

The code logic shows that Log thread needs to be started even
log_early is false.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37315 from changchengx/paxos_extend

mon: define paxos services indexes using enum and cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.com>

Merge pull request #36536 from matthewoliver/pick_address_warning_doc_update

common/pick_address: support IPv4 and IPv6 dual stack mode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37670 from changchengx/crypto_plugin

crypto: remove useless code and unify file name variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37635 from jecluis/wip-46816

mon: have 'mon stat' output json as well

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33392 from alram/master

osdmaptool: add --mark-{up,in} and --adjust-crush-weight options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #36603 into master

* refs/pull/36603/head:
test: add test for setting ceph mirror virtual xattr
tests/pybind/cephfs: cleanup xattrs before starting tests
client: filter ceph.* xattrs from listing
client: force an attr fetch for ceph xattrs
client: changes for ceph.mirror.info xattr
mds: restrict setting/removing certain xattrs in ceph namespace
mds: introduce ceph.mirror.info virtual xattr
mds: customize xattr handling using dispatch handlers
mds: introduce is_ceph_vxattr() helper

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37699 from dillaman/wip-47880

journal: possible race condition between flush and append callback

Reviewed-by: Mykola Golub <mgolub@suse.com>

build-integration-branch: change to be pep8 compliant

Signed-off-by: Kefu Chai <kchai@redhat.com>

build-integration-branch: send token in the header

see
https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37698 from tchaikov/wip-crimson-do-until

crimson/common: schedule action only if the future is not available

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #37695 from tchaikov/wip-dashboard-pep8

mgr/dashboard: do not use "l" for variable name

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37645 from dillaman/wip-librbd-aio-completion-race

librbd: update AioCompletion return value before evaluating pending count

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37643 from dillaman/wip-47840

librbd: ignore -ENOENT error when disabling object-map

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37444 from dillaman/wip-librbd-copyup-api

librbd: support the ability to process parent data prior to copyup

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Or Ozeri <oro@il.ibm.com>

journal: possible race condition between flush and append callback

When notifying the journal recorder of an overflow or if the object
close request has completed due to no more in-flight IO, it was
possible for a race between a flush request and the processing of
an append completion to attempt to kick off duplicate notifications.
Since the overflowed and closed callbacks are properly protected from
duplicates, use a counter instead of a boolean to track possible
in-flight handler callbacks.

Fixes: https://tracker.ceph.com/issues/47880
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

crimson/common: schedule action only if the future is not available

otherwise we could call do_until() recursively if we have other tasks
which need to prempt the reactor and current future's state is actually
always available.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: do not use "l" for variable name

see also https://www.flake8rules.com/rules/E741.html

also silences flake8 warnings like:

2: {tty:'./controllers/home.py:90:26: E741 ambiguous variable name 'l'':'./controllers/home.py:90:26: E741 ambiguous variable name 'l''}
2: {tty:'./controllers/logs.py:52:13: E741 ambiguous variable name 'l'':'./controllers/logs.py:52:13: E741 ambiguous variable name 'l''}
2: {tty:'./services/ganesha.py:36:40: E741 ambiguous variable name 'l'':'./services/ganesha.py:36:40: E741 ambiguous variable name 'l''}
2: 3     E741 ambiguous variable name 'l'

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37687 from tchaikov/wip-crimson-errorator

crimson/common: do not take from a future twice

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37602 from tspmelo/wip-flake8-ignore

mgr/dashboard: Remove some Flake8 ignore rules

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/common: do not take from a future twice

before this change, in our specialization of seastar::do_until(),
we access `f` after calling `f.get()`, this is not correct. as `f.get()`
actually moves `f._state` away and detaches the associated promise if any.
so we cannot call `f._then()` anymore after calling `f.get()`. as
`f._then()` schedules `f` by detaching the future from promise and
attaching the scheduled task to the promise. but `future_base::detach_promise()`
does not check `_promise` before accessing it, hence the segfault.

after this change, the order of the checks is rearranged so that
`f.get()` is called at the end. and also use `f.get0()` to be more
explicit, as we are accessing the only element of the returned
value.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37684 from ivancich/rgw-yummy-orphan-testing

rgw: use yum rather than dnf for teuthology testing of rgw-orphan-list

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: use yum rather than dnf for testing rgw-orphan-list

The teuthology testing for rgw-orphan-list needs to install
`s3cmd`. Switch from using dnf to yum to work on a wider variety of
platforms.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37660 from adamemerson/wip-datalog-fix

cls/fifo: Switch use CLS_ERR for errors
rgw/fifo: Fix a few missed return value assignments
rgw/fifo: Add some error logging
rgw/fifo: Catch two instances journaling a new part
rgw/fifo: Use unique_ptr and explicit release for callbacks

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37575 from bk201/wip-47742

qa/suites/rados/cephadm/dashboard: fix creating OSD failure

Merge pull request #37669 from lixiaoy1/fix_cache_state_ut_1015

rbd/cache: fix UTs related to image cache state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

rbd/cache: fix UTs related to image cache state

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37633 from tchaikov/wip-pybind-rados-cleanup

pybind/rados: add more type hintings and cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37624 from lxbsz/cram

qa/tasks: add a 'parallel' option support for the cram task

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37621 from dsavineau/cephadm_ntpsec

cephadm: allow ntpsec service

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #37601 from tspmelo/wip-rm-wait-for-expected-get-result

qa/mgr: Remove _wait_for_expected_get_result

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crypto/isa-l: remove redundant declare shared library

isal_crypto_plugin_objs is same as ceph_crypto_isal

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common/plugin: correct output filename path

The around code use fname as output in some places.
Unify the file name variable to be fname.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common/plugin: remove stale commented code

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37609 from lixiaoy1/remove_writelog

rbd/cache: remove unnecessary WriteLogCache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #37625 from lixiaoy1/rwl_path

rbd/cache: store full cache path to image metadata

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

rgw/fifo: Use unique_ptr and explicit release for callbacks

To simplify and clarify memory allocation in AIO cases.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Catch two instances journaling a new part

If we see another part journaled, just run the journal ourselves.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Add some error logging

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Fix a few missed return value assignments

Slipped in when I converted from call-back based to sequential.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Switch use CLS_ERR for errors

So things going wrong show up in the log by default. Also use
__PRETTY_FUNCTION__ instead of __func__ and print some more context.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #34702 into master

* refs/pull/34702/head:
doc: Make time-related osd default values in documentation big-endian.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37587 from anthonyeleven/yet-more-doc-polishing

doc: clarity, detail, modernization, capitalization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37168 from ifed01/wip-ifed-fix-compress-csum

os/bluestore: attach csum for compressed blobs

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #37611 from tchaikov/wip-cmake-tox

cmake: do not always add py3 to TOX_ENVS

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37594 from trociny/wip-rbd-quiesce-cancel

librbd: fix race on watcher unregister

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37650 from jschmid1/gs

mgr/cephadm: adapt <placement> usage string

mgr/dashboard: set 'vendor/variables' SASS module as the single source of truth

Fixes: https://tracker.ceph.com/issues/46869
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: add an empty line to appease flake8

silence the warnings like

./controllers/mgr_modules.py:38:5: E301 expected 1 blank line, found 0'
./controllers/user.py:69:5: E301 expected 1 blank line, found 0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37394 from rosinL/cleanup-queue-option

common/options: remove unavailable values of osd_op_queue

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37627 from changchengx/msg

msg: remove stale comment and unnecessary code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37626 from tchaikov/wip-drop-std-iterator

include/interval_set: do not inherit from std::iterator

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #37455 from smithfarm/wip-spec-file-cleanup

rpm: three spec file cleanups

Reviewed-by: Tim Serong <tserong@suse.com>

pybind/rados: correct the declarations of C APIs

so they are consistent with the ones declared in include/rados/librados.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: add more type hintings

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not always add py3 to TOX_ENVS

before this change add_tox_test() always add "py3" to testenv, even the
caller specifies TOX_ENVS explicitly.

after this change, py3 is added only if the caller does not specify any
TOX_ENVS.

this change helps with the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pick_address: Warn and continue when you find at least 1 IPv4 or IPv6 address

Currently if specify a single public or cluster network, yet have both
`ms bind ipv4` and `ms bind ipv6` set daemons crash when they can't find
both IPs from the same network:

    unable to find any IPv4 address in networks '2001:db8:11d::/120' interfaces ''

And rightly so, of course it can't find an IPv4 network in an IPv6
network.
This patch, adds a new helper method, networks_address_family_coverage,
that takes the list of networks and returns a bitmap of address families
supported.
We then check to see if we have enough networks defined and if you don't
it'll warn and then continue.

Also update the network-config-ref to mention having to define both
address family addresses for cluster and or public networks.

As well as a warning about `ms bind ipv4` being enabled by default which
is easy to miss, there by enabling dual stack when you may only be
expect single stack IPv6.

Thee is also a drive by to fix a `note` that wan't being displayed due
to missing RST syntax.

Signed-off-by: Matthew Oliver <moliver@suse.com>
Fixes: https://tracker.ceph.com/issues/46845
Fixes: https://tracker.ceph.com/issues/39711

Merge PR #37583 into master

* refs/pull/37583/head:
mgr/volumes/nfs: Fix wrong error message for pseudo path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37585 into master

* refs/pull/37585/head:
doc/cephfs/nfs: Remove outdated doc related to rook

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/crontab/teuthology-cronjobs: add crimson-rados

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37610 from anthonyeleven/doc-rados-troubleshooting

doc/rados/troubleshooting: clarity and modernization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #36537 into master

* refs/pull/36537/head:
qa/cephfs: update ephemeral pin tests
mds: distribute dirfrags for ephemeral distributed directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/crimson-rados: add .qa helper

Fixes:
OSError: /home/nojha/src/github.com_ceph_ceph_master/qa/suites/crimson-rados/basic/centos_latest.yaml
does not exist (abs /home/nojha/src/github.com_ceph_ceph_master/qa/suites/crimson-rados/basic/centos_latest.yaml)

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/cephfs: update ephemeral pin tests

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: distribute dirfrags for ephemeral distributed directory

Instead of distribute individual dir inodes inside the ephemeral
distributed dir. Distributing dirfrags can limit number of subtrees
created by the ephemeral dist pin.

This patch also unifies codes that handle export pin and ephemeral pin.

Fixes: https://tracker.ceph.com/issues/46696
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

librbd: support preprocessing source object data prior to deep-copy

Let object dispatch layers potentially mutate the data read from the
source image prior to issuing the actual deep-copy operations against
the destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: continue to simplify deep-copy object copy write ops types

Replace the custom WriteOp data structure and just re-use the new
io::SnapshotSparseBufferlist structure to record write and zero ops.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify deep-copy object copy write ops types

The write-ops now only stores write vs zero ops and the type of
zero operation is delayed until the actual op is sent. This will
make the state machine compatible with the copyup process hook.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support preprocessing parent data prior to copyup

Let object dispatch layers potentially mutate the copyup data read
from the parent prior to issuing the actual copyup operation. This
can allow for a layer like the crypto layer to re-encrypt the parent
image data using the child image's encryption keys, for example.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new hook for pre-processing copyup data

This will permit the crypto layer to properly encrypt and potentially
align the sparse copyup data prior to it being written. It passes
potentially multiple sets of data in one pass to permit the deep-copy
state machine to utilize the same API and allow the crypto layer to
potentially handle layered alignment issues.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>