test/crimson/seastore/test_btree_lba_manager: fix get_overlap

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_btree_lba_manager: add single transaction split/merge test

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: always return absolute or record relative addrs

insert and lookup_range should return absolute or record-relative addrs,
never block relative.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../lba_btree_node_impl: use node_[un]resolve_vals

Implement node_[un]resolve_vals to transform values to/from block relative
representations when copying out of/in to newly created blocks.  This can
happen when splitting a node which has had entries added during the
same transaction, or itself was created during the transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common/fixed_kv_node_layout: add support for node dependent values

node_[un]resolve_vals can be used by implementations to transform vals
during split and merge.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: add test for large single-transaction split/merge

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: fix lookup_range, scan_* references with weak transactions

Weak transactions won't keep references in memory as we traverse
the tree, update each recursive call to ensure lifecycle of
this.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: extract common test logic for transaction_maanager initialization

This should make it somewhat easier to avoid breaking tests that
depend on transaction_manager.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: add online gc

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore/test_transaction_manager: close and reopen tm after mkfs

We'll just require this generally.  Most of the time, mkfs will be a
standalone operation anyway.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add get_physical_extent_if_live

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add public, cache aware get_extent_by_type variants

Signed-off-by: Samuel Just <sjust@redhat.com>

os/seastore/journal: replay_segment: skip replay on deltas to released segments

See comment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: fix get_journal_seq

current_journal_segment_seq was misleading -- it's actually the value
for the next one.  Fix naming and update get_journal_seq.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add journal_replay check and debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: add segment_header_t operator<<

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add duplicate_for_write debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/journal: check offset against segment length in replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: no need for p2roundup, to_write must already be aligned

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: simplify new extent paddr mapping during complete_transaction

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: improve encode_record debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use scan_segment to add extent scan

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: generalize replay scan

We'll need to do at least two forms of scan:
- deltas for replay
- extents for gc

Most of the mechanics are common, however, so this patch
hoists the common machinery into scan_segemnt.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: record extents types and laddr in record metadata

We'll need this to scan segments for live blocks.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/journal: use metadata size for block base rather than assuming one block

Uncaught because it's rare for the metadata prefix to exceed 4k.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add space accounting to segment_cleaner and wire in

Adds support for space accounting to SegmentCleaner and wires into
Journal, Cache, and tests.

SegmentCleaner has two tracking implementations, SpaceTrackerSimple
and SpaceTrackerDetailed.  SpaceTrackerSimple simply keeps a count
of live bytes and is intended to be the normal implementation.
SpaceTrackerDetailed maintains a bitmap and is simply useful
for debugging unit tests.  It may be removed in the future.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/.../test_transaction_manager: reset ephemeral segment_manager state on replay

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: introduce weak transactions

This way, we can do a bulk scan of the store without building up
an unbounded amount of state in Transaction::read_set.  Note that
such transactions will not be snapshot isolated.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager/btree: hold a reference to parent until added to cache

Currently, we need to rely on the Transaction::read_set to ensure cache
residence of a leaf node until TransactionManager adds the logical
extent to the cache.  The next patch, however, will introduce a lazy
flag for Transaction's to enable doing snapshot inconsistent scans
without populating the read_set, so we'll want this to work without
it.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: rearrange to put private members at end

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager: add scan_mappings

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager: expose segment_state_t

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_manager/ephemeral: fix segment bounds checks

Also adds some debugging.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #37644 from mgfritch/orch-device-lsm_data

mgr/orchestrator: set `lsm_data` to default of Dict type

Merge pull request #37175 from changchengx/log_start

common: remove log_early configuration option

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37617 from rhcs-dashboard/46869-sass-modules

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37700 from tchaikov/wip-build-integration-branch

build-integration-branch: send token in the header

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #37564 from s0nea/wip-dashboard-nfs-daemons-param

mgr/dashboard/qa: consider `daemons` parameter

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #37658 from neha-ojha/wip-crimson-suite

qa/crontab/teuthology-cronjobs: add crimson-rados

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

common: remove log_early configuration option

After deciding to always enable tracking log in early phase, there's no
need to keep "log_early" option here and remove it directly.

Suggested-by: Kefu Chai <kefu@redhat.com>
Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common: remove useless log_early check condition here

The code logic shows that Log thread needs to be started even
log_early is false.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37315 from changchengx/paxos_extend

mon: define paxos services indexes using enum and cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.com>

Merge pull request #36536 from matthewoliver/pick_address_warning_doc_update

common/pick_address: support IPv4 and IPv6 dual stack mode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37670 from changchengx/crypto_plugin

crypto: remove useless code and unify file name variable

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37635 from jecluis/wip-46816

mon: have 'mon stat' output json as well

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33392 from alram/master

osdmaptool: add --mark-{up,in} and --adjust-crush-weight options

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #36603 into master

* refs/pull/36603/head:
test: add test for setting ceph mirror virtual xattr
tests/pybind/cephfs: cleanup xattrs before starting tests
client: filter ceph.* xattrs from listing
client: force an attr fetch for ceph xattrs
client: changes for ceph.mirror.info xattr
mds: restrict setting/removing certain xattrs in ceph namespace
mds: introduce ceph.mirror.info virtual xattr
mds: customize xattr handling using dispatch handlers
mds: introduce is_ceph_vxattr() helper

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37699 from dillaman/wip-47880

journal: possible race condition between flush and append callback

Reviewed-by: Mykola Golub <mgolub@suse.com>

build-integration-branch: change to be pep8 compliant

Signed-off-by: Kefu Chai <kchai@redhat.com>

build-integration-branch: send token in the header

see
https://developer.github.com/changes/2020-02-10-deprecating-auth-through-query-param/

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37698 from tchaikov/wip-crimson-do-until

crimson/common: schedule action only if the future is not available

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #37695 from tchaikov/wip-dashboard-pep8

mgr/dashboard: do not use "l" for variable name

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37645 from dillaman/wip-librbd-aio-completion-race

librbd: update AioCompletion return value before evaluating pending count

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37643 from dillaman/wip-47840

librbd: ignore -ENOENT error when disabling object-map

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37444 from dillaman/wip-librbd-copyup-api

librbd: support the ability to process parent data prior to copyup

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Or Ozeri <oro@il.ibm.com>

journal: possible race condition between flush and append callback

When notifying the journal recorder of an overflow or if the object
close request has completed due to no more in-flight IO, it was
possible for a race between a flush request and the processing of
an append completion to attempt to kick off duplicate notifications.
Since the overflowed and closed callbacks are properly protected from
duplicates, use a counter instead of a boolean to track possible
in-flight handler callbacks.

Fixes: https://tracker.ceph.com/issues/47880
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

crimson/common: schedule action only if the future is not available

otherwise we could call do_until() recursively if we have other tasks
which need to prempt the reactor and current future's state is actually
always available.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: do not use "l" for variable name

see also https://www.flake8rules.com/rules/E741.html

also silences flake8 warnings like:

2: {tty:'./controllers/home.py:90:26: E741 ambiguous variable name 'l'':'./controllers/home.py:90:26: E741 ambiguous variable name 'l''}
2: {tty:'./controllers/logs.py:52:13: E741 ambiguous variable name 'l'':'./controllers/logs.py:52:13: E741 ambiguous variable name 'l''}
2: {tty:'./services/ganesha.py:36:40: E741 ambiguous variable name 'l'':'./services/ganesha.py:36:40: E741 ambiguous variable name 'l''}
2: 3     E741 ambiguous variable name 'l'

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37687 from tchaikov/wip-crimson-errorator

crimson/common: do not take from a future twice

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37602 from tspmelo/wip-flake8-ignore

mgr/dashboard: Remove some Flake8 ignore rules

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/common: do not take from a future twice

before this change, in our specialization of seastar::do_until(),
we access `f` after calling `f.get()`, this is not correct. as `f.get()`
actually moves `f._state` away and detaches the associated promise if any.
so we cannot call `f._then()` anymore after calling `f.get()`. as
`f._then()` schedules `f` by detaching the future from promise and
attaching the scheduled task to the promise. but `future_base::detach_promise()`
does not check `_promise` before accessing it, hence the segfault.

after this change, the order of the checks is rearranged so that
`f.get()` is called at the end. and also use `f.get0()` to be more
explicit, as we are accessing the only element of the returned
value.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37684 from ivancich/rgw-yummy-orphan-testing

rgw: use yum rather than dnf for teuthology testing of rgw-orphan-list

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: use yum rather than dnf for testing rgw-orphan-list

The teuthology testing for rgw-orphan-list needs to install
`s3cmd`. Switch from using dnf to yum to work on a wider variety of
platforms.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37660 from adamemerson/wip-datalog-fix

cls/fifo: Switch use CLS_ERR for errors
rgw/fifo: Fix a few missed return value assignments
rgw/fifo: Add some error logging
rgw/fifo: Catch two instances journaling a new part
rgw/fifo: Use unique_ptr and explicit release for callbacks

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37575 from bk201/wip-47742

qa/suites/rados/cephadm/dashboard: fix creating OSD failure

Merge pull request #37669 from lixiaoy1/fix_cache_state_ut_1015

rbd/cache: fix UTs related to image cache state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

rbd/cache: fix UTs related to image cache state

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37633 from tchaikov/wip-pybind-rados-cleanup

pybind/rados: add more type hintings and cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37624 from lxbsz/cram

qa/tasks: add a 'parallel' option support for the cram task

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37621 from dsavineau/cephadm_ntpsec

cephadm: allow ntpsec service

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #37601 from tspmelo/wip-rm-wait-for-expected-get-result

qa/mgr: Remove _wait_for_expected_get_result

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crypto/isa-l: remove redundant declare shared library

isal_crypto_plugin_objs is same as ceph_crypto_isal

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common/plugin: correct output filename path

The around code use fname as output in some places.
Unify the file name variable to be fname.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

common/plugin: remove stale commented code

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #37609 from lixiaoy1/remove_writelog

rbd/cache: remove unnecessary WriteLogCache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #37625 from lixiaoy1/rwl_path

rbd/cache: store full cache path to image metadata

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

rgw/fifo: Use unique_ptr and explicit release for callbacks

To simplify and clarify memory allocation in AIO cases.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Catch two instances journaling a new part

If we see another part journaled, just run the journal ourselves.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Add some error logging

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Fix a few missed return value assignments

Slipped in when I converted from call-back based to sequential.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Switch use CLS_ERR for errors

So things going wrong show up in the log by default. Also use
__PRETTY_FUNCTION__ instead of __func__ and print some more context.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #34702 into master

* refs/pull/34702/head:
doc: Make time-related osd default values in documentation big-endian.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37587 from anthonyeleven/yet-more-doc-polishing

doc: clarity, detail, modernization, capitalization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37168 from ifed01/wip-ifed-fix-compress-csum

os/bluestore: attach csum for compressed blobs

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #37611 from tchaikov/wip-cmake-tox

cmake: do not always add py3 to TOX_ENVS

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37594 from trociny/wip-rbd-quiesce-cancel

librbd: fix race on watcher unregister

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37650 from jschmid1/gs

mgr/cephadm: adapt <placement> usage string

mgr/dashboard: set 'vendor/variables' SASS module as the single source of truth

Fixes: https://tracker.ceph.com/issues/46869
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

mgr/dashboard: add an empty line to appease flake8

silence the warnings like

./controllers/mgr_modules.py:38:5: E301 expected 1 blank line, found 0'
./controllers/user.py:69:5: E301 expected 1 blank line, found 0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37394 from rosinL/cleanup-queue-option

common/options: remove unavailable values of osd_op_queue

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37627 from changchengx/msg

msg: remove stale comment and unnecessary code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37626 from tchaikov/wip-drop-std-iterator

include/interval_set: do not inherit from std::iterator

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #37455 from smithfarm/wip-spec-file-cleanup

rpm: three spec file cleanups

Reviewed-by: Tim Serong <tserong@suse.com>

pybind/rados: correct the declarations of C APIs

so they are consistent with the ones declared in include/rados/librados.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: add more type hintings

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not always add py3 to TOX_ENVS

before this change add_tox_test() always add "py3" to testenv, even the
caller specifies TOX_ENVS explicitly.

after this change, py3 is added only if the caller does not specify any
TOX_ENVS.

this change helps with the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pick_address: Warn and continue when you find at least 1 IPv4 or IPv6 address

Currently if specify a single public or cluster network, yet have both
`ms bind ipv4` and `ms bind ipv6` set daemons crash when they can't find
both IPs from the same network:

    unable to find any IPv4 address in networks '2001:db8:11d::/120' interfaces ''

And rightly so, of course it can't find an IPv4 network in an IPv6
network.
This patch, adds a new helper method, networks_address_family_coverage,
that takes the list of networks and returns a bitmap of address families
supported.
We then check to see if we have enough networks defined and if you don't
it'll warn and then continue.

Also update the network-config-ref to mention having to define both
address family addresses for cluster and or public networks.

As well as a warning about `ms bind ipv4` being enabled by default which
is easy to miss, there by enabling dual stack when you may only be
expect single stack IPv6.

Thee is also a drive by to fix a `note` that wan't being displayed due
to missing RST syntax.

Signed-off-by: Matthew Oliver <moliver@suse.com>
Fixes: https://tracker.ceph.com/issues/46845
Fixes: https://tracker.ceph.com/issues/39711

Merge PR #37583 into master

* refs/pull/37583/head:
mgr/volumes/nfs: Fix wrong error message for pseudo path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37585 into master

* refs/pull/37585/head:
doc/cephfs/nfs: Remove outdated doc related to rook

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/crontab/teuthology-cronjobs: add crimson-rados

Signed-off-by: Neha Ojha <nojha@redhat.com>