README.md: remove extra 'the' and format para related to ninja

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/install: replace do_cmake note with link to ceph README

Ceph README.md doc already has the do_cmake note and other build related info.

Signed-off-by: Varsha Rao <varao@redhat.com>

doc/install: Update about ninja

'ninja' replaced 'make' in this PR[1]. This patch updates the doc about it.

[1] https://github.com/ceph/ceph/pull/39826

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #40334 from dvanders/pick

common/ipaddr: skip loopback interfaces named 'lo' and test it

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40292 from tchaikov/wip-39191

debian,cmake,cephsqlite: hide non-public symbols

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40171 from swegener/bluefs-assert-always-true

bluefs: fix assert always being true

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40304 from tchaikov/wip-telegraph-ann

pybind/mgr/telegraf: add type annotations

Reviewed-by: Wido den Hollander <wido@widodh.nl>

Merge pull request #40403 from vshankar/wip-cephfs-top-add-metrics

cephfs-top: include additional metrics reported by `fs perf stats`

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40167 from singuliere/wip-49781

common/mempool: only fail tests if sharding is very bad

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40331 from athanatos/sjust/wip-seatore-gc-refactor

crimson/seastore: refactor gc into concurrent task

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #40336 from tchaikov/wip-cmake-fio

cmake: pass compile options by fio interface library

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40323 from aclamk/wip-onode-resiliant-to-split-cache

os/bluestore: Make Onode::put/get resiliant to split_cache

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>

crimson/os/seastore/segment_cleaner: bump reclaim_ratio limits

GC behavior still needs a ton of work, but this adjustment should
reduce gc activity for now until we have time to refine the params
and introduce the ability to write cold extents to a segment other
than the current journal segment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../btree_range_pin: use ceph_assert

There's a performance overhead, but seastore is quite immature and
detecting these logic errors saves a ton of debugging time.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/.../block.cc: add error message on alloc error to segment_write

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #39776 from alimaredia/wip-rgw-log-req-latency

rgw: add latency to the request summary of an op

Merge PR #40366 into master

* refs/pull/40366/head:
mgr/cephadm: Fix dashboard gateway configuration when using IPV6

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40395 from rhcs-dashboard/fix-49952-master

mgr/dashboard:Simplify some complex calculations in test_alerts.yml

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

cephfs-top: include additional metrics reported by `fs perf stats`.

Without this, `cephfs-top` hits an exception since the additional
metrics keys were not configured.

Also, include a validation suring selftest that checks if `ceph
fs perf stats` metrics match what cephfs-top is configured to
report.

Fixes: http://tracker.ceph.com/issues/49974
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #40400 from tchaikov/wip-ctest-jenkins

run-make-check.sh: let ctest generate XML output

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #40327 from racpatel/wip-cephfs-top-refresh-interval

cephfs-top: allow configurable stats refresh interval

Merge pull request #40210 from joscollin/wip-B49736-missing-client-metadata

cephfs-top: be resilient to missing client metadata keys

run-make-check.sh: let ctest generate XML output

to enable XUnit plugin of jenkins to consume the ctest output and
publish it in the dashboard, we need to

* let ctest generate XML output instead of plain text output
* do not fail the test if any test case fails. this allows the publisher
  to do its job by checking the XML output.
* prevent ctest from compressing the output. see
  https://issues.jenkins.io/browse/JENKINS-21737

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard:Simplify some complex calculations in test_alerts.yml

run-promtool-unittests is failing with difference in floating point values in some complex calculations. This PR intends to simplify those calculations and fix this issue.

Fixes: https://tracker.ceph.com/issues/49952
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #40377 from tchaikov/wip-doc-fixes

doc/man: fix formatting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38044 from dvanders/dvanders_buffered

common/options: bluefs_buffered_io=true by default

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40380 from adamemerson/wip-num-rados-handles-die-die-die

Die, `num_rados_handles`, die.

Reviewed-by: Ali Maredia <amaredia@redhat.com>

rgw: Remove spurious `num_rados_handles` option

While any code that used the option is gone, it appears the option
itself was accidentally resurrected in various options.cc refactors
and transitions.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

crimson/os/seastore/segment_cleaner: introduce byte limit on rewrite_dirty

Signed-off-by: Samuel Just <sjust@redhat.com>

doc/install/windows-troubleshooting: fix typo

and tweak the format a little bit

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/man/8/ceph-volume: fix formatting

Signed-off-by: Kefu Chai <kchai@redhat.com>

admin/build-doc: build man page in proper directory

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #40350 into master

* refs/pull/40350/head:
doc/cephadm: explicitly show host requirments in adding host section

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40235 from tchaikov/wip-cmake-test-fixture

cmake: use fixture for preparing venv

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mgr/cephadm: Fix dashboard gateway configuration when using IPV6

Fixes: https://tracker.ceph.com/issues/49957
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

cmake: use fixture for preparing venv

this change should allow us to decouple "ninja tests" from "ctest".
in other words, we can just run

ctest -R run-tox-python-common -V

without running "ninja tests" first. before this change

${name}-venv is added as a dependency of "tests" target.

after this change,

setup-venv-for-${name} is added as a test, which is in turn a test of
run-tox-${name}, so we can just

ctest -R run-tox-${name}

now for preparing the venv and then testing the tox test of ${name}.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40344 from zdover23/wip-doc-cephadm-index-rewrite-2021-Mar-23

doc/cephadm: rewrite "index.rst"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #40353 into master

* refs/pull/40353/head:
cephsqlite: add julian day offset in milliseconds

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40291 from tchaikov/wip-debian-cleanups

debian: cleanups and exclude ceph_crypto from dh_shlibdep

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>

tools/cephfs: fix missing keys in cephfs-top client_metadata

Fixes: https://tracker.ceph.com/issues/49736
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #39683 from maydaycc/wip-49487

osd: fix scrub reschedule bug

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

debian/rules: add with_system_libs option

so the maintainer can choose to use the packages shipped by distro.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian/control: remove cython from Build-Depends

as cython3 is enough. and we've dropped the python2 support.

Signed-off-by: Kefu Chai <kchai@redhat.com>

debian/rules: exclude ceph_crypto from dh_shlibdep

as libceph_crypto* are plugins, and they are not self-contained. they
reference symbols offered by the executable loading them. dh_shlibdep
should not complain when checking them, so add them to the exclude list.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40346 from zdover23/wip-doc-cephadm-osd-service-2021-Mar-23-first

doc/cephadm: rewrite osd.rst - list devices

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/cephadm: rewrite "index.rst"

This PR rewrites the three paragraphs at the
front of the cephadm guide, increasing their
elegance and removing ambiguities.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40357 from ybwang0211/quota_doc

doc: remove duplicated 'quota' parameter in URL

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/journal: add error output for submit_record erange

Signed-off-by: Samuel Just <sjust@redhat.com>

debian/control: remove libsqlite3-mod-ceph from dep of ceph-mgr

as the subvar of ${shlibs:Depends} is able to take care of it

Fixes: https://tracker.ceph.com/issues/49899
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: remove duplicated 'quota' parameter in URL

Only one "quota" parameter is needed in the URL to indicate that it is a quota request.

Signed-off-by: wangyingbin <ybwang0211@163.com>

cephfs-top: signal main thread to exit on SIGINT/SIGTERM

Fixes: http://tracker.ceph.com/issues/49953
Signed-off-by: Rachana Patel <racpatel@redhat.com>

cephfs-top: allow configurable stats refresh interval

Signed-off-by: Rachana Patel <racpatel@redhat.com>

crimson/os/seastore/segment_cleaner: add more method comments

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: don't consider journal segments reclaimable

Otherwise, we might end up looping trying to gc based on the reclaimable
metric but be unable to actually find a reclaimable segment.  Mainly a
problem in unit tests that artificially force replay prior rolling the
first journal segment.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../transaction_manager: wait_io in get_extent_if_cached

Exposing wait_io to TransactionManager directly would complicate
any future users.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store-nbd: randomize uuid for BlockSegmentManager

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: more debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_manager/btree: handle invalid extents in rewrite_extent

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store-nbd: add information to assert_all calls

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/lba_manager/btree: add debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store-nbd: permit IO sizes in any multiple of block size

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: add_to_read_set in update_extent_from_transaction

GC transactions need the read set populated.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: stop at any newly dirty extents in get_next_dirty_extents

Any extent with dirty_from == journal_seq_t() is in the process of being
dirtied for the first time and hasn't had dirty_from filled in yet.  As
such, its dirty_from will end up being after the current journal_head.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: track uncached released extents

We need to properly release the space after commit.

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/seastore: disable concurrent gc

The concurrent gc configuration is simpler for the most part, but
greatly complicates these unit tests as they might hit an eagain or not
based on gc interleaving.  For the purposes of these unit tests (and
other basically functional tests), we'll instead disable the background
behavior and invoke it directly after each submitted transaction.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: update journal_tail in submit_transaction

Rather than relying on rewrite_dirty, wait until the transaction actually
clears and update directly in TransactionManager.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: root may not be loaded in duplicate_for_write

In particular, SegmentCleaner::rewite_dirty may operate on root as the
first operation.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/segment_cleaner: remove now unneeded machinery

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: introduce background gc framework

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #40314 into master

* refs/pull/40314/head:
qa/tasks/cephadm: use 'orch apply mon' to deploy mons

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge PR #40324 into master

* refs/pull/40324/head:
mgr/cephadm/upgrade: do not crash if error races with user cancellation

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge PR #40305 into master

* refs/pull/40305/head:
doc/cephfs/nfs: Add note about cephadm NFS-Ganesha daemon port

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

cephsqlite: add julian day offset in milliseconds

This magic number was copied from another VFS but was not adjusted for
the xCurrentTimeInt64 interface.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40320 into master

* refs/pull/40320/head:
mgr/orchestrator: move PORTS column in 'orch ps' output

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #40322 into master

* refs/pull/40322/head:
cephadm: only bootstrap using image that matches cephadm version

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #40216 from rhcs-dashboard/fix-role-cloning

mgr/dashboard: Fix for broken User management role cloning

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>

Merge pull request #36402 from rhcs-dashboard/wip-45965-users-quota-usage

mgr/dashboard: Display users current quota usage

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

doc/cephadm: explicitly show host requirments in adding host section

Signed-off-by: Daniel Pivonka <dpivonka@redhat.com>

doc/cephadm: rewrite osd.rst - list devices

This PR rewrites the "List Devices" section of
the OSD chapter of the Cephadm guide. This PR
is a simple grammar-and-elegance improvement.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #40208 from CongMinYin/fix-flush-block-bug

librbd/cache/pwl: fix bug of flush request blocked by defered IO

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

test_ipaddr: check that we correctly skip loopback

We should skip devices named 'lo' or of the form 'lo:0' regardless
of their IP address.

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Related-to: https://tracker.ceph.com/issues/49938

common/ipaddr: also skip just `lo`

Skip iface's with name like 'lo' or of the form 'lo:0', 'lo:1'. This
brings back the original behavior from b6d0fc9e0e515e50894c08217d688a8c94db7570

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
Fixes: https://tracker.ceph.com/issues/49938

Merge pull request #40005 from rhcs-dashboard/49655-handle-no-rgw-daemons-running

mgr/dashboard: fix error shown when no rgw daemons are running.

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix error notification shown when no rgw daemons are running.

- Adapted code to changes introduced in: https://github.com/ceph/ceph/pull/40220
- Improved error handling.
- Increased test coverage.
- Some refactoring.
- Simplified documentation about setting default daemon host and port.

Fixes: https://tracker.ceph.com/issues/49655
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

os/bluestore: acquire proper lock in split_cache()

Fixes: https://tracker.ceph.com/issues/49900
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #40300 from tchaikov/test-promtool

ceph.spec,install-deps: use golang-github-prometheus for promtools

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

pybind/mgr/telegraf: add type annotations

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass compile options by fio interface library

for better readability and maintainability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/segment_cleaner: introduce ExtentCallbackInterface::submit_transaction_direct

Signed-off-by: Samuel Just <sjust@redhat.com>

Revert "crimson/os/seastore: add releasing state for segments pending close"

This won't be necessary with the refactor in this series.

This reverts commit 3768ef692fc47b629b3ac7a9dd09762172fbe646.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #40318 from batrick/libcephsqite-label

.github/labeler: add libcephsqlite label

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/libsqlite3-mod-ceph: add .symbols file

for tracking the public symbols

see also dh_makeshlibs(1) and dpkg-gensymbols(1)

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephsqlite: hide non-public symbols

mark public interfaces using `[[gnu::visibility("default")]]`, so
we can expose them when the default visibility is "hidden".

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: hide non-public symbols in libcephsqlite

should not expose non-public symbols.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #40192 into master

* refs/pull/40192/head:
test: reduce number of threads to 32 in LibCephFS.ShutdownRace

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40174 into master

* refs/pull/40174/head:
qa/vstart_runner: fix the admin_socket args parsing

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #40158 into master

* refs/pull/40158/head:
test: add test for removing non-existent xattr
mds: fix removexattr logic when there aren't any

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #40146 into master

* refs/pull/40146/head:
test: pass peer uuid when adding cephfs mirror peers
mon: check cephfs mirror peer based on remote cluster spec and file system name

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40145 into master

* refs/pull/40145/head:
doc: add note about disabling standby-replay during upgrades
qa: add test for standby-replay disable
mon: fail standby-replay daemons when flag is turned off

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #38443 into master

* refs/pull/38443/head:
qa: set "shell" to False for run_ceph_w()
vstart_runner: make "shell" a default argument

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>