Merge PR #29705 into master

* refs/pull/29705/head:
client: fix lazyio_synchronize() to update file size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #29569 into master

* refs/pull/29569/head:
mds: Reorganize class members in DamageTable header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29888 into master

* refs/pull/29888/head:
mds: Reorganize class members in JournalPointer header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30029 into master

* refs/pull/30029/head:
mds: use set to store to evict client

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30058 into master

* refs/pull/30058/head:
qa/tasks: test for prevent negative subvolume size
mgr/volumes: prevent negative subvolume size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge PR #30265 into master

* refs/pull/30265/head:
doc: add a new document on CephFS distributed metadata cache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: add a new document on CephFS distributed metadata cache

Fixes: https://tracker.ceph.com/issues/41451
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #30399 from tchaikov/wip-crimson-peering-query

crimson: handle MOSDPGQuery2 properly

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #28830 from albin-antony/indv2

rgw: fix indentation for listobjectsv2

Reviewed-by: Ali Maredia <amaredia@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29672 into master

* refs/pull/29672/head:
common/RefCountedObj: cleanup con/des
common: add make_ref factory for RefCountedObject
common: refactor RefCountedObj and reduce header dependencies
include: convert FunctionContext usage to generic LambdaContext
osd: add missing header
cls/rgw: add missing header dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30377 from tspmelo/wip-e2e-hosts-mds

mgr/dashboard: Fix e2e host test

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29819 from Devp00l/wip-41396

mgr/dashboard: Document UiApiController with ApiController usage

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

common/RefCountedObj: cleanup con/des

Also, don't allow children to set nref (to 0). This is the more significant
change as it required fixing various code to not do this:

    <reftype> ptr = new RefCountedObjectFoo(..., 0);

as a way to create a starting reference with nref==1. This is a pretty
bad code smell so I've converted all the code doing this to use the new
factory method which produces the reference safely:

    auto ptr = ceph::make_ref<RefCountedObjectFoo>(...);

libradosstriper was particularly egregious in its abuse of setting the starting
nref. :(

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add make_ref factory for RefCountedObject

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: refactor RefCountedObj and reduce header dependencies

- Protect constructors/destructors; RefCountedObj is an abstract class.

- Move some helpers to separate unit to avoid circular dependency with
  CephContext which could otherwise use RefCountedObj.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

include: convert FunctionContext usage to generic LambdaContext

The main motivation for this change is to avoid copies due to the use of
boost::function/std::function where captures of std::unique_ptr (in
subsequent commits) would fail to compile.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

osd: add missing header

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cls/rgw: add missing header dependency

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30368 from liewegas/wip-snapmapper-cleanup

osd/SnapMapper: remove pre-octopus snapmapper keys after conversion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30356 from majianpeng/msg-reduece-duplicated-code

msg: add func is_blackhole to reduce duplicated code.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30350 from liewegas/wip-bluestore-poolstats-configs

os/bluestore: simplify per-pool-stat config options

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

crimson/osd: change log level to warn if pg is absent

it's not fatal or indicating a bug, but it's unusual. so let's make it a
warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: reply if pg is absent when handling MQuery

in previous commit, the support of `MOSDPGQuery` was dropped. but
if we want to handle `MOSDPGQuery2` properly, we should take
`on_pg_absent()` into consideration when handling `MOSDPGQuery2`.

in this change, we handle `MOSDPGQuery2` by checking the boost event
against `MQuery`, so we don't need to create a new class derived from
`RemotePeeringEvent` for specializing the behavior.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not handle MOSDPGQuery anymore

this is a follow-up of 260ab1a5, as MOSDPGQuery is deprecated in favor
of MOSDPGQuery2 in octopus and up. let's drop the support of
MOSDPGQuery.

Signed-off-by: Kefu Chai <kchai@redhat.com>

osd/OSD.h: remove unused OSD::discard_context()

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: do not use __func__ in a lambda for logging

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: verbose logging in OSD::_send_alive()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30398 from tspmelo/wip-backport-head

script/ceph-backport.sh: Fix verification of git repository

script/ceph-backport.sh: Fix verification of git repository

In git 2.16 running "git show-ref HEAD" returns an error and
prevents the script to procced.

Signed-off-by: Tiago Melo <tmelo@suse.com>

rgw: fix indentation for listobjectsv2

https://tracker.ceph.com/issues/41840

Signed-off-by: Albin Antony <aantony@redhat.com>

Merge pull request #29549 from smanjara/wip-multisite-metadata

rgw: Allow admin APIs that write metadata to be executed first on the mast…

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30174 from mkogan1/wip-rgw-40700

rgw: fix memory growth while deleting objects with

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30379 from smithfarm/wip-41821

tools/setup-virtualenv.sh: do not default to python2.7

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #30362 from uweigand/fix-unaligned-check

cmake: Fix unaligned check on big-endian systems

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30374 from tchaikov/wip-cmake-brotli

cmake: s/bortli_libs/brotli_libs/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30366 from liu-chunmei/ceph_seastar_MonClient

crimson/mon: use shared_future for waiting MauthReply

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30380 from rzarzynski/wip-crimson-cyanread-returns-bl

crimson/os: Object::read() returns bufferlist instead of never used errcode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30339 from rzarzynski/wip-crimson-cls-2

crimson: serve basic RBD traffic coming from fio

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/mon:use shared_future for waiting MauthReply

when monitor-side close the connection, msgr call MonClient
ms_handle_reset cause reply.get_future be called twice then
assert happen in promise.get_future.
promise<T...>::get_future() noexcept {
    assert(!this->_future && this->_state && !this->_task);
    return future<T...>(this);
}
use shared_promise instead of promise to solve it.

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

Merge pull request #30363 from idryomov/wip-krbd-huge-osdmap

qa: add script to test how libceph handles huge osdmaps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd: add dummy support for CEPH_OSD_OP_WATCH.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_get_client_features().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: switch get_orig_source_inst() to get_message() in OpsExecuter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_cxx_map_set_val().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_get_request_origin().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETVALS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #28560 into master

* refs/pull/28560/head:
cephfs-shell: handle du's arguments elsewhere outside do_du()
cephfs-shell: reuse code
cephfs-shell: rewrite call to perror in do_du
pybind/cephfs: define variable for hexcode used in stat()
test_cephfs_shell: test cephfs-shell command at invocation
cephfs-shell: refactor do_du()
cephfs-shell: option -r is not for reverse
cephfs-shell: extend to_bytes()
test_cephfs_shell: test du with no args
test_cephfs_shell: test du with multiple paths in args
test_cephfs_shell: test behaviour of "du -r"
test_cephfs_shell: test du's output for softlinks
qa/cephfs: add convenience method lstat()
qa/cephfs: add option to make stat() unfollow symlinks
test_cephfs_shell: test du's output for hardlinks
test_cephfs_shell: test du's output for directories
test_cephfs_shell: test du's output for regular files
test_cephfs_shell: add a method to get command output
test_cephfs_shell: allow cmd as list too
test_cephfs_shell: rename and rewrite _cephfs_shell()
test_cephfs_shell: copy humanize() from cephfs-shell
cephfs-shell: print disk usage for non-directory files too
pybind/cephfs: add method that stats symlinks without following
cephfs-shell: Fix 'du' command error

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29907 into master

* refs/pull/29907/head:
doc: add a doc for vstart_runner.py

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30185 into master

* refs/pull/30185/head:
mgr/volumes: drop unnecessary size

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os: fix ENODATA handling in CyanStore::get_attrs().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETKEYS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PGBackend::write() alters object size now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_cxx_write2().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPSETVALS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETVALSBYKEYS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29037 from zhangsw/rgw-reshard-numshards

rgw: shard number must be non-negative when resharding the bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_CREATE.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: ignore CEPH_OSD_OP_TMAPUP like ceph-osd does for EC pools.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa: add script to test how libceph handles huge osdmaps

That code will also handle moderately-sized osdmaps when the memory is
fragmented.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #29623 from majianpeng/bluestore-combine-deferred_finisher-and-finisher

os/bluestore: Merge deferred_finisher and finisher.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30342 from ifed01/wip-ifed-fix-blues-alerts

mon/pgmap: fix bluestore alerts output.

Reviewed-by: Sage Weil <sage@redhat.com>

crimson/os: Object::read() returns bufferlist instead of never used errcode.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30372 from tchaikov/wip-crimson-single-pg-peering

crimson/osd: use single-pg peering ops

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #24794 into master

* refs/pull/24794/head:
mds: move BatchOp to separate translation unit
mds: use auto ref during iteration
mds: answering all pending getattr/lookups targeting the same inode in one go

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28702 into master

* refs/pull/28702/head:
qa: update json format from session listing
mds: recall caps from quiescent sessions
mds: use Session::dump method uniformly
mds: use auto to deduce iterator type
mds: simplify method definition
mds: remove useless debug message
mds: use const get_request_count
mds: use session_info_t socket inst for addr
mds: refactor session lookup
mds: add explicit trim flag
mds: alphabetize tracked config keys
common: provide method to get half-life
common: correct variable name

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #29005 into master

* refs/pull/29005/head:
client: nfs-ganesha with cephfs client, removing dir reports not empty

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #29594 into master

* refs/pull/29594/head:
mds: reject sessionless messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29752 into master

* refs/pull/29752/head:
mds: make MDSIOContextBase delete itself when shutting down

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30361 into master

* refs/pull/30361/head:
qa/tasks/mon_thrash: sync force requires some force flags

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #30357 from zhangsw/rgw-cleanup-bilog-status

rgw: add missing bilog status to help info.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tools/setup-virtualenv.sh: do not default to python2.7

This commit changes the script's behavior when run without a --python option.
Before, we were running virtualenv with "--python python2.7" in that case.

That is not a reasonable default in September 2019, given that Python 2 EOL is
coming on January 1, 2020. Also, it breaks setup-virtualenv.sh in py3-only
environments.

With this patch, the script will not pass any "--python" option to virtualenv
(i.e., it will let virtualenv do whatever it thinks is best) in this case.

Fixes: https://tracker.ceph.com/issues/41821
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29081 from wintamute/patch-1

Documentation: Add missing ceph-volume lvm batch argument to ceph-volume.rst

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #30352 into master

* refs/pull/30352/head:
mon: disable min pg per osd warning

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30284 from Damenly/missing_cmath

tests: add missing header cmath to test/mon/test_mon_memory_target.cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix e2e host test

There is a random problem with mds service listing
that is causing a e2e test to fail.

This will temporarily fix this problem.

Signed-off-by: Tiago Melo <tmelo@suse.com>

doc: Add missing ceph-volume lvm batch argument to ceph-volume.rst

The optional argument '--osds-per-device' is missing. Since this is quite helpful when creating NVMe based OSDs, it should be documented.

Signed-off-by: Andreas Krebs <wintamute@gmail.com>

mgr/dashboard: Increase the global value of jasmine timeout (#30359)

mgr/dashboard: Increase the global value of jasmine timeout

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

cmake: s/bortli_libs/brotli_libs/

it's merely a typo.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks: test for prevent negative subvolume size

Fixes: https://tracker.ceph.com/issues/41617
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #29761 from wjwithagen/wjw-fix-venv-python-version

test/venv: do not hardwire to py2.7 for tox tests

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix calculation of PG Status percentage (#30343)

mgr/dashboard: Fix calculation of PG Status percentage

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

crimson/osd: require_osd_release >= octopus to boot

we assume that crimson will GA after octopus+2, so no need to be
compatible with pre-octopus releases.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use single-pg peering ops

classic OSD started to use single-pg peering ops since ce05c172, and it
switched over to these ops since octopus. and it's assumed that
crimson-osd won't be GA until octopus+2 release, so it's not needed to
support pre-octopus releases.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: pass octopus to PeeringCtx's constructor

to accommodate the change of 77ecc327

classic OSD started to use single-pg peering ops since ce05c172, and it
switched over to these ops since octopus. and it's assumed that
crimson-osd won't be GA until octopus+2 release, so it's not needed to
support pre-octopus releases. hence "octopus" is passed to PeeringCtx's
constructor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30285 from IlsooByun/slow_op_report

mgr: do not reset reported if a new metric is not collected

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephfs-shell: handle du's arguments elsewhere outside do_du()

Create new action class for do_du()'s args.paths and push the code that
handles arguments for do_du() in there. The intention is to keep only
the code that does some disk usage stuff in do_du().

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: reuse code

Make path_to_bytes.__call__() call to_bytes() and get rid of the code
within path_to_bytes.__call__().

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: rewrite call to perror in do_du

Adapt to new signature of cmd2.Cmd.perror.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

pybind/cephfs: define variable for hexcode used in stat()

Instead of using the same hexcode at multiple locations, define it as a
variable and use the variable instead.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test cephfs-shell command at invocation

Verify that "cephfs-shell -c ceph.conf <cephfs-shell-cmd>" works as
expected.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: refactor do_du()

Improve code's readability and keep all paths as bytes.
Specifically, define a function within do do_du() to simplify the code
that handles recursion, rename args.dirs to args.paths, get rid of
dir_passed and use args.paths directly instead,  change the default
value of args.dirs to a list with one member that contains current
working directory, rewrite the docstring, and few more similar changes.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: option -r is not for reverse

Option "-r" for du command stands for recursive and not reverse.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: extend to_bytes()

Make to_bytes() handle lists, None as well as other simple types as
well. In case of None, to_bytes() returns None instead of returning
b'None'.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du with no args

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du with multiple paths in args

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test behaviour of "du -r"

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du's output for softlinks

Test that CephFS shell command du prints output for softlinks as
expected

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: add convenience method lstat()

It's more convenient lstat(path) than stat(path, follow_symlink=False).

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: add option to make stat() unfollow symlinks

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du's output for hardlinks

Test that CephFS shell command du prints output for directories as
expected.

Signed-off-by: Rishabh Dave <ridave@redhat.com>