Merge pull request #30377 from tspmelo/wip-e2e-hosts-mds

mgr/dashboard: Fix e2e host test

Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29819 from Devp00l/wip-41396

mgr/dashboard: Document UiApiController with ApiController usage

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30368 from liewegas/wip-snapmapper-cleanup

osd/SnapMapper: remove pre-octopus snapmapper keys after conversion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30356 from majianpeng/msg-reduece-duplicated-code

msg: add func is_blackhole to reduce duplicated code.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #30350 from liewegas/wip-bluestore-poolstats-configs

os/bluestore: simplify per-pool-stat config options

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30398 from tspmelo/wip-backport-head

script/ceph-backport.sh: Fix verification of git repository

script/ceph-backport.sh: Fix verification of git repository

In git 2.16 running "git show-ref HEAD" returns an error and
prevents the script to procced.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29549 from smanjara/wip-multisite-metadata

rgw: Allow admin APIs that write metadata to be executed first on the mast…

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30174 from mkogan1/wip-rgw-40700

rgw: fix memory growth while deleting objects with

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30379 from smithfarm/wip-41821

tools/setup-virtualenv.sh: do not default to python2.7

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #30362 from uweigand/fix-unaligned-check

cmake: Fix unaligned check on big-endian systems

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30374 from tchaikov/wip-cmake-brotli

cmake: s/bortli_libs/brotli_libs/

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30366 from liu-chunmei/ceph_seastar_MonClient

crimson/mon: use shared_future for waiting MauthReply

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30380 from rzarzynski/wip-crimson-cyanread-returns-bl

crimson/os: Object::read() returns bufferlist instead of never used errcode

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30339 from rzarzynski/wip-crimson-cls-2

crimson: serve basic RBD traffic coming from fio

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/mon:use shared_future for waiting MauthReply

when monitor-side close the connection, msgr call MonClient
ms_handle_reset cause reply.get_future be called twice then
assert happen in promise.get_future.
promise<T...>::get_future() noexcept {
    assert(!this->_future && this->_state && !this->_task);
    return future<T...>(this);
}
use shared_promise instead of promise to solve it.

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

Merge pull request #30363 from idryomov/wip-krbd-huge-osdmap

qa: add script to test how libceph handles huge osdmaps

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson/osd: add dummy support for CEPH_OSD_OP_WATCH.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_get_client_features().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: switch get_orig_source_inst() to get_message() in OpsExecuter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_cxx_map_set_val().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_get_request_origin().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETVALS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #28560 into master

* refs/pull/28560/head:
cephfs-shell: handle du's arguments elsewhere outside do_du()
cephfs-shell: reuse code
cephfs-shell: rewrite call to perror in do_du
pybind/cephfs: define variable for hexcode used in stat()
test_cephfs_shell: test cephfs-shell command at invocation
cephfs-shell: refactor do_du()
cephfs-shell: option -r is not for reverse
cephfs-shell: extend to_bytes()
test_cephfs_shell: test du with no args
test_cephfs_shell: test du with multiple paths in args
test_cephfs_shell: test behaviour of "du -r"
test_cephfs_shell: test du's output for softlinks
qa/cephfs: add convenience method lstat()
qa/cephfs: add option to make stat() unfollow symlinks
test_cephfs_shell: test du's output for hardlinks
test_cephfs_shell: test du's output for directories
test_cephfs_shell: test du's output for regular files
test_cephfs_shell: add a method to get command output
test_cephfs_shell: allow cmd as list too
test_cephfs_shell: rename and rewrite _cephfs_shell()
test_cephfs_shell: copy humanize() from cephfs-shell
cephfs-shell: print disk usage for non-directory files too
pybind/cephfs: add method that stats symlinks without following
cephfs-shell: Fix 'du' command error

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29907 into master

* refs/pull/29907/head:
doc: add a doc for vstart_runner.py

Reviewed-by: Varsha Rao <varao@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30185 into master

* refs/pull/30185/head:
mgr/volumes: drop unnecessary size

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os: fix ENODATA handling in CyanStore::get_attrs().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETKEYS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PGBackend::write() alters object size now.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement cls_cxx_write2().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPSETVALS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_OMAPGETVALSBYKEYS.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #29037 from zhangsw/rgw-reshard-numshards

rgw: shard number must be non-negative when resharding the bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

crimson/osd: add support for CEPH_OSD_OP_CREATE.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: ignore CEPH_OSD_OP_TMAPUP like ceph-osd does for EC pools.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa: add script to test how libceph handles huge osdmaps

That code will also handle moderately-sized osdmaps when the memory is
fragmented.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #29623 from majianpeng/bluestore-combine-deferred_finisher-and-finisher

os/bluestore: Merge deferred_finisher and finisher.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #30342 from ifed01/wip-ifed-fix-blues-alerts

mon/pgmap: fix bluestore alerts output.

Reviewed-by: Sage Weil <sage@redhat.com>

crimson/os: Object::read() returns bufferlist instead of never used errcode.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30372 from tchaikov/wip-crimson-single-pg-peering

crimson/osd: use single-pg peering ops

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #24794 into master

* refs/pull/24794/head:
mds: move BatchOp to separate translation unit
mds: use auto ref during iteration
mds: answering all pending getattr/lookups targeting the same inode in one go

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28702 into master

* refs/pull/28702/head:
qa: update json format from session listing
mds: recall caps from quiescent sessions
mds: use Session::dump method uniformly
mds: use auto to deduce iterator type
mds: simplify method definition
mds: remove useless debug message
mds: use const get_request_count
mds: use session_info_t socket inst for addr
mds: refactor session lookup
mds: add explicit trim flag
mds: alphabetize tracked config keys
common: provide method to get half-life
common: correct variable name

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #29005 into master

* refs/pull/29005/head:
client: nfs-ganesha with cephfs client, removing dir reports not empty

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #29594 into master

* refs/pull/29594/head:
mds: reject sessionless messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29752 into master

* refs/pull/29752/head:
mds: make MDSIOContextBase delete itself when shutting down

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #30361 into master

* refs/pull/30361/head:
qa/tasks/mon_thrash: sync force requires some force flags

Reviewed-by: João Eduardo Luís <joao@suse.de>

Merge pull request #30357 from zhangsw/rgw-cleanup-bilog-status

rgw: add missing bilog status to help info.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

tools/setup-virtualenv.sh: do not default to python2.7

This commit changes the script's behavior when run without a --python option.
Before, we were running virtualenv with "--python python2.7" in that case.

That is not a reasonable default in September 2019, given that Python 2 EOL is
coming on January 1, 2020. Also, it breaks setup-virtualenv.sh in py3-only
environments.

With this patch, the script will not pass any "--python" option to virtualenv
(i.e., it will let virtualenv do whatever it thinks is best) in this case.

Fixes: https://tracker.ceph.com/issues/41821
Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #29081 from wintamute/patch-1

Documentation: Add missing ceph-volume lvm batch argument to ceph-volume.rst

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge PR #30352 into master

* refs/pull/30352/head:
mon: disable min pg per osd warning

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #30284 from Damenly/missing_cmath

tests: add missing header cmath to test/mon/test_mon_memory_target.cc

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix e2e host test

There is a random problem with mds service listing
that is causing a e2e test to fail.

This will temporarily fix this problem.

Signed-off-by: Tiago Melo <tmelo@suse.com>

doc: Add missing ceph-volume lvm batch argument to ceph-volume.rst

The optional argument '--osds-per-device' is missing. Since this is quite helpful when creating NVMe based OSDs, it should be documented.

Signed-off-by: Andreas Krebs <wintamute@gmail.com>

mgr/dashboard: Increase the global value of jasmine timeout (#30359)

mgr/dashboard: Increase the global value of jasmine timeout

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

cmake: s/bortli_libs/brotli_libs/

it's merely a typo.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29761 from wjwithagen/wjw-fix-venv-python-version

test/venv: do not hardwire to py2.7 for tox tests

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: Fix calculation of PG Status percentage (#30343)

mgr/dashboard: Fix calculation of PG Status percentage

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

crimson/osd: require_osd_release >= octopus to boot

we assume that crimson will GA after octopus+2, so no need to be
compatible with pre-octopus releases.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: use single-pg peering ops

classic OSD started to use single-pg peering ops since ce05c172, and it
switched over to these ops since octopus. and it's assumed that
crimson-osd won't be GA until octopus+2 release, so it's not needed to
support pre-octopus releases.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: pass octopus to PeeringCtx's constructor

to accommodate the change of 77ecc327

classic OSD started to use single-pg peering ops since ce05c172, and it
switched over to these ops since octopus. and it's assumed that
crimson-osd won't be GA until octopus+2 release, so it's not needed to
support pre-octopus releases. hence "octopus" is passed to PeeringCtx's
constructor.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30285 from IlsooByun/slow_op_report

mgr: do not reset reported if a new metric is not collected

Reviewed-by: Kefu Chai <kchai@redhat.com>

cephfs-shell: handle du's arguments elsewhere outside do_du()

Create new action class for do_du()'s args.paths and push the code that
handles arguments for do_du() in there. The intention is to keep only
the code that does some disk usage stuff in do_du().

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: reuse code

Make path_to_bytes.__call__() call to_bytes() and get rid of the code
within path_to_bytes.__call__().

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: rewrite call to perror in do_du

Adapt to new signature of cmd2.Cmd.perror.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

pybind/cephfs: define variable for hexcode used in stat()

Instead of using the same hexcode at multiple locations, define it as a
variable and use the variable instead.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test cephfs-shell command at invocation

Verify that "cephfs-shell -c ceph.conf <cephfs-shell-cmd>" works as
expected.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: refactor do_du()

Improve code's readability and keep all paths as bytes.
Specifically, define a function within do do_du() to simplify the code
that handles recursion, rename args.dirs to args.paths, get rid of
dir_passed and use args.paths directly instead,  change the default
value of args.dirs to a list with one member that contains current
working directory, rewrite the docstring, and few more similar changes.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: option -r is not for reverse

Option "-r" for du command stands for recursive and not reverse.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: extend to_bytes()

Make to_bytes() handle lists, None as well as other simple types as
well. In case of None, to_bytes() returns None instead of returning
b'None'.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du with no args

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du with multiple paths in args

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test behaviour of "du -r"

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du's output for softlinks

Test that CephFS shell command du prints output for softlinks as
expected

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: add convenience method lstat()

It's more convenient lstat(path) than stat(path, follow_symlink=False).

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: add option to make stat() unfollow symlinks

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du's output for hardlinks

Test that CephFS shell command du prints output for directories as
expected.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du's output for directories

Test that CephFS shell command du prints output for directories as
expected.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du's output for regular files

Test that CephFS shell command du prints an output for regular files
correctly.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: add a method to get command output

Let run_cephfs_shell_cmd() return the object instead of output in string
format so that stderr, return value, etc. of the command executed
remain available. And add a new method that returns only output of the
CephFS shell command executed.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: allow cmd as list too

Convert cmd to string if it is passed as list to
run_cephfs_shell_cmd(). Allowing cmd to be list would be nice since
elsewhere, in tests, command to be executed must be a list of arguments
rather than string of arguments separated by spaces.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: rename and rewrite _cephfs_shell()

Rename _cephfs_shell() to run_cephfs_shell_cmd() and add a default
client for run_cephfs_shell_cmd() since most of tests in this
testsuite will need only one client.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: copy humanize() from cephfs-shell

Copy the method named humanize from cephfs-shell so that output can be
compared. Unfortunately, importing this method isn't possible since the
dash in "cephfs-shell" leads to an syntax error.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa: update json format from session listing

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: print disk usage for non-directory files too

Currently when du crashes when it comes across a non-directory files.
Instead of doing that, print disk usage for regular files.

Fixes: http://tracker.ceph.com/issues/40371
Signed-off-by: Rishabh Dave <ridave@redhat.com>

pybind/cephfs: add method that stats symlinks without following

Add a new Python binding equivalent to lstat so that information about
the symlink itself can be also obtained, along with other type of files.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #30112 from liewegas/wip-autoscaler-on

common: default pg_autoscale_mode=on for new pools

Reviewed-by: Kefu Chai <kchai@redhat.com>

mds: recall caps from quiescent sessions

This introduces two new config options [1,2] that dictate when a session
is considered quiescent by the MDS. (Options are documented fully in
options.cc.) When a session is quiescent, the MDS will preemptively
recall caps to reduce the outstanding capabilities which optimizes for
reducing work during failover.

[1] mds_session_cache_liveness_magnitude
[2] mds_session_cache_liveness_decay_rate

Fixes: https://tracker.ceph.com/issues/22446
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use Session::dump method uniformly

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use auto to deduce iterator type

This prevents accidental type mismatches.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: simplify method definition

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: remove useless debug message

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use const get_request_count

So it can be used from a const dump method. Make the elist mutable
instead.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: use session_info_t socket inst for addr

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: refactor session lookup

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: add explicit trim flag

So that the intent of the caller is clear to trim due to the cache being
too full.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: alphabetize tracked config keys

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: provide method to get half-life

Print the half-life in Formatter output as it is human understandable
and the value used when configuring the counters.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: correct variable name

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

msg: add func is_blackhole to reduce duplicated code.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge PR #29574 into master

* refs/pull/29574/head:
mds: Reorganize class members in FSMapUser header

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>