cephfs-shell: handle du's arguments elsewhere outside do_du()

Create new action class for do_du()'s args.paths and push the code that
handles arguments for do_du() in there. The intention is to keep only
the code that does some disk usage stuff in do_du().

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: reuse code

Make path_to_bytes.__call__() call to_bytes() and get rid of the code
within path_to_bytes.__call__().

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: rewrite call to perror in do_du

Adapt to new signature of cmd2.Cmd.perror.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

pybind/cephfs: define variable for hexcode used in stat()

Instead of using the same hexcode at multiple locations, define it as a
variable and use the variable instead.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test cephfs-shell command at invocation

Verify that "cephfs-shell -c ceph.conf <cephfs-shell-cmd>" works as
expected.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: refactor do_du()

Improve code's readability and keep all paths as bytes.
Specifically, define a function within do do_du() to simplify the code
that handles recursion, rename args.dirs to args.paths, get rid of
dir_passed and use args.paths directly instead,  change the default
value of args.dirs to a list with one member that contains current
working directory, rewrite the docstring, and few more similar changes.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: option -r is not for reverse

Option "-r" for du command stands for recursive and not reverse.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: extend to_bytes()

Make to_bytes() handle lists, None as well as other simple types as
well. In case of None, to_bytes() returns None instead of returning
b'None'.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du with no args

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du with multiple paths in args

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test behaviour of "du -r"

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du's output for softlinks

Test that CephFS shell command du prints output for softlinks as
expected

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: add convenience method lstat()

It's more convenient lstat(path) than stat(path, follow_symlink=False).

Signed-off-by: Rishabh Dave <ridave@redhat.com>

qa/cephfs: add option to make stat() unfollow symlinks

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du's output for hardlinks

Test that CephFS shell command du prints output for directories as
expected.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du's output for directories

Test that CephFS shell command du prints output for directories as
expected.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: test du's output for regular files

Test that CephFS shell command du prints an output for regular files
correctly.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: add a method to get command output

Let run_cephfs_shell_cmd() return the object instead of output in string
format so that stderr, return value, etc. of the command executed
remain available. And add a new method that returns only output of the
CephFS shell command executed.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: allow cmd as list too

Convert cmd to string if it is passed as list to
run_cephfs_shell_cmd(). Allowing cmd to be list would be nice since
elsewhere, in tests, command to be executed must be a list of arguments
rather than string of arguments separated by spaces.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: rename and rewrite _cephfs_shell()

Rename _cephfs_shell() to run_cephfs_shell_cmd() and add a default
client for run_cephfs_shell_cmd() since most of tests in this
testsuite will need only one client.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_cephfs_shell: copy humanize() from cephfs-shell

Copy the method named humanize from cephfs-shell so that output can be
compared. Unfortunately, importing this method isn't possible since the
dash in "cephfs-shell" leads to an syntax error.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: print disk usage for non-directory files too

Currently when du crashes when it comes across a non-directory files.
Instead of doing that, print disk usage for regular files.

Fixes: http://tracker.ceph.com/issues/40371
Signed-off-by: Rishabh Dave <ridave@redhat.com>

pybind/cephfs: add method that stats symlinks without following

Add a new Python binding equivalent to lstat so that information about
the symlink itself can be also obtained, along with other type of files.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

cephfs-shell: Fix 'du' command error

This patch fixes the following:
* No error message outputed for non existing directories.
* Takes only single argument.
* When no arguments are passed, existing directories in the current directory
  are not displayed.
* Always '.' is concated to the path.

Fixes: https://tracker.ceph.com/issues/39641
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #29985 from xiexingguo/wip-41514-plus

osd/PrimaryLogPG: cancel in-flight manifest ops on interval changing; fix race with scrub

Reviewed-by: Myoungwon Oh <ommw@sk.com>

Merge pull request #30184 from tchaikov/wip-make-check

run-make-check.sh: extract run-make.sh

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #30085 from xiexingguo/wip-41601

osd/PrimaryLogPG: update oi.size on write op implicitly truncating object up

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29842 into master

* refs/pull/29842/head:
kv/KeyValueDB: fix estimate_prefix_size()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #29855 into master

* refs/pull/29855/head:
pybind/mgr: remove type conversions for ceph native options
mgr: return get_ceph_option result as typed Py object (not string)

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #30013 from ivancich/wip-beanjoy-update-attrib-tags

rgw: fix refcount tags to match and update object's idtag

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #30062 from david-z/wip-fix-test-assert

test: remove useless ASSERT_XXX macros for rgw test

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29673 from rafaelquint/imagetrash

mgr/dashboard: RBD Image Purge Trash, Move to Trash and Restore

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28755 from dzafman/wip-network

feature: Health warnings on long network ping times, add "dump_osd_network" to get a report

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

run-make-check.sh: extract run-make.sh

so we can reuse run-make.sh for building the artifact used by other
tests than "make check", for instance, dashboard's E2E test and
crimson's performance test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29402 from Devp00l/wip-40828

mgr/dashboard: Use onCancel on any modal event

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30064 from bk201/wip-41368

mgr/dashboard: fix cdEncode decorator is not working on class

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #30145 from theanalyst/doc/releases/14.2.3

doc: releases: 14.2.3 dashboard note

https://github.com/ceph/ceph/pull/24179

mgr/dashboard: Empty datatable rendered before data has been f… (#30103)

mgr/dashboard: Empty datatable rendered before data has been fetched

Reviewed-by: Patrick Seidensal <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

test: remove useless ASSERT_XXX macros for rgw test

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>

Merge pull request #29981 from falcon78921/wip-ceph-conf-usage

ceph-conf: added --show-config-value to usage

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-conf: added --show-config-value to ceph-conf usage

--show-config-value is another way to lookup ceph.conf
values. Users may want to utilize --show-config-value instead of
--lookup because --show-config-value also shows global defaults.

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>

doc: Document network performance monitoring

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #29847 from iotcg/mon_doc

doc: set ceph_perf_msgr_server arguments

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #30136 from yuvalif/build_docs_on_fedora30

build-doc: allow building docs on fedora 30

Merge pull request #26755 from cbodley/wip-rgw-bilog-complete

rgw multisite: avoid writing bilog entries on PREPARE and CANCEL

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

osd doc mon mgr: To milliseconds for config value, user input and threshold out

Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #29692 from mkogan1/wip-rgw-37615

rgw: fix cls_bucket_list_unordered() partial results

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/dashboard: Refactoring

I came across a yoda case and a test case that didn't use 'done' while
using an async call.

Fixes: https://tracker.ceph.com/issues/40828
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Use onCancel on any modal event

The problem was, that if 'onCancel' was provided, it wasn't used in all
modal exit cases. Only for the close buttons (the X and the labeled
one), but not if you clicked outside the modal or hit escape.

'onCancel' is currently only used inside the user management form, where
the problem caused, that the submit button was not longer click able if
the modal was hidden through clicking outside of it or hitting escape.

Fixes: https://tracker.ceph.com/issues/40828
Signed-off-by: Stephan Müller <smueller@suse.com>

mgr/dashboard: Add tests for confirmation modal

Fixes: https://tracker.ceph.com/issues/40828
Signed-off-by: Stephan Müller <smueller@suse.com>

doc: releases: 14.2.3 dashboard note

The comment unfortunately got buried in a resolved blog

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29973 from theanalyst/doc/releases/14.2.3

doc: release notes for v14.2.3 nautilus

Reviewed-By: Nathan Cutler <nculter@suse.com>
Reviewed-By: Neha Ojha <nojha@redhat.com>
Reviewed-By: Josh Durgin <jdurgin@redhat.com>
Reviewed-By: Lenz Grimmer <lgrimmer@suse.com>

doc: add a final set of notable changes to 14.2.3

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

build-doc: allow building docs on fedora 30

Signed-off-by: Yuval Lifshitz <yuvalif@yahoo.com>

Merge pull request #29882 from cyx1231st/wip-seastar-msgr-test-lossless-crimson

crimson/net: bug fixes from v2 failover tests

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30075 from rhcs-dashboard/fix-41600-master

mgr/dashboard: change warn_explicit to warn

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #29929 from tspmelo/wip-ng-analytics

mgr/dashboard: Prevent angular of getting stuck during installation

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #29557 from Devp00l/wip-41166

mgr/dashboard: Fix CephFS chart

Reviewed-by: Ni-Feng Chang <kiefer.chang@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #29996 from iliul/export-standby-metadata

mgr/prometheus: export standby mds metadata

Merge pull request #30122 from tchaikov/wip-40646

install-deps: revert 47d4351d

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #30119 from neha-ojha/wip-41385

osd/MissingLoc, PeeringState: remove osd from missing loc in purge_strays()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #30072 from wjwithagen/wjw-wip_split_run-rbd-unit-tests

test/librbd: allow parallel runs of run-rbd-unit-tests

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

install-deps: revert 47d4351d

this change reverts 47d4351d3ad2787ae7781d868608eb0e3c7797c7

see also 33dfffc8

Fixes: https://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #30018 from xiexingguo/wip-38483

osd: prime splits/merges for any potential fabricated split/merge participant

Reviewed-by: Sage Weil <sage@redhat.com>

test/librbd: allow parallel runs of run-rbd-unit-tests

Running all tests sequential makes it the longest test of
`make check`, with each partial test taking around 500 sec.
Running 6 tests thus takes almost an hour.

Cut this down if ctest runs tests in parallel

Default behaviour of src/test/run-rbd-unit-tests.sh is kept:
  Without parameters the tests are run in sequence

To run unitttest_librbd with RBD_FEATURES, use `N` as parameter

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #29990 from aclamk/fix-doc-ceph_kvstore_tool-stats

doc/ceph-kvstore-tool: add description for 'stats' command

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/man/ceph-kvstore-tool: fix typo

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

osd/MissingLoc, PeeringState: remove osd from missing loc in purge_strays()

We should always try to keep osds in missing_loc consistent with peer_missing
and peer_info. When we remove an osd from peer_missing and peer_info, we
should also remove it from missing_loc during purging strays.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #29639 from baixueyu/wip-crypto-bug

rgw: get barbican secret key request maybe return error code

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29776 from cbodley/wip-rgw-beast-static-flat-buffer

rgw: increase beast parse buffer size to 64k

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29852 from cbodley/wip-41395

rgw: move bucket reshard checks out of write path

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #29856 from cbodley/wip-41412

rgw: RGWCoroutine::call(nullptr) sets retcode=0

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #29894 from markhpc/wip-rgw-reshard-json

rgw/rgw_reshard: Don't dump RGWBucketReshard JSON in process_single_logshard

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #30088 from tchaikov/wip-40646

ceph.spec.in: use g++ >= 8.3.1-3.1

Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29742 from dang/push

rgw: make sure modelines are correct for all files

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29199 from tianshan/fix_gc_tag_remove

rgw: gc remove tag after all sub io finish

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29628 from NancySu05/lihuanhuan

RGW: add bucket permission verify when copy obj

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29943 from markhpc/wip-rgw-optimize-get_val2

rgw/rgw_op: Remove get_val from hotpath via legacy options

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #29654 from Songweibin/wip-rbd-list-descendants

 librbd: skip stale child with non-existent pool for list descendants

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #30113 from tchaikov/wip-dmclock

dmclock:  pick up change to use specified C++ settings if any

Reviewed-by: Casey Bodley <cbodley@redhat.com>

dmclock:  pick up change to use specified C++ settings if any

see https://github.com/ceph/dmclock/pull/73

so dmclock won't add `-std=c++11` to CXX_FLAGS even if the parent project already
have `-std=c++17` in CXX_FLAGS.

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/mgr: remove type conversions for ceph native options

We now return the correct Py types, just like we do for module options.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: return get_ceph_option result as typed Py object (not string)

We already return module options as the proper Py type; do the same for
native ceph options.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Prevent angular of getting stuck during installation

During npm packages installation Angular was getting stuck waiting for input.
Using "NG_CLI_ANALYTICS=false" will prevent that,

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29613 from trociny/wip-41206

librbd: behave more gracefully when data pool removed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #29835 into master

* refs/pull/29835/head:
osd/PeeringState: fix wrong history of merge target

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/dashboard: Empty datatable rendered before data has been fetched

Fixes: https://tracker.ceph.com/issues/41615
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #30097 from jan--f/ceph-backport-cp-indiviudal-commits

ceph-backport.sh: cherry-pick individual commits

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #30063 from Rjerk/wip-silence-unused-warning

rgw/services: silence -Wunused-variable warning

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30065 from tchaikov/wip-rgw-test-wo-global

test: do not include unnecessary includes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

ceph-backport.sh: cherry-pick individual commits

In git 2.23.0 (possibly earlier too) cherry-pick seems to interact with
rev-list and submodules in unexpected ways, where the cherry-pick of a
commit range (<sha1>..<sha1>) fails with "changes would be overwritten"
in a submodule. CP'ing individual commits avoids this.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #30060 from c744402859/master

rgw: fixed "unrecognized arg" error when using "radosgw-admin zone rm".

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30091 from joke-lee/doc-rgw-object-version

rgw: add --object-version in radosgw-admin help info

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29957 from smithfarm/wip-ceph-backport-sh-improvements

script/ceph-backport.sh: wholesale refactor

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

doc/ceph-kvsore-tool: add description for 'stats' command

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #29665 from rishabh-d-dave/cv-remove-redundant-calls

ceph-volume: minor optimizations related to class Volumes's use

mgr/dashboard: fix cdEncode decorator is not working on class

Object.keys does not return all methods of a class.

Fixes: https://tracker.ceph.com/issues/41368
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

ceph.spec.in: use g++ >= 8.3.1-3.1

since https://bugzilla.redhat.com/show_bug.cgi?id=1726630 has been
fixed. and we have devtoolset-8-gcc-c++-8.3.1-3.1.el7.x86_64.rpm.

no reason to stick with devtoolset-8-gcc-c++-8.2.1-3 anymore.

Fixes: https://tracker.ceph.com/issues/40646
Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: add --object-version in radosgw-admin help info

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #30086 from uniqueyehu/fix-complie-errors-dpdk

msg/async/dpdk: fix complie errors from fix FTBFS

Reviewed-by: Kefu Chai <kchai@redhat.com>