qa/tests: reduced distro to run to be random

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #28313 from Songweibin/wip-fix-get-args

rbd/action: fix error getting positional argument

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28253 from Songweibin/wip-39973-rbd

osd/OSDCap: rbd profile permits use of rbd_info

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #27902 from mikechristie/rbd-nbd-netlink-support

rbd-nbd: add netlink map/unmap support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28370 from linuxbox2/wip-rgw-tntbucket

rgw_file: include tenant when hashing bucket names

mgr/dashboard: show degraded/misplaced/unfound objects. (#28104)

mgr/dashboard: show degraded/misplaced/unfound objects.

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28039 from majianpeng/make-tcmalloc/jemalloc-work-well

make perf_async_msgr link jemalloc/tcmalloc

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28373 from xiexingguo/wip-40104

mon, osd: parallel clean_pg_upmaps

Reviewed-by: Sage Weil <sage@redhat.com>

test/crimson: make perf_async_msgr link tcmalloc/jemalloc.

For 1M performance, perf_async_msgr w/ tcmalloc/jemalloc is better than
libc.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

test: add parallel clean_pg_upmaps test

With parallel clean_pg_upmaps feature on, the total time cost
of the performance test which now can utilize up to 8 threads for
parallel upmap validating decreased from:

maybe_remove_pg_upmaps (~10000 pg_upmap_items) latency:104s

to:

clean_pg_upmaps (~10000 pg_upmap_items) latency:7s

Note that by default the mon uses only 4 worker threads for
CPU intensive background work, you could further increase
the "mon_cpu_threads" option value if you decided the
time-consuming of clean_pg_upmaps still matters.

Fixes: http://tracker.ceph.com/issues/40104
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mon/OSDMonitor: do clean_pg_upmaps the parallel way if necessary

There could definitely be some certain cases we could reliably
skip this kind of checking, but there is no easy way to separate
those out.
However, this is clearly the general way to do the massive pg
upmap clean-up job more efficiently and hence should make sense
in all cases.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

osd/OSDMap: split clean_pg_upmaps into smaller helpers

- it's good to read.
- the updating pending_inc part should be made independent
  since it is going to be racy while running in parallel.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28402 from tchaikov/wip-crimson-cleanup

crimson/osd: some cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28302 from dzafman/wip-40078

test: Make sure that extra scheduled scrubs don't confuse test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw_file: pretty-print fh_key

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: include tenant in hashes of object

Because bucket names are taken as object names in the top
of an export.  Make hashing by tenant general to avoid disjoint
hashing of bucket.

Fixes: http://tracker.ceph.com/issues/40118
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw_file: include tenant when hashing bucket names

Prevent identical paths from distinct tenants from colliding in
RGW NFS handle cache.

Fixes: http://tracker.ceph.com/issues/40118
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #28353 into master

* refs/pull/28353/head:
qa/suites/rados/multimon: skew clocks 2s (< paxos lease)
qa/suites/rados/multimon: create_rbd_pool: false
qa/suites/rados/multimon: no osds when skewing clock

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #28365 into master

* refs/pull/28365/head:
global/global_context: always add '\0' after strncpy()

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28391 from cbodley/wip-39984

cls/rgw: test before accessing pkeys->rbegin()

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

crimson/osd/osd.cc: add editor variables/options

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: s/ostream/std::ostream/

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: mark one of PG::do_peering_event()s private

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27995 from richael02/fix_indent

vstart.sh: unify the indent

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28358 from liu-chunmei/ceph_seastar_futurized_store

crimson: add FuturizedStore to encapsulate CyanStore

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27735 from xiexingguo/wip-device-class-noout

osd: revamp {noup,nodown,noin,noout} related commands

Reviewed-by: Sage Weil <sage@redhat.com>

rbd nbd: always try to load module on map

nl and ioctl both need the module loaded so just always load it during
mapping.

Signed-off-by: Mike Christie <mchristi@redhat.com>

rbd nbd: add support for netlink interface

This adds support for mapping and unmapping using the netlink
interface. If the user passes in --try-netlink during mapping then we
will try to use the netlink interface and if that is not supported use
the ioctl interface. For unmap we will always try netlink and if that is
not supported drop down to the ioctl.

Signed-off-by: Mike Christie <mchristi@redhat.com>

cmake and build: add netlink lib requirement detection

The next patch adds netlink support to rbd-nbd. This patch just adds the
build related changes to bring in the netlink lib and test and build
against it.

Signed-off-by: Mike Christie <mchristi@redhat.com>

cmake: Move WITH_RBD option earlier

The next patch will add a WITH_RBD check to the LINUX check block so we
only check for the netlink libs when those 2 conditions are met, so this
moves the WIT_RBD option setup to before the LINUX check.

Signed-off-by: Mike Christie <mchristi@redhat.com>

rbd nbd: Add cond to signal nbd disconnection

With the ioctl interface we know when a disconnect has been executed
because the NBD_DO_IO ioctl will return. The netlink interface returns
after the nl connect cmd call has completed, so this patch adds a cond
that signals when a nl disconnect/NBD_DISCONNECT has been executed and
the server thread has handled the resulting NBD_CMD_DISC.

Signed-off-by: Mike Christie <mchristi@redhat.com>

rbd nbd: just move signal handler

This just moves the signal handler to make it easier to call the netlink
disconnect helpers in the next patches.

Signed-off-by: Mike Christie <mchristi@redhat.com>

rbd nbd: move nbd index parsing to function

Make the nbd index parsing into a functio so we can use in the ioctl and
netlink paths.

Signed-off-by: Mike Christie <mchristi@redhat.com>

rbd nbd: move server startup code to function

The netlink connect cmd interface requires the server to be running when
the connect call completes or commands will timeout/fail. This breaks the
server code to 2 functions so we can control when it starts and runs.

Right now it looks odd in that start_server will allocate and return
the server only for it be freed after run, and it is not touched. The
netlink enablement patch at the end will use the returned server.

Signed-off-by: Mike Christie <mchristi@redhat.com>

rbd nbd: move nbd map ioctls to new function

The next patches will add nbd map netlink support so this moves the ioctl
code to a new function to keep things cleaner and not overloading do_map.

Signed-off-by: Mike Christie <mchristi@redhat.com>

Merge pull request #28366 from albert-github/feature/bug_docu_formula_invocation

common/hobject: Error invocation of formula in documentation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27994 from xiexingguo/wip-balancer-fudge

mgr/balancer: fix fudge

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28342 from rzarzynski/wip-messages-std-for-pgcreate2

messages: MOSDPGCreate2 doesn't assume using namespace std.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28340 from elliswuuuuuuuu/remove_unused_variable

mgr: remove unused variable pool_name

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28268 from liewegas/wip-progress-bar

mon: fix off-by-one rendering progress bar

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

vstart.sh: add editor variables/options

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add an extra param when "run 'mds'"

Signed-off-by: Kefu Chai <kchai@redhat.com>

vstart.sh: unify the indent

Change-Id: Ie1dbe25f81609e5b2f6751da9e87e191a2f78bc2
Signed-off-by: Richael Zhuang <richael.zhuang@arm.com>

osd/OSDMapMapping: make ParallelPGMapper can accept input pgs

The existing "prime temp" machinism is a good inspiration
for cluster with a large number of pgs that need to do various
calculations quickly.
I am planning to do the upmap tidy-up work the same way, hence
the need for an alternate way of specifying pgs to process other
than taking directly from the map.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #28324 into master

* refs/pull/28324/head:
client: clean up error checking and return of _lookup_parent
client: drop Client::lookup_parent

Reviewed-by: huanwen ren <ren.huanwen@zte.com.cn>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #28116 into master

* refs/pull/28116/head:
client: fix "ceph.snap.btime" vxattr value
client: fix "ceph.dir.rctime" vxattr value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28371 into master

* refs/pull/28371/head:
qa: fix kclient symlink to use latest rhel7
qa: fix latest distros and add helper symlinks

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #28148 into master

* refs/pull/28148/head:
common/secret.c: fix key parsing when doing a remount

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

crimson: add FuturizedStore to encapsulate cyanstore

only add encapsulation. alien store will be implement later.
exception handling will be implement later.

Signed-off-by: chunmei Liu <chunmei.liu@intel.com>

Merge pull request #26415 from zmc/wip-populate-grafana

mgr/dashboard: Push Grafana dashboards on startup

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #28294 from alfredodeza/wip-rm40063

ceph-volume: pass --ssh-config to pytest to resolve hosts when connecting

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #28228 from jan--f/spec-grafana-dashboards-require-grafana

ceph.spec.in: grafana-dashboards package depends on grafana

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

cls/rgw: test before accessing pkeys->rbegin()

if pkeys is empty here, dereferencing rbegin() will crash

Fixes: http://tracker.ceph.com/issues/39984
Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Push Grafana dashboards on startup

Because we didn't have viable way to populate Grafana dashboards in a
fully containerized context, we needed a different method. This will
work in packaged and containerized deployments.

We provide three methods to push the dashboard JSON files to the Grafana
instance: a command (ceph dashboard update-grafana-dashboards); an API
endpoint (/api/grafana/update_dashboards); and an automatic on-startup
feature that is disabled by default (ceph dashboard
set-grafana-update-dashboards true). The on-startup method will also
retry periodically to avoid racing with the startup of Grafana itself.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #28385 from theanalyst/doc/releases/13.2.6

doc: add release notes for v13.2.6 mimic

Reviewed-By: Sage Weil <sage@redhat.com>

Merge PR #28236 into master

* refs/pull/28236/head:
doc: Fix incorrect indentations in cephfs-shell
doc: Add usage for stat command in cephfs-shell

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28386 from cbodley/wip-vstart-rgw-run

vstart: fix run() invocation for rgw

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

vstart: fix run() invocation for rgw

add an extra argument to run() for rgw so that --redirect-output
sends stdout to rgw.{port}.stdout

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #28384 into master

* refs/pull/28384/head:
mgr/volumes: fix subvolume creation with quota

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/volumes: fix subvolume creation with quota

A quota is set on a subvolume by setting a quota virtual
xattr on a fs subdirectory. The value of the quota xattr
should be a bytes object. The quota value wasn't correctly
converted to a bytes object. Fix this.

Fixes: http://tracker.ceph.com/issues/40152
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #28252 from tchaikov/wip-cmake-cleanup

cmake: cleanups

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #28380 into master

* refs/pull/28380/head:
mgr/volumes: convert string to bytes object

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

doc: add release notes for v13.2.6 mimic

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

common/hobject.h: fix syntax in doxygen comment

fix the error invocation of formula in documentation

Error invocation of formula in documentation, the second formula was
started with $f instead of the correct \f$

see also http://www.doxygen.nl/manual/formulas.html

Signed-off-by: Albert <albert.tests@gmail.com>

mgr/volumes: convert string to bytes object

mount method in libcephfs's cython modules expects filesystem_name to
be a bytes object. Therefore, convert string to bytes object
beforehand.

Fixes: http://tracker.ceph.com/issues/39750
Signed-off-by: Rishabh Dave <ridave@redhat.com>

cmake: extract rgw_*_scheduler.cc into rgw_schedulers

i was running into following issue:

../../../lib/libradosgw_a.a(rgw_loadgen_process.cc.o): In function
`RGWLoadGenProcess::run()':
/home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/rgw_loadgen_process.cc:102:
undefined reference to `signal_shutdown()'
collect2: error: ld returned 1 exit status

signal_shutdown() is only used by rgw_main.cc to notify the main thread,
but we don't need this facility in unit test exercising the schedulers,
so, in this change, rgw schedulers are extracted into librgw_schedulers,
so this library can be consumed by the unit test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28243 from cbodley/wip-40022

radosgw-admin: fix syncs_from in 'bucket sync status'

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #28156 from cbodley/wip-rgw-realm-list-api

rgw: add 'GET /admin/realm?list' api to list realms

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #28345 from tianshan/fix_http_return_check

rgw: fix miss handle curl error return

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28303 from cbodley/wip-39992

rgw: fetch_remote_obj() compares expected object size

Reviewed-by: Xiaoxi Chen <xiaoxchen@ebay.com>

Merge pull request #28188 from tianshan/fix_39984

rgw: fix bucket may redundantly list keys after BI_PREFIX_CHAR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27993 from liuchang0812/rgw-bucket-tagging

rgw: bucket tagging

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: cleanup bucket tags code

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: show bucket tagging when user stats bucket

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

test/rgw: test truncated reads in RGWHTTPManager

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix miss handle curl error return

if meet tcp packet loss, curl return will be result=18(CURLE_PARTIAL_FILE)
and http_status=200, so sync will continue and cause content miss match.

Fixes: https://tracker.ceph.com/issues/39992
Signed-off-by: Tianshan Qu <tianshan@xsky.com>

Merge pull request #28362 from cyx1231st/wip-seastar-fix-random-failure

test/crimson: fix msgr test of ref counter racing

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson: fix msgr test of ref counter racing

Foreign-dispatch in crimson-messenger is not implemented yet, and no
need to test this feature now.

Fixes: http://tracker.ceph.com/issues/36405
Signed-off-by: Yingxin Cheng <yingxincheng@gmail.com>

rgw: simplify RGWObjTags dump

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

doc: bucket tagging is supported now

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: forwards to master and uses gw_bucket_set_attrs

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: make max number of bucket/object tags configurble

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: iam policy about bucket tagging

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: impl put_bucket_tags and delete_bucket_tags

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: add set_bucket_tags api

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: uses 404 http code when there is NoSuchTagSetError

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: get bucket tagging

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw: declares bucket tagging op classes

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

rgw:new op type and error code about bucket tagging

Signed-off-by: Chang Liu <liuchang0812@gmail.com>

qa: fix kclient symlink to use latest rhel7

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix latest distros and add helper symlinks

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

client: clean up error checking and return of _lookup_parent

ll_lookup_inode can end up getting back 0 from _lookup_parent,
without zeroing out the parent pointer, which ends up remaining
uninitialized.

Fix this by moving most of the sanity checks in _lookup_parent
into ll_lookup_inode, and only have it issue the call to the MDS.

This also allows us to do the checks in a more sane order.

Fixes: http://tracker.ceph.com/issues/40085
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge PR #26547 into master

* refs/pull/26547/head:
Added validation of zabbix_host to support hostnames, IPv4 and IPv6.
mgr/zabbix: Documentation added.
mgr/zabbix: Adds possibility to send data to multiple zabbix servers.

Reviewed-by: Wido den Hollander <wido@42on.com>

Merge PR #27667 into master

* refs/pull/27667/head:
src/test: fix creating two different objects for testing chunked object

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28200 into master

* refs/pull/28200/head:
pybind/mgr: add slash to prometheus illegal metric name characters
pybind/mgr: fix format for rbd-mirror prometheus metrics

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28180 from athanatos/sjust/wip-crimson-peering

Integrate PeeringState into crimson, fix related bugs

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28068 from cbodley/wip-qa-rgw-s3a

qa/rgw: don't use ceph-ansible in s3a-hadoop suite

Reviewed-by: Vasu Kulkarni <vasu@redhat.com>

Merge PR #28300 into master

* refs/pull/28300/head:
doc: cephfs: add section on fsync error reporting to posix.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

client: drop Client::lookup_parent

Nothing calls this method.

Signed-off-by: Jeff Layton <jlayton@redhat.com>

cmake: link rbd_reply against rbd_replay_types

should make it `PUBLIC` because, for instance,

src/rbd_replay/actions.hpp uses
`rbd_replay::action::IoActionBase::dump()` and
`rbd_replay::action::ActionBase::dump()`

Signed-off-by: Kefu Chai <kchai@redhat.com>