cephfs_features: add metric collection feature support

In client side we need to known whether mds support the metric
collection or not, or when mds server receieves the metric message
from client side and if it couldn't recognise, the mgr will close
the socket connection directly.

Fixes: https://tracker.ceph.com/issues/46041
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #35544 from neha-ojha/wip-reduce-noise

qa/tasks/ceph_manager.py: dump more useful info before failing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35545 from LenzGr/doc-troubleshooting-community

doc: Updated mailing list addresses

doc: Updated mailing list addresses

The "Community" chapter of the RADOS troubleshooting section
was still pointing to the old mailing list addresses.

Updated to point to the new ceph.io addresses.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

qa/tasks/ceph_manager.py: dump more useful info before failing

Add helpers that dump information only about PGs that haven't reached
the desired state when we fail. Previously we dumped the output of
"ceph pg dump" before failing, which prints a lot of unnecessary information
about PGs that are not responsible for the failure, making debugging harder.

Also, try to make the failure messages distinct.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34380 from jdkruzr/patch-1

doc/man: Update mount.ceph.rst for clarity

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35504 from zdover23/wip-doc-cephadm-ssh-key-setup-2020-Jun

doc/cephadm: Importing existing ssh keys

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35526 from tchaikov/wip-qa-krbd-whitelist-mon-down

qa/suites/krbd: whitelist MON_DOWN health warning

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #35521 from sebastian-philipp/dashboard-mgr-module-option-default-value-none-for-it-or-float

mgr/dashboard: fix None value for ints and floats

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35318 from ifed01/wip-ifed-fix-ex-frag-test

test/objectstore/store_test: kill ExcessiveFragmentation test case.

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #35502 from wjwithagen/wjw-fix-cls-le32

cls: fix missing __le32 definition

Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #35486 from jdurgin/wip-radosbench-qa

qa/tasks/radosbench: use long form of option for compatibility

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #35258 from majianpeng/bluefs-flush-use-force

os/bluestore/BlueFS: force flush when call flush().

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #35478 from idryomov/wip-populate-watchers-replica

osd/PrimaryLogPG: don't populate watchers if replica

Reviewed-by: Samuel Just <sjust@redhat.com>

qa/suites/krbd: whitelist MON_DOWN health warning

see also

- 93de19adcf1233ce4c68f1253e0b9abef6d97a9d
- 608e002195638e80323780f1907db40c0b9768f0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #35530 into master

* refs/pull/35530/head:
vstart: set $CEPH_CONF when running ganesha-rados-grace

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

vstart: set $CEPH_CONF when running ganesha-rados-grace

...otherwise we could pick up the wrong ceph.conf file. Also, use the
correct $test_user userid for these ops.

Fixes: https://tracker.ceph.com/issues/45971
Signed-off-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #35459 from sebastian-philipp/orch-apply-unknown-type

mgr/orch: apply: Fix proper error when passed unknown type

Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: pylint: reduce number of branches

fixes: `R0912: Too many branches (14/12) (too-many-branches)`
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: fix None value for ints and floats

Fixes: https://tracker.ceph.com/issues/45963
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35007 from rhcs-dashboard/fix-36354-master

mgr/dashboard: work with v1 RBD images

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35371 from yuriw/wip-yuriw-crontab-master

qa/tests: Changed -t from `py2` to default for `octopus`

Merge pull request #35455 from Conan-Kudo/ceph-libexec

rpm, deb: Use /usr/libexec for libexecdir whenever possible

Reviewed-by: Nathan Cutler <ncutler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35357 from yison/rwl-fix-deadlock-on-ut-issue

rbd/cache: fix deadlock on RWL unit test

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35425 from dzafman/wip-44314

test: osd-backfill-stats.sh use nobackfill to avoid races in remainin…

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/radosbench: use long form of option for compatibility

Since the short version of --object-size changed from -o to -O, it
does not work with upgrade tests.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Revert "qa/tasks/radosbench: only set object size if != block size"

This reverts commit fb991fa5f62c4aaa6c564a6d15aeee4bfaa33b39.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
qa/tasks/radosbench.py - extra context from -t, trivial resolution

Revert "qa/tasks/radosbench: fix usage of -O"

This reverts commit da19fd14dbe8bdb62ce376d9df7bf5578f169b9c.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>
Conflicts:
qa/tasks/radosbench.py - concurrency arg added in context, trival resolution

Merge pull request #35099 from sebastian-philipp/doc-orch-update-specs

doc/orch: How to update an existing specification

Reviewed-by: Alexandra Settle <asettle@suse.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

cls: fix missing __le32 definition

On platforms that do not the __le* definitions it throws a compiler error.

fixes: https://github.com/ceph/ceph/pull/35186
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #35506 from rzarzynski/wip-crimson-thread-comment-fix

crimson/thread: fix comment phrasing in ThreadPool.h.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34492 from xxhdx1985126/wip-crimson-recovery-block-io-pr

crimson/net: fix system shutdown problems

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: Import existing ssh key docs.

This commit supersedes the commit below, which was suggested by
mohnewald:

https://github.com/ceph/ceph/pull/34453/commits/1cd278b08a342df1acf9b43dbc5c27ee686149c0

Co-author: https://github.com/mohnewald

Signed-off-by: Zac Dover <zac.dover@gmail.com>

test/objectstore/store_test: kill ExcessiveFragmentation test case.
This test case was introduced by https://github.com/ceph/ceph/pull/18494
to verify allocation failure handling while gifting during bluefs rebalance
Not it looks outdated as there is no periodic gifting any more.

Fixes: https://tracker.ceph.com/issues/45788
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

crimson/thread: fix comment phrasing in ThreadPool.h.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35423 from scoopex/master

tools/histogram_dump: improve io-latency and -size statistics

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35187 from CodingSpiderFox/patch-3

doc: Introduce the concept of a pool before talking about it.

Merge pull request #32705 from petrutlucian94/windows.3

include,common: Windows support [part 3]

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35422 from adamemerson/wip-45878

osdc: Fix crash on OSD failure

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33720 from yunfeiguan/wip-44409

log: fix timestap precision of log can't set to millisecond.

Reviewed-by: Adam Emersen <aemerson@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #34667 from aclamk/wip-rocksdb-reshard

Resharding tool for sharded rocksdb

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #35186 from liewegas/wip-cas

cls_cas: misc cleanup, and dynamic chunk_refs_t

Reviewed-by: Samuel Just <sjust@redhat.com>

tools/histogram_dump: improve io-latency and -size statistics

The script is part of the ceph distribution but not
part of the official tooling. Nevertheless it was useful to understand
the usage scenarios of my osds.

- add colors to the output to ease interpretation
- add possibility to specify one ore more sockets
- aggregate statistics over multiple sockets
- prevent stacktrace if tool is interrupted by SIGINT
- add possibility to specify the number of data collection
  seconds between loops
- raise loop_seconds default

Signed-off-by: Marc Schoechlin <ms-github@256bit.org>

doc/orch: How to update an existing specification

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34633 from sebastian-philipp/cephadm-total-scheduler

mgr/cephadm: make scheduling make sense

Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35460 from agayev/fix-cmake

blk: Add missing PUBLIC keyword to target_link_libraries command.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35095 from tchaikov/wip-kefu-onode-v1

crimson/os: add onode node implementation

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os: add OnodeBlock

used for representing the an node of onode tree

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: add crimson::gtest target

instead re-compiling gtest_seastar.cc, better off creating a static
library and link against it instead.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson: add node implementation for onode tree

* node_t and friends are representations of in-mem and on-disk onode
  block/page/node. it might have different names in different contexts.
  but normally, they are the same thing and are loaded as a logical
  cached extent.
* delta_t holds the mutations against a given node onode. they are
  applied to memory once they are committed to disk as part of the
  log record.

regarding to the name of "simple-fltree", it is a short for simple
Flexible Layout Fixed Tree, which was designed by Samuel Just. My
implementation is a naive implementation of this design. But it
tries to be extensive and should be able to adapt to different policies
for rebalancing nodes when overflow/underflow. But there could be
other variant of flexible layout tree implementations in future,
so i added "simple" prefix.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: extract onode.h out

so the concrete onode manager implementation can include it without
including the onode_manager.h interface as a whole

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: add ObjectStore and OnodeManager stubs

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #33878 from theanalyst/rgw/cleanup/user-policy-append

rgw: append user policies to existing policies

Reviewed-by: Pritha Srivastava <prsivas@redhat.com>
Reviewed-by: Adam Emersen <aemerson@redhat.com>

Merge pull request #33521 from linuxbox2/wip-getset-ver-nosuch

rgw: fail when get/set-bucket-versioning attempted on a non-existent …

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #33511 from theanalyst/rgw/append-gc-fixes

rgw: append obj: prevent tail from being GC'ed

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33492 from prasakri/multisite-sync-check

RGW:Multisite: Verify if the synced object is identical to source

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #35470 into master

* refs/pull/35470/head:
mds: fix compile error with CephContext declaration

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests: removed ceph-deploy suite run on mimic as EOL

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge branch 'master' of https://github.com/ceph/ceph into wip-yuriw-crontab-master

Merge PR #35344 into master

* refs/pull/35344/head:
vstart: Support deployment of ganesha daemon by cephadm with NFS option.
mgr/test_orchestrator: List nfs daemon

Reviewed-by: Ramana Raja <rraja@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

debian: Use /usr/libexec for libexecdir

Debian has adopted FHS 3.0 with Debian Policy Standard version 4.4.0,
and now defaults to /usr/libexec for libexecdir as well. This is
present in Ubuntu 20.04 LTS, for example.

Signed-off-by: Neal Gompa <ngompa13@gmail.com>

rpm: Drop override on %_libexecdir

We no longer need it, as distributions are converging on /usr/libexec
since FHS 3.0 has specified it as the acceptable path for libexecdir.

Fedora has been using /usr/libexec since the beginning, and
openSUSE has accepted software installed into /usr/libexec since
September 2019 in openSUSE:Factory. openSUSE Leap 15.x still uses
/usr/lib for %_libexecdir, but the rest of the spec should handle the
change gracefully.

Signed-off-by: Neal Gompa <ngompa13@gmail.com>

ceph.spec.in: use %{_presetdir} for preset settings

the systemd preset settings should not be located under /usr/libexec
```
$ rpm --eval "%{_presetdir}"
/usr/lib/systemd/system-preset
```

see also
https://www.freedesktop.org/software/systemd/man/systemd.preset.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

rpm: Add missing CMake parameter to path to systemd unit dir

This mimics the setting passed to CMake in other package builds,
and ensures that systemd units are installed into the right place.

Signed-off-by: Neal Gompa <ngompa13@gmail.com>

Merge pull request #35298 from tspmelo/wip-ng-progressbar

mgr/dashboard: Use ng-bootstrap for Progressbar

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #33083 from mkogan1/wip-rgw-add-beast-access-logs

rgw: add access log to the beast frontend

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

src: Rename "zns" directory to "zoned".

"zoned" is a more appropriate name for both HDDs and SSDs that support the zone
interface.  ZNS is an NVMe standard for zoned SSDs only.  Zoned HDDs, on the
other hand, use a SATA standard called ZBC/ZAC.

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>

src: Add missing PRIVATE keyword to target_link_libraries command.

This fixes the following cmake error that happens if you have libzbc installed:

CMake Error at src/blk/CMakeLists.txt:40 (target_link_libraries):
  The keyword signature for target_link_libraries has already been used with
  the target "blk".  All uses of target_link_libraries with a target must be
  either all-keyword or all-plain.

  The uses of the keyword signature are here:

   * src/blk/CMakeLists.txt:32 (target_link_libraries)

Signed-off-by: Abutalib Aghayev <agayev@cs.cmu.edu>

Merge pull request #35379 from votdev/issue_45856_show_host_labels

mgr/dashboard: Show labels in hosts page

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mds: fix compile error with CephContext declaration

SimilarTo: https://github.com/ceph/ceph/pull/35209
Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

rgw: add access log to the beast frontend

Add to the Beast frontend an access log line similar to CivetWeb.
attempting to adhere as much as possible to the Apache Combined Log
Format.

Fixes: https://tracker.ceph.com/issues/45920
rgw: use beast message for access log
(cherry picked from commit 44ec38933cbe6cc864c56332cc99502d86568fdc)

Co-authored-by: Casey Bodley <cbodley@redhat.com>
Signed-off-by: Mark Kogan <mkogan@redhat.com>

osd/PrimaryLogPG: don't populate watchers if replica

If an object has an established watch, serving a read from replica
eventually leads to an assert in is_degraded_or_backfilling_object()
called from handle_watch_timeout().

The issue is that after can_serve_replica_read() check is satisfied,
we look up the object context.  If not found, we fetch the object info
and then call populate_obc_watchers() which sees the recorded watch and
treats it as unconnected, arming HandleWatchTimeout on replica.

Fixes: https://tracker.ceph.com/issues/45795
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #35381 from tspmelo/wip-fix-cephfs-directories

mgr/dashboard: Fix CephFS's Directories not showing

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35373 from bk201/wip-45870

mgr/dashboard: remove space after service name in the Hosts List table

Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #35457 from tchaikov/wip-rpm-unitdir

test/crimson: avoid using variadic future

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

test/crimson: avoid using variadic future

see also 3cee25ba41069e9e9b4855f99be20910759440ca

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: apply: Fix proper error when passed unknown type

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35405 from ivancich/wip-fix-truncated-bucket-listing

rgw: bucket list/stats truncates for user w/ >1000 buckets

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #33762 from clwluvw/wildcard-subuser

rgw: Add support wildcard subuser for bucket policy

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #32393 from clwluvw/opa-access-key

rgw: Add subuser to OPA request

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35203 from linuxbox2/wip-lc-perfctr

rgwlc: add lifecycle perf counters

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35073 from cbodley/wip-45382

rgw: add quota enforcement to CopyObj

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Or Friedmann <ofriedma@redhat.com>

Merge pull request #34041 from mdw-at-linuxbox/wip-rgw-swstat-loops-2

rgw: fix loop problem with swift stat on account.

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: convert rgw_user_max_buckets to new config options

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: clean up code around buckets listing

This commit adds some const correctness, avoids some string/structure
copying, and reformats some code.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

rgw: bucket list/stats truncates for user w/ >1000 buckets

The is_truncated flag was not being handled properly and therefore
limited the output on these radosgw-admin commands to 1000 buckets.

Fixes: https://tracker.ceph.com/issues/45630
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

test: osd-backfill-stats.sh use nobackfill to avoid races in remaining test

Fixes: https://tracker.ceph.com/issues/44314
Signed-off-by: David Zafman <dzafman@redhat.com>

Merge pull request #35380 from sebastian-philipp/cephadm-logs-daemon-id

cephadm: More robust way to deduce the systemd unit

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35411 from sebastian-philipp/adopt-9cd9ad020d93b0b420924fec55da307aff8bd422

qa/cephadm: update adopt corpus

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35372 from mgfritch/orch-service-spec-mypy

python-common: add ServiceSpec type checking

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

osdc: Fix crash on OSD failure

OSDSession::lock should not use lockdep, as we violate its rules in a
place that's safe. Revert it to std::mutex.

Thanks to Or Friedmann <ofriedma@redhat.com> for finding the problem.

Fixes: https://tracker.ceph.com/issues/45878
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #34275 from pritha-srivastava/wip-rgw-sts-oidc-provider

rgw: adds code for creating and managing oidc provider entities in rgw and for offline validation of OpenID Connect Access and ID Token.

rgw: adding code for policy evaluation for ops
like getbucketversioning, putbucketversioning etc

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: Correcting pool name for oidc provider for older
versions of osd.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: fix for make failure in test folder.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds code for role session to be used an ARN.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adding new iam ops for iam oidc provider
to correct unittest faliure.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: modifying documentation to include explanation of
open id connect provider related REST APIs and removing
references to token introspection for validating incoming
web token.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds a commom method to remove prefix(www, http and https)
from a url.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

rgw: adds code to check for client id and cert thumbprint
while validating incoming web token in AssumeRoleWithWebIdentity.

Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>