mgr/cephadm: tolerate old host inventory without 'hostname' key

Older cephadm clusters lack the 'hostname' key in the host spec.  e.g.,

 "cpach": {"addr": "cpach", "labels": ["mon"]}, "eutow": {"addr": "eutow", "labels": ["mon"]}, "stud": {"addr": "stud", "labels": ["mon"]}}

Populate hostname from the dict key if necessary for compatibility.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #38768 from votdev/fix_path_to_dash_devel_doc

mgr/dashboard: Fix path to dash_devel.rst

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38790 into master

* refs/pull/38790/head:
common: remove dead option

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38788 into master

* refs/pull/38788/head:
mount.ceph: collect v2 addresses for non-legacy ms_mode options

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38640 into master

* refs/pull/38640/head:
qa: add test for reserved keyword feature
qa: use no client for required client feature tests
mds: do not allow setting a reserved feature by name
mds: return sv for efficiency

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #38766 from sebastian-philipp/cephadm-service-action-async

mgr/cephadm: make `ceph orch {restart|...}` asynchronous

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #38255 from linuxbox2/wip-fio-librgw

rgw:fio: introduce libfio_rgw

Merge pull request #36120 from smithfarm/wip-rpath-ceph-diff-sorted

cmake: add empty RPATH to ceph-diff-sorted

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #38671 from rhcs-dashboard/48685-upgrade-grafana

mgr/dashboard/monitoring: upgrade Grafana version due to CVE-2020-13379

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: b-ranto <branto@redhat.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: p-se <pseidensal@suse.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #38108 into master

* refs/pull/38108/head:
doc, man: man page for `cephfs-top` utility
doc: document `cephfs-top` utility
test: selftest for `cephfs-top` utility
spec, deb: package cephfs-top utility
cephfs-top: top(1) like utility for Ceph Filesystem
mgr/stats: include kernel version (for kclients) in `perf stats` command output
mgr/stats: include version with `perf stats` output

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38836 into master

* refs/pull/38836/head:
client: add debug messages for osdmap wait

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #38141 from Daniel-Pivonka/nodeexporterpullauth

cephadm: fix podman failure to pull authenticated registry image from systemd unit

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38712 from orozery/librbd-encryption-api

librbd: encryption API

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cmake: add empty RPATH to ceph-diff-sorted

This fixes a transient FTBFS on openSUSE:

[ 5365s] -- Installing: /home/abuild/rpmbuild/BUILDROOT/ceph-15.2.4.337+g55cec95eaf-1.1.x86_64/usr/bin/ceph-diff-sorted
[ 5365s] CMake Error at src/tools/cmake_install.cmake:230 (file):
[ 5365s]   file RPATH_CHANGE could not write new RPATH:
[ 5365s]
[ 5365s]     /usr/lib64/ceph
[ 5365s]
[ 5365s]   to the file:
[ 5365s]
[ 5365s]     /home/abuild/rpmbuild/BUILDROOT/ceph-15.2.4.337+g55cec95eaf-1.1.x86_64/usr/bin/ceph-diff-sorted
[ 5365s]
[ 5365s]   No valid ELF RPATH or RUNPATH entry exists in the file;

Fixes: https://tracker.ceph.com/issues/46553
Signed-off-by: Nathan Cutler <ncutler@suse.com>

librbd: add encryption api

This commit exposes librbd encryption api

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd: change remap_extents to lvalue reference

This commit changes the remap_extents api to use lvalue instead of ravlue references

Signed-off-by: Or Ozeri <oro@il.ibm.com>

librbd: fix wrong delete type in DataCryptor

This commit fixes a delete which does not match its allocation

Signed-off-by: Or Ozeri <oro@il.ibm.com>

doc, man: man page for `cephfs-top` utility

Fixes: http://tracker.ceph.com/issues/48602
Signed-off-by: Venky Shankar <vshankar@redhat.com>

doc: document `cephfs-top` utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: selftest for `cephfs-top` utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

spec, deb: package cephfs-top utility

Signed-off-by: Venky Shankar <vshankar@redhat.com>

cephfs-top: top(1) like utility for Ceph Filesystem

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/stats: include kernel version (for kclients) in `perf stats` command output

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/stats: include version with `perf stats` output

So that `fstop` infers the version of mgr/stats to parse the
JSON command output.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #38804 from sebastian-philipp/cephadm-silence-evict-container

cephadm: silence "Failed to evict container" log msg

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #38615 from adk3798/haproxy-stuff

mgr/cephadm: HA for RGW endpoints

Reviewed-by: Daniel-Pivonka <dpivonka@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix path to dash_devel.rst

Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #38543 from zjcmszh/cmpext_op

pybind/rados: add WriteOp::cmpext() and ReadOp::cmpext()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38531 from gregsfortytwo/wip-stretch-updates

More stretch updates for overly-aggressive safety checks that broke clients

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

client: add debug messages for osdmap wait

To help debug i47294.

See-also: https://tracker.ceph.com/issues/47294
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: move object create before set alloc hint

This commit fixes the order of ops in write requests to have create before set alloc hint, so that exclusive create will not falsely fail

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38701 from rosinL/fix-memory-over-use

commom/buffer: use small page aligned when bufferlist prealloc memory

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38747 from tchaikov/wip-scrub-event

osd/scrub_machine: pass an instance of event to post_event()

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

librbd: fix crypto related memory leaks

This commit fixes memory leaks introduced by the new librbd encryption feature

Signed-off-by: Or Ozeri <oro@il.ibm.com>

Merge pull request #38796 from MahatiC/wip-ssd-read

librbd/cache: Implement aio_read operation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38272 from vbashkirtsev/master

test/test_arch.cc: search "pni" in CPU flags also when detecting SSE3

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/test_arch.cc: search "pni" in CPU flags also when detecting SSE3

as sometimes a SSE-3 CPU reports "pni" (Prescott New Instructions) in /proc/cpuinfo.

Signed-off-by: Vladimir Bashkirtsev <vladimir@bashkirtsev.com>

do_cmake.sh: do not perform 'git submodule update' if building from tarball

Signed-off-by: Vladimir Bashkirtsev <vladimir@bashkirtsev.com>

Merge pull request #38753 from Rethan/style-ceph-df-output

mon/PGMap: align to same side when output ceph df / ceph df detail

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38775 into master

* refs/pull/38775/head:
qa: remove ceph file systems on completion

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #38791 into master

* refs/pull/38791/head:
mon/MDSMonitor: add missing frozen checks

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #38778 into master

* refs/pull/38778/head:
mds: don't use dentry_key_t in C_IO_Dir_Commit_Ops

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #38773 into master

* refs/pull/38773/head:
mds: avoid spurious sleeps

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>

Merge PR #38698 into master

* refs/pull/38698/head:
client: fix the cache size check
client: add more detail info for the _put_inode debug log
client: skip putting inodes if map is empty

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38683 into master

* refs/pull/38683/head:
mds: add scrub perf counters support

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38642 into master

* refs/pull/38642/head:
rpm/deb/cmake: install mount.fuse.ceph man page

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

osd: osdmap: do not assert target_v when encoding in stretch mode

Again, this was an overly-strict safety check. It passed testing against
(even old) userspace clients, but the kernel clients don't have all the
same feature bits and so when encoding for them, we see lower target_v values
and assert out erroneously.

As before, monitors and OSDs that actually need to interpret this data
are gated on cluster entry, so we don't need this safety check here.

Signed-off-by: Greg Farnum <gfarnum@redhat.com>

cephadm: HA for RGW endpoints

Cephadm deploying keepalived and HAproxy for providing High availability for RGW endpoints

Fixes: https://tracker.ceph.com/issues/45116
Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #37882 from cbodley/wip-rgw-data-sync-notification-fix

rgw: data sync: fix obligation check

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #38720 from bk201/wip-48717

mgr/dashboard: fix Reads/Writes ratio of Clients IOPS donut chart

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge PR #38776 into master

* refs/pull/38776/head:
qa: get pool holding file data from file layout

Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge PR #38664 into master

* refs/pull/38664/head:
qa: bump scrub timeout
qa: move cephfs_ec_profile under cephfs
qa: do not use ec pools for default data pool
qa: skip check-counters for light workloads
qa: remove empty multimds suite
qa: merge multimds:verify with fs:verify
qa: merge multimds:thrash to fs:thrash
qa: remove dead multimds:basic
qa: move functional multimds tests to fs:functional
qa: migrate multimds workloads to fs:workloads
qa: only run valgrind on cephfs daemons
qa: stop testing filestore on cephfs suites
qa: load data pools before deleting fs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Tested-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38693 into master

* refs/pull/38693/head:
qa: execute scrubs only on rank 0
client: print debug information about resolved MDS
qa: let Client::resolve_mds lookup the rank

Reviewed-by: Xiubo Li <xiubli@redhat.com>

qa: remove ceph file systems on completion

So that we can avoid MDS replacement warnings.

Fixes: https://tracker.ceph.com/issues/48757
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38510 into master

* refs/pull/38510/head:
PendingReleaseNotes: Update about volume/nfs cluster id changes
mgr/volumes/nfs: remove 'ganesha-' prefix from cluster id

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38537 from IlsooByun/fix_ret_invalid_err

rgw: return error when trying to copy encrypted object without key

Merge pull request #38283 from dvanders/dvanders_throttle

rgw: add an outstanding_requests perf counter

Merge pull request #38242 from theanalyst/rgw/reset-stats-1-fix

rgw: cls/user: set from_index for reset stats calls

qa: add test for reserved keyword feature

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38725 from CongMinYin/immutable-obejct-cache-doc

doc/rbd: Immutable object cache doc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38667 from pcuzner/fix-cephadm-stdout

cephadm: make host add failure message more friendly

Merge pull request #37920 from votdev/issue_48068_timezone

cephadm: Various properties like 'last_refresh' do not contain timezone

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38807 from tchaikov/wip-mon-cmd-doc

doc: build mon_command_api.rst using a homebrew extension

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

pybind/rados: add test_cmpext_op()

Signed-off-by: Zhang Jiao <zhangjiao_yewu@cmss.chinamobile.com>

pybind/rados: add ReadOp::cmpext()

Signed-off-by: Zhang Jiao <zhangjiao_yewu@cmss.chinamobile.com>

pybind/rados:add WriteOp::cmpext()

Signed-off-by: Zhang Jiao <zhangjiao_yewu@cmss.chinamobile.com>

PendingReleaseNotes: Update about volume/nfs cluster id changes

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/volumes/nfs: remove 'ganesha-' prefix from cluster id

Orchestrator defines service name as "<service_type>.<service_id>". The ganesha
common config object name in orchestrator is "conf-<service_name>". Volume's
nfs plugin deploys nfs-ganesha clusters with 'ganesha' prefixed to cluster id
and common config object.  It can cause unecessary issues in rook and cephadm.
So let's remove the prefix.

Fixes: https://tracker.ceph.com/issues/48514
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #38754 from tchaikov/wip-confutil-string-view

common/ConfUtils: pass string_view instead of string

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/_ext: accept params separated with ", "

"mon enable_stretch_mode" command's arg descriptors are delimited with
", ", so we should ignore empty kv when split the params with ",".

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: build mon_command_api.rst using a homebrew extension

so we can build the command doc using ReadTheDoc infra, without adding
the generated rst file to the source repo.

before this change, all the commands are ordered alphabetically. after
this change, command docs are generated by two directives, and are
ordered separately. we could restructure the directives and merge them.

but let's leave it for a future change if this is important.

for more details on writing sphinx directives, see
https://www.sphinx-doc.org/en/master/extdev/markupapi.html and
https://docutils.sourceforge.io/docs/howto/rst-directives.html

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa: use no client for required client feature tests

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: bump scrub timeout

To avoid timeouts for large workloads.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: move cephfs_ec_profile under cephfs

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: do not use ec pools for default data pool

This configuration unnecessarily slows down backtrace writes and bloats
the used disk space.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: skip check-counters for light workloads

None of these are likely to generate exports.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove empty multimds suite

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: merge multimds:verify with fs:verify

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: merge multimds:thrash to fs:thrash

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: remove dead multimds:basic

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: move functional multimds tests to fs:functional

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: migrate multimds workloads to fs:workloads

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
Fixes: https://tracker.ceph.com/issues/48121

qa: only run valgrind on cephfs daemons

OSD valgrind slows things down too much to the point where some tasks
fail to complete.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: stop testing filestore on cephfs suites

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: load data pools before deleting fs

Otherwise current data pools at time of deletion may not be known.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: do not allow setting a reserved feature by name

Fixes: https://tracker.ceph.com/issues/48661
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: return sv for efficiency

This is a trivial refactor.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38794 into master

* refs/pull/38794/head:
qa: add debug vstart_runner option

Reviewed-by: Rishabh Dave <ridave@redhat.com>

mgr/dashboard/monitoring: upgrade Grafana version due to CVE-2020-13379

Fixes: https://tracker.ceph.com/issues/48685
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

cephadm: silence "Failed to evict container" log msg

Right now, we're printing some evil looking messages in the log:

```
systemd[1]: Starting Ceph mgr.node2.ankmgz for ...
podman[32354]: Error: no container with name or ID ceph-... found: no such container
bash[32363]: Error: Failed to evict container: "": Failed to find container "ceph-..." in state: no container with name or ID ceph-... found: no such container
bash[32363]: Error: no container with ID or name "ceph-..." found: no such container
````

Also, the unit.run command already removes the container. No need
for ExecStartPre to do the same.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38377 from yanghonggang/master

rgw/rgw_http_client.cc: Introduce a new configuration item - rgw_curl_buffersize

Merge pull request #38800 from sebastian-philipp/doc-cephadm-rm-link-to-de

doc/cephadm: some minor updates

Reviewed-by: Varsha Rao <varao@redhat.com>

mount.ceph: collect v2 addresses for non-legacy ms_mode options

The kclient recently had support for msgr2 merged, and with that we have
a new ms_mode mount option. The mount helper's support for autodiscovery
of mons currently always ignores v2 addresses.

Change the helper to look for the ms_mode in the provided options and if
it's set to anything but "legacy", have it collect v2 addresses instead
of v1 addrs. The default is still "legacy" so if the option is not
provided, we will still collect v1 addrs.

Fixes: https://tracker.ceph.com/issues/48765
Signed-off-by: Jeff Layton <jlayton@redhat.com>

cephadm: fix podman failure to pull authenticated registry image from systemd unit

have podman keep auth file in /var/lic/ceph/<fsid> becasue podman commands run from systemd do not have access to the default location and the default location is not consistant depending on the user logged in as, where your using sudo or not.

Signed-off-by: Daniel-Pivonka <dpivonka@redhat.com>

doc/cephadm: /etc/ceph is created automaticlly now

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: CephFS: Encourage users to call `ceph fs volume create`

Manually deploying MDS daemons is an advanced topic.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: update stability notes

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/cephadm: Rm doc/dev/cephadm from user docs

We should not link the dev guide form the user documentation. The user docs are already convoluted and hard to follow. Adding the dev guide is not helpful.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #38784 from sebastian-philipp/resurect-doc-dev-cephadm

doc/dev: resurrect hidden dev/cephadm.rst

Reviewed-by: Varsha Rao <varao@redhat.com>

Merge pull request #38560 from pcuzner/cephadm-maintenance-mode

cephadm: add host-maintenance subcommand

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>