mgr/nfs: take optional virtual_ip for deploying ingress

For 'nfs cluster create', optionally take a virtual_ip to deploy ingress.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: remove 'nfs cluster update'

This command is very awkward to implement unless all service spec fields
are always required.  That will soon mean both the placement *and*
virtual_ip (if any), making it much less useful for a human to make use
of.

Instead, let them update yaml, or adjust the nfs and/or ingress specs
directly.  I don't think this command is needed.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: factor out ganesha pool creation

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: delete -> rm for CLI

The rest of the CLI uses 'rm' in place of 'remove' or 'delete', so let's
deprecate 'delete' and add 'rm'.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/nfs: add some type annotations

Signed-off-by: Sage Weil <sage@newdream.net>

python-common: fix IngressSpec yaml dump

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: ingress: remove eth0 default

Better to raise an error; eth0 will never be correct.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: allow mounting volumes in shell

Signed-off-by: Sage Weil <sage@newdream.net>

cephadm: add -v arg to shell

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/vip: add 'vip.exec' task

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: add --port arg to 'orch apply nfs'

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: nfs: add purge

Remove the grace object if we purge the service.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: ingress: support nfs

- use consistent hashing
- statically map across ranks
- disable backend checks so that clients don't move

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: do not reconfigure daemons on deleted services

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: nfs: shell out to rados tool for conf creation

This avoids any hangs due to rados.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: nfs: add rank to grace file from mgr module

Do the grace file manipulation from the mgr module.  For add, this isn't
especially important, but for remove it is very important.  Clean out
old ranks from the grace table before we record that the rank has been
purged from the rank_map.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: nfs: bind ganesha to appropriate ip:port

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: enable ranked daemons for nfs

Use ranked daemons for NFS. Ganesha does not like it if multiple
instances start up with the same rank, but we need stable ranks so that
a rank can "fail over" to a new instance of a new daemon on another host
(with the same rank) for NFS client reclaim to work.

Specify a nodeid of '{service_name}.{rank}' for ganesha.

Include a unique id in the daemon_id just because this avoids some issues
with the create/destroy ordering, and because the daemon_id doesn't matter
much anymore since we are using a stable rank.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: support creation of daemons with ranks

- we need to assign all names and update the rank_map before we start
creating daemons.
- if we are using ranks, we should delete old daemons first, and
fence them from the cluster (where possible).

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: make _plan show removed daemon names

This is more informative than just the hostnames.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/schedule: assign/map ranks

If we are passed a rank_map, use it maintain one daemon per rank, where
the ranks are consecutive non-negative integers starting from 0.

A bit of refactoring in place() so that we only do the rank allocations
on slots we are going to use (no more than count).

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: add rank[_generation] properties

DaemonDescription
CephadmDaemonDeploySpec
DaemonPlacement
unit.meta
get_unique_name() (we include it in the daemon_id)

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/inventory: store optional rank_map along with specs

The rank_map is a bit of state to keep track of which ranks are
occupied by which generation and daemon_id.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: include service_name is generated DaemonDescription

This makes 'orch ls' match up daemosn to services (and probably cleans up
other bits and pieces) when the old daemon id -> service name calc code
can't do its thing.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/orchestrator: include service_name in DaemonDescription dump

('orch ps')

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/inventory: fix deleted check

Look in dict, not encoded JSON string

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: simplify

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm/schedule: make placement shuffle deterministic

hash(str) is non-deterministic, probably because it is using the internal
object ID or something and not the string content?

In any case, explicitly hash the string content and use that instead.

Also, sort the input pre-shuffle to ensure that variations in the original
host list ordering don't screw with the result.

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: document CephadmService flags

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41399 from idryomov/wip-mrun-if-posix

mrun: avoid == in [ command

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41393 from rosinL/fix-50835

common/crc32c_aarch64: fix crc32c unittest failed on aarch64

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41388 into master

* refs/pull/41388/head:
cephadm: write config files as utf-8

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge PR #41286 into master

* refs/pull/41286/head:
qa/suites/orch/rook: disable centos for now
qa/suites/orch/rook/smoke: initial smoke suite
qa/tasks/rook: ROOK_HOSTPATH_REQUIRES_PRIVILEGED=true on centos
qa/tasks/rook: simplify shutdown
qa/tasks/rook: archive logs
qa/tasks/rook: more orderly cluster teardown
qa/tasks/rook: deploy ceph via rook on top of kubernetes
qa/tasks/kubeadm: install kubernetes with kubeadm
qa/suites: move rados/cephadm -> orch/cephadm; symlink
qa/tasks/cephadm: add whitespace between functions
qa/tasks/cephadm: clean up ctx.manager setup

Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #41370 into master

* refs/pull/41370/head:
mgr/cephadm: pin haproxy image to 2.3.z

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #41345 from rhcs-dashboard/fix-CVE-2021-3509-master

mgr/dashboard: fix cookie injection issue (CVE-2021-3509)

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mrun: avoid == in [ command

== operator is not supported in POSIX sh (e.g. dash):

  ./mrun: 24: [: unexpected operator

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41104 from rhcs-dashboard/fix-ESOCKETTIMEDOUT-e2e-failure

mgr/dashboard: fix ESOCKETTIMEDOUT E2E failure

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41273 from rhcs-dashboard/update-frontend-deps

mgr/dashboard: update frontend deps due to security vulnerabilities

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

common/crc32c_aarch64: fix crc32c unittest failed on aarch64

On centos 8.2 for aarch64 with gcc 8.3, the complier will use
register v0 conflicting with the register v0 be usded in inline
asm code. Adding the related registers into clobber list to inform
complier avoiding the confict.

Fixes: https://tracker.ceph.com/issues/50835
Signed-off-by: luo rixin <luorixin@huawei.com>

cephadm: write config files as utf-8

If we don't specify the encoding, python may use something like 'latin-1',
which cannot handle unicode characters.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41390 from batrick/i50865

doc: use sphinx .dot include

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41357 into master

* refs/pull/41357/head:
mds: do not assert when receiving a unknow metric type

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #41347 into master

* refs/pull/41347/head:
mon,doc: deprecate CephFS min_compat_client
doc: add alternate_name cephfs feature to table

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge PR #41332 into master

* refs/pull/41332/head:
mds: place the journaler pointer under the mds_lock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41171 into master

* refs/pull/41171/head:
test: disable mirroring module for certain tests

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41097 into master

* refs/pull/41097/head:
cephfs-mirror: allow connecting to local cluster using mon address

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sébastien Han <seb@redhat.com>

Merge PR #40885 into master

* refs/pull/40885/head:
doc: document cephfs-mirror configuration options
cephfs-mirror: use sensible mount timeout when mounting local/remote fs
test: add tests for settting mount timeout
pybind/cephfs: add interface to set mount timeout
libcephfs: add interface to set mount timeout

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41371 into master

* refs/pull/41371/head:
mds/Mutation: initialize MutationImpl object's initiated_at

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/orch/rook: disable centos for now

We are hitting an selinux denial:

type=AVC msg=audit(1621353663.595:6016): avc: denied { write } for pid=28812 comm="cephcsi" name="pids.max" dev="cgroup" ino=2933 scontext=system_u:system_r:container_t:s0:c43,c177 tcontext=system_u:object_r:cgroup_t:s0 tclass=file permissive=1

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites/orch/rook/smoke: initial smoke suite

- os: 18.04, 20.04, centos, centos+kubic
- nodes: 1 or 3
- rook: 1.6.2 or master
- k8s: 1.21 always (for now)
- net: calico always (flannel + centos is wonky)
- workload: none, rados bench

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: ROOK_HOSTPATH_REQUIRES_PRIVILEGED=true on centos

This fixes OSD creation on centos.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: simplify shutdown

For some reason deleting common.yaml sometimes fails.  Not really
sure why, but since we will tear down kubernetes anyway this
cleanup isn't really needed.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: archive logs

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: more orderly cluster teardown

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: deploy ceph via rook on top of kubernetes

This assumes that k8s is installed and kubectl works.

The ceph container to use is selected the same way the cephadm
task does it.

All scratch devices are consumed as OSDs.

A ceph.conf and client.admin keyring are deployed on all test
nodes, so normal tasks should work (if/when packages are installed).

Fixes: https://tracker.ceph.com/issues/47507
Signed-off-by: Sage Weil <sage@newdream.net>

doc: use sphinx .dot include

To compile svg automatically into docs.

Fixed: https://tracker.ceph.com/issues/50865
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40921 from kamoltat/wip-pool-start-out-with-1-pg

mon/OSDMonitor: prepare_new_pool() starts out with 1 pg

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #41383 into master

* refs/pull/41383/head:
doc/security: Add single GPG key in lieu of existing three GPG keys

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/kubeadm: install kubernetes with kubeadm

- install k8s with kubeadm
- initial support for flannel only
- remove taint from bootstrap/master node
- create PVs for all scratch_devs + a 'scratch' SC
- kubeadm.kubectl task

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites: move rados/cephadm -> orch/cephadm; symlink

Move cephadm under orch/ top-level suite.  Symlink so that we
still include it in a rados run.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41352 from tchaikov/wip-crimson-metrics

crimson/os/seastore: use seastar::metrics for metrics reporting

Reviewed-by: Amnon Hanuhov <ahanukov@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41377 from rhcs-dashboard/change-password-preserve-url-fix

mgr/dashboard: Fix for query params resetting on change-password

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

qa/tasks/cephadm: add whitespace between functions

Signed-off-by: Sage Weil <sage@newdream.net>

doc/security: Add single GPG key in lieu of existing three GPG keys

Replace existing three GPG keys with new Ceph Security Team GPG key

Signed-off-by: Hardik Vyas <hvyas@redhat.com>

Merge PR #40640 into master

* refs/pull/40640/head:
common: send SYSLOG_IDENTIFIER to journald
cephadm: enable log to journald by default

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #39580 from ifed01/wip-ifed-migrate

ceph-volume: implement bluefs volume migration.

mgr/cephadm: pin haproxy image to 2.3.z

The 2.4 version fails for reasons that aren't immediate clear:

May 17 15:28:42 dael conmon[3007013]: [NOTICE]   (8) : haproxy version is 2.4.0-6cbbecf
May 17 15:28:42 dael conmon[3007013]: [NOTICE]   (8) : path to executable is /usr/local/sbin/haproxy
May 17 15:28:42 dael conmon[3007013]: [ALERT]    (8) : Cannot open configuration file/directory /var/lib/haproxy/haproxy.cfg : Permission denied

Even with mode 777 on those files, there is an error about the chroot to
/var/lib/haproxy.

For now, just stick with a working version.

Fixes: https://tracker.ceph.com/issues/50830
Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41282 from cbodley/wip-rgw-rm-civetweb

rgw: remove the civetweb and fcgi frontends

crimson/admin: reindent SeastarMetricsHook

we don't use 1-space indent.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: add optional prefix arg to "perf dump_seastar"

so we can filter the dumped metrics by their prefix

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: do not construct AdminSocketHook with osd

osd is not used at all.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: add desc for DumpPerfCountersHook

logger and counter are used when executing the hook

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: use seastar::metrics for metrics reporting

to be better prepeare for the m:n mapping and to attach more labels
to a metric for better off-line data analysis using ETL tools in future.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: print different metric type in different way

and use registered_metric::operator() for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/admin: use better variable names

for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: define SeastoreCollection as a class

to be consistent with the forward declaration. C++ standard does not
differentiate class from struct in this perspective. but Clang warngs at
seeing it. so silence the warning.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41364 from rzarzynski/wip-crimson-monc-pending_messages-assert

crimson/monc: fix send_message() racing with reopen_session().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41366 from tchaikov/wip-crimson-os-debug

crimson/os: use compile-time validation

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

mgr/dashboard: Fix for query params resetting on change-password

Fixes: https://tracker.ceph.com/issues/50857
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41354 from idryomov/wip-rbd-pwl-ssd-recovery

librbd/cache/pwl/ssd: fix some crash recovery issues

Reviewed-by: Yin Congmin <congmin.yin@intel.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #39772 from xxhdx1985126/wip-crimson-client-req-pipeline-parallelism

crimson/osd: optimize crimson-osd's client requests process parallelism

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41362 from Aran85/crimson-diagrams

crimson/seastore: add string_kv_node_layout diagrams

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #40234 into master

* refs/pull/40234/head:
client: always register callbacks before mount()
client: move SnapRealm methods to ClientSnapRealm.cc

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40842 into master

* refs/pull/40842/head:
qa: update the ffsb.sh to clone it from git://git.ceph.com/ffsb.git

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41235 into master

* refs/pull/41235/head:
mds: PurgeQueue.cc fix for 32bit compilation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41239 into master

* refs/pull/41239/head:
librbd: use uint64_t instead of size_t for SparseExtent::length
mgr/PyModule: use Py_ssize_t for the PyList index
os/bluestore: print size_t using %xz
client: print int64_t using PRId64

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge PR #41254 into master

* refs/pull/41254/head:
mds: save the metadata pool id MDSRank class's private member

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41267 into master

* refs/pull/41267/head:
mds: defer the journal recovered success log

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41268 into master

* refs/pull/41268/head:
mds: fix possible mds_lock not locked assert

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41341 from tchaikov/wip-dmclock

dmclock: pick up change to fix run_sched_ahead() scheduling issue

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

mds/Mutation: initialize MutationImpl object's initiated_at

... to current time, ceph_clock_now(). This is a more useful
initialized value than utime_t().

Fixes: https://tracker.ceph.com/issues/42516
Signed-off-by: Ramana Raja <rraja@redhat.com>
Co-Authored-By: Patrick Donnelly <pdonnell@redhat.com>

mon,doc: deprecate CephFS min_compat_client

Instead admins should specify specific features to require.

Fixes: https://tracker.ceph.com/issues/50819
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: building the beast frontend is no longer optional

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove the fcgi frontend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove the civetweb frontend from src and qa

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41262 from cbodley/wip-rgw-civetweb-deprecate

rgw: deprecate the civetweb frontend

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

crimson/os: pass log level to LOG()

instead of passing function name to the underlying macro, pass log
level for better readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: use compile-time validation

libfmt does compile-time format argument validation of the format string
and the argument when the the user-defined literal is used. but the
downside is that the formatter materialize the whole formatted string
into a std::string, before printing them argument into seastar's log buffer
inserter. presumably, the inserter would be more efficient in
comparision to the pre-format approach. so this validation is only
enabled for non NDEBUG build. so it is able to help us to identify
errors like

DEBUG("{} {}", 1, 2, 3)

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41348 from jdurgin/wip-release-notes-fixes

script/ceph-release-notes: work with py3 and remove backport release names from PRs

Merge PR #41314 into master

* refs/pull/41314/head:
qa/tasks/nfs: add test to check if cmds fail on not passing required arguments
mgr/nfs: fix flake8 missing whitespace around parameter equals error
mgr/nfs: annotate _cmd_nfs_* methods return value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>