qa/tasks/rook: simplify shutdown

For some reason deleting common.yaml sometimes fails.  Not really
sure why, but since we will tear down kubernetes anyway this
cleanup isn't really needed.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: archive logs

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: more orderly cluster teardown

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/rook: deploy ceph via rook on top of kubernetes

This assumes that k8s is installed and kubectl works.

The ceph container to use is selected the same way the cephadm
task does it.

All scratch devices are consumed as OSDs.

A ceph.conf and client.admin keyring are deployed on all test
nodes, so normal tasks should work (if/when packages are installed).

Fixes: https://tracker.ceph.com/issues/47507
Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/kubeadm: install kubernetes with kubeadm

- install k8s with kubeadm
- initial support for flannel only
- remove taint from bootstrap/master node
- create PVs for all scratch_devs + a 'scratch' SC
- kubeadm.kubectl task

Signed-off-by: Sage Weil <sage@newdream.net>

qa/suites: move rados/cephadm -> orch/cephadm; symlink

Move cephadm under orch/ top-level suite.  Symlink so that we
still include it in a rados run.

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: add whitespace between functions

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm: clean up ctx.manager setup

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41003 from tchaikov/wip-more-confval

common/options: extract osd and mgr settings out

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

common/options: dedup Option::services

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: extract osd and mgr settings out

the purposes of this change are:

* faster compilation, so the cmake generator can process .yaml.in files
  in parallel.
* allow daemons to include a subset of options which it is interested
  in.
* better maintainability. by grouping options in different .yaml.in
  files, developers understand who are the consumers of an option.

in this change, options only read by mgr are extracted into mgr.yaml.in,
and options only read by osd are extracted into osd.yaml.in.

so all options in mgr.yaml.in should have "services: mgr" in their
definition by default. the ones in osd.yaml.in have "services: osd".

in the case where options are consumed by multiple services or tools,
the option should add "common" to its "services" if it is supposed to be
consumed by a tool, or "mon" if it is read by monitor as well.

but it takes time to audit all the options, so only part of them are
processed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Revert "cmake: use file(GLOB ..) to find .yaml.in files"

This reverts commit 4e58b5899e98397ea928cbde1451b0734fa0892b.

the list of .yaml.in file might change over time before we finish the
.yaml.in file split, but cmake would fail to figure out the list without
rerunning "cmake", so when a new .yaml.in file is introduced, developer
might end up with a FTBFS after pulling the change from remote repo.

so, we need to revert the file(GLOB ..) change, until all .yaml.in file
are created.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41326 from tchaikov/wip-seastar

seastar: bump seastar to latest upstream master

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41322 from AmnonHanuhov/wip-fix_messageuref_memory_leak

crimson/net: Set add_ref to false when creating a MessageRef in conn::send()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: Set add_ref to false when creating a MessageRef in conn::send()

we should not increase the ref count when transferring the ownership from a
MessageURef instance to a MessageRef instance.

to address the memory leak introduced by
1f6e66470504fe5aec4cae3fc74d6e9927aae7aa.

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #41132 from liu-chunmei/crimson-perfcouter-dump

crimson/seastore: dump crimson perf counters

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41317 from neha-ojha/wip-commandwq-cleanup

common/options/global.yaml.in: remove osd_command_thread* timeouts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #41329 into master

* refs/pull/41329/head:
doc: 15.2.12 Release Notes
doc: 14.2.21 Release Notes
doc/security: summarize CVEs
doc: 16.2.4 Release Notes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: 15.2.12 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

doc: 14.2.21 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

doc/security: summarize CVEs

Signed-off-by: Sage Weil <sage@newdream.net>

doc: 16.2.4 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

seastar: bump seastar to latest upstream master

for silencing GCC warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net/Socket: do not reset FixedCPUServerSocket::shutdown_gate

the copy constructor of seastar::gate is deleted explicitly. so we
cannot reset FixedCPUServerSocket::shutdown_gate by assigning a new
seastar::gate to it.

since we don't reuse a FixedCPUServerSocket after calling
FixedCPUServerSocket::destroy(), it's safe to leave a closed gate after
calling FixedCPUServerSocket::reset()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41305 from zdover23/wip-doc-cephadm-monitoring-disabling-monitoring-2021-05-12

doc/cephadm: rewrite "disabling monitoring"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41027 from mkogan1/wip-rgw-gc-max-objsf2

radosgw-admin: skip GC init on read-only admin ops

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #39550 into master

* refs/pull/39550/head:
mgr/cephadm: induce retune of osd memory on osd creation
qa/tasks/cephadm.conf: autotune osd memory by default
mgr/cephadm: do not autotune when _no_autotune_memory label is present
mgr/cephadm: autotune osd memory
common: add osd_memory_target_autotune
mgr/cephadm: report memory usage, request (limit) in 'orch ps'
doc/cephadm/host-management: document _admin group
mgr/orchestrator: fix help formatting

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #41319 from tchaikov/wip-crimson-osd-cleanups

crimson/osd: cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41095 from liu-chunmei/crimson-perfcouter

crimson/seastore: add perfcounters in seastore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41270 from trociny/wip-50351

osd: don't assert in-flight backfill is always in recovery list

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #40660 from dvanders/stash

mon: load stashed map before mkfs monmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41233 from tchaikov/wip-osd-pg-cleanup

osd/PG: do not check for empty before loop

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41257 from liewegas/cephadm-filter-net-message

mgr/cephadm/schedule: fix message

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41290 from Aran85/fix-crc

osd: clear data digest when write_trunc

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: forward declare OSDOp and ObjectState

simpler this way, also, silence clang warning, as OSDOp is defined as
struct, let's declare it as struct as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: construct future state in-place

instead of constructing the value of state, pass the parameters for
building it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40646 from rzarzynski/wip-crimson-watch-timeout

crimson/osd: implement timeout support for watches

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40608 from tschoonj/etag-spelling

rgw: fix spelling of eTag in S3 message structure

Merge pull request #41245 from tchaikov/wip-doc-confval-ext

doc/_ext: fixes related to mgr option rendering

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/seastore: add perfcoutners in seastore

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/seastore: fix decode build error in journal.cc

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

doc/rados/configuration/osd-config-ref: remove osd_command_thread_timeout

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41309 from yuriw/wip-yuriw-crontab-master

qa/tests: added pacific-p2p

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41303 from tchaikov/wip-crimson-os-cleanups

crimson/os/seastore: do not capture unused variables

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41261 from cbodley/wip-rgw-beast-sync

rgw: add config option to disable beast's async process_request()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

common/options/global.yaml.in: remove osd_command_thread* timeouts

These are no longer used after 817cca779db24b9ef08138a546ccb339271a3d9c

Signed-off-by: Neha Ojha <nojha@redhat.com>

crimson/osd: unify the interruption handling between {Internal,}ClientRequest.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: share do_recover_missing() between {Internal,}ClientRequest.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tests: added pacific-p2p

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #41094 from dvanders/negative_progress

mgr/progress: ensure progress stays between [0,1]

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41092 from ifed01/wip-ifed-fix-alloc-init-add-free-0-len

os/bluestore: tolerate zero length for allocators' init_[add/rm]_free()

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: induce retune of osd memory on osd creation

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm.conf: autotune osd memory by default

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: do not autotune when _no_autotune_memory label is present

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: autotune osd memory

- set osd_memory_target_autotune=true to enable
- tuning is periodic (check every 10m by default)
- tuned values are reflected by osd_memory_target config options scoped
  to the host
- only make a change if it appears that we will affect at least 1 of the
  relevant OSDs
- attempt to clean out conflicting options.  (This is imperfect, since any
  manner of weirdly-scoped config options could be responsible; we only
  attempt to clean out one scoped directly to the osd name.)

Signed-off-by: Sage Weil <sage@newdream.net>

common: add osd_memory_target_autotune

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: report memory usage, request (limit) in 'orch ps'

Fill in from {osd,mon}_memory_target if no container limit is set.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40864 from wjwithagen/wjw-fix-KernelDevice-locking

include, blk: fallback to basic locking if a OS does not have F_OFD_SETLK

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephadm/host-management: document _admin group

Signed-off-by: Sage Weil <sage@newdream.net>

crimson/common: use parameter pack for building future<>

it mirrors the way how seastar::make_ready_future() works. and more
importantly, it allows us to build interruptible_future in the same
way as we build plain seastar::future<> in-place. so we can, for
instance, create a future<tuepl<int,int>> using:

make_ready_future<tuple<int,int>>(1, 2)

instead of using

make_ready_future<tuple<int,int>>(make_tuple(1, 2))

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: ClientRequest::do_recover_missing doesn't depend on OSD anymore.

This commit enables the unification of missing objects between
`ClientRequest` and `InternalClientRequest`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: sending EVENT_DISCONNECT becomes implementation detail of Watch.

In contrast to ceph-osd crimson sends CEPH_WATCH_EVENT_DISCONNECT directly
from the timeout handler and after CEPH_WATCH_EVENT_NOTIFY_COMPLETE.
This simplifies the Watch::remove() interface as callers aren't obliged
anymore to decide whether EVENT_DISCONNECT needs to be send or not -- it
becomes an implementation detail of Watch.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: wire up handling of watch timeouts.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: s/do_timeout/do_notify_timeout/ per the upcoming do_watch_timeout().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: introduce the InternalClientRequest infrastructure.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PG::with_locked_obc() doesn't depend on MOSDOp anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: rewrite "disabling monitoring"

* Simplified the text.
* Correctly formatted the prompts.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41161 from rhcs-dashboard/ingress-creation-follow-up

mgr/dashboard: ingress service creation follow-up

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41174 from pritha-srivastava/wip-rgw-sts-cert-url-fix

rgw/sts: code to fetch certs using .well-known/openid-configuration url

Merge pull request #41221 from rhcs-dashboard/50686-fix-osd-host-grafana

mgr/dashboard: fix OSDs Host details/overview grafana graphs

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: ingress service creation follow-up

Fixes: https://tracker.ceph.com/issues/50568
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Pre-populating the service id(read-only) with the value same as backend service.

Merge pull request #41103 from navinbarnwal/RenameFilesystems

mgr/dashboard: fix typo: Filesystems to File Systems

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41023 from rhcs-dashboard/bucket-name-async-validator-performance

 mgr/dashboard: RGW buckets async validator performance enhancement and name constraints

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #41196 from rhcs-dashboard/add-services-e2e

mgr/dashboard: add Services e2e tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41292 from tchaikov/wip-crimson-alienstore-cleanup

crimson/os/alienstore: cleanups

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #41291 from athanatos/sjust/wip-store-nbd-fs

crimson: add seastore and bluestore to crimson-store-nbd

Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: clear data digest when write_trunc

like CEPH_OSD_OP_TRUNCATE, the truncate within CEPH_OSD_OP_WRITE should
clear oi.data_digest too.

consider such circumstance:
[write 0~10]
[write 5~10 [1@5]]

the first write will generate data_digest, if the second write don't
clear digest, it will continue calc digest and make crc mismatch.

Fixes: https://tracker.ceph.com/issues/50763
Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

crimson/tools/store_nbd: ensure RequestWriter outlives any requests on error

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore.cc: swap stop and umount

start/stop seem to be intended for things like thread pools, etc that
seastore doesn't have.  Umount maps more cleanly here.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd: add FuturizedStore driver

Allows usage of bluestore and seastore.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/alienstore: create tuple in-place

no need to use make_tuple<> when constructing a future whose value is
available. as future<> can be constructed by perfect forwarding the
parameters to its state constructor.

also, wrap the lines whose length is over 80 chars.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/alienstore: add curly brackets

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/alienstore: use get_conf() helper

do not use cct->_conf.get_val<>, unless we are in alien threads.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: add debugging on shutdown for cache, pin contents

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store-nbd: break into multiple files

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #41271 from cyx1231st/wip-seastore-onode-tree-logger

crimson/onode-staged-tree: switch to seastore logger macros

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41256 from tchaikov/wip-doc-rbd-confval

doc/rbd/rbd-config-ref: use confval directive to define options

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: fix typo: Filesystems to File Systems

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

mgr/dashboard: fix typo: Filesystems to File Systems

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

Merge pull request #41226 from yuriw/wip-yuriw-crontab-master

qa/tests: removed most of runs for nautilus as it's almost EOL

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Detailed error messages in rgw bucket name validation

Explain the rgw bucket name constrains for each bucket name validation
errors.

Fixes: https://tracker.ceph.com/issues/50516
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix typo: Filesystems to File Systems

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

mgr/dashboard: add Services e2e tests

Fixes: https://tracker.ceph.com/issues/50567
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Introducing e2e tests for service creation for Ingress and RGW service types.

qa/tests: removed most of runs for nautilus as it's almost EOL

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #41272 from Aran85/fix-assert

crimson/os: cleanup wrongly assertion

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41274 from tchaikov/wip-crimson-os-cleanups

crimson/os/seastore: do not capture unused variables

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: RGW buckets async validator performance enhancement

The rgw bucket creation form has the Name field which have an async
validator. The validator calls all the bucket name and check if the
entered name is unique or not. This happens on every keystroke. So if
100 or more buckets are there, then the async validation can be real
    slow and causes misvalidations in different fields.

I changed the validation logic and did some cleanups to improve the
performance of the async validation.

Fixes: https://tracker.ceph.com/issues/50514
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41193 from sebastian-philipp/cephadm-cephfs-mirror-key-profile-cephfs-mirror

mgr/cephadm: Use `cephfs-mirror` caps profile

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41181 from adk3798/iscsi-timeout

mgr/cephadm: add timeout when removing iscsi gateway.conf

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>