rgw: building the beast frontend is no longer optional

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove the fcgi frontend

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove the civetweb frontend from src and qa

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41262 from cbodley/wip-rgw-civetweb-deprecate

rgw: deprecate the civetweb frontend

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #41348 from jdurgin/wip-release-notes-fixes

script/ceph-release-notes: work with py3 and remove backport release names from PRs

Merge PR #41314 into master

* refs/pull/41314/head:
qa/tasks/nfs: add test to check if cmds fail on not passing required arguments
mgr/nfs: fix flake8 missing whitespace around parameter equals error
mgr/nfs: annotate _cmd_nfs_* methods return value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #40172 from p-se/pse-fix-cephadm-prom-alerts-missing

mgr/cephadm: fix missing prometheus alerts

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41361 from zdover23/wip-doc-rados-gateway-spelling-embeddding-2021-05-17

doc/radosgw: s/embeddding/embedding/

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/radosgw: s/embeddding/embedding/

res ipsa loquitur

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41360 from mflehmig/patch-1

doc/rados: Fix typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados: Fix typo

Signed-off-by: Martin Flehmig <martin.flehmig@tu-dresden.de>

Merge pull request #41315 from adk3798/check-version

mgr/cephadm: check version in upgrade check

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #41328 from jmolmo/osd_replacement_in_fqdn_hosts

mgr/cephadm: Fix OSD replacement in hosts with FQDN host name

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #41343 from dsavineau/issue_50717

mgr/cephadm: fix prometheus jinja template

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #41306 from liewegas/udpate-isa-l

isa-l: incorporate fix for aarch64 text relocation

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41342 from yuriw/wip-yuriw-crontab-master

qa/tests: added client-upgrade-nautilus-pacific tests

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #41350 from liu-chunmei/seastore-fix-omap-manager-testcase

crimson/seastore: fix omap manager testcase bug

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/seastore: fix omap manager testcase bug

can't erase map while iterating it.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

script/ceph-release-notes: remove release name from backport PR titles

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

script/ceph-release-notes: remove EOL releases

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

script/ceph-release-notes: do not encode to bytes

This is no longer needed with python3, and adds b'' around strings.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: Fix OSD replacement in hosts with FQDN host name

Resolves: https://tracker.ceph.com/issues/50805

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge PR #40848 into master

* refs/pull/40848/head:
SECURITY.md: update security policy

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix prometheus jinja template

The prometheus yaml file shouldn't use "tab" character but only spaces.

Fixes: https://tracker.ceph.com/issues/50717
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #41307 from dang/wip-dang-zipper-bucket-remove

RGW Zipper - consolidate two bucket remove APIs

qa/tests: added client-upgrade-nautilus-pacific tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tasks/nfs: add test to check if cmds fail on not passing required arguments

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: fix flake8 missing whitespace around parameter equals error

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: annotate _cmd_nfs_* methods return value

Fixes: 7c57d1634
Fixes: https://tracker.ceph.com/issues/50783
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #41003 from tchaikov/wip-more-confval

common/options: extract osd and mgr settings out

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

SECURITY.md: update security policy

Update Ceph Security Team GPG key and fix indentation issues

Signed-off-by: Hardik Vyas <hvyas@redhat.com>

common/options: dedup Option::services

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: extract osd and mgr settings out

the purposes of this change are:

* faster compilation, so the cmake generator can process .yaml.in files
  in parallel.
* allow daemons to include a subset of options which it is interested
  in.
* better maintainability. by grouping options in different .yaml.in
  files, developers understand who are the consumers of an option.

in this change, options only read by mgr are extracted into mgr.yaml.in,
and options only read by osd are extracted into osd.yaml.in.

so all options in mgr.yaml.in should have "services: mgr" in their
definition by default. the ones in osd.yaml.in have "services: osd".

in the case where options are consumed by multiple services or tools,
the option should add "common" to its "services" if it is supposed to be
consumed by a tool, or "mon" if it is read by monitor as well.

but it takes time to audit all the options, so only part of them are
processed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Revert "cmake: use file(GLOB ..) to find .yaml.in files"

This reverts commit 4e58b5899e98397ea928cbde1451b0734fa0892b.

the list of .yaml.in file might change over time before we finish the
.yaml.in file split, but cmake would fail to figure out the list without
rerunning "cmake", so when a new .yaml.in file is introduced, developer
might end up with a FTBFS after pulling the change from remote repo.

so, we need to revert the file(GLOB ..) change, until all .yaml.in file
are created.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41326 from tchaikov/wip-seastar

seastar: bump seastar to latest upstream master

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41322 from AmnonHanuhov/wip-fix_messageuref_memory_leak

crimson/net: Set add_ref to false when creating a MessageRef in conn::send()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: Set add_ref to false when creating a MessageRef in conn::send()

we should not increase the ref count when transferring the ownership from a
MessageURef instance to a MessageRef instance.

to address the memory leak introduced by
1f6e66470504fe5aec4cae3fc74d6e9927aae7aa.

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #41132 from liu-chunmei/crimson-perfcouter-dump

crimson/seastore: dump crimson perf counters

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41317 from neha-ojha/wip-commandwq-cleanup

common/options/global.yaml.in: remove osd_command_thread* timeouts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #41329 into master

* refs/pull/41329/head:
doc: 15.2.12 Release Notes
doc: 14.2.21 Release Notes
doc/security: summarize CVEs
doc: 16.2.4 Release Notes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: 15.2.12 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

doc: 14.2.21 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

doc/security: summarize CVEs

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: check version in upgrade check

Fixes: https://tracker.ceph.com/issues/50784
Signed-off-by: Adam King <adking@redhat.com>

doc: 16.2.4 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

seastar: bump seastar to latest upstream master

for silencing GCC warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net/Socket: do not reset FixedCPUServerSocket::shutdown_gate

the copy constructor of seastar::gate is deleted explicitly. so we
cannot reset FixedCPUServerSocket::shutdown_gate by assigning a new
seastar::gate to it.

since we don't reuse a FixedCPUServerSocket after calling
FixedCPUServerSocket::destroy(), it's safe to leave a closed gate after
calling FixedCPUServerSocket::reset()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41305 from zdover23/wip-doc-cephadm-monitoring-disabling-monitoring-2021-05-12

doc/cephadm: rewrite "disabling monitoring"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41027 from mkogan1/wip-rgw-gc-max-objsf2

radosgw-admin: skip GC init on read-only admin ops

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #39550 into master

* refs/pull/39550/head:
mgr/cephadm: induce retune of osd memory on osd creation
qa/tasks/cephadm.conf: autotune osd memory by default
mgr/cephadm: do not autotune when _no_autotune_memory label is present
mgr/cephadm: autotune osd memory
common: add osd_memory_target_autotune
mgr/cephadm: report memory usage, request (limit) in 'orch ps'
doc/cephadm/host-management: document _admin group
mgr/orchestrator: fix help formatting

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #41319 from tchaikov/wip-crimson-osd-cleanups

crimson/osd: cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41095 from liu-chunmei/crimson-perfcouter

crimson/seastore: add perfcounters in seastore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41270 from trociny/wip-50351

osd: don't assert in-flight backfill is always in recovery list

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #40660 from dvanders/stash

mon: load stashed map before mkfs monmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41233 from tchaikov/wip-osd-pg-cleanup

osd/PG: do not check for empty before loop

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41257 from liewegas/cephadm-filter-net-message

mgr/cephadm/schedule: fix message

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41290 from Aran85/fix-crc

osd: clear data digest when write_trunc

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: forward declare OSDOp and ObjectState

simpler this way, also, silence clang warning, as OSDOp is defined as
struct, let's declare it as struct as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: construct future state in-place

instead of constructing the value of state, pass the parameters for
building it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40646 from rzarzynski/wip-crimson-watch-timeout

crimson/osd: implement timeout support for watches

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40608 from tschoonj/etag-spelling

rgw: fix spelling of eTag in S3 message structure

Merge pull request #41245 from tchaikov/wip-doc-confval-ext

doc/_ext: fixes related to mgr option rendering

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/seastore: add perfcoutners in seastore

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/seastore: fix decode build error in journal.cc

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

doc/rados/configuration/osd-config-ref: remove osd_command_thread_timeout

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41309 from yuriw/wip-yuriw-crontab-master

qa/tests: added pacific-p2p

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41303 from tchaikov/wip-crimson-os-cleanups

crimson/os/seastore: do not capture unused variables

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41261 from cbodley/wip-rgw-beast-sync

rgw: add config option to disable beast's async process_request()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

common/options/global.yaml.in: remove osd_command_thread* timeouts

These are no longer used after 817cca779db24b9ef08138a546ccb339271a3d9c

Signed-off-by: Neha Ojha <nojha@redhat.com>

RGW Zipper - consolidate two bucket remove APIs

Bucket::remove_entrypoint() and Bucket::remove_instance_info() are only
used together, and take the same args.  Combine them into a single call.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

crimson/osd: unify the interruption handling between {Internal,}ClientRequest.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: share do_recover_missing() between {Internal,}ClientRequest.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tests: added pacific-p2p

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #41094 from dvanders/negative_progress

mgr/progress: ensure progress stays between [0,1]

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41092 from ifed01/wip-ifed-fix-alloc-init-add-free-0-len

os/bluestore: tolerate zero length for allocators' init_[add/rm]_free()

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/cephadm: induce retune of osd memory on osd creation

Signed-off-by: Sage Weil <sage@newdream.net>

qa/tasks/cephadm.conf: autotune osd memory by default

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: do not autotune when _no_autotune_memory label is present

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: autotune osd memory

- set osd_memory_target_autotune=true to enable
- tuning is periodic (check every 10m by default)
- tuned values are reflected by osd_memory_target config options scoped
  to the host
- only make a change if it appears that we will affect at least 1 of the
  relevant OSDs
- attempt to clean out conflicting options.  (This is imperfect, since any
  manner of weirdly-scoped config options could be responsible; we only
  attempt to clean out one scoped directly to the osd name.)

Signed-off-by: Sage Weil <sage@newdream.net>

common: add osd_memory_target_autotune

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: report memory usage, request (limit) in 'orch ps'

Fill in from {osd,mon}_memory_target if no container limit is set.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #40864 from wjwithagen/wjw-fix-KernelDevice-locking

include, blk: fallback to basic locking if a OS does not have F_OFD_SETLK

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/cephadm/host-management: document _admin group

Signed-off-by: Sage Weil <sage@newdream.net>

crimson/common: use parameter pack for building future<>

it mirrors the way how seastar::make_ready_future() works. and more
importantly, it allows us to build interruptible_future in the same
way as we build plain seastar::future<> in-place. so we can, for
instance, create a future<tuepl<int,int>> using:

make_ready_future<tuple<int,int>>(1, 2)

instead of using

make_ready_future<tuple<int,int>>(make_tuple(1, 2))

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

isa-l: incorporate fix for aarch64 text relocation

Fixes: https://tracker.ceph.com/issues/48681
Fixes: https://tracker.ceph.com/issues/50384
Signed-off-by: Sage Weil <sage@newdream.net>

crimson/osd: ClientRequest::do_recover_missing doesn't depend on OSD anymore.

This commit enables the unification of missing objects between
`ClientRequest` and `InternalClientRequest`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: sending EVENT_DISCONNECT becomes implementation detail of Watch.

In contrast to ceph-osd crimson sends CEPH_WATCH_EVENT_DISCONNECT directly
from the timeout handler and after CEPH_WATCH_EVENT_NOTIFY_COMPLETE.
This simplifies the Watch::remove() interface as callers aren't obliged
anymore to decide whether EVENT_DISCONNECT needs to be send or not -- it
becomes an implementation detail of Watch.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: wire up handling of watch timeouts.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: s/do_timeout/do_notify_timeout/ per the upcoming do_watch_timeout().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: introduce the InternalClientRequest infrastructure.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: PG::with_locked_obc() doesn't depend on MOSDOp anymore.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/cephadm: rewrite "disabling monitoring"

* Simplified the text.
* Correctly formatted the prompts.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41161 from rhcs-dashboard/ingress-creation-follow-up

mgr/dashboard: ingress service creation follow-up

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41174 from pritha-srivastava/wip-rgw-sts-cert-url-fix

rgw/sts: code to fetch certs using .well-known/openid-configuration url

Merge pull request #41221 from rhcs-dashboard/50686-fix-osd-host-grafana

mgr/dashboard: fix OSDs Host details/overview grafana graphs

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: ingress service creation follow-up

Fixes: https://tracker.ceph.com/issues/50568
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Pre-populating the service id(read-only) with the value same as backend service.

Merge pull request #41103 from navinbarnwal/RenameFilesystems

mgr/dashboard: fix typo: Filesystems to File Systems

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41023 from rhcs-dashboard/bucket-name-async-validator-performance

 mgr/dashboard: RGW buckets async validator performance enhancement and name constraints

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #41196 from rhcs-dashboard/add-services-e2e

mgr/dashboard: add Services e2e tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>