mrun: avoid == in [ command

== operator is not supported in POSIX sh (e.g. dash):

  ./mrun: 24: [: unexpected operator

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41354 from idryomov/wip-rbd-pwl-ssd-recovery

librbd/cache/pwl/ssd: fix some crash recovery issues

Reviewed-by: Yin Congmin <congmin.yin@intel.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #39772 from xxhdx1985126/wip-crimson-client-req-pipeline-parallelism

crimson/osd: optimize crimson-osd's client requests process parallelism

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41362 from Aran85/crimson-diagrams

crimson/seastore: add string_kv_node_layout diagrams

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge PR #40234 into master

* refs/pull/40234/head:
client: always register callbacks before mount()
client: move SnapRealm methods to ClientSnapRealm.cc

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #40842 into master

* refs/pull/40842/head:
qa: update the ffsb.sh to clone it from git://git.ceph.com/ffsb.git

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41235 into master

* refs/pull/41235/head:
mds: PurgeQueue.cc fix for 32bit compilation

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #41239 into master

* refs/pull/41239/head:
librbd: use uint64_t instead of size_t for SparseExtent::length
mgr/PyModule: use Py_ssize_t for the PyList index
os/bluestore: print size_t using %xz
client: print int64_t using PRId64

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

Merge PR #41254 into master

* refs/pull/41254/head:
mds: save the metadata pool id MDSRank class's private member

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41267 into master

* refs/pull/41267/head:
mds: defer the journal recovered success log

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41268 into master

* refs/pull/41268/head:
mds: fix possible mds_lock not locked assert

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41341 from tchaikov/wip-dmclock

dmclock: pick up change to fix run_sched_ahead() scheduling issue

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>

Merge pull request #41262 from cbodley/wip-rgw-civetweb-deprecate

rgw: deprecate the civetweb frontend

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #41348 from jdurgin/wip-release-notes-fixes

script/ceph-release-notes: work with py3 and remove backport release names from PRs

Merge PR #41314 into master

* refs/pull/41314/head:
qa/tasks/nfs: add test to check if cmds fail on not passing required arguments
mgr/nfs: fix flake8 missing whitespace around parameter equals error
mgr/nfs: annotate _cmd_nfs_* methods return value

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

crimson/osd: make do_osd_ops receive lvalue reference to osd ops vector

otherwise any async execution of lambdas in PG::do_osd_ops_execute() may
reference a outdated osd_op

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #40172 from p-se/pse-fix-cephadm-prom-alerts-missing

mgr/cephadm: fix missing prometheus alerts

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41361 from zdover23/wip-doc-rados-gateway-spelling-embeddding-2021-05-17

doc/radosgw: s/embeddding/embedding/

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/seastore: add string_kv_node_layout diagrams

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

doc/radosgw: s/embeddding/embedding/

res ipsa loquitur

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41360 from mflehmig/patch-1

doc/rados: Fix typo

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rados: Fix typo

Signed-off-by: Martin Flehmig <martin.flehmig@tu-dresden.de>

Merge pull request #41315 from adk3798/check-version

mgr/cephadm: check version in upgrade check

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #41328 from jmolmo/osd_replacement_in_fqdn_hosts

mgr/cephadm: Fix OSD replacement in hosts with FQDN host name

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #41343 from dsavineau/issue_50717

mgr/cephadm: fix prometheus jinja template

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #41306 from liewegas/udpate-isa-l

isa-l: incorporate fix for aarch64 text relocation

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: optimize crimson-osd's client requests process parallelism

Make client requests go to the concurrent pipeline stage "wait_repop" once they
are "submitted" to the underlying objectstore, which means their on-disk order
is guaranteed, so that successive client requests can go into the "process"
pipeline stage.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: add new facilities to interruptible future

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/osd: add two more stages into pg's client request pipeline

These two stages are used to provide more parallelism in the pipeline,
while preserving client requests order.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #41342 from yuriw/wip-yuriw-crontab-master

qa/tests: added client-upgrade-nautilus-pacific tests

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

librbd/cache/pwl/ssd: stronger assert in aio_read_data_blocks()

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd: rename aio_read_data_block() overload

Rename the overload that deals with multiple data blocks to
aio_read_data_blocks().

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd: persist correct write_data_pos

WriteLogCacheEntry gets appended to persist_log_entries before
write_data_pos is updated with the actual media offset.  Because
push_back() makes a copy, the updated write_data_pos value never
makes it to media, making recovery impossible.

Fixes: https://tracker.ceph.com/issues/50669
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd: set m_bytes_allocated_cap on recovery

Currently it's set only when a new cache is formatted.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd: actually use first_{valid,free}_entry on recovery

first_valid_entry and first_free_entry pointers are read from media
but not actually used: both m_first_valid_entry and m_first_free_entry
get assigned 0 (or garbage).  next_log_pos gets the same value as well
meaning that not only no recovery is attempted but the cache also gets
corrupted because DATA_RING_BUFFER_OFFSET is not applied.

Fixes: https://tracker.ceph.com/issues/50669
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd: don't count log entries

In ssd mode log entries are variable size.  Attempting to count and
impose watermarks on the number of log entries is bogus because the
total number of entries it would take to fill the cache to capacity
is also variable and can't be precisely estimated.

Fixes: https://tracker.ceph.com/issues/50669
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: fix AbstractWriteLog::check_allocation() signature

All parameters are integers and none of them are (in-)out, so don't
take them by reference.  Additionally num_lanes, num_log_entries and
num_unpublished_reserves don't need to be 64-bit as their respective
fields in AbstractWriteLog are 32-bit.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: rename m_log_pool_config_size to m_log_pool_size

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl: get rid of AbstractWriteLog::m_log_pool_actual_size

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd: get rid of WriteLog::pool_size

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41350 from liu-chunmei/seastore-fix-omap-manager-testcase

crimson/seastore: fix omap manager testcase bug

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/seastore: fix omap manager testcase bug

can't erase map while iterating it.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

script/ceph-release-notes: remove release name from backport PR titles

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

script/ceph-release-notes: remove EOL releases

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

script/ceph-release-notes: do not encode to bytes

This is no longer needed with python3, and adds b'' around strings.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

mgr/cephadm: Fix OSD replacement in hosts with FQDN host name

Resolves: https://tracker.ceph.com/issues/50805

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge PR #40848 into master

* refs/pull/40848/head:
SECURITY.md: update security policy

Reviewed-by: Sage Weil <sage@redhat.com>

mgr/cephadm: fix prometheus jinja template

The prometheus yaml file shouldn't use "tab" character but only spaces.

Fixes: https://tracker.ceph.com/issues/50717
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #41307 from dang/wip-dang-zipper-bucket-remove

RGW Zipper - consolidate two bucket remove APIs

qa/tests: added client-upgrade-nautilus-pacific tests

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/tasks/nfs: add test to check if cmds fail on not passing required arguments

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: fix flake8 missing whitespace around parameter equals error

Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/nfs: annotate _cmd_nfs_* methods return value

Fixes: 7c57d1634
Fixes: https://tracker.ceph.com/issues/50783
Signed-off-by: Varsha Rao <varao@redhat.com>

dmclock: pick up change to fix run_sched_ahead() scheduling issue

see also https://tracker.ceph.com/issues/48317

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41003 from tchaikov/wip-more-confval

common/options: extract osd and mgr settings out

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

SECURITY.md: update security policy

Update Ceph Security Team GPG key and fix indentation issues

Signed-off-by: Hardik Vyas <hvyas@redhat.com>

common/options: dedup Option::services

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: extract osd and mgr settings out

the purposes of this change are:

* faster compilation, so the cmake generator can process .yaml.in files
  in parallel.
* allow daemons to include a subset of options which it is interested
  in.
* better maintainability. by grouping options in different .yaml.in
  files, developers understand who are the consumers of an option.

in this change, options only read by mgr are extracted into mgr.yaml.in,
and options only read by osd are extracted into osd.yaml.in.

so all options in mgr.yaml.in should have "services: mgr" in their
definition by default. the ones in osd.yaml.in have "services: osd".

in the case where options are consumed by multiple services or tools,
the option should add "common" to its "services" if it is supposed to be
consumed by a tool, or "mon" if it is read by monitor as well.

but it takes time to audit all the options, so only part of them are
processed.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: Revert "cmake: use file(GLOB ..) to find .yaml.in files"

This reverts commit 4e58b5899e98397ea928cbde1451b0734fa0892b.

the list of .yaml.in file might change over time before we finish the
.yaml.in file split, but cmake would fail to figure out the list without
rerunning "cmake", so when a new .yaml.in file is introduced, developer
might end up with a FTBFS after pulling the change from remote repo.

so, we need to revert the file(GLOB ..) change, until all .yaml.in file
are created.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41326 from tchaikov/wip-seastar

seastar: bump seastar to latest upstream master

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41322 from AmnonHanuhov/wip-fix_messageuref_memory_leak

crimson/net: Set add_ref to false when creating a MessageRef in conn::send()

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: Set add_ref to false when creating a MessageRef in conn::send()

we should not increase the ref count when transferring the ownership from a
MessageURef instance to a MessageRef instance.

to address the memory leak introduced by
1f6e66470504fe5aec4cae3fc74d6e9927aae7aa.

Signed-off-by: Amnon Hanuhov <ahanukov@redhat.com>

Merge pull request #41132 from liu-chunmei/crimson-perfcouter-dump

crimson/seastore: dump crimson perf counters

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41317 from neha-ojha/wip-commandwq-cleanup

common/options/global.yaml.in: remove osd_command_thread* timeouts

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #41329 into master

* refs/pull/41329/head:
doc: 15.2.12 Release Notes
doc: 14.2.21 Release Notes
doc/security: summarize CVEs
doc: 16.2.4 Release Notes

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc: 15.2.12 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

doc: 14.2.21 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

doc/security: summarize CVEs

Signed-off-by: Sage Weil <sage@newdream.net>

mgr/cephadm: check version in upgrade check

Fixes: https://tracker.ceph.com/issues/50784
Signed-off-by: Adam King <adking@redhat.com>

doc: 16.2.4 Release Notes

Signed-off-by: David Galloway <dgallowa@redhat.com>

seastar: bump seastar to latest upstream master

for silencing GCC warnings

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net/Socket: do not reset FixedCPUServerSocket::shutdown_gate

the copy constructor of seastar::gate is deleted explicitly. so we
cannot reset FixedCPUServerSocket::shutdown_gate by assigning a new
seastar::gate to it.

since we don't reuse a FixedCPUServerSocket after calling
FixedCPUServerSocket::destroy(), it's safe to leave a closed gate after
calling FixedCPUServerSocket::reset()

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41305 from zdover23/wip-doc-cephadm-monitoring-disabling-monitoring-2021-05-12

doc/cephadm: rewrite "disabling monitoring"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41027 from mkogan1/wip-rgw-gc-max-objsf2

radosgw-admin: skip GC init on read-only admin ops

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge PR #39550 into master

* refs/pull/39550/head:
mgr/cephadm: induce retune of osd memory on osd creation
qa/tasks/cephadm.conf: autotune osd memory by default
mgr/cephadm: do not autotune when _no_autotune_memory label is present
mgr/cephadm: autotune osd memory
common: add osd_memory_target_autotune
mgr/cephadm: report memory usage, request (limit) in 'orch ps'
doc/cephadm/host-management: document _admin group
mgr/orchestrator: fix help formatting

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #41319 from tchaikov/wip-crimson-osd-cleanups

crimson/osd: cleanups

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41095 from liu-chunmei/crimson-perfcouter

crimson/seastore: add perfcounters in seastore

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41270 from trociny/wip-50351

osd: don't assert in-flight backfill is always in recovery list

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #40660 from dvanders/stash

mon: load stashed map before mkfs monmap

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41233 from tchaikov/wip-osd-pg-cleanup

osd/PG: do not check for empty before loop

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41257 from liewegas/cephadm-filter-net-message

mgr/cephadm/schedule: fix message

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41290 from Aran85/fix-crc

osd: clear data digest when write_trunc

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: forward declare OSDOp and ObjectState

simpler this way, also, silence clang warning, as OSDOp is defined as
struct, let's declare it as struct as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: construct future state in-place

instead of constructing the value of state, pass the parameters for
building it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40646 from rzarzynski/wip-crimson-watch-timeout

crimson/osd: implement timeout support for watches

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40608 from tschoonj/etag-spelling

rgw: fix spelling of eTag in S3 message structure

Merge pull request #41245 from tchaikov/wip-doc-confval-ext

doc/_ext: fixes related to mgr option rendering

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/seastore: add perfcoutners in seastore

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson/seastore: fix decode build error in journal.cc

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

doc/rados/configuration/osd-config-ref: remove osd_command_thread_timeout

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41309 from yuriw/wip-yuriw-crontab-master

qa/tests: added pacific-p2p

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41303 from tchaikov/wip-crimson-os-cleanups

crimson/os/seastore: do not capture unused variables

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41261 from cbodley/wip-rgw-beast-sync

rgw: add config option to disable beast's async process_request()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

common/options/global.yaml.in: remove osd_command_thread* timeouts

These are no longer used after 817cca779db24b9ef08138a546ccb339271a3d9c

Signed-off-by: Neha Ojha <nojha@redhat.com>

RGW Zipper - consolidate two bucket remove APIs

Bucket::remove_entrypoint() and Bucket::remove_instance_info() are only
used together, and take the same args.  Combine them into a single call.

Signed-off-by: Daniel Gryniewicz <dang@redhat.com>

crimson/osd: unify the interruption handling between {Internal,}ClientRequest.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: share do_recover_missing() between {Internal,}ClientRequest.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tests: added pacific-p2p

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #41094 from dvanders/negative_progress

mgr/progress: ensure progress stays between [0,1]

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41092 from ifed01/wip-ifed-fix-alloc-init-add-free-0-len

os/bluestore: tolerate zero length for allocators' init_[add/rm]_free()

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>