crimson/seastore: fix omap manager testcase bug

can't erase map while iterating it.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #41303 from tchaikov/wip-crimson-os-cleanups

crimson/os/seastore: do not capture unused variables

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41261 from cbodley/wip-rgw-beast-sync

rgw: add config option to disable beast's async process_request()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #41094 from dvanders/negative_progress

mgr/progress: ensure progress stays between [0,1]

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41092 from ifed01/wip-ifed-fix-alloc-init-add-free-0-len

os/bluestore: tolerate zero length for allocators' init_[add/rm]_free()

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #40864 from wjwithagen/wjw-fix-KernelDevice-locking

include, blk: fallback to basic locking if a OS does not have F_OFD_SETLK

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/common: use parameter pack for building future<>

it mirrors the way how seastar::make_ready_future() works. and more
importantly, it allows us to build interruptible_future in the same
way as we build plain seastar::future<> in-place. so we can, for
instance, create a future<tuepl<int,int>> using:

make_ready_future<tuple<int,int>>(1, 2)

instead of using

make_ready_future<tuple<int,int>>(make_tuple(1, 2))

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not capture unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41161 from rhcs-dashboard/ingress-creation-follow-up

mgr/dashboard: ingress service creation follow-up

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41174 from pritha-srivastava/wip-rgw-sts-cert-url-fix

rgw/sts: code to fetch certs using .well-known/openid-configuration url

Merge pull request #41221 from rhcs-dashboard/50686-fix-osd-host-grafana

mgr/dashboard: fix OSDs Host details/overview grafana graphs

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: ingress service creation follow-up

Fixes: https://tracker.ceph.com/issues/50568
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Pre-populating the service id(read-only) with the value same as backend service.

Merge pull request #41103 from navinbarnwal/RenameFilesystems

mgr/dashboard: fix typo: Filesystems to File Systems

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41023 from rhcs-dashboard/bucket-name-async-validator-performance

 mgr/dashboard: RGW buckets async validator performance enhancement and name constraints

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #41196 from rhcs-dashboard/add-services-e2e

mgr/dashboard: add Services e2e tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41292 from tchaikov/wip-crimson-alienstore-cleanup

crimson/os/alienstore: cleanups

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #41291 from athanatos/sjust/wip-store-nbd-fs

crimson: add seastore and bluestore to crimson-store-nbd

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: ensure RequestWriter outlives any requests on error

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/seastore.cc: swap stop and umount

start/stop seem to be intended for things like thread pools, etc that
seastore doesn't have.  Umount maps more cleanly here.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store_nbd: add FuturizedStore driver

Allows usage of bluestore and seastore.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/alienstore: create tuple in-place

no need to use make_tuple<> when constructing a future whose value is
available. as future<> can be constructed by perfect forwarding the
parameters to its state constructor.

also, wrap the lines whose length is over 80 chars.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/alienstore: add curly brackets

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/alienstore: use get_conf() helper

do not use cct->_conf.get_val<>, unless we are in alien threads.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: add debugging on shutdown for cache, pin contents

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/tools/store-nbd: break into multiple files

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #41271 from cyx1231st/wip-seastore-onode-tree-logger

crimson/onode-staged-tree: switch to seastore logger macros

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41256 from tchaikov/wip-doc-rbd-confval

doc/rbd/rbd-config-ref: use confval directive to define options

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

mgr/dashboard: fix typo: Filesystems to File Systems

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

mgr/dashboard: fix typo: Filesystems to File Systems

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

Merge pull request #41226 from yuriw/wip-yuriw-crontab-master

qa/tests: removed most of runs for nautilus as it's almost EOL

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Detailed error messages in rgw bucket name validation

Explain the rgw bucket name constrains for each bucket name validation
errors.

Fixes: https://tracker.ceph.com/issues/50516
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix typo: Filesystems to File Systems

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

mgr/dashboard: add Services e2e tests

Fixes: https://tracker.ceph.com/issues/50567
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
Introducing e2e tests for service creation for Ingress and RGW service types.

qa/tests: removed most of runs for nautilus as it's almost EOL

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #41272 from Aran85/fix-assert

crimson/os: cleanup wrongly assertion

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41274 from tchaikov/wip-crimson-os-cleanups

crimson/os/seastore: do not capture unused variables

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: RGW buckets async validator performance enhancement

The rgw bucket creation form has the Name field which have an async
validator. The validator calls all the bucket name and check if the
entered name is unique or not. This happens on every keystroke. So if
100 or more buckets are there, then the async validation can be real
    slow and causes misvalidations in different fields.

I changed the validation logic and did some cleanups to improve the
performance of the async validation.

Fixes: https://tracker.ceph.com/issues/50514
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41193 from sebastian-philipp/cephadm-cephfs-mirror-key-profile-cephfs-mirror

mgr/cephadm: Use `cephfs-mirror` caps profile

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #41181 from adk3798/iscsi-timeout

mgr/cephadm: add timeout when removing iscsi gateway.conf

Reviewed-by: Daniel Pivonka <dpivonka@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #41175 from mgfritch/cephadm-autopep8

cephadm: introduce autopep8

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #41250 from VasishtaShastry/tune_err_msg

mgr: Fix orch osd rm stop help message

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41240 from tchaikov/wip-debian-install-cleanup

debian/ceph-common.postinst: fix indent and cleanups

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #41276 from tchaikov/wip-docs-responsive-button

doc/_theme: show the menu button

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #41265 from zdover23/wip-doc-config-ssl-tls-for-grafana-2021-05-11

doc/cephadm: rewrite "config ssl/tls f. grafana"

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #41194 from idryomov/wip-rbd-pwl-status

rbd: don't attempt to interpret image cache state json

Reviewed-by: Yin Congmin <congmin.yin@intel.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

mgr/dashboard: fix typo: Filesystems to File Systems
Added breadcrumbs and edited Filesystem -> File Systems in dashboard
Navigation.

Fixes: https://tracker.ceph.com/issues/50341
Signed-off-by: Navin Barnwal <knbarnwal@gmail.com>

doc/_theme: show the menu button

because we have a top-nav bar, which is setting on top of the bar
containing the menu button when the docs is displayed wit a device with
smaller width. in this change, the container of the menu button is moved
down a little bit, so it is visible again.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore: do not capture unused variables

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41100 from navinbarnwal/MinorIssueFeature

mgr/dashboard: fix rename inventory to disks

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

crimson/os: cleanup wrongly assertion

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

crimson/onode-staged-tree: switch to use seastore logger macros

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: remove duplicated log prefix

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: change to test logger in TreeBuilder

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41269 from tchaikov/wip-mgr-max_misplaced

doc/rados/operations: s/max_misplaced/target_max_misplaced_ratio/

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41075 from cyx1231st/wip-seastore-onode-mgr

crimson/seastore: integrate and test onode-erase/list features

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/rados/operations: s/max_misplaced/target_max_misplaced_ratio/

max_misplaced with replaced by in target_max_misplaced_ratio
edbd592ee44e02a5328e1510879555c2f9dcfc9e, but the document was not
sync'ed. let's update it accordingly.

Fixes: https://tracker.ceph.com/issues/50745
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41207 from paulreece42/wip-doc-balancer-options

doc: added doc on more throttling options for the PG balancer module

Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/rbd/rbd-config-ref: use confval directive to define options

less repeating this way.

also fix a typo of "rbd_qos_writ_bps_limit", it should be
"rbd_qos_write_bps_limit".

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/cephadm: rewrite "config ssl/tls f. grafana"

This PR streamlines the grammar in the subsection
called "Configuring SSL/TLS for Grafana" in the
monitoring.rst file. It also corrects the prompt
rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

rgw: add config option to disable beast's async process_request()

when disabled, a null_yield is given to process_request() so that it
runs synchronously. mostly intended for debugging

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge PR #41128 into master

* refs/pull/41128/head:
qa/crontab: reduce frequency of pacific nightlies

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #40563 from BryceCao/wip_add_check_for_sync_url

rgw : add check empty for sync url

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #38729 from rosinL/fix-rgw-file-read

rgw/rgw_file: Fix the return value of read() and readlink()

Reviewed-by: Matt Benjamin mbenjamin@redhat.com

Merge pull request #36305 from ivancich/wip-ordered-list-map-efficiency

rgw: ordered list map efficiency

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41108 from dang/wip-dang-zipper-link

RGW Zipper - Remove link/unlink from API

Merge pull request #37720 from ifed01/wip-ifed-alloc-tool-fixes

os/bluestore: some minor fixes/improvements for allocator's stats inquiries

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

doc/_ext: render constraint and policies if available

they are used by rbd options.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr: Fix orch osd rm stop help message

Corrects help message of `ceph orch osd rm --help`
Doc link - https://docs.ceph.com/en/latest/cephadm/osd/#stopping-osd-removal

Signed-off-by: VasishtaShastry <vipin.indiasmg@gmail.com>

Merge pull request #40931 from AmnonHanuhov/wip-refactor_conn_send

crimson/net: Refactor conn::send()

rgw/sts: code to fetch certs using .well-known/openid-configuration url
which is oidc standard compliant.

The bug and fix were suggested by Pietari Hyvarinen <pietari.hyvarinen@csc.fi>

Fixes: https://tracker.ceph.com/issues/50721
Signed-off-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #41218 from rhcs-dashboard/revert-base-href

mgr/dashboard: fix base-href: revert it to previous approach

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #41185 from idryomov/wip-rbd-pwl-reopen

librbd/cache/pwl: fix parsing of cache_type in create_image_cache_state()

Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>
Reviewed-by: Yin Congmin <congmin.yin@intel.com>

blk/kernel: explicit assign to fields in struct

Clang on FreeBSD reports:
```
Building CXX object src/global/CMakeFiles/libglobal_objs.dir/pidfile.cc.o
../src/global/pidfile.cc:170:5: warning: ISO C++ requires field designators to be specified in declaration order; field 'l_whence' will be initialized after field 'l_start' [-Wreorder-init-list]
    .l_start = 0,
    ^~~~~~~~~~~~
../src/global/pidfile.cc:169:17: note: previous initialization for field 'l_whence' is here
    .l_whence = SEEK_SET,
                ^~~~~~~~
```
And Linux and BSD have different orders in their `struct flock`.
It also prevents the wrong initialisation on FreeBSD

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

include/compat: revert to basic locking if a OS does not have F_OFD_SETLK

FreeBSD does not have the F_OFD_* versions for fcntl().
So instead F_SETLK is used instead of F_OFD_SETLK

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

debian/ceph-common.postinst: fix indent and cleanups

* add editor variables for emacs
* replace tab with 8 spaces
* move "then" to previous line to be more consistent

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41141 from ivancich/wip-listing-initial-marker

rgw: fix bucket object listing when marker matches prefix

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #41140 from ivancich/wip-bucket-purge-paging

rgw: radosgw_admin remove bucket not purging past 1,000 objects

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #40886 from pritha-srivastava/wip-rgw-mfa-pin-check

rgw: fix for mfa resync crash when supplied with only one totp_pin.

Reviewed-by: Matt Benjamin mbenjamin@redhat.com

Merge pull request #41166 from tchaikov/wip-cmake-cython-cflags

cmake: remove cflags from CC

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41234 from tchaikov/wip-crimson-common

crimson/common: use string_view when appropriate

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/common: use string_view when appropriate

the typical use case of get_val() passes a literal string as the key,
in that case, there is no need to create a std::string. as
md_config_t::get_val() always accepts a string_view as the option name.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41080 from t-msn/readdir-fix2

os/FileStore: fix to handle readdir error correctly

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #40993 from neha-ojha/wip-50466

osd/PG.cc: handle removal of pgmeta object

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #41143 from idryomov/wip-posix-memalign-fix

common/buffer: adjust align before calling posix_memalign()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41155 from rzarzynski/wip-global-backtrace-bug-50653

log: fix the formatting when dumping thread IDs.

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/seastore: integrate and test remove/list to SeaStore

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: integrate and test oid list to OnodeManager

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: make sure oids in tree are always valid

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: use const reference in test_fltree_onode_manager

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: define and use _MIN/_MAX_OID internally

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: declare and use max key indexes

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: print the signed shard_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix potential overflow in toMatchKindCMP()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: integrate and test onode erase to OnodeManager

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: cleanup OnodeManager curly braces

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: switch tab mode of OnodeManager files to nil

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: make OnodeManager interfaces pure virtual

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41220 from rzarzynski/wip-crimson-monc-honor-cancel

crimson/monc: honor auth_result_t::canceled as the result of do_auth().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41222 from tchaikov/wip-crimson-cleanups

crimson/os: cleanups

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #41223 from rzarzynski/wip-crimson-alienstore-sighup

crimson/alienstore: block SIGHUP to coexist with Seastar's signal handling

Reviewed-by: Kefu Chai <kchai@redhat.com>