qa/tasks/cephadm: Include bootstrap registry options for downstream
- registry-url, registry-username and registry-password bootstrap options are
supported now. This is needed to access monitoring service container images.
- usage of RHEL distribution based cephadm in download_cephadm task.

Signed-off-by: sunilkumarn417 <sunnagar@redhat.com>

Merge pull request #41363 from Aran85/crimson-fix-syntax

crimson/seastore: remove unused method

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/seastore: remove unused method

Signed-off-by: Zengran Zhang <zhangzengran@sangfor.com.cn>

Merge pull request #41480 from MrFreezeex/fix-segfault-replayer-snapshot-shutdown

rbd-mirror: fix segfault in snapshot replayer shutdown

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41433 from tchaikov/wip-50891

os/bluestore/bluestore_tool: compare retval stat() with -1

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #41429 from ifed01/wip-ifed-fix-repair-multithreading

os/bluestore: introduce multithireading sync for bluestore's repairer

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #41436 from runsisi/wip-fix-unit

common/options: fix option type for bluestore_block_db_size

Reviewed-by: Igor Fedotov <ifedotov@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41466 from ansiwen/install-cepfs-headers

include/cephfs: add cephfs headers to CMakeLists.txt

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #40910 from galsalomon66/update_s3select_submodule_and_s3tests_head_10apr

rgw/s3select: update s3select submodule to last commit, (new features), update for test coverage(s3test)

Merge PR #41479 into master

* refs/pull/41479/head:
qa/tasks/cephadm.conf: log_to_journald=false

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #41472 from cyx1231st/wip-seastore-onode-tree-errorhandling

crimson/onode-staged-tree: tolerate eagain and add proper errorhandling

Reviewed-by: Samuel Just <sjust@redhat.com>

update to s3select/master (new features)

Signed-off-by: gal salomon <gal.salomon@gmail.com>
force-branch to s3test/master

Signed-off-by: gal salomon <gal.salomon@gmail.com>
update of s3select (== -> =)

Signed-off-by: gal salomon <gal.salomon@gmail.com>
update of s3select (== -> =)

Signed-off-by: gal salomon <gal.salomon@gmail.com>
closing documentation gaps between previous and current functionalities(WIP)

Signed-off-by: gal salomon <gal.salomon@gmail.com>
Update s3select.rst

additional features

Update s3select.rst

Signed-off-by: gal salomon <gal.salomon@gmail.com>
Update s3select.rst

editorial
Signed-off-by: gal salomon <gal.salomon@gmail.com>
Update s3select.rst

editorial
Signed-off-by: gal salomon <gal.salomon@gmail.com>
Update s3select.rst

adding cast to bool
Signed-off-by: gal salomon <gal.salomon@gmail.com>
editorial; more description for substring and trim

Signed-off-by: gal salomon <gal.salomon@gmail.com>

Merge PR #41441 into master

* refs/pull/41441/head:
.github/labeler: add nfs label

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

rbd-mirror: fix segfault in snapshot replayer shutdown

If an error arises in the init flow of the snapshot replayer and the
function returns before the call on `register_local_update_watcher`
the value of `m_update_watch_ctx` will not be initialized. Therefore,
on the shutdown phase, the replayer will try to free this pointer
and segfault.

This commit fixes this issue by setting `m_update_watch_ctx` to
`nullptr`.

Fixes: https://tracker.ceph.com/issues/50931
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

Merge pull request #41464 from jdurgin/wip-bib

script/build-integration-branch: always generate merge commits

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #41476 from tchaikov/wip-crimson-options

crimson/osd: disable allow_guessing when parsing command line options

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/tasks/cephadm.conf: log_to_journald=false

For teuthology runs, we set log_to_stderr=false, so that we only see
derr-level events in the container log (and teuthology.log).  Now that we
log directly to journald, set log_to_journald=false too, so that we don't
see level-20 logs in teuthology.log.

Signed-off-by: Sage Weil <sage@newdream.net>

Merge pull request #41474 from rhcs-dashboard/fix-50918-master

mgr/dashboard: remove non-null id in Grafana dashboard

Reviewed-by: Guillaume Abrioux <gabrioux@redhat.com>

crimson/osd: disable allow_guessing when parsing command line options

we pass "--id <n>" to ceph-osd for specifying the osd id, but seastar
app template also provides an option of "--idle-poll-time-us arg".
boost::program_option::command_line_parser() uses default_style when
parsing options. and default_style includes allow_guessing, which in
turn matches partial option as well, so "--id" matches with "--idle"
when we are trying to figure out which options are consumed by seastar
app template, and which are not. see
https://www.boost.org/doc/libs/1_76_0/doc/html/boost/program_options/command_line_style/style_t.html

so, in this change, stype is specified explicitly, and "allow_guessing"
is removed from the "default_style" before being passed to style(), so
that only the full option name are matched.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: remove non-null id in Grafana dashb

Testing added to prevent this situation.

Fixes: https://tracker.ceph.com/issues/50918
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

os/bluestore/bluestore_tool: use std::filesystem

for better readability

Signed-off-by: Kefu Chai <kchai@redhat.com>

os/bluestore/bluestore_tool: use boost::filesystem as an alternative

the libstdc++ shipped with GCC 7.5 does not have good support of
std::filesystem, among other things, it does not offer
std::filesystem::weakly_canonical(). but boost::filesystem does.
and boost::filesystem is compatible with std::filesystem to some
degree. so let's use it if <filesystem> is not available, we can
take it as a signal that std::filesystem is not quite ready yet.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/onode-staged-tree: detect errors from seastore backend

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: add asserts

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: tolerate eagain during fix_index()

Fix the one-directional link during fix_index() which causes errors
during node destruction under interruptive eagain.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup Node tracking logic for eagain

Introduce deref_super/parent() to make sure the bi-directional links
are reset together to survive eagain.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: reduce unit test efforts

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: validate insert/lookup/erase with eagain

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: fix initialization in perf tool

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: free resources when call submit_transaction()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: tolerate eagain between extent allocation and node initialization

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: distinguish extent state between retired and invalid

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: extend SeastoreNodeExtentManager to test eagain

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: cleanup SeastoreSuper::write_root_laddr()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: change FLTreeOnodeManager to use create_seastore()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: simplify errors in OnodeManager

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: use eagain_future/ertr internally

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/onode-staged-tree: handle fatal errors when call NodeExtentManager

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41467 from anthonyeleven/anthonyeleven/rgw-doc-tweaks

doc/radosgw: update and clarify layout and placement

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

os/bluestore/bluestore_tool: compare retval stat() with -1

before this change, stat() is always called to check if the
file specified by --dev-target exists even if this option is not
specified. also, we compare the retval of stat() with ENOENT, while
state() returns -1 on error.

after this change, stat() is called only if --dev-target is specified,
and we compare the retval of stat() with -1 and 0 only, so if
--dev-target option is not specified, the tool still hehaves.

this change addresses a regression introduced by
94a91f54fe30a4dd113fbc1b02bc3f3d52c82a92

Fixes: https://tracker.ceph.com/issues/50891
Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41457 from rzarzynski/wip-crimson-test-ertr-return-int

test/crimson: verify the futurization in an errorated future.

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41456 from athanatos/sjust/wip-seastore-logging

crimson/os/seastore: fix build failure

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

include/cephfs: add cephfs headers to CMakeLists.txt

If WITH_LIBCEPHFS is set, the cephfs headers are installed as well by
the install target.

Signed-off-by: Sven Anderson <sven@redhat.com>

Merge pull request #41382 from tchaikov/wip-doc-msgr-fsm-diagram

doc/dev/msgr2: add state machine diagrams

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41408 from zdover23/wip-doc-security-2021-05-19-fourth-item

doc/security: updating fourth listitem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

crimson/os/seastore/logging.h: fix LOG/LOGT macros

Signed-off-by: Samuel Just <sjust@redhat.com>

Revert "crimson/os/seastore/logging: remove comma"

This wasn't the correct fix.

This reverts commit 0ce58c2066d69ce82d5c7728a085921bc5cec759.

Signed-off-by: Samuel Just <sjust@redhat.com>

script/build-integration-branch: always generate merge commits

This guarantees we'll always get a new sha1, so shaman will generate a
new build even if the same PRs are pushed to more than one branch.

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41407 from zdover23/wip-doc-security-2021-05-19-second-item

Wip doc security 2021 05 19 second item

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #41358 into master

* refs/pull/41358/head:
mds: add heartbeat_reset() in start_files_to_reover() to avoid timedout issue

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41338 into master

* refs/pull/41338/head:
mds: reset code after cpu_profiler
cpu_profiler: fix asok command crash

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #41084 into master

* refs/pull/41084/head:
test: test to verify dir path removal when no mirror daemons are running
pybind/mirroring: advance state machine from stalled state
pybind/mirroring: start from correct state during policy init

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

test/crimson: verify the futurization in an errorated future.

`seastar::future` allows a lambda passed to e.g. `then()` to
return not only a future but also any non-future type like
a plain integer:

  ```cpp
  seastar::now().then([] {
    return 42;
  }).then([] (int futurized_life) {
    // ...
  });
  ```

In such case, the plain type is being wrapped in a future.
This process is called _futurization_ and is performed by
`seastar::futurize` called from the internals of `future`.

In this commit we want to ensure analogous facility is offered
by errorated futures.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/security: updating fourth item

This PR makes minor changes (nitpicking, really)
to make the sentence a little easier to read.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/security: enriching second point

This PR breaks a long sentence into two shorter
sentences and improves the language.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41411 from zdover23/wip-doc-security-2021-05-19-eighth-item

doc/security: enriching eighth listitem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41437 from tchaikov/wip-crimson-logging

crimson/os/seastore/logging: remove comma

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #41070 from sunnyku/wip-rbd-50522

rbd: fix default pool handling for nbd map/unmap

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #41340 from tchaikov/wip-ceph-allocator

librbd: do not use ceph::allocator<>

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/security: enriching eighth listitem

This PR improves the language in the eighth
listitem in the Vulnerability Management Process.

(This one's pretty nitpicky.)

Signed-off-by: Zac Dover <zac.dover@gmail.com>

.github/labeler: add nfs label

Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #40328 from rhcs-dashboard/fix-49262-master

mgr/dashboard: provide the service events when showing a service in the UI

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os/seastore/logging: remove comma

to address the FTBFS of the release builds.

fix a regression introduced by c39d64d7bbd7d7cf1d476a44f60c70edce55f3d1

Signed-off-by: Kefu Chai <kchai@redhat.com>

common/options: fix option type for bluestore_block_db_size

the type should be `size` to support both SI and IEC units

Signed-off-by: luo.runbing <luo.runbing@zte.com.cn>

Merge pull request #41396 from tchaikov/wip-crimson-metrics

crimson/os: replace perfcounter with seastar::metrics

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #41258 from ronen-fr/wip-ronenf-repair-stat

osd/scrub: separate between PG state flags and internal scrubber operation

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41264 from mgfritch/cephadm-sysctl-osd

cephadm: apply sysctl settings

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #41368 from idryomov/wip-monc-rotating-timeout

mon/MonClient: tolerate a rotating key that is slightly out of date

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #41277 from david-caro/consider_dev_root_mounted

ceph-volume: Consider /dev/root as mounted

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #41287 from adk3798/daemon-for-service-warn

mgr/cephadm: warn users if they try to pass a daemon to 'orch rm'

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #41356 from tchaikov/wip-options-mgr-mon

common/options: extract mgr and mon options out

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #41373 from myoungwon/wip-fix-50806

 osd: fix wrong input when calling recover_object()

Reviewed-by: Samuel Just <sjust@redhat.com>

test/crimson: remove unnecessary local variable

to return the return value of "make_unique<>()" is more readable, and more
consistent with other part of this source file.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson/seastore: teardown in reactor

otherwise, we rely on the destructor of TMTestState to teardown the
fixuture created in TMTestState::_init(), but TMTestState::_init() is
called in reactor. the objects like seastar::metric_groups are
supposed to be destroyed on the same thread where they are created.
because they use thread local storage of storing persisting their status.
if we destroy objects like seastar::metric_groups on different reactor
or thread where they are created, we would have memory leak and
unexpected behavior.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: drop seastore_perf_counters.h

it is not used anymore after the seastar::metrics migration

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/tools/store_nbd: remove PerfService

since PerfCounter is not used in seastore, there is no need to keep an
instance of PerfService.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: do not create PerfService for SeaStore

because we are replacing PerfCounter with seastart::metrics in crimson,
and the former has already a sharded service builtin in seastar, there
is no need to create PerfService or its counterpart for SeaStore
anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: move ctor of SeaStore into .cc

so the ctor and dtor are colocated in the same file. for better
readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: replace perfcounter with metrics in SegmentCleaner

for better support of m:n mapping, and better maintainability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: register metrics() in TransactionManager's ctor

this change should have been a part of
2356da856784e78750ba7f3b381b31b1c8ad5672

also do not include the unused header file

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/os: move ctor of SegmentCleaner into .cc

will add more stuff into it. and presumably, smaller header file, faster
compilation.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #41423 from athanatos/sjust/wip-lba-merge-fix

crimson/.../lba_btree_node_impl: handle relative addr in merge

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/onode-staged-tree: use fine-grained errorator in NodeExtentManager

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #41392 from tchaikov/wip-librados

include/rados/librados.h: use UINT64_C to define constants

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #41409 from zdover23/wip-doc-security-2021-05-19-fifth-item

doc/security: updating fifth listitem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41410 from zdover23/wip-doc-security-2021-05-19-seventh-item

doc/security: enrich seventh listitem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #41424 from athanatos/sjust/wip-interruptor-unittest-name

test/crimson/CMakeLists.txt: fix name of unittest-interruptible-future

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@360.cn>

Merge pull request #41425 from athanatos/sjust/wip-seastore-build

crimson: fix misc build issues

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore: introduce multithireading sync for bluestore's repairer

In quick-fix mode bluestore uses 2 threads by default to perform the
repair. Due to lacking synchronization they might corrupt repair
transaction batch.

Fixes: https://tracker.ceph.com/issues/50017
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

test/bluestore: add test case to reproduce #50017

This issue is caused by the lack of multithreading sync when doing
bluestore's quick-fix.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

crimson/tools/store_nbd/tm_driver: remove perfcounter argument

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/logging: fix log() argument

Signed-off-by: Samuel Just <sjust@redhat.com>

test/crimson/CMakeLists.txt: fix name of unittest-interruptible-future

unittest-interruptible-future matches the other unit tests here.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/.../lba_btree_node_impl: handle relative addr in merge

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #40836 from kamoltat/wip-autoscaler-new-alg-turn-on-off

pybind/mgr/pg_autoscaler: Added autoscale-profile feature

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

pybind/mgr/pg_autoscaler: Added autoscale-profile feature

The autoscaler by default will start out each pool with minimal
pgs and `scale-up` the pgs when there is more usage in each pool.

Users can now use the commands:

`osd pool set autoscale-profile scale-down` to make the pools
start out with a full complement of pgs and only `scale-down`
when usage ratio across the pools are not even.

`osd pool set autoscale-profile scale-up` (by default) to make the pools
start out with minimal pgs and `scale-up` the pgs when there
is more usage in each pool.

Edited KVMonitor.cc file to make the `autoscale_profile` variable
persistent.

Edited tests/test_cal_final_pg_target.py so that it takes into account
the new `profile` argument when calling cal_final_pg_target(). Also,
added some new test cases for when profile is `scale-up`

Renamed tests/test_autoscaler.py to a more appropriate name:
tests/test_cal_ratio.py

Signed-off-by: Kamoltat <ksirivad@redhat.com>

doc/security: enrich seventh listitem

This PR improves the language of the seventh listitem
in the Vulnerability Management Process in the security
documentation.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41406 from zdover23/wip-doc-security-2021-05-19-first-item

doc/security: removing pleonasm

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

doc/security: updating fifth listitem

This PR enriches the text in the fifth listitem
of the Vulnerability Management Process.

Signed-off-by: Zac Dover <zac.dover@gmail.com>