crimson/os/seastore: adjust logs to FixedKVBtree

* Fix the subsys name.
* Change FixedKVBtree logs to trace level -- seems too verbose when the
  global level is debug.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: add metrics for extent rewrite version

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: improve and fix tree metrics

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: add metrics about journal sizes

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: add metrics about why IO is blocking

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #46292 from anthonyeleven/anthonyeleven/clarify-min-alloc-size2

doc/rados/configuration: clarify VMware note in bluestore-config-ref.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/rados/configuration: clarify VMware note in bluestore-config-ref.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #46265 from ivancich/wip-misplaced-dout_subsys-def

rgw: remove potentially conficting definition of dout_subsys

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #46158 from cyx1231st/wip-seastore-cleaner

crimson/os/seastore/segment_cleaner: misc cleanups and improvements

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #45662 from zenomri/wip-cond-tracing

rgw: support conditional tracing using lua script

Reviewed-by: Casey Bodley <cbodley@redhat.com>, Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #46274 from xxhdx1985126/wip-admin-sock

crimson/admin: deal with seastar's new metric types/logics

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/admin: deal with seastar's new metric types/logics

Fixes: https://tracker.ceph.com/issues/55666
Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #45962 from ronen-fr/wip-rf-sbe-test2

test/osd: test environment for the scrubber "backend"

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #46270 from anthonyeleven/anthonyeleven/clarify-min-alloc-size

doc/rados/configuration: Enhance BlueStore min_alloc_size section

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #45947 from optimistyzy/418_add_nvme_tcp_support

blk/spdk:  add support to use NVMe device provided by NVMe-oF target

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46226 from ronen-fr/wip-rf-cln-typesh

osd: cleanup object_stat_sum_t initialization

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

blk/spdk: Add the support to use nvme device provided by NVMe-of Target

This patch is used to add the support to use the nvmedevice provided
by NVMe-oF target.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>

Merge pull request #45886 from lxbsz/wip-55283

qa: add filesystem/file sync stuck test support

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45955 from lxbsz/wip-55240

mds: trigger to flush the mdlog in handle_find_ino()

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46063 from vshankar/wip-55170

mds: fetch all dirfrag entries when no dentry keys are requested

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46153 from lxbsz/wip-54411-1

client: force send global open_files/metadata metrics

Reviewed-by: Venky Shankar <vshankar@redhat.com>

rgw: remove potentially conficting definition of dout_subsys

dout_subsys should strictly be defined in .cc files. There was a
definition of dout_subsys in rgw_log.h, which is included in a variety
of .cc files due to transitive includes. When a subsystem wants a
conflicting definition, a compiler warning is then generated due to
the redefinition of dout_subsys.

The definition is removed from rgw_log.h and follow-on fixes are made.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #46258 from xxhdx1985126/wip-backref-minor-bug-fixes

crimson/os/seastore/cache: backref buffer minor bug fix

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #46154 from zdover23/wip-doc-2022-04-14-dev-guide-basic-workflow

doc/dev: update basic-workflow.rst

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #45581 from MrFreezeex/jsonnet-multiline-cleanup

ceph-mixin: rewrite promql queries to multiline

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #46142 from tchaikov/wip-cmake-fuse

cmake: mark FUSE REQUIRED if WITH_FUSE

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46237 from vshankar/wip-add-multids-stray-check-for-upgrade

doc: add note about checking pending stray directory entries before initiating upgrade

Reviewed-by: Venky Shankar <vshankar@redhat.com>

crimson/os/seastore/cache: backref buffer minor bug fix

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

doc: add note about checking pending stray directory entries before initiating upgrade

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46236 from vshankar/wip-manual-upgrade-add-check

doc: add setting/unsetting `mon_mds_skip_sanity` config for manual ceph-mds upgrade

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46206 from votdev/issue_55578_prom_am_silence

mgr/dashboard: Creating and editing Prometheus AlertManager silences is buggy

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/os/seastore: fix metrics with TRIM_BACKREF transaction

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: measure the initial reclaim ratio

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: distinguish journal/ool when count open/close/release

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: distinguish in-journal and type-journal segments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: exclude utilizations from empty/open segments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: refine metrics

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: fix an issue that caused forever-GC

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: improve space calculations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: classify space into available, unreclaimable and reclaimable

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: cleanup, track available space in open segments

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: unify update_segment_avail_bytes()

* unify update_segment_avail_bytes() and set_journal_head() interfaces.
* reuse segment_info_t::written_to to get the current journal head.
* more strict validations about journal head maintainence.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: misc assert reinforcements

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: add missing maybe_wake_on_space_used()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: cleanup, drop the unnecessary stopping flag

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: fix journal tail calculations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_allocator: simplify close_segment()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: cleanup and validate config_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/segment_cleaner: drop unused code

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #46256 from xxhdx1985126/wip-backref-big-buffer

crimson/os/seastore: use a whole large backref buffer in cache instea…

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: use a whole large backref buffer in cache instead of several small bufs

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #45368 from a16bitsysop/nasm

nasm-wrapper: ignore -O flags passed

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46244 from tchaikov/wip-bump-up-seastar

crimson: bump up seastar

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #46227 from rhcs-dashboard/angular-production-fix

mgr/dashboard: avoid overriding the configuration in package.json

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

seastar: bump up seastar submodule to recent master

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

crimson: replace DERIVE metrics with COUNTER metrics

in a recent change in Seastar, DERIVE metric was dropped in favor
of COUNTER.

in this change, all DERIVE metrics are replaced with COUNTER metrics.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46239 from tchaikov/wip-crimson-nbd

crimson/tools/store_nbd: s/get_paddr/get_val/

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #46231 from rhcs-dashboard/update-id-email

githubmap: Email and Id updated

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #45950 from rishabh-d-dave/qa-cephfs-test_acls

qa/cephfs: minor fixes for xfstests_dev.py and test_acls.py

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45829 from dparmar18/fix_put

cephfs-shell: fix put and get cmd

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45834 from ajarr/wip-53996

qa/suites/fs/upgrade: change octopus installations and references

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46161 from joscollin/wip-B55516-extra-data

qa: fix is_addr_blocklisted() to get blocklisted clients from 'osd dump'

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46192 from rishabh-d-dave/qa-cephfs-mount-omit_sudo-no-effect

qa/cephfs: omit_sudo must be passed to underlying method...

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45185 from a16bitsysop/quincy-32bit

src/os/bluestore/BlueStore.cc: Use static_cast<uint64_t> fix for 32bit

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson/tools/store_nbd: s/get_paddr/get_val/

it is a follow-up change of ec89b17f2a1724addee7531c97c616fe399cd1cd
which breaks the build of the nbd tool.

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

src/os/bluestore/BlueStore.cc: Use static_cast<uint64_t> to fix 32bit compilation

Fixes: https://tracker.ceph.com/issues/54415
Signed-off-by: Duncan Bellamy <dunk@denkimushi.com>

doc: add setting/unsetting `mon_mds_skip_sanity` config for manual ceph-mds upgrade

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46136 from athanatos/sjust/wip-crimson-operation-refactor-for-review

crimson: refactor osd operation pipeline

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #45390 from xxhdx1985126/wip-backref2

crimson/os/seastore: add back-reference

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #46126 from soumyakoduri/wip-skoduri-dbstore-vstart

rgw/dbstore: Support user creation via `radosgw-admin`

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

githubmap: Email and Id upadted

Signed-off-by: Sarthak0702 <sarthak.dev.0702@gmail.com>

Merge pull request #45064 from s-t-e-v-e-n-k/no-direct-use-of-nose

test: No direct use of nose

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #45762 from NitzanMordhai/wip-nitzan-test-cls-lock-expiered-locks

cls_lock: check expired lock before unlock

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #45529 from NitzanMordhai/wip-nitzan-pglog-dups-not-trimmed

osd/PGLog.cc: Trim duplicates by number of entries

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #44839 from NitzanMordhai/wip-nitzan-mon-ok-to-stop

mon/Monitor: Validate monitor ids during 'mon ok-to-stop' cmd

Reviewed-by: Sridhar Seshasayee <sseshasa@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: avoid overriding the configuration in package.json

the warning in npm run build is `Option "configuration" was already specified with value "en-US". The
new value "production" will override it.`

This causes the dist for production env to be created in dist/ whereas
it should be dist/en-US (or whatever npm_config_locale. That prevented
from dashboard to load and cause the cephadm e2e checks to fail.

https://jenkins.ceph.com/job/ceph-dashboard-cephadm-e2e-nightly-master/185/consoleFull#57532618373a8703b-5adb-41c5-84a0-8cf4e065ba3d

Related to https://github.com/ceph/ceph/pull/46143

Signed-off-by: Nizamudeen A <nia@redhat.com>

qa: testcase test_get_without_target_name fixed

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

doc/man/cephfs-shell.rst: get cmd must accept both path mandatorily

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

cephfs-shell: get cmd must get both path and should validate them

Description:
- While using `get` command, `local_path` parameter is optional. Changing it
  to mandatory.
  - Rationale: Till now, there used to be a default path of `local_path` as
               `default='.'` but wasn't mentioned anywhere. It led to confusion.
               On top of it, considering get command to be a ssh inspired utlity,
               or any other CLI tool that copies file between filesystems, source
               and destination path are always mandatory. Therefore in order to
               simulate this behavior in cephfs-shell`s command(s), my opinion is
               to make get command accept both the paths.

- Added checks to make sure:
1) File does exist at `remote_path`
2) File with the same name doesn't exist in `local_path`
3) Removed code that would run through the directory and if it finds
   nothing in `root_src_dir`, then it will try to do:
   `os.makedirs(root_dst_dir + b'/' + root_src_dir)`, but it will
   never be empty as 1) takes care of it.

Fixes: https://tracker.ceph.com/issues/55216
Signed-off-by: dparmar18 <dparmar@redhat.com>

qa: added testcase to check get cmd doesn't create subdirs on its own

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

qa: path in test_get_with_target_name() fixed

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

cephfs-shell: get command should never create a directory on its own

Descrption: When using the get command on a single file, it would append the remote path to
            source path and create directories that shouldn't be created. For instance,
            file 'foo.txt' resides at `/dir1/dir2/` and `get` command is used to copy it to
            /tmp/foo then it would do `/tmp/dir1/dir2/foo.txt` which is not the expected
            behavior. Therefore this PR intends to correct this behavior.

Fixes: https://tracker.ceph.com/issues/55112
Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

qa: removed test_without_target_dir

Rationale: get and put now demand both the paths mandatorily.
           Also testing of get and put without target paths
           have been take of in other tests in class TestGetAndPut().

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

doc/man/cephfs-shell.rst: put cmd must accept both path mandatorily

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

qa: added tests to make sure put accepts both paths and validate local path

Signed-off-by: Dhairya Parmar <dparmar@redhat.com>

cephfs-shell: put cmd must accept both paths and validate local_path

Description:

- `put` command didn't display any error when file at local_path was not
   found. This PR intends to add that check.
   - Rationale: Till now, there used to be a default path of `remote_path` as
                `default='.'` but wasn't mentioned anywhere. It could lead to confusion.
                On top of it, considering put command to be a ssh inspired utlity,
                or any other CLI tool that copies file between filesystems, source
                and destination path are always mandatory. Therefore in order to
                simulate this behavior in cephfs-shell`s command(s), my opinion is
                to make put command accept both the paths.

Fixes: https://tracker.ceph.com/issues/55242
Signed-off-by: dparmar18 <dparmar@redhat.com>

osd: cleanup object_stat_sum_t initialization

Fix the confusing combination of default-member-initializers
for some members and direct initialization of others in
the ctor.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

rgw: support conditional tracing using lua script

before lua script is being executed, we keep the tracer runtime configuration value, and then decides whether to trace or not the request based on the value that maybe changed during lua exeuction, so we can disable/enable tracing for request even if the tracer is in the opposite state at the same time

Signed-off-by: Omri Zeneva <ozeneva@redhat.com>

test: fixing compilation issues with kv_store_bench

Existing code relied on an include path that had
specified 'using std'.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

test/osd/scrub: scrubber backend test files

introducing the scrubber_generators to create
scrubber test data, and the scrubber_test_datasets
for pre-prepared test configurations.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

test/scrub/osd: scrubber-related interface changes

required to facilitate testing

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #46164 from rishabh-d-dave/vstart-minor-fixes

qa/vstart_runner: minor bug fix and minor adjustments

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46217 from Matan-B/wip-matanb-c-shard_services

crimson/osd: Moving shard_services to PGBackend

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #46208 from rzarzynski/wip-crimson-main-using-cleanups

crimson/osd: drop "using ..." from main.cc

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #46110 from dparmar18/fix_onecmd

cephfs-shell: make onecmd() print proper error msg

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #46047 from dparmar18/qa_testcephfs_shell_pep8_fixes

qa/cephfs/test_cephfs_shell: Fixed pep8 formatting violations

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #45808 from rhcs-dashboard/add-cpu/mem-usage

mgr/dashboard: introduce memory and cpu usage for daemons

Reviewed-by: Sarthak0702 <NOT@FOUND>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: ceph-jenkins <NOT@FOUND>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>
Reviewed-by: sunilangadi2 <NOT@FOUND>

Merge pull request #46190 from rhcs-dashboard/smart-data-fix

mgr/dashboard: fix smart data error

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Paul Cuzner <pcuzner@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

doc/dev: update basic-workflow.rst

This PR updates the basic-workflow.rst file
to serve the needs of people in 2022 who were not
present at jump street.

The text has been refined up to the section called
"Integration Tests" (non-inclusive).

Signed-off-by: Zac Dover <zac.dover@gmail.com>