crimson/os/seastore/lba_manager: introduce subsys seastore_lba_details

We will share the lba tree implementation, so classify the related logs
and rename to seastore_lba_details for now.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: consolidate logs

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: convert to seastore logging

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/lba_manager: move lba_tree_inner_stats into LBAManager

Cleanup, change stats from LBATree static member to BtreeLBAManager
class member.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44842 from idryomov/wip-krbd-rxbounce-option

rbd: recognize rxbounce map option

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43910 from sunnyku/wip-rbd-snap-sched

mgr/rbd_support: fix schedule remove

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44893 from cfsnyder/wip-54130-oplog-segfault

rgw: fix segfault in OpsLogRados::log when realm is reloaded

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44851 from batrick/i54081

mds: add inline flag to MDS bootstrap compat in MDSMap

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44887 from josephsawaya/vstart-dev

src/vstart.sh: Add --no-restart to vstart & add mkfs logs to osd.stdout

Reviewed-by: Samuel Just <sjust@redhat.com>

src/vstart.sh: Add --no-restart to vstart & add mkfs logs to osd.stdout

The output of mkfs wasn't also being included in the OSD's log before this which
can make it more difficult to debug issues with mkfs.

ceph-run restarting every 5 seconds can make it difficult to read the osd's stdout.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

Merge pull request #44868 from neha-ojha/wip-move-to-stream

qa/distros: remove centos8

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #44866 from yuvalif/wip-yuval-fix-datalog-list

rgw/admin: fix radosgw-admin datalog list max-entries issue

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #44386 from ivancich/wip-reshard-tentative-goal

rgw: in bucket reshard list, clarify new num shards is tentative

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #42330 from SWJTU-ZhangLei/wip-rgw-postobj

rgw: RGWPostObj::execute() may lost data.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44743 from linuxbox2/wip-rgwlc-lcgetv

rgwlc: fix compat-decoding of cls_rgw_lc_get_entry_ret

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44137 from ivancich/wip-fix-reshard-cancel

rgw: "reshard cancel" errors with "invalid argument"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44892 from trociny/wip-54119

rgw: check bucket shard init status in RGWRadosBILogTrimCR

Reviewed-by: Casey Bodley <cbodley@redhat.com>

qa/suites/krbd: add legacy+rxbounce and crc+rxbounce coverage

For basic, rbd and rbd-nomount subsuites, replace legacy and crc
facets with "legacy or legacy+rxbounce" and "crc or crc+rxbounce"
facets (chosen at random).

For fsx, singleton and thrash subsuites, add legacy+rxbounce and
crc+rxbounce facets and drop prefer-crc facet.  The expected behaviour
of the latter depends on cluster configuration and should be tested
separately.

The total number of jobs remains the same.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa: krbd rxbounce test

Lives in its own directory since ms_mode doesn't need to be permuted
here.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rbd: recognize rxbounce map option

Fixes: https://tracker.ceph.com/issues/54063
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44563 from benhanokh/NCB_new_alloc_map

BlueStore:NCB:Bug-Fix for recovery code with shared blobs

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #44796 from pcuzner/remove-old-mib

monitoring: remove old MIB

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44857 from rhcs-dashboard/test-doc-dash-cephadm

doc: update kcli test env documentation

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44059 from MrFreezeex/mixins

mgr/dashboard: monitoring: refactor into ceph-mixin

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #44862 from andriytk/fix-put-memleak2

rgw_sal_motr: fix possible memleak on put

qa/workunits/rbd: improve schedule add/remove cli test

This patch adds few tests to cover schedule add/remove with invalid
inputs.

Signed-off-by: Sunny Kumar <sunkumar@redhat.com>

mgr/rbd_support: fix schedule remove

Issue:

If we provide a random string in the schedule remove
command the entire schedule at specified level gets
removed.

Fixes: https://tracker.ceph.com/issues/53250
Signed-off-by: Sunny Kumar <sunkumar@redhat.com>

Merge pull request #44348 from soumyakoduri/wip-skoduri-lc-teuthology

rgw/qa: Add test suite for lifecycle cases

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

os/BlueStore: NCB fixes recovery code with shared blobs

Replaces the BitmapAllocator used by NCB Recovery code with a dedicated SimpleBitmap.
The SimpleBitmap allows for bits to be set multiple times without any adverse effect.
This is needed beacuse shared-blobs will report the same allocation multiple times.

Fixes: https://tracker.ceph.com/issues/53678
Signed-off-by: Gabriel Benhanokh <gbenhano@redhat.com>

Merge pull request #44003 from zdover23/wip-doc-2021-11-17-start-documenting-ceph-python-pip-update

doc/start: update python-pip to python3-pip

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #44560 from jdurgin/wip-edit-on-github

doc/conf.py: use links to the 'edit' page for edit on github

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44888 from josephsawaya/zns-dev

crimson/os/seastore: change segment_off_t to seastore_off_t in ZNSSegmentManager

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #44691 into master

* refs/pull/44691/head:
doc: document PI .qa symlinks
qa: fix or add missing .qa links
qa: move .qa terminator

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

rgw: fix segfault in OpsLogRados::log when realm is reloaded

We weren't previously handling the deallocation of the store when
a realm was reloaded. Now passing a const reference to the pointer.

Fixes: https://tracker.ceph.com/issues/54130
Signed-off-by: Cory Snyder <csnyder@iland.com>

rgw_sal_motr: fix possible memleak on put

Currently, the MotrAtomicWriter::cleanup() is called from
MotrAtomicWriter::commit(), which may not be called at all
by rgw in case of md5 checksum failure.

Solution: call cleanup() from process() when data is zero.
rgw calls Writer::process(data, off) with zero data at the
end of the loop to allow writes to flush the data. From:
src/rgw/rgw_op.cc:RGWPutObj::execute():

      op_ret = filter->process(std::move(data), ofs);
      ...
      ofs += len;
    } while (len > 0);

    // flush any data in filters
    op_ret = filter->process({}, ofs);

Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>
Reviewed-by: Sining Wu <sining.wu@seagate.com>

rgw_sal_motr: fix cosmetic issues

- rename acc_bl to acc_data
- drop DEBUG prefix from logs

Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>

Merge pull request #44878 from aclamk/fix-git-id-aclamk

github: fix id and mail

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/os/seastore: change segment_off_t to seastore_off_t

Signed-off-by: Joseph Sawaya <josephsawaya938@gmail.com>

mds: add inline feature to MDS bootstrap incompat

File systems that had inline data enabled at some point would have this
bit in the CompatSet "incompat" set. This would conflict during upgrade
with the default v16.2.4 CompatSet assigned to existing (16.2.4-) MDS.
Subsequently, this would cause an assertion in FSMap::sanity during
pending map creation.

This bit will get added anyway during the upgrade process so might as
well add it to the MDS CompatSet during bootstrap.

Fixes: https://tracker.ceph.com/issues/54081
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mds: throw some feature definitions in static memory

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test inline compat set on older MDSMap

Reproduced here:

    /ceph/teuthology-archive/pdonnell-2022-01-31_19:13:02-fs:upgrade-master-distro-default-smithi/6651572/teuthology.log

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: document PI .qa symlinks

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix or add missing .qa links

Using this command:

    find qa/suites/ -type d -execdir ln -sfT ../.qa/ {}/.qa \;

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: move .qa terminator

So links can be elsewhere in the qa suite (not used yet) and to simplify
a find command in a follow-up commit.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

github: fix id and mail

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

monitoring: build jsonnet/jb only for testing

Build jsonnet and jb in the testso that we can build ceph without
internet access and still be able to run the test needed for monitoring
using jsonnet tools.

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

spec: debian: monitoring: build jsonnet from source to use 0.18.0

As this new version is recently released it's still not in every distro
we use. We now build jsonnet from source so that we can use this new
version of jsonnet. This commit could be reverted later on when the new
version would be available everywhere.

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

mgr/dashboard: monitoring: refactor into ceph-mixin

Mixin is a way to bundle dashboards, prometheus rules and alerts into
jsonnet package. Shifting to mixin will allow easier integration with
monitoring automation that some users may use.

This commit moves `/monitoring/grafana/dashboards` and
`/monitoring/prometheus` to `/monitoring/ceph-mixin`. Prometheus alerts
was also converted to Jsonnet using an automated way (from yaml to json
to jsonnet). This commit minimises any change made to the generated files
and should not change neithers the dashboards nor the Prometheus alerts.

In the future some configuration will also be added to jsonnet to add
more functionalities to the dashboards or alerts (i.e.: multi cluster).

Fixes: https://tracker.ceph.com/issues/53374
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

spec: debian: add golang as build dependency

Add golang as a build dependency to build golang project in the test
for monitoring/ceph-mixin.

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

rgw: check bucket shard init status in RGWRadosBILogTrimCR

to avoid using not properly initialized bs.bucket_obj.

Fixes: https://tracker.ceph.com/issues/54119
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #44812 from rhcs-dashboard/box-remove-error

cephadm/box: fix remove image tar error

Merge pull request #44832 from cyx1231st/wip-crimson-msgr-test-timeout

test/crimson/test_messenger: increase timeout

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/rados/thrash-old-clients: remove centos_8.3_container_tools_3.0

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/distros/container-hosts: remove centos_8.2 and centos_8.3

Related to https://tracker.ceph.com/issues/54087

Signed-off-by: Neha Ojha <nojha@redhat.com>

qa/distros/podman: remove centos_8.2 and centos_8.3

Related to https://tracker.ceph.com/issues/54087

Signed-off-by: Neha Ojha <nojha@redhat.com>

rgw/qa: Add test suite for lifecycle cases

Execute lifecycle s3-tests in the teuthology test-suite by configuring
required storage classes and 'rgw lc debug interval' option.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #43707 from BenoitKnecht/ceph-mgr-service-id

mgr: Fix ceph_daemon label in ceph_rgw_* metrics

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

test/crimson/test_messenger: increase timeout

120 seconds are still too short for make check, change to 300.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

rgw/admin: fix radosgw-admin datalog list max-entries issue

Fixes: https://tracker.ceph.com/issues/54116
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #44609 from rhcs-dashboard/daemon_action

mgr/dashboard: perform daemon actions

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

doc: update dashboard kcli test env documentation

Fixes: https://tracker.ceph.com/issues/54105
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44850 from cbodley/wip-qa-rgw-centos8-eol

qa/rgw: rgw/verify no longer pins centos 8.0

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #44810 from rkachach/fix_doc_issue_54018

doc/cephadm: fixing cluster purging section

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44834 from idryomov/wip-rbd-image-id-spec

mgr/rbd_support: fix mon command API regressions

Reviewed-by: Mykola Golub <mgolub@suse.com>

cephadm/box: fix remove image tar error and cleanups

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44820 from NitzanMordhai/nitzan-add-email-contact

Nitzan add email contact

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgwlc:  add cls_rgw_lc_get_entry_ret to rgw_types.h (dencoder)

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #44408 from linuxbox2/wip-rgwlc-warn-attr

rgwlc:  warn on missing RGW_ATTR_LC

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

qa/rgw: rgw/verify no longer pins centos 8.0

the symlink rgw/verify/centos_latest.yaml already selects centos

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44384 from rhcs-dashboard/cephadm-e2e-static-ip

mgr/dashboard: cephadm e2e job: display info on error & other improvements

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44846 from cbodley/wip-qa-centos8-eol

qa: remove centos8 from supported distros

Merge pull request #38750 from xxhdx1985126/wip-rgw-setattr-write-bug-fix

librgw: make rgw file handle versioned

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

qa/distros: remove duplicate centos_8.stream.yaml from supported

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/distros: centos_8.yaml is now a symlink to centos_8.stream.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44805 from Matan-B/wip-matanb-deps-unicode

install-deps.sh: set a UTF-8 locale when running pip

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #35849 from inspur-wyq/wip-decode-user-policy

rgw: add try catch for user policies decode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44785 from xijiacun/default-placement

rgw: Match decode_json with dump for default-placement in RGWZoneGroup.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44562 from ivancich/wip-fix-minor-bi-list-calc

rgw: fix bucket index list minor calculation bug

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #37665 from clwluvw/role-policy-doc

rgw: add infile arg to role-policy put

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #44357 from mfatemipour/rgw-bucket-chown-bad-memory-usage

rgw: bucket chown bad memory usage

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #44395 from BryceCao/wip-add-error-message-when-rm-key

rgw : add error message when rm key

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44399 from BryceCao/wip-modify-error-message-when-user-info-by-access-key

rgw : modify error message when user info by access-key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44833 from fstachura/wip-doc-more-pdf-links

doc: Fix links to CRUSH, RADOS and DSP research papers

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc: Fix links to CRUSH, RADOS and DSP research papers.

Signed-off-by: Franciszek Stachura <fbstachura@gmail.com>

mailmap: Add Nitzan Mordechai

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

qa/suites/rbd: add cram-based mon command API test

With mon (rbd_support mgr module in this case) command definitions
generated automatically by @CLI{Read,Write}Command decorator, it's
very easy to accidentally break the external facing API.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/rbd_support: level_spec is optional for schedule list/status

Commit fea6fdff4c74 ("mgr/rbd_support: level_spec passed to some
commands is not optional") is wrong.  While it is true that a valid
level_spec is needed to create a LevelSpec instance, an empty string
is very much a valid level spec -- it signifies "all levels".

This wasn't caught because within Ceph these commands are wrapped by
rbd CLI which injects an empty string in get_level_spec_args().

Fixes: https://tracker.ceph.com/issues/54058
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/rbd_support: "trash remove" takes image_id_spec, not image_spec

Because of @CLIWriteCommand, the parameter name has to adhere to
the mon command API.  Commit dcb51b067a49 ("mgr/rbd_support: define
commands using CLICommand") accidentally changed image_id_spec to
image_spec, breaking external users such as go-ceph.

Fixes: https://tracker.ceph.com/issues/54057
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44707 from liumiaomiaoIntel/qatdoc

doc/radosgw: update QAT acceleration to specify the configuration

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc/radosgw: update QAT acceleration to specify the configuration

add and change the configuration steps to enable latest version of QAT
driver on QAT based encryption and compression

Signed-off-by: Miaomiao Liu <miaomiao.liu@intel.com>

rgwlc: fix compat-decoding of cls_rgw_lc_get_entry_ret

Fix compat-decode of cls_rgw_lc_get_entry_ret, which had changed
earlier in 394750597.  While not initially a problem, the more
recent change to allow radosgw-admin lc process to operate on a single
bucket created a way to decode an un-upgraded structure.

Fixes: https://tracker.ceph.com/issues/53927
Reported by Jeegn Chen <jeegnchen@gmail.com>.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #44816 from cbodley/wip-rgw-cmake-system

cmake/rgw: src/rgw and src/rgw/services are not system headers

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: in bucket reshard list, clarify new num shards is tentative

With dynamic bucket index resharding, when the average number of
objects per shard exceeds the configured value, that bucket is
scheduled for reshard. That bucket may receive more new objects before
the resharding takes place. As a result, the existing code
re-calculates the number of new shards just prior to resharding,
rather than waste a resharding opportunity with too low a value.

The same holds true for a user-scheduled resharding.

A user reported confusion that the number reported in `radosgw-admin
reshard list` wasn't the number that the reshard operation ultimately
used. This commit makes it clear that the new number of shards is
"tentative". And test_rgw_reshard.py is updated to reflect this
altered output.

Additionally this commit adds some modernization and efficiency to the
"reshard list" subcommand.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

doc/cephadm: fixing cluster purging section
https://tracker.ceph.com/issues/54018
ceph orch is not enough to stop all cephadm operations

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #44801 from sebastian-philipp/doc-cephadm-count-per-host

doc/cephadm: Co-location of daemons

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by:Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #44823 from andriytk/fix-install-deps2

install-deps.sh: fix cortx-motr installation condition

install-deps.sh: fix cortx-motr installation condition

Don't try to install cortx-motr if it is already installed
or not needed. (Currently, it installs on any run from an
interactive terminal session).

Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>

mgr/dashboard: perform daemon actions on cluster->services

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Fixes: https://tracker.ceph.com/issues/50322

Merge pull request #44711 from zdover23/wip-doc-2022-01-21-rados-bluestore-cache-autotune-interval-rewrite-2

src/rados: rewrite bs cache autotune instructions

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #44703 from phlogistonjohn/jjm-issue46773

cephadm: validate that the constructed YumDnf baseurl is usable

Reviewed-by: Michael Fritch <mfritch@suse.com>