monitoring: build jsonnet/jb only for testing

Build jsonnet and jb in the testso that we can build ceph without
internet access and still be able to run the test needed for monitoring
using jsonnet tools.

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

spec: debian: monitoring: build jsonnet from source to use 0.18.0

As this new version is recently released it's still not in every distro
we use. We now build jsonnet from source so that we can use this new
version of jsonnet. This commit could be reverted later on when the new
version would be available everywhere.

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

mgr/dashboard: monitoring: refactor into ceph-mixin

Mixin is a way to bundle dashboards, prometheus rules and alerts into
jsonnet package. Shifting to mixin will allow easier integration with
monitoring automation that some users may use.

This commit moves `/monitoring/grafana/dashboards` and
`/monitoring/prometheus` to `/monitoring/ceph-mixin`. Prometheus alerts
was also converted to Jsonnet using an automated way (from yaml to json
to jsonnet). This commit minimises any change made to the generated files
and should not change neithers the dashboards nor the Prometheus alerts.

In the future some configuration will also be added to jsonnet to add
more functionalities to the dashboards or alerts (i.e.: multi cluster).

Fixes: https://tracker.ceph.com/issues/53374
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

spec: debian: add golang as build dependency

Add golang as a build dependency to build golang project in the test
for monitoring/ceph-mixin.

Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

Merge pull request #44812 from rhcs-dashboard/box-remove-error

cephadm/box: fix remove image tar error

Merge pull request #44832 from cyx1231st/wip-crimson-msgr-test-timeout

test/crimson/test_messenger: increase timeout

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43707 from BenoitKnecht/ceph-mgr-service-id

mgr: Fix ceph_daemon label in ceph_rgw_* metrics

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

test/crimson/test_messenger: increase timeout

120 seconds are still too short for make check, change to 300.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #44609 from rhcs-dashboard/daemon_action

mgr/dashboard: perform daemon actions

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44850 from cbodley/wip-qa-rgw-centos8-eol

qa/rgw: rgw/verify no longer pins centos 8.0

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #44810 from rkachach/fix_doc_issue_54018

doc/cephadm: fixing cluster purging section

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #44834 from idryomov/wip-rbd-image-id-spec

mgr/rbd_support: fix mon command API regressions

Reviewed-by: Mykola Golub <mgolub@suse.com>

cephadm/box: fix remove image tar error and cleanups

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44820 from NitzanMordhai/nitzan-add-email-contact

Nitzan add email contact

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #44408 from linuxbox2/wip-rgwlc-warn-attr

rgwlc:  warn on missing RGW_ATTR_LC

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

qa/rgw: rgw/verify no longer pins centos 8.0

the symlink rgw/verify/centos_latest.yaml already selects centos

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44384 from rhcs-dashboard/cephadm-e2e-static-ip

mgr/dashboard: cephadm e2e job: display info on error & other improvements

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44846 from cbodley/wip-qa-centos8-eol

qa: remove centos8 from supported distros

Merge pull request #38750 from xxhdx1985126/wip-rgw-setattr-write-bug-fix

librgw: make rgw file handle versioned

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

qa/distros: remove duplicate centos_8.stream.yaml from supported

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/distros: centos_8.yaml is now a symlink to centos_8.stream.yaml

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44805 from Matan-B/wip-matanb-deps-unicode

install-deps.sh: set a UTF-8 locale when running pip

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #35849 from inspur-wyq/wip-decode-user-policy

rgw: add try catch for user policies decode

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44785 from xijiacun/default-placement

rgw: Match decode_json with dump for default-placement in RGWZoneGroup.

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44562 from ivancich/wip-fix-minor-bi-list-calc

rgw: fix bucket index list minor calculation bug

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #37665 from clwluvw/role-policy-doc

rgw: add infile arg to role-policy put

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #44357 from mfatemipour/rgw-bucket-chown-bad-memory-usage

rgw: bucket chown bad memory usage

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #44395 from BryceCao/wip-add-error-message-when-rm-key

rgw : add error message when rm key

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44399 from BryceCao/wip-modify-error-message-when-user-info-by-access-key

rgw : modify error message when user info by access-key

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44833 from fstachura/wip-doc-more-pdf-links

doc: Fix links to CRUSH, RADOS and DSP research papers

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc: Fix links to CRUSH, RADOS and DSP research papers.

Signed-off-by: Franciszek Stachura <fbstachura@gmail.com>

mailmap: Add Nitzan Mordechai

Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

qa/suites/rbd: add cram-based mon command API test

With mon (rbd_support mgr module in this case) command definitions
generated automatically by @CLI{Read,Write}Command decorator, it's
very easy to accidentally break the external facing API.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/rbd_support: level_spec is optional for schedule list/status

Commit fea6fdff4c74 ("mgr/rbd_support: level_spec passed to some
commands is not optional") is wrong.  While it is true that a valid
level_spec is needed to create a LevelSpec instance, an empty string
is very much a valid level spec -- it signifies "all levels".

This wasn't caught because within Ceph these commands are wrapped by
rbd CLI which injects an empty string in get_level_spec_args().

Fixes: https://tracker.ceph.com/issues/54058
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

mgr/rbd_support: "trash remove" takes image_id_spec, not image_spec

Because of @CLIWriteCommand, the parameter name has to adhere to
the mon command API.  Commit dcb51b067a49 ("mgr/rbd_support: define
commands using CLICommand") accidentally changed image_id_spec to
image_spec, breaking external users such as go-ceph.

Fixes: https://tracker.ceph.com/issues/54057
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44707 from liumiaomiaoIntel/qatdoc

doc/radosgw: update QAT acceleration to specify the configuration

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

doc/radosgw: update QAT acceleration to specify the configuration

add and change the configuration steps to enable latest version of QAT
driver on QAT based encryption and compression

Signed-off-by: Miaomiao Liu <miaomiao.liu@intel.com>

Merge pull request #44816 from cbodley/wip-rgw-cmake-system

cmake/rgw: src/rgw and src/rgw/services are not system headers

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

doc/cephadm: fixing cluster purging section
https://tracker.ceph.com/issues/54018
ceph orch is not enough to stop all cephadm operations

Signed-off-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #44801 from sebastian-philipp/doc-cephadm-count-per-host

doc/cephadm: Co-location of daemons

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by:Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #44823 from andriytk/fix-install-deps2

install-deps.sh: fix cortx-motr installation condition

install-deps.sh: fix cortx-motr installation condition

Don't try to install cortx-motr if it is already installed
or not needed. (Currently, it installs on any run from an
interactive terminal session).

Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>

mgr/dashboard: perform daemon actions on cluster->services

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Fixes: https://tracker.ceph.com/issues/50322

Merge pull request #44711 from zdover23/wip-doc-2022-01-21-rados-bluestore-cache-autotune-interval-rewrite-2

src/rados: rewrite bs cache autotune instructions

Reviewed-by: Laura Flores <lflores@redhat.com>

Merge pull request #44703 from phlogistonjohn/jjm-issue46773

cephadm: validate that the constructed YumDnf baseurl is usable

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44517 from adk3798/agent-cache

mgr/cephadm: store agent metadata in its own cache

Reviewed-By: Sebastian Wagner <sewagner@redhat.com>

cmake/rgw: src/rgw and src/rgw/services are not system headers

include these with -I instead of -isystem so we don't hide their
compiler warnings

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44719 from zdover23/wip-doc-2021-01-22-man-page-radosgw-multitenancy-remove-double-quotes

doc/radosgw: replace double quotes with single

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #44379 from andriytk/add-motr-rgw-sal

rgw: add CORTX Motr SAL implementation

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Soumya Koduri <skoduri@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #44654 from rhcs-dashboard/page-title-feature

mgr/dashboard: Page title should indicate current location

Reviewed-by: Sarthak0702 <NOT@FOUND>
Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44793 from anthonyeleven/anthonyeleven/cephfs-mirroring

doc/cephfs: Clean up mirroring

Reviewed-by: Venky Shankar <vshankar@redhat.com>

install-deps.sh: set a UTF-8 locale when running pip

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

rgw: add CORTX Motr SAL implementation

To build with Motr backend, use -DWITH_RADOSGW_MOTR=YES cmake
option. cortx-motr-devel rpm should be installed beforehand.

To connect to Motr cluster, add the following configuration
parameters to ceph.conf:

    [client]
            ...
            rgw backend store = motr
            motr profile fid  = 0x7000000000000001:0x4f
            motr ha endpoint  = inet:tcp:10.0.0.1@2001
    ...
    [client.rgw.8000]
            ...
            motr my endpoint  = inet:tcp:10.0.0.1@5001
            motr my fid       = 0x7200000000000001:0x29

The correct values for the Motr connection parameters can be
taken from `hctl status` cmd output after Motr cluster is
bootstrapped. In the example above, the values were taken from
the following output:

    Profile:
        0x7000000000000001:0x4f
    Services:
        centos8n1
        [started] hax       0x7200000000000001:0x6  inet:tcp:10.0.0.1@5001
        ...
        [unknown] m0_client 0x7200000000000001:0x29 inet:tcp:10.0.0.1@5001

Motr pkgs for the build/run can be taken from
https://github.com/Seagate/cortx-motr/releases/tag/2.0.0-rgw.

Co-authored-by: Sining Wu <sining.wu@seagate.com>
Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>

mgr/dashboard: Page title should indicate current location

Added titleService to BreadcrumbsComponent.

Fixes: https://tracker.ceph.com/issues/36587
Signed-off-by: Sarthak0702 <sarthak.0702@gmail.com>

doc/cephadm: Co-location of daemons

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44693 from rhcs-dashboard/configuration-test

mgr/dashboard: navigation page e2e fix

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #44748 from cyx1231st/wip-crimson-improve-log-tm

crimson/os/seastore/transaction_manager: cleanups and logs

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #44760 from mgfritch/fixup-asyncssh-mypy

mgr/cephadm: improve asyncssh type checking

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Melissa Li <melissali@redhat.com>

Merge pull request #44792 from Matan-B/wip-matanb-email-contact

mailmap: Add Matan Breizman

Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/cephfs: Clean up mirroring

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

mailmap: Add Matan Breizman

Add Matan Breizman to mailmap.

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #44304 from guits/guits-fix-cv-batch-mpath

ceph-volume: improve mpath devices support

rgw: Match decode_json with dump for default-placement in RGWZoneGroup.

Fixes: https://tracker.ceph.com/issues/54016
Signed-off-by: zhiming zhang <zhangzhm1@chinatelecom.cn>

Merge pull request #44782 from liu-chunmei/crimson-fix-content-corrupt

crimson: fix qa/RadosModel content corrupt

reviewed by: Samuel Just <sjust@redhat.com>

Merge pull request #44718 from liu-chunmei/crimson-fix-split_pin_right-assert

crimson: fix split_pin_left assert_aligned

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson: fix qa/RadosModel content corrupt

RadosModel test read more length than object size, if truncate
not append zero to bufferlist, RadosMode will read more data and
it will check if there is content after the object size. So append
zero after truncate size to end of extent.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson: eliminate warning for unused variable

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

crimson: fix split_pin_left assert_aligned

hint passed to lba alloc_extent may not be block size aligned
in a certain case, this hint insert lba tree as key, so make
laddr in lba tree is not aligned.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

ceph-volume: fix typo in tests

This fixes 2 typo in ceph-volume tests.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

doc/ceph-volume: fix a typo

This fixes a typo in ceph-volume documentation.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: add a test `test_mpath_device_is_device`

This test checks that Device.is_device() returns True for a mpath device.

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

ceph-volume: improve mpath devices support

ee8887f4c0ff4f91117f31b621b95c8d08019130 was intended for adding
mpath devices support in ceph-volume but it has missed the lvm batch scenario.
This also fixes the zapping of mpath devices prepared with `ceph-volume raw`

Fixes: https://tracker.ceph.com/issues/52908
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #44549 from melissa-kun-li/remove-iscsi

mgr/cephadm/iscsi: use `mon_command` in `post_remove` instead of `check_mon_command`

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44682 from sebastian-philipp/cephadm-ipv6-parse-ipv6-addr-wo-route

cephadm: _parse_ipv6_route: Fix parsing ifs w/o route

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Flores <lflores@redhat.com>

mgr/cephadm: improve asyncssh type checking

asyncssh 2.9.0 introduces additional type hints

Fixes: https://tracker.ceph.com/issues/54003
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #44735 from neha-ojha/wip-reduce-cephadm

qa/suites/rados: reduce the number of cephadm tests

Reviewed-by: Adam King adking@redhat.com
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #44662 from guits/cv-shared_ceph_folder

cephadm: add shared_ceph_folder opt to ceph-volume subcommand

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44695 from liewegas/fix-cephadm-pretty-timedelta

pybind/mgr/mgr_util: fix to_pretty_timedelta()

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44459 from adk3798/misc-args

mgr/cephadm: allow miscellaneous container args at service level

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #44192 from liuqinfei/docfix

doc: typo fixes in ceph-volume and install

cephadm: add shared_ceph_folder opt to ceph-volume subcommand

This commit adds the `--shared_ceph_folder` option to `ceph-volume`
subcommand, just like `shell` and `bootstrap` subcommands.

Fixes: https://tracker.ceph.com/issues/53931
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #44282 from orozery/qa-qemu-nbd-ide-interface

qa/tasks/qemu: switch nbd devices from virtio to ide

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #44066 from guits/guits-fix-rollback-osd

ceph-volume: zap osds in rollback_osd()

Merge pull request #44755 from ronen-fr/wip-rf-be2-clang1

osd/scrub: fix ceph::mutex usage

Reviewed-by: Aishwarya Mathuria <amathuri@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #44740 from liu-chunmei/crimson-fix-remove_from_lru-assert

crimson: fix remove_from_lru assert

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #44653 from cyx1231st/wip-crimson-osd-empty-transaction

crimson/osd: ShardServices::dispatch_context_transaction to flush

Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: consolidate logs in levels

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction_manager: abort when error

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/transaction_manager: drop the extent metrics

These metrics won't distinguish extent operations from invalidated and
successful transactions.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

cephadm: fully replace agent files when writing them

otherwise, if the new file is shorter than the old one
we will end up with a malformed file retaining the end bit
of the old version

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: store agent metadata in its own cache

To avoid having the host cache getting too big
due to having to store this additional info

Fixes: https://tracker.ceph.com/issues/53624
Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #44761 from mgfritch/cephadm-asyncssh-2_8

mgr/cephadm: require asyncssh 2.8

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Melissa Li <melissali@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

mgr/cephadm: require asyncssh 2.8

Fixes: https://tracker.ceph.com/issues/54003
Signed-off-by: Michael Fritch <mfritch@suse.com>

osd/scrub: fix clang compilation issues

Issues created by the scrub-backend refactoring PR.

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #44737 from adamemerson/wip-wrong-modelines

cleanup: Change incorrect modelines

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

crimson: fix remove_from_lru assert

RETIRED_PLACEHOLDER extent also put into retired set, when do
commit_retire_extent need check if it is RETIRED_PLACEHOLDER,
otherwise will cause remove_from_lru assert.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge PR #44574 into master

* refs/pull/44574/head:
qa: Add syslog ignore list

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: cephadm e2e job: display info on error & other improvements

- Fix: ensure that on_error trap is called (display more info on error).
- Set static IPs to VMs.
- Remove domain in cluster definition to avoid side effects of potential dns misconfiguration.
- Minor improvements.

Fixes: https://tracker.ceph.com/issues/53991
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #44714 from idryomov/wip-rbd-mirror-delprop-races

rbd-mirror: fix races in snapshot-based mirroring deletion propagation

Reviewed-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>
Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: navigation page e2e fix

Looks like the newly added relative position for the sidebar is
causing cypress to verify that the sidebar is hidden from the user view.

Fixes: https://tracker.ceph.com/issues/53960
Signed-off-by: Nizamudeen A <nia@redhat.com>