Merge PR #35081 into master

* refs/pull/35081/head:
qa: do not save/restore ip_forward sysctl

Reviewed-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34764 from ianwatsonrh/master

Docs: vault kv and transit engine updates

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35025 from dillaman/wip-45518

librbd: copy API should not inherit v1 image format by default

Merge pull request #34789 from trociny/wip-rbd-nbd-quiesce

rbd-nbd: add quiesce/unquiesce hooks

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35023 from yuvalif/fix_issue_45513

cls/2pc_queue: fix timing issue in test

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34959 from yison/rwl-image-writeback-cache-seq10

rbd/cache: Replicated Write Log core codes - load existing cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34930 from trociny/wip-45409

rbd-mirror: wait for events to replay before shut down journal replay

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #35042 from tchaikov/wip-cmake-remove-iwyu-cppcheck

cmake: remove cppcheck and iwyu targets

Reviewed-by: Yuval Lifshitz <ylifshit@redhat.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #35070 from Devp00l/wip-45545-2

mgr/dashboard: Updates jest to fix cache failure

Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34913 from athanatos/sjust/wip-seastore-journal

crimson/os/seastore: initial PR

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35052 from tspmelo/wip-unittest-warnings

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #35013 from matthewoliver/cephadm-iscsi-log-to-stderr

cephadm: enable logging to stderr in ceph-iscsi

Reviewed-by: Michael Fritch <mfritch@suse.com>

dec/dev/seastore.rst: add more detailed design information and status

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: add initial cache support with tests

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: introduce initial journal implementation and tests

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/gtest_seastar: add helpers for building seastar gtest unit tests

Also adds unsafe_get methods for errorator for use within unit tests.

Signed-off-by: Samuel Just <sjust@redhat.com>

qa: do not save/restore ip_forward sysctl

We don't really care if this is left on during the course of the mount.
These settings don't persist across tests anyway.

Fixes: https://tracker.ceph.com/issues/45525
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #35077 from yuriw/wip-yuriw-crontab-master

qa/tests: reduced powercycle jobs amount

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: reduced powercycle jobs amount

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #32829 from Devp00l/wip-43765

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #35076 from yuriw/wip-yuriw-distro-master

qa/tests: run `nautilus-x` on `pacific` on `ubuntu 18.04`

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: run `nautilus-x` on `pacific` on `ubuntu 18.04`

...as we don't build `nauiltus` on `20.04`

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #32829 from Devp00l/wip-43765

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Updates jest to fix cache failure

Fixes: https://tracker.ceph.com/issues/45545
Signed-off-by: Stephan Müller <smueller@suse.com>

Merge pull request #31889 from ofriedma/wip-ofriedma-reshard-debug

rgw: Fix reshard log so we can know why the reshard is returning error

Merge pull request #34003 from AmnonHanuhov/wip-radosbench_4k_rw_yaml

test/crimson: Added CPU cycles per operation as an acceptance criteria to radosbench_4k_r/w yaml

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34981 from sebastian-philipp/doc-dev-cephadm-cuncurrency

doc/dev/cephadm: Note regarding network calls form CLI handlers

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #35066 from rzarzynski/wip-crimson-omap_get_keys-num

crimson/osd: fix key num handling in PGBackend::omap_get_keys().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34668 from zdover23/wip-doc-ceph-orch-zap-45128

doc/ceph orch: Add ceph orch zap command

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34944 from jschmid1/fix_osdspec_affinity_passing

mgr/cephadm: fix env_var passing to cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35030 from mgfritch/cephadm-shell-mount-special

cephadm: fix shell mount for special files

Reviewed-by: Matthew Oliver <moliver@suse.com>

crimson/osd: fix key num handling in PGBackend::omap_get_keys().

The patch fixes running-out-of-buffer when decoding the data
from `omap_get_keys()`. In some cases, the declared number
of encoded keys is greater by 1 from the actual causing execption
`buffer::end_of_buffer` to be thrown at decoder.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35011 from dillaman/wip-45504

pybind/rbd: RBD.create() method's 'old_format' parameter now defaults to False

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35022 from sebastian-philipp/cephadm-add-host-oserror

mgr/cephadm: Remove AsyncCompletion

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34976 from changchengx/bufferlist_ut

test/bufferlist: fix and cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35056 from agayev/fix-do-make

do_cmake.sh: do not unnecessarily warn about debug build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35029 from yuriw/wip-yuriw-subsets-master

qa/tests: reduce number of jobs for rados to 309

do_cmake.sh: do not unnecessarily warn about debug build

do_cmake.sh forgets to check the arguments passed to it and ends up warning
about debug build even if a user specifies a release build.

Signed-off-by: Abutalib Aghayev <agayev@gmail.com>

Merge pull request #34969 from pritha-srivastava/wip-rgw-gc-fix

cls/rgw_gc: Fixing the iterator used to access urgent data map

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: Remove unit tests warnings

Fixes: https://tracker.ceph.com/issues/45536
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34965 from Devp00l/wip-45433

mgr/dashboard: Always use fast angular unit tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35041 from smithfarm/wip-tcmalloc-nit

doc: fix TCMalloc spelling

Reviewed-by: Zac Dover <zac.dover@gmail.com>

cmake: remove cppcheck and iwyu targets

these targets are not used by any other targets, and are not integrated with CI.
so let's drop them.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33121 from tspmelo/wip-angular-9

mgr/dashboard: Update to Angular 9

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

rbd-nbd: don't abuse rbd_cache_writethrough_until_flush

If rbd_cache and rbd_cache_writethrough_until_flush options are
set, do not use flush for internal purposes unless the user
issued flush, so it won't unintentionally enable writeback cache
when it is still unsafe.

Signed-off-by: Mykola Golub <mgolub@suse.com>

doc: fix TCMalloc spelling

The google/tcmalloc page on GitHub refers to "TCMalloc", not
"tc_malloc".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #35010 from pcuzner/k8sevents-sanitise

mgr/k8sevents:sanitise kubernetes events

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34972 from tspmelo/wip-fix-45445

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33886 from sebastian-philipp/doc-mon-command-api

doc: Add a generated reference of all mon commands.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35017 from tspmelo/wip-fix-fakefs

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34956 from tchaikov/wip-qa-tasks-cephfs-py3

qa/tasks/cephfs: remove sys.exc_clear() calls

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

librbd: load existing cache

Signed-off-by: Peterson, Scott <scott.d.peterson@intel.com>
Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>
Signed-off-by: Lu, Yuan <yuan.y.lu@intel.com>
Signed-off-by: Chamarthy, Mahati <mahati.chamarthy@intel.com>

Merge pull request #34857 from tchaikov/wip-qa-tasks-no-filter

qa/tasks: use list comprehension for checking the length

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephadm: enable logging to stderr in ceph-iscsi

All the other ceph servies when deployed are deployed with:

  --default-log-to-stderr=true --default-log-stderr-prefix=debug
  --default-log-to-file=false

Ceph-iscsi doesn't have these as commandline params but once
https://github.com/ceph/ceph-iscsi/pull/186 lands it'll have something
similar as config options. This patch sets these options

Signed-off-by: Matthew Oliver <moliver@suse.com>

mgr/dashboard: Add dashboard node script

This file should be used to agregate all external scripts we need.
Multiple flags can be used in the same call.

Available flags:

--env: Generates angular environment files.

--pre: Modifies 'angular.json' to enable the build of custom locales using
       angular --localize.
       Languages can be defined using the environment variable LANGS,
       if no value is provided all languages will be build.
       Default language is always build, even if not provided.
       p.e.: 'LANGS="pt" node cd --pre', will build EN and PT.
       For backward compatibility we accept both long and short version of
       languages, p.e.: 'pt' and 'pt-BR'

--post: Restores 'angular.json' to its original.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove ngcc warnings

Some packages are still not updated to be compatible with Angular 9
and can cause warnings.

I have whitelisted some that work well, so we don't see the warning every time.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update Node.js to latest LTS: v12.16.2

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update npm dependencies

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix SCSS imports

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix i18n configuration

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Few linting fixes

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update Angular to v9

This was done with "ng update" cli command.

Fixes: https://tracker.ceph.com/issues/44829
Signed-off-by: Tiago Melo <tmelo@suse.com>

cephadm: return absolute path from pathify

normalizes the user defined paths a bit better to avoid inconsistent
results with os.path.basename(), os.path.join(), etc.

For example:
$ cephadm shell --mount ~/

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/tests: reduce number of jobs for rados to 309

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: add shell mount test

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: fix shell mount for special files

block/char devs, named pipes, sockets etc

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35009 from ivancich/wip-orphan-pending-rel-notes

PendingReleaseNotes: rgw-orphan-list, deprecation, and fixes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge PR #34978 into master

* refs/pull/34978/head:
qa/tasks/cephfs/mount.py: always setup the NAT rules
qa/tasks/cephfs/mount.py: fall back to use ip command to setup the bridge

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Incorporating sebastian-philipp's suggestions.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35021 from tchaikov/wip-cephfs-shell-pep8

cephfs-shell: fix warnings reported by flake8

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

cls/2pc_queue: fix timing issue in test

Fixes: https://tracker.ceph.com/issues/45513
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/cephadm: Remove AsyncCompletion

Simplyfy things a lot by not using multiprocessing.pool.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

librbd: copy API should not inherit v1 image format by default

When copying from a v1 image, by default the new destination image
would be created using the v1 format. Since the creation of v1 images
is disallowed, this has been updated to default to using the v2
image format.

Fixes: https://tracker.ceph.com/issues/45518
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cephfs-shell: fix warnings reported by flake8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34746 from tspmelo/wip-form-loading

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/cephadm: Proper error for OSErrors

Previsously, an OSError resulted in

    OSError: cannot send (already closed?)

which is not helpful.

Fixes: https://tracker.ceph.com/issues/45032
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix HomeTest setup

'setup_controllers' was being called outside of the fake fs mock,
so the tests would fail if there was no real dist folder.

Fixes: https://tracker.ceph.com/issues/45516
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/cephadm: disable env_var passing until pr#34835 is merged

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: generate and pass env vars to cephadm

Signed-off-by: Joshua Schmid <jschmid@suse.de>

python-common: do not pass envs after entrypoint

Signed-off-by: Joshua Schmid <jschmid@suse.de>

bin/cephadm: move --env-var handling to global argparser

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: adapt env_var handling

Signed-off-by: Joshua Schmid <jschmid@suse.de>

doc/dev/cephadm: Note regarding network calls form CLI handlers

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34888 from jmolmo/fix_command_parse

cephadm: Fix error parsing shell command arguments

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34945 from sebastian-philipp/cephadm-remove-async-map-resurrect

mgr/cephadm: replace async_map_completion with a simple wrapper (again)

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34970 from sebastian-philipp/bluestore-revert-sharding

Revert "os/bluestore: Enable sharding by default"

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34977 from mgfritch/cephadm-prom-encoding

mgr/cephadm: ceph_default_alerts.yml can be utf-8 encoded

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

test/bufferlist: add space between string and time info

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

test/bufferlist: test move assignment operator not move constructor

"bufferlist move = std::move(bl)" exercises the "move constructor".
But the purpose of this test is to verify the functionality of
"move assignment operator".
So this PR construct "move" first, and then assign rvalue reference to it.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

test/bufferlist: correct the usage of string constructor

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

qa/tasks/cephfs/mount.py: always setup the NAT rules

If the last test failed just after the 'ceph-brx' bridge is created
but with setuping the NAT rules in iptables, so if we run the test
case again, it will just skip seting the 'ceph-brx' and the NAT rules.
Then deleting the NAT rules will get the errors like:

iptables: Bad rule (does a matching rule exist in that chain?).

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/mount.py: fall back to use ip command to setup the bridge

CentOS/RHEL8 have abandoned the bridge-utils package so the brctl
cmd is none avalible, and on Ubuntu 18.04 and 20.04 the nmcli is
buggy to setup the bridge. So this will fall back to use ip cmd
to setup the bridge stuff.

Fixes: https://tracker.ceph.com/issues/45459
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34955 from tchaikov/wip-cbt-with-bluestore

scripts/run-cbt.sh: use bluestore for testing classic osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #34962 into master

* refs/pull/34962/head:
vstart_runner.py: use tuple instead of set

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/rbd: RBD.create() method's 'old_format' parameter now defaults to False

The RBD v1 format has been deprecated for numerous releases and creation of
v1 format images has been disabled since the Mimic release. This fixes
the Python API's image create() method to ensure v2 images are created by
default (and no longer throw an exception that creation of v1 images are
disabled).

Fixes: https://tracker.ceph.com/issues/45504
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/k8sevents:sanitise kubernetes events

Kubernetes can generate events without a timestamp
or an event count. When this occurs the k8sevents 'ls'
command fails due to None type values. This patch
sanitises the event received before adding to the
internal data structures to account for these
issues.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

PendingReleaseNotes: add missing quote character

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

PendingReleaseNotes: `rgw-orphan-list` and deprecated `radosgw-admin orphans`

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #34938 from mgfritch/cephadm-systemd-stop-timeout

cephadm: extend systemd stop timeout

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34905 from tspmelo/wip-mirroring

Reviewed-by: Alfonso Martínez <almartin@redhat.com>