kv/RocksDBStore: Added resharding control

Added possibility to control batch size and iterator refresh time for resharding process.
Replaced getenv() with new control for resharding unittests.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/RocksDBStore: Improved debug output quality during resharding.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/RocksDBStore: Now resharding properly deletes all copies of key transferred

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

ceph_test_keyvaluedb: Add tests for resharding

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/RocksDBStore: Fix db closing after reshard is completed

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/RocksDBStore: Added getenv() hookups for testing of resharding

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/RocksDBStore: Fix reacquiring of iterator

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/RocksDBStore: Add protection against mounting database that is during resharding

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/RocksDBStore: Store RocksDB env

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/RocksDBStore: Modified merge operator name

Now merge operator name also includes operators from dedicated to shards.
This makes it possible to reshard that will move prefix with merge operator
away from default column family.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/RocksDBStore: Add reshard ability to database.

Reshard ability allows to change current sharding schema, without change to data.

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

kv/KeyValueDB: Make pair open/close idempotent

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluestore: Split _open_db function

Created function that prepares environment for _open_db.
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34981 from sebastian-philipp/doc-dev-cephadm-cuncurrency

doc/dev/cephadm: Note regarding network calls form CLI handlers

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #35066 from rzarzynski/wip-crimson-omap_get_keys-num

crimson/osd: fix key num handling in PGBackend::omap_get_keys().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34668 from zdover23/wip-doc-ceph-orch-zap-45128

doc/ceph orch: Add ceph orch zap command

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34944 from jschmid1/fix_osdspec_affinity_passing

mgr/cephadm: fix env_var passing to cephadm

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #35030 from mgfritch/cephadm-shell-mount-special

cephadm: fix shell mount for special files

Reviewed-by: Matthew Oliver <moliver@suse.com>

crimson/osd: fix key num handling in PGBackend::omap_get_keys().

The patch fixes running-out-of-buffer when decoding the data
from `omap_get_keys()`. In some cases, the declared number
of encoded keys is greater by 1 from the actual causing execption
`buffer::end_of_buffer` to be thrown at decoder.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #35011 from dillaman/wip-45504

pybind/rbd: RBD.create() method's 'old_format' parameter now defaults to False

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #35022 from sebastian-philipp/cephadm-add-host-oserror

mgr/cephadm: Remove AsyncCompletion

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34976 from changchengx/bufferlist_ut

test/bufferlist: fix and cleanups

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35056 from agayev/fix-do-make

do_cmake.sh: do not unnecessarily warn about debug build

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #35029 from yuriw/wip-yuriw-subsets-master

qa/tests: reduce number of jobs for rados to 309

do_cmake.sh: do not unnecessarily warn about debug build

do_cmake.sh forgets to check the arguments passed to it and ends up warning
about debug build even if a user specifies a release build.

Signed-off-by: Abutalib Aghayev <agayev@gmail.com>

Merge pull request #34969 from pritha-srivastava/wip-rgw-gc-fix

cls/rgw_gc: Fixing the iterator used to access urgent data map

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34965 from Devp00l/wip-45433

mgr/dashboard: Always use fast angular unit tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #35041 from smithfarm/wip-tcmalloc-nit

doc: fix TCMalloc spelling

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #33121 from tspmelo/wip-angular-9

mgr/dashboard: Update to Angular 9

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

doc: fix TCMalloc spelling

The google/tcmalloc page on GitHub refers to "TCMalloc", not
"tc_malloc".

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #35010 from pcuzner/k8sevents-sanitise

mgr/k8sevents:sanitise kubernetes events

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34972 from tspmelo/wip-fix-45445

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #33886 from sebastian-philipp/doc-mon-command-api

doc: Add a generated reference of all mon commands.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35017 from tspmelo/wip-fix-fakefs

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #34956 from tchaikov/wip-qa-tasks-cephfs-py3

qa/tasks/cephfs: remove sys.exc_clear() calls

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #34857 from tchaikov/wip-qa-tasks-no-filter

qa/tasks: use list comprehension for checking the length

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Add dashboard node script

This file should be used to agregate all external scripts we need.
Multiple flags can be used in the same call.

Available flags:

--env: Generates angular environment files.

--pre: Modifies 'angular.json' to enable the build of custom locales using
       angular --localize.
       Languages can be defined using the environment variable LANGS,
       if no value is provided all languages will be build.
       Default language is always build, even if not provided.
       p.e.: 'LANGS="pt" node cd --pre', will build EN and PT.
       For backward compatibility we accept both long and short version of
       languages, p.e.: 'pt' and 'pt-BR'

--post: Restores 'angular.json' to its original.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Remove ngcc warnings

Some packages are still not updated to be compatible with Angular 9
and can cause warnings.

I have whitelisted some that work well, so we don't see the warning every time.

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update Node.js to latest LTS: v12.16.2

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update npm dependencies

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix SCSS imports

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Fix i18n configuration

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Few linting fixes

Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Update Angular to v9

This was done with "ng update" cli command.

Fixes: https://tracker.ceph.com/issues/44829
Signed-off-by: Tiago Melo <tmelo@suse.com>

cephadm: return absolute path from pathify

normalizes the user defined paths a bit better to avoid inconsistent
results with os.path.basename(), os.path.join(), etc.

For example:
$ cephadm shell --mount ~/

Signed-off-by: Michael Fritch <mfritch@suse.com>

qa/tests: reduce number of jobs for rados to 309

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

qa/workunits/cephadm/test_cephadm.sh: add shell mount test

Signed-off-by: Michael Fritch <mfritch@suse.com>

cephadm: fix shell mount for special files

block/char devs, named pipes, sockets etc

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #35009 from ivancich/wip-orphan-pending-rel-notes

PendingReleaseNotes: rgw-orphan-list, deprecation, and fixes

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge PR #34978 into master

* refs/pull/34978/head:
qa/tasks/cephfs/mount.py: always setup the NAT rules
qa/tasks/cephfs/mount.py: fall back to use ip command to setup the bridge

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Incorporating sebastian-philipp's suggestions.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #35021 from tchaikov/wip-cephfs-shell-pep8

cephfs-shell: fix warnings reported by flake8

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Kyr Shatskyy <kyrylo.shatskyy@suse.com>

mgr/cephadm: Remove AsyncCompletion

Simplyfy things a lot by not using multiprocessing.pool.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

cephfs-shell: fix warnings reported by flake8

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34746 from tspmelo/wip-form-loading

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

mgr/cephadm: Proper error for OSErrors

Previsously, an OSError resulted in

    OSError: cannot send (already closed?)

which is not helpful.

Fixes: https://tracker.ceph.com/issues/45032
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/dashboard: Fix HomeTest setup

'setup_controllers' was being called outside of the fake fs mock,
so the tests would fail if there was no real dist folder.

Fixes: https://tracker.ceph.com/issues/45516
Signed-off-by: Tiago Melo <tmelo@suse.com>

mgr/cephadm: disable env_var passing until pr#34835 is merged

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: generate and pass env vars to cephadm

Signed-off-by: Joshua Schmid <jschmid@suse.de>

python-common: do not pass envs after entrypoint

Signed-off-by: Joshua Schmid <jschmid@suse.de>

bin/cephadm: move --env-var handling to global argparser

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: adapt env_var handling

Signed-off-by: Joshua Schmid <jschmid@suse.de>

doc/dev/cephadm: Note regarding network calls form CLI handlers

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34888 from jmolmo/fix_command_parse

cephadm: Fix error parsing shell command arguments

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34945 from sebastian-philipp/cephadm-remove-async-map-resurrect

mgr/cephadm: replace async_map_completion with a simple wrapper (again)

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34970 from sebastian-philipp/bluestore-revert-sharding

Revert "os/bluestore: Enable sharding by default"

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #34977 from mgfritch/cephadm-prom-encoding

mgr/cephadm: ceph_default_alerts.yml can be utf-8 encoded

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

test/bufferlist: add space between string and time info

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

test/bufferlist: test move assignment operator not move constructor

"bufferlist move = std::move(bl)" exercises the "move constructor".
But the purpose of this test is to verify the functionality of
"move assignment operator".
So this PR construct "move" first, and then assign rvalue reference to it.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

test/bufferlist: correct the usage of string constructor

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

qa/tasks/cephfs/mount.py: always setup the NAT rules

If the last test failed just after the 'ceph-brx' bridge is created
but with setuping the NAT rules in iptables, so if we run the test
case again, it will just skip seting the 'ceph-brx' and the NAT rules.
Then deleting the NAT rules will get the errors like:

iptables: Bad rule (does a matching rule exist in that chain?).

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/mount.py: fall back to use ip command to setup the bridge

CentOS/RHEL8 have abandoned the bridge-utils package so the brctl
cmd is none avalible, and on Ubuntu 18.04 and 20.04 the nmcli is
buggy to setup the bridge. So this will fall back to use ip cmd
to setup the bridge stuff.

Fixes: https://tracker.ceph.com/issues/45459
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #34955 from tchaikov/wip-cbt-with-bluestore

scripts/run-cbt.sh: use bluestore for testing classic osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #34962 into master

* refs/pull/34962/head:
vstart_runner.py: use tuple instead of set

Reviewed-by: Sidharth Anupkrishnan <sanupkri@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/rbd: RBD.create() method's 'old_format' parameter now defaults to False

The RBD v1 format has been deprecated for numerous releases and creation of
v1 format images has been disabled since the Mimic release. This fixes
the Python API's image create() method to ensure v2 images are created by
default (and no longer throw an exception that creation of v1 images are
disabled).

Fixes: https://tracker.ceph.com/issues/45504
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/k8sevents:sanitise kubernetes events

Kubernetes can generate events without a timestamp
or an event count. When this occurs the k8sevents 'ls'
command fails due to None type values. This patch
sanitises the event received before adding to the
internal data structures to account for these
issues.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

PendingReleaseNotes: add missing quote character

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

PendingReleaseNotes: `rgw-orphan-list` and deprecated `radosgw-admin orphans`

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #34938 from mgfritch/cephadm-systemd-stop-timeout

cephadm: extend systemd stop timeout

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34905 from tspmelo/wip-mirroring

Reviewed-by: Alfonso Martínez <almartin@redhat.com>

vstart_runner.py: use tuple instead of set

Using python3 leads to an error that says
"TypeError: unhashable type: 'Raw'".

Fixes: https://tracker.ceph.com/issues/45446
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34926 from sebastian-philipp/mgr-mock-mon-store

pybind/mgr: properly mock the mon store for pytest

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Revert "os/bluestore: Enable sharding by default"

This reverts commit aa1a116f9eef5d6fbe09d6ed99910223451431f1.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/cephadm: ceph_default_alerts.yml can be utf-8 encoded

non-ascii chars introduced by 653c3f66823

Fixes: https://tracker.ceph.com/issues/45458
Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge PR #34948 into master

* refs/pull/34948/head:
doc/cephfs-shell: Add quota command usage
doc/cephfs-shell: Add du command usage
doc/cephfs-shell: Add df command usage

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34958 from yuvalif/fix_versioning_in_notif

rgw/notifications: version id was not sent in versioned buckets

Merge pull request #34967 from tchaikov/wip-rpm-fmt

rpm: BuildRequires fmt-devel on el8

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34781 from lxbsz/run_shell1

qa/tasks/cephfs/mount.py: run_shell to use StringIO as default

Reviewed-by: Jeff Layton <jlayton@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34834 from sidharthanup/qa-test-exports-fix

qa/test_exports: fix TestExports failure under new py3 compability changes

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34871 from tchaikov/wip-qa-tasks-mgr-set-application

qa/tasks/mgr: always add "application" label to pool

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #34895 from mgfritch/mgr-insights-py3-warn

mgr/insights: fix python3 DeprecationWarning

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34875 from tchaikov/wip-qa-tasks-qemu

qa/tasks/qemu: open file in text mode

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #34941 into master

* refs/pull/34941/head:
qa/cephfs: use StringIO instead of BytesIO

Reviewed-by: Xiubo Li <xiubli@redhat.com>

mgr/dashboard: Fix random E2E error in mgr-modules

This test failed at random times when it tried to find the new value of pool_ids
in the balancer module.
This happened because the value of pool_ids is automatically reverted by ceph,
so in some situations when we tried to read the new value,
it was already reverted and failed.

Enhanced the tests to be able to use any text input, not only the ones with
empty default values.

Fixes: https://tracker.ceph.com/issues/45445
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34037 from ivancich/wip-adjust-ordered-list-attempts

rgw: clean up attempt loop in ordered listing

Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #33865 from ivancich/wip-fix-ordered-marker-bug

rgw: fix bug where bucket listing end marker not always set correctly

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #30511 from tianshan/fix_41953

rgw: skip add empty multipart chain to gc in abort_multipart_upload

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #29886 from tianshan/fix_bucket_handle_leak

rgw: fix some list buckets handle leak

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #34961 from sebastian-philipp/cephadm-fix-node-exporter-auth-entity

mgr/cephadm: node-exporter is not a ceph service

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34932 from sebastian-philipp/cephadm-hostassignment-schedule.py

mgr/cephadm: move HostAssignment to new module

Reviewed-by: Joshua Schmid <jschmid@suse.de>