kv/rocksdb: support rmrange unconditionally

removes the config options rocksdb_enable_rmrange and
rocksdb_max_items_rmrange that avoid calls to DeleteRange()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: rgw_bi_log_trim() uses cls_cxx_map_remove_range()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/log: cls_log_trim() uses cls_cxx_map_remove_range()

uses cls_cxx_map_get_keys() instead of cls_cxx_map_get_vals() to limit
the number of keys removed

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cls: add cls_log.trim_by_marker test

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cls: test_cls_log doesn't allocate ObjectOperations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cls: test_cls_log uses fixture for temporary pool

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cls: add cls_rgw.bi_log_trim test

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: expose cls_rgw_bilog_list/trim() for single shard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cls: test_cls_rgw uses cls_rgw_obj_key

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cls: test_cls_rgw doesn't allocate ObjectOperations

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/cls: test_cls_rgw uses fixture for temporary pool

Signed-off-by: Casey Bodley <cbodley@redhat.com>

objclass: add cls_cxx_map_remove_range()

Fixes: http://tracker.ceph.com/issues/19975
Signed-off-by: Casey Bodley <cbodley@redhat.com>

librados: add rados_write_op_omap_rm_range2()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

osdc: add Objecter omap_rm_range()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

osd: add CEPH_OSD_OP_OMAPRMKEYRANGE to do_osd_ops()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

osd: add omap_rmkeyrange() to PGTransaction

Signed-off-by: Casey Bodley <cbodley@redhat.com>

os: add bufferlist overload for omap_rmkeyrange()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

tracing: add do_osd_op_pre_omaprmkeyrange

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rados: add CEPH_OSD_OP_OMAPRMKEYRANGE

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28331 from linuxbox2/wip-rgw-uinfo-by-acc

rgw: permit rgw-admin to populate user info by access-key

Merge pull request #29413 from tchaikov/wip-remove-test/python

test/python: remove stale tests

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29445 from ricardoasmarques/fix-doc-rbd-namespace

doc: Fix rbd namespace documentation

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

doc: Fix rbd namespace documentation

Pool and namespace should be separated by a slash.

Fixes: https://tracker.ceph.com/issues/41046
Signed-off-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #29446 from tchaikov/wip-cmake-tox-test

cmake: add add_tox_test()

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29423 from rhcs-dashboard/wip-41023-master

doc: Show Jenkins commands

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29297 from iotcg/starlingx

ceph-objectstore-tool: set log date format

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: add add_tox_test()

to consolidate the tox tests

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29364 from capri1989/doc-pg-num

doc: pg_num should always be a power of two

Reviewed-By: Jan Fajerski <jfajerski@suse.com>
Reviewed-By: Sage Weil <sage@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29063 from sebastian-philipp/orchestrator-get_feature_set

mgr/orchestrator: Split *_stateless_service and add get_feature_set

Merge pull request #29131 from sebastian-philipp/rook-caching

mgr/rook: Add caching for the Dashboard

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #29334 into master

* refs/pull/29334/head:
qa/tasks/mgr/dashboard/test_mgr_module: adjust expected schema
mgr/telemetry: separate out cluster config vs running daemons
mgr/telemetry: include any config options that are customized

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge PR #29340 into master

* refs/pull/29340/head:
mgr/telemetry: specify license when opting in

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #29419 from vshankar/wip-fix-rbd-support

pybind/rbd: fix call to unregister_osd_perf_queries

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29406 from yuriw/wip-yuriw-upgrade-rwg-master_2

qa/tests: added rgw into upgrade sequence to improve coverage

Merge pull request #29428 from yuriw/wip-yuriw-crontab-master

qa/tests: added mimic-x on master run

Merge PR #29104 into master

* refs/pull/29104/head:
qa/cephfs: test case for timeout config of individual session
mds: add command that config individual client session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29430 from uweigand/fix-no-boost-context-build

rgw: tests: Fix building with -DWITH_BOOST_CONTEXT=OFF

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29412 into master

* refs/pull/29412/head:
doc: mds-config-ref: update 'mds_log_max_segments' value.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29355 into master

* refs/pull/29355/head:
mgr/volumes: set uid/gid of FS client's mount as 0/0

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29133 from ricardoasmarques/prevent-iscsi-target-delete-with-sessions

mgr/dashboard: Prevent deletion of iSCSI IQNs with open sessions

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #29388 into master

* refs/pull/29388/head:
qa: sleep briefly after resetting kclient

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge PR #29377 into master

* refs/pull/29377/head:
mds: add perf counter for finisher of MDSRank

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests: added rgw into upgrade sequence to improve coverage

added rgw into upgrade sequence to improve coverage - splits
See https://github.com/ceph/ceph/pull/29234 https://github.com/ceph/ceph/pull/29282

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw: tests: Fix building with -DWITH_BOOST_CONTEXT=OFF

Attempting to build with -DWITH_BOOST_CONTEXT=OFF will result in
a "conflicting declaration" error when building
test/rgw/test_rgw_dmclock_scheduler.cc

Fixed by avoiding the boost/asio/spawn.hpp include in that case.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

qa/tests: added mimic-x on master run

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc: Address further comments on choosing pg_num

Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>

doc: adjust examples to use 2^n pg_num

The examples used pg_num set to 300 or 1000. This misled some users.

Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>

Merge pull request #29405 from wjwithagen/wjw-fix-env-vs-args.t

test/cli: Make the ceph-conf test more liberal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29414 from tchaikov/wip-mgr-api-test

pybind/mgr: install setuptools >= 12

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #27754 from jan--f/c-v-validate-device-lists

ceph-volume: batch ensure device lists are disjoint

doc: Show Jenkins commands

Fixes: https://tracker.ceph.com/issues/41023
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

test/cli: Make the ceph-conf test more liberal

The tests look for the exact error text.
But they are OS dependant, and thus on FreeBSD the output is
slightly different:
2019-07-30T19:04:16.581+0000 80bc48000 -1 parse_file: filesystem error: in file_size: No such file or directory [from-env]

This stems from the fact that different OSes use different
standard libraries: libc++ <> libstd++

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

pybind/rbd: fix call to unregister_osd_perf_queries

.. should be invoked as self.unregister_osd_perf_queries()
rather than self.module.unregister_osd_perf_queries().

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29318 from tspmelo/wip-backport-component

script/ceph-backport.sh: Allow to set component label

pybind/mgr: install setuptools >= 12

otherwise we have
```
  Running setup.py egg_info for package apache-libcloud

    libcloud/utils/py3.py:76: UserWarning: Missing backports.ssl_match_hostname package
      warnings.warn("Missing backports.ssl_match_hostname package")
    Traceback (most recent call last):
...
      File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66, in _warn_if_setuptools_outdated

...
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
...
Command python setup.py egg_info failed with error code 1 in /tmp/tmp.i95Fg82Ea4/venv/build/apache-libcloud
```

and more important thing is, setuptools >= 12 allows us to install the
python modules in venv, without uninstalling existing modules if they
are also installed in by system packages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rook: monkey-patch for CoreV1Api.list_node

`kubernetes.client.apis.core_v1_api.CoreV1Api#list_node` is
broken and needs a monkey patch. Might be related to
an incompatible k8s cluster version.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: better format `service ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Implement fallback for CentOS 7

`kubernetes-client` requires `urllib3` > 1.15, but CentOS 7 ships 1.10

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Move KubernetesResource to rook_cluser.py

Use `KubernetesResource` as cache for pods, inventory maps and nodes.
This is required for the dashboard.

Also:
  properly return K8s nodes
  fix minor issues

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Add feature to gather k8s storageclass information

Provide ability to fetch or watch k8s resources,
and hold the resulting objects in a dict. A method
has been added to the rook module to allow other
mgr modules to fetch the state of specific k8s
resources.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/orchestrator_clli: Always return str

Also: Don't return `"None"`
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ansible: Moved RGW defaults to ansible

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Fix RGW creation

* Moved RGWSpec defaults to a new method

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Split *_stateless_service and add get_feature_set

Split *_stateless_service, because every orchestrator
did some method dispatching internally anyway.

Also: Added test

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-volume: batch ensure device lists are disjoint

Fixes: https://tracker.ceph.com/issues/41018
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

test/python: remove test for ceph cli

we have abundant test for exercising `ceph` cli, like

qa/standalone/mon/misc.sh.

and `test_ceph.py` does not test any functionality of this python script
other than importing it, which basically runs it through python's syntax
check.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/python: remove test for brag/client

ceph-brag is now take replaced by telemetry. and it was removed in
cf97b5c1 .

so no need to keep its test.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #29363 into master

* refs/pull/29363/head:
qa/suites/multimds/basic/tasks/ceph_test_snapshots: disable RECENT_CRASH
qa/suites/kcephfs/recovery/failover.yaml: disable RECENT_CRASH
qa/suites/fs/multifs/tasks/failover.yaml: disable RECENT_CRASH

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/mgr/dashboard/test_mgr_module: adjust expected schema

Signed-off-by: Sage Weil <sage@redhat.com>

doc: mds-config-ref: update 'mds_log_max_segments' value.

Also drop options that not exists on code:

* mds_log_max_expiring
* mds_log_eopen_size

Signed-off-by: Konstantin Shalygin <k0ste@k0ste.ru>

Merge PR #29185 into master

* refs/pull/29185/head:
os/bluestore: add tail length when caculating reserved buffer size
os/bluestore: record next_offs when offset is not sequential.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29356 into master

* refs/pull/29356/head:
pybind/mgr/restful: use dict.items() for py3 compatibility

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #29098 from liewegas/wip-unhexdump-c

src/script/unhexdump-C: script to reverse a 'hexdump -C' style hexdump

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29394 from mkogan1/wip-rgw-bi-list-shard-id

rgw: add option to specify shard-id for "bi list" admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #29362 into master

* refs/pull/29362/head:
Update nautilus release for 14.2.2
Updated schedule with latest Nautilus release

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28239 into master

* refs/pull/28239/head:
cephfs-shell: Add tox for testing with flake8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29386 from alfredodeza/wip-rm40987

ceph-volume  simple: better detection when 'type' file is not present

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

cephfs-shell: Add tox for testing with flake8

Fixes: https://tracker.ceph.com/issues/39947
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge PR #29331 into master

* refs/pull/29331/head:
common: add comment about pod memory requests/limits

Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #29161 from valentinbajrami/improve-grep-regex

src/script/check_commands.sh: fix grep regex class range

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: add option to specify shard-id for "bi list" admin command

related to: https://tracker.ceph.com/issues/40700
Signed-off-by: Mark Kogan <mkogan@redhat.com>

src/script/check_commands.sh: fix grep regex class range

Correct the faulty regex class.

Signed-off-by: Valentin Bajrami <valentin.bajrami@gmail.com>

Merge pull request #28915 from sebastian-philipp/python-common-package-2

python-common:  Python common package

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #27336 from SUSE/wip-orchestrator-deepsea-iscsi-ganesha

mgr/deepsea: return ganesha and iscsi endpoint URLs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: pg_num should always be a power of two

Signed-off-by: Kai Wagner <kwagner@suse.com>

Merge pull request #29387 from ceph/wip-ceph-volume-tests-no-dashboard

ceph-volume: when testing disable the dashboard

Merge pull request #29347 from smithfarm/wip-40974

rpm: improve ceph-mgr plugin package summaries

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29346 from smithfarm/wip-40973

rpm: fdupes in SUSE builds to conform with packaging guidelines

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29367 from tchaikov/wip-doc-crush-rule

doc/rados/operations: crush_rule is a name

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29368 from tchaikov/wip-40618

qa/tasks/mgr/test_progress.py: s/ev/new_event/

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/telemetry: specify license when opting in

Choosing not to include this in the docs so that the user is more likely
to see this interactively.  (That is...probably good?)

Choose sharing-1-0.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/multimds/basic/tasks/ceph_test_snapshots: disable RECENT_CRASH

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28804 into master

* refs/pull/28804/head:
mds: delay exporting directory whose pin value exceeds max rank id

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29339 into master

* refs/pull/29339/head:
client: don't report any vxattrs to listxattr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge PR #29304 into master

* refs/pull/29304/head:
cephfs: avoid map been inserted by mistake

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29321 into master

* refs/pull/29321/head:
Client: unlink dentry for inode with llref=0

Reviewed-by: Zheng Yan <zyan@redhat.com>

qa: sleep briefly after resetting kclient

Continuation of 6b83f43ba0a99ad86850974d354b17c3e23877be.

Fixes: https://tracker.ceph.com/issues/40999
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: when testing disable the dashboard

ceph-ansible has started enabling the dashboard by
default as of https://github.com/ceph/ceph-ansible/pull/4268

Disabling the dashboard gets us around needing to run
the grafana_server role which is required for the dashboard
and not needed for ceph-volume functional tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #29338 into master

* refs/pull/29338/head:
qa: ignore expected MDS_CLIENT_LATE_RELEASE warning

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28825 into master

* refs/pull/28825/head:
qa: wait for kernel client death
qa: use hard_reset to reboot kclient

Reviewed-by: David Galloway <dgallowa@redhat.com>