Merge pull request #29297 from iotcg/starlingx

ceph-objectstore-tool: set log date format

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29364 from capri1989/doc-pg-num

doc: pg_num should always be a power of two

Reviewed-By: Jan Fajerski <jfajerski@suse.com>
Reviewed-By: Sage Weil <sage@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #29063 from sebastian-philipp/orchestrator-get_feature_set

mgr/orchestrator: Split *_stateless_service and add get_feature_set

Merge pull request #29131 from sebastian-philipp/rook-caching

mgr/rook: Add caching for the Dashboard

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge PR #29334 into master

* refs/pull/29334/head:
qa/tasks/mgr/dashboard/test_mgr_module: adjust expected schema
mgr/telemetry: separate out cluster config vs running daemons
mgr/telemetry: include any config options that are customized

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge PR #29340 into master

* refs/pull/29340/head:
mgr/telemetry: specify license when opting in

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #29419 from vshankar/wip-fix-rbd-support

pybind/rbd: fix call to unregister_osd_perf_queries

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29406 from yuriw/wip-yuriw-upgrade-rwg-master_2

qa/tests: added rgw into upgrade sequence to improve coverage

Merge pull request #29428 from yuriw/wip-yuriw-crontab-master

qa/tests: added mimic-x on master run

Merge PR #29104 into master

* refs/pull/29104/head:
qa/cephfs: test case for timeout config of individual session
mds: add command that config individual client session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29430 from uweigand/fix-no-boost-context-build

rgw: tests: Fix building with -DWITH_BOOST_CONTEXT=OFF

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29412 into master

* refs/pull/29412/head:
doc: mds-config-ref: update 'mds_log_max_segments' value.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29355 into master

* refs/pull/29355/head:
mgr/volumes: set uid/gid of FS client's mount as 0/0

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29133 from ricardoasmarques/prevent-iscsi-target-delete-with-sessions

mgr/dashboard: Prevent deletion of iSCSI IQNs with open sessions

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #29388 into master

* refs/pull/29388/head:
qa: sleep briefly after resetting kclient

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge PR #29377 into master

* refs/pull/29377/head:
mds: add perf counter for finisher of MDSRank

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests: added rgw into upgrade sequence to improve coverage

added rgw into upgrade sequence to improve coverage - splits
See https://github.com/ceph/ceph/pull/29234 https://github.com/ceph/ceph/pull/29282

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw: tests: Fix building with -DWITH_BOOST_CONTEXT=OFF

Attempting to build with -DWITH_BOOST_CONTEXT=OFF will result in
a "conflicting declaration" error when building
test/rgw/test_rgw_dmclock_scheduler.cc

Fixed by avoiding the boost/asio/spawn.hpp include in that case.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

qa/tests: added mimic-x on master run

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

doc: Address further comments on choosing pg_num

Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>

doc: adjust examples to use 2^n pg_num

The examples used pg_num set to 300 or 1000. This misled some users.

Signed-off-by: Lars Marowsky-Bree <lmb@suse.com>

Merge pull request #29405 from wjwithagen/wjw-fix-env-vs-args.t

test/cli: Make the ceph-conf test more liberal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29414 from tchaikov/wip-mgr-api-test

pybind/mgr: install setuptools >= 12

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #27754 from jan--f/c-v-validate-device-lists

ceph-volume: batch ensure device lists are disjoint

test/cli: Make the ceph-conf test more liberal

The tests look for the exact error text.
But they are OS dependant, and thus on FreeBSD the output is
slightly different:
2019-07-30T19:04:16.581+0000 80bc48000 -1 parse_file: filesystem error: in file_size: No such file or directory [from-env]

This stems from the fact that different OSes use different
standard libraries: libc++ <> libstd++

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

pybind/rbd: fix call to unregister_osd_perf_queries

.. should be invoked as self.unregister_osd_perf_queries()
rather than self.module.unregister_osd_perf_queries().

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29318 from tspmelo/wip-backport-component

script/ceph-backport.sh: Allow to set component label

pybind/mgr: install setuptools >= 12

otherwise we have
```
  Running setup.py egg_info for package apache-libcloud

    libcloud/utils/py3.py:76: UserWarning: Missing backports.ssl_match_hostname package
      warnings.warn("Missing backports.ssl_match_hostname package")
    Traceback (most recent call last):
...
      File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66, in _warn_if_setuptools_outdated

...
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
...
Command python setup.py egg_info failed with error code 1 in /tmp/tmp.i95Fg82Ea4/venv/build/apache-libcloud
```

and more important thing is, setuptools >= 12 allows us to install the
python modules in venv, without uninstalling existing modules if they
are also installed in by system packages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/rook: monkey-patch for CoreV1Api.list_node

`kubernetes.client.apis.core_v1_api.CoreV1Api#list_node` is
broken and needs a monkey patch. Might be related to
an incompatible k8s cluster version.

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: better format `service ls`

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Implement fallback for CentOS 7

`kubernetes-client` requires `urllib3` > 1.15, but CentOS 7 ships 1.10

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Move KubernetesResource to rook_cluser.py

Use `KubernetesResource` as cache for pods, inventory maps and nodes.
This is required for the dashboard.

Also:
  properly return K8s nodes
  fix minor issues

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Add feature to gather k8s storageclass information

Provide ability to fetch or watch k8s resources,
and hold the resulting objects in a dict. A method
has been added to the rook module to allow other
mgr modules to fetch the state of specific k8s
resources.

Signed-off-by: Paul Cuzner <pcuzner@redhat.com>

mgr/orchestrator_clli: Always return str

Also: Don't return `"None"`
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/ansible: Moved RGW defaults to ansible

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/rook: Fix RGW creation

* Moved RGWSpec defaults to a new method

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr/orchestrator: Split *_stateless_service and add get_feature_set

Split *_stateless_service, because every orchestrator
did some method dispatching internally anyway.

Also: Added test

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

ceph-volume: batch ensure device lists are disjoint

Fixes: https://tracker.ceph.com/issues/41018
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #29363 into master

* refs/pull/29363/head:
qa/suites/multimds/basic/tasks/ceph_test_snapshots: disable RECENT_CRASH
qa/suites/kcephfs/recovery/failover.yaml: disable RECENT_CRASH
qa/suites/fs/multifs/tasks/failover.yaml: disable RECENT_CRASH

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/mgr/dashboard/test_mgr_module: adjust expected schema

Signed-off-by: Sage Weil <sage@redhat.com>

doc: mds-config-ref: update 'mds_log_max_segments' value.

Also drop options that not exists on code:

* mds_log_max_expiring
* mds_log_eopen_size

Signed-off-by: Konstantin Shalygin <k0ste@k0ste.ru>

Merge PR #29185 into master

* refs/pull/29185/head:
os/bluestore: add tail length when caculating reserved buffer size
os/bluestore: record next_offs when offset is not sequential.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29356 into master

* refs/pull/29356/head:
pybind/mgr/restful: use dict.items() for py3 compatibility

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #29098 from liewegas/wip-unhexdump-c

src/script/unhexdump-C: script to reverse a 'hexdump -C' style hexdump

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29394 from mkogan1/wip-rgw-bi-list-shard-id

rgw: add option to specify shard-id for "bi list" admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #29362 into master

* refs/pull/29362/head:
Update nautilus release for 14.2.2
Updated schedule with latest Nautilus release

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28239 into master

* refs/pull/28239/head:
cephfs-shell: Add tox for testing with flake8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29386 from alfredodeza/wip-rm40987

ceph-volume  simple: better detection when 'type' file is not present

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

cephfs-shell: Add tox for testing with flake8

Fixes: https://tracker.ceph.com/issues/39947
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge PR #29331 into master

* refs/pull/29331/head:
common: add comment about pod memory requests/limits

Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #29161 from valentinbajrami/improve-grep-regex

src/script/check_commands.sh: fix grep regex class range

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: add option to specify shard-id for "bi list" admin command

related to: https://tracker.ceph.com/issues/40700
Signed-off-by: Mark Kogan <mkogan@redhat.com>

src/script/check_commands.sh: fix grep regex class range

Correct the faulty regex class.

Signed-off-by: Valentin Bajrami <valentin.bajrami@gmail.com>

Merge pull request #28915 from sebastian-philipp/python-common-package-2

python-common:  Python common package

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #27336 from SUSE/wip-orchestrator-deepsea-iscsi-ganesha

mgr/deepsea: return ganesha and iscsi endpoint URLs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc: pg_num should always be a power of two

Signed-off-by: Kai Wagner <kwagner@suse.com>

Merge pull request #29387 from ceph/wip-ceph-volume-tests-no-dashboard

ceph-volume: when testing disable the dashboard

Merge pull request #29347 from smithfarm/wip-40974

rpm: improve ceph-mgr plugin package summaries

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29346 from smithfarm/wip-40973

rpm: fdupes in SUSE builds to conform with packaging guidelines

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29367 from tchaikov/wip-doc-crush-rule

doc/rados/operations: crush_rule is a name

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29368 from tchaikov/wip-40618

qa/tasks/mgr/test_progress.py: s/ev/new_event/

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/telemetry: specify license when opting in

Choosing not to include this in the docs so that the user is more likely
to see this interactively.  (That is...probably good?)

Choose sharing-1-0.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/multimds/basic/tasks/ceph_test_snapshots: disable RECENT_CRASH

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28804 into master

* refs/pull/28804/head:
mds: delay exporting directory whose pin value exceeds max rank id

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29339 into master

* refs/pull/29339/head:
client: don't report any vxattrs to listxattr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge PR #29304 into master

* refs/pull/29304/head:
cephfs: avoid map been inserted by mistake

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29321 into master

* refs/pull/29321/head:
Client: unlink dentry for inode with llref=0

Reviewed-by: Zheng Yan <zyan@redhat.com>

qa: sleep briefly after resetting kclient

Continuation of 6b83f43ba0a99ad86850974d354b17c3e23877be.

Fixes: https://tracker.ceph.com/issues/40999
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: when testing disable the dashboard

ceph-ansible has started enabling the dashboard by
default as of https://github.com/ceph/ceph-ansible/pull/4268

Disabling the dashboard gets us around needing to run
the grafana_server role which is required for the dashboard
and not needed for ceph-volume functional tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #29338 into master

* refs/pull/29338/head:
qa: ignore expected MDS_CLIENT_LATE_RELEASE warning

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28825 into master

* refs/pull/28825/head:
qa: wait for kernel client death
qa: use hard_reset to reboot kclient

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-volume tests: ensure that better heuristics exist for objectstore detection

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate better detect bluestore/filestore when type is not found

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Update nautilus release for 14.2.2

Nautilus v14.2.2 added to releases

Signed-off-by: JuanJose Galvez <juanjose.galvez@gmail.com>

Updated schedule with latest Nautilus release

Signed-off-by: JuanJose Galvez <juanjose.galvez@gmail.com>

Merge PR #29374 into master

* refs/pull/29374/head:
cephfs-shell: Fix flake8 line too long error
cephfs-shell: Fix flake8 line break after binary operator error
cephfs-shell: Fix flake8 continuation line over-indented for hanging indent error
cephfs-shell: Fix flake8 missing whitespace around arithmetic operator error
cephfs-shell: Fix multiple spaces after ':' flake8 error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27780 from theanalyst/cls-timeindex-denc

cls: add timeindex types to ceph-dencoder

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29182 from dcasier/mgr-dashboard-ganesha-fix-cephfs-name

mgr/dashboard: ganesha: Specify the name of the filesystem (create_path)

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #29138 from Reperator/dashboard-pw-hash

mgr/dashboard: support setting password hashes

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

cephfs-shell: Fix flake8 line too long error

https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 line break after binary operator error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 continuation line over-indented for hanging indent error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 missing whitespace around arithmetic operator error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix multiple spaces after ':' flake8 error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #29272 from votdev/issue_40925

mgr/dashboard: Various UI issues related to CephFS

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29216 from rafaelquint/configtest

mgr/dashboard: E2E test to verify Configuration editing functionality

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29324 from adk3798/typepages

mgr/dashboard: Explicitly type page variables

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29371 from bk201/wip-40971

mgr/dashboard: fix MDS counter chart is not displayed

Reviewed-by: Ricardo Dias <rdias@suse.com>

pybind/mgr/restful: use dict.items() for py3 compatibility

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: add perf counter for finisher of MDSRank
        in all flash cluster, there are a lot of reqs in the queue.

Signed-off-by: simon gao <simon29rock@gmail.com>

qa/cephfs: test case for timeout config of individual session

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #29341 from zhangbingyi1/master

test/librados: avoid residual crush rule after test case execution

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix MDS counter chart is not displayed

Unit of timetamp for performance counters is changed to nanoseconds [1].
Adapt this for CephFS MDS performance counters in backend.

[1]: https://github.com/ceph/ceph/pull/28882

Fixes: https://tracker.ceph.com/issues/40971
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

test/librados: avoid residual crush rule after test case execution
Fixes: http://tracker.ceph.com/issues/40970
Signed-off-by: Bingyi Zhang <zhangbingyi@umcloud.com>

os/bluestore: add tail length when caculating reserved buffer size

Signed-off-by: Xiangyang Yu <penglaiyxy@gmail.com>

os/bluestore: record next_offs when offset is not sequential.

Signed-off-by: Xiangyang Yu <penglaiyxy@gmail.com>

Merge pull request #29348 from myoungwon/fix-wrong-class

osd: fix wrong arguments when dropping refcount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29345 from majianpeng/bluestore-flush-waiter

os/bluestore: avoid unnecessary notify.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29126 from xiexingguo/wip-clamp-ratio-to-0

mgr/progress: clamp pg recovery ratio to 0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28012 from rouming/bootstrap-after-setuid

global/global_init: do first transport connection after setuid()

Reviewed-by: Kefu Chai <kchai@redhat.com>