mds: split the dir if the op makes it oversized

some ops maybe in flight, so that, split dir in finisher thread ,may affect inflight ops.

Fixes: https://tracker.ceph.com/issues/41880
Signed-off-by: simon gao <simon29rock@gmail.com>

Merge PR #29334 into master

* refs/pull/29334/head:
qa/tasks/mgr/dashboard/test_mgr_module: adjust expected schema
mgr/telemetry: separate out cluster config vs running daemons
mgr/telemetry: include any config options that are customized

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge PR #29340 into master

* refs/pull/29340/head:
mgr/telemetry: specify license when opting in

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #29419 from vshankar/wip-fix-rbd-support

pybind/rbd: fix call to unregister_osd_perf_queries

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #29406 from yuriw/wip-yuriw-upgrade-rwg-master_2

qa/tests: added rgw into upgrade sequence to improve coverage

Merge pull request #29428 from yuriw/wip-yuriw-crontab-master

qa/tests: added mimic-x on master run

Merge PR #29104 into master

* refs/pull/29104/head:
qa/cephfs: test case for timeout config of individual session
mds: add command that config individual client session

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29430 from uweigand/fix-no-boost-context-build

rgw: tests: Fix building with -DWITH_BOOST_CONTEXT=OFF

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #29412 into master

* refs/pull/29412/head:
doc: mds-config-ref: update 'mds_log_max_segments' value.

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29355 into master

* refs/pull/29355/head:
mgr/volumes: set uid/gid of FS client's mount as 0/0

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29133 from ricardoasmarques/prevent-iscsi-target-delete-with-sessions

mgr/dashboard: Prevent deletion of iSCSI IQNs with open sessions

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge PR #29388 into master

* refs/pull/29388/head:
qa: sleep briefly after resetting kclient

Reviewed-by: David Galloway <dgallowa@redhat.com>

Merge PR #29377 into master

* refs/pull/29377/head:
mds: add perf counter for finisher of MDSRank

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tests: added rgw into upgrade sequence to improve coverage

added rgw into upgrade sequence to improve coverage - splits
See https://github.com/ceph/ceph/pull/29234 https://github.com/ceph/ceph/pull/29282

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

rgw: tests: Fix building with -DWITH_BOOST_CONTEXT=OFF

Attempting to build with -DWITH_BOOST_CONTEXT=OFF will result in
a "conflicting declaration" error when building
test/rgw/test_rgw_dmclock_scheduler.cc

Fixed by avoiding the boost/asio/spawn.hpp include in that case.

Signed-off-by: Ulrich Weigand <ulrich.weigand@de.ibm.com>

qa/tests: added mimic-x on master run

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #29405 from wjwithagen/wjw-fix-env-vs-args.t

test/cli: Make the ceph-conf test more liberal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29414 from tchaikov/wip-mgr-api-test

pybind/mgr: install setuptools >= 12

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #27754 from jan--f/c-v-validate-device-lists

ceph-volume: batch ensure device lists are disjoint

test/cli: Make the ceph-conf test more liberal

The tests look for the exact error text.
But they are OS dependant, and thus on FreeBSD the output is
slightly different:
2019-07-30T19:04:16.581+0000 80bc48000 -1 parse_file: filesystem error: in file_size: No such file or directory [from-env]

This stems from the fact that different OSes use different
standard libraries: libc++ <> libstd++

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

pybind/rbd: fix call to unregister_osd_perf_queries

.. should be invoked as self.unregister_osd_perf_queries()
rather than self.module.unregister_osd_perf_queries().

Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #29318 from tspmelo/wip-backport-component

script/ceph-backport.sh: Allow to set component label

pybind/mgr: install setuptools >= 12

otherwise we have
```
  Running setup.py egg_info for package apache-libcloud

    libcloud/utils/py3.py:76: UserWarning: Missing backports.ssl_match_hostname package
      warnings.warn("Missing backports.ssl_match_hostname package")
    Traceback (most recent call last):
...
      File "build/bdist.linux-x86_64/egg/setuptools_scm/version.py", line 66, in _warn_if_setuptools_outdated

...
setuptools_scm.version.SetuptoolsOutdatedWarning: your setuptools is too old (<12)
...
Command python setup.py egg_info failed with error code 1 in /tmp/tmp.i95Fg82Ea4/venv/build/apache-libcloud
```

and more important thing is, setuptools >= 12 allows us to install the
python modules in venv, without uninstalling existing modules if they
are also installed in by system packages.

Signed-off-by: Kefu Chai <kchai@redhat.com>

ceph-volume: batch ensure device lists are disjoint

Fixes: https://tracker.ceph.com/issues/41018
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #29363 into master

* refs/pull/29363/head:
qa/suites/multimds/basic/tasks/ceph_test_snapshots: disable RECENT_CRASH
qa/suites/kcephfs/recovery/failover.yaml: disable RECENT_CRASH
qa/suites/fs/multifs/tasks/failover.yaml: disable RECENT_CRASH

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks/mgr/dashboard/test_mgr_module: adjust expected schema

Signed-off-by: Sage Weil <sage@redhat.com>

doc: mds-config-ref: update 'mds_log_max_segments' value.

Also drop options that not exists on code:

* mds_log_max_expiring
* mds_log_eopen_size

Signed-off-by: Konstantin Shalygin <k0ste@k0ste.ru>

Merge PR #29185 into master

* refs/pull/29185/head:
os/bluestore: add tail length when caculating reserved buffer size
os/bluestore: record next_offs when offset is not sequential.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29356 into master

* refs/pull/29356/head:
pybind/mgr/restful: use dict.items() for py3 compatibility

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #29098 from liewegas/wip-unhexdump-c

src/script/unhexdump-C: script to reverse a 'hexdump -C' style hexdump

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #29394 from mkogan1/wip-rgw-bi-list-shard-id

rgw: add option to specify shard-id for "bi list" admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #29362 into master

* refs/pull/29362/head:
Update nautilus release for 14.2.2
Updated schedule with latest Nautilus release

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28239 into master

* refs/pull/28239/head:
cephfs-shell: Add tox for testing with flake8

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #29386 from alfredodeza/wip-rm40987

ceph-volume  simple: better detection when 'type' file is not present

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

cephfs-shell: Add tox for testing with flake8

Fixes: https://tracker.ceph.com/issues/39947
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge PR #29331 into master

* refs/pull/29331/head:
common: add comment about pod memory requests/limits

Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #29161 from valentinbajrami/improve-grep-regex

src/script/check_commands.sh: fix grep regex class range

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>
Reviewed-by: Sebastian Krah <skrah@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

rgw: add option to specify shard-id for "bi list" admin command

related to: https://tracker.ceph.com/issues/40700
Signed-off-by: Mark Kogan <mkogan@redhat.com>

src/script/check_commands.sh: fix grep regex class range

Correct the faulty regex class.

Signed-off-by: Valentin Bajrami <valentin.bajrami@gmail.com>

Merge pull request #28915 from sebastian-philipp/python-common-package-2

python-common:  Python common package

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #27336 from SUSE/wip-orchestrator-deepsea-iscsi-ganesha

mgr/deepsea: return ganesha and iscsi endpoint URLs

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29387 from ceph/wip-ceph-volume-tests-no-dashboard

ceph-volume: when testing disable the dashboard

Merge pull request #29347 from smithfarm/wip-40974

rpm: improve ceph-mgr plugin package summaries

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #29346 from smithfarm/wip-40973

rpm: fdupes in SUSE builds to conform with packaging guidelines

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #29367 from tchaikov/wip-doc-crush-rule

doc/rados/operations: crush_rule is a name

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #29368 from tchaikov/wip-40618

qa/tasks/mgr/test_progress.py: s/ev/new_event/

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

mgr/telemetry: specify license when opting in

Choosing not to include this in the docs so that the user is more likely
to see this interactively.  (That is...probably good?)

Choose sharing-1-0.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/multimds/basic/tasks/ceph_test_snapshots: disable RECENT_CRASH

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28804 into master

* refs/pull/28804/head:
mds: delay exporting directory whose pin value exceeds max rank id

Reviewed-by: Zheng Yan <zyan@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29339 into master

* refs/pull/29339/head:
client: don't report any vxattrs to listxattr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: David Disseldorp <ddiss@suse.de>

Merge PR #29304 into master

* refs/pull/29304/head:
cephfs: avoid map been inserted by mistake

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29321 into master

* refs/pull/29321/head:
Client: unlink dentry for inode with llref=0

Reviewed-by: Zheng Yan <zyan@redhat.com>

qa: sleep briefly after resetting kclient

Continuation of 6b83f43ba0a99ad86850974d354b17c3e23877be.

Fixes: https://tracker.ceph.com/issues/40999
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph-volume: when testing disable the dashboard

ceph-ansible has started enabling the dashboard by
default as of https://github.com/ceph/ceph-ansible/pull/4268

Disabling the dashboard gets us around needing to run
the grafana_server role which is required for the dashboard
and not needed for ceph-volume functional tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #29338 into master

* refs/pull/29338/head:
qa: ignore expected MDS_CLIENT_LATE_RELEASE warning

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28825 into master

* refs/pull/28825/head:
qa: wait for kernel client death
qa: use hard_reset to reboot kclient

Reviewed-by: David Galloway <dgallowa@redhat.com>

ceph-volume tests: ensure that better heuristics exist for objectstore detection

Signed-off-by: Alfredo Deza <adeza@redhat.com>

ceph-volume simple.activate better detect bluestore/filestore when type is not found

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Update nautilus release for 14.2.2

Nautilus v14.2.2 added to releases

Signed-off-by: JuanJose Galvez <juanjose.galvez@gmail.com>

Updated schedule with latest Nautilus release

Signed-off-by: JuanJose Galvez <juanjose.galvez@gmail.com>

Merge PR #29374 into master

* refs/pull/29374/head:
cephfs-shell: Fix flake8 line too long error
cephfs-shell: Fix flake8 line break after binary operator error
cephfs-shell: Fix flake8 continuation line over-indented for hanging indent error
cephfs-shell: Fix flake8 missing whitespace around arithmetic operator error
cephfs-shell: Fix multiple spaces after ':' flake8 error

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #27780 from theanalyst/cls-timeindex-denc

cls: add timeindex types to ceph-dencoder

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #29182 from dcasier/mgr-dashboard-ganesha-fix-cephfs-name

mgr/dashboard: ganesha: Specify the name of the filesystem (create_path)

Reviewed-by: Ricardo Dias <rdias@suse.com>

Merge pull request #29138 from Reperator/dashboard-pw-hash

mgr/dashboard: support setting password hashes

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

cephfs-shell: Fix flake8 line too long error

https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 line break after binary operator error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 continuation line over-indented for hanging indent error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix flake8 missing whitespace around arithmetic operator error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

cephfs-shell: Fix multiple spaces after ':' flake8 error

Fixes: https://tracker.ceph.com/issues/40992
Signed-off-by: Varsha Rao <varao@redhat.com>

Merge pull request #29272 from votdev/issue_40925

mgr/dashboard: Various UI issues related to CephFS

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29216 from rafaelquint/configtest

mgr/dashboard: E2E test to verify Configuration editing functionality

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #29324 from adk3798/typepages

mgr/dashboard: Explicitly type page variables

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #29371 from bk201/wip-40971

mgr/dashboard: fix MDS counter chart is not displayed

Reviewed-by: Ricardo Dias <rdias@suse.com>

pybind/mgr/restful: use dict.items() for py3 compatibility

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: add perf counter for finisher of MDSRank
        in all flash cluster, there are a lot of reqs in the queue.

Signed-off-by: simon gao <simon29rock@gmail.com>

qa/cephfs: test case for timeout config of individual session

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #29341 from zhangbingyi1/master

test/librados: avoid residual crush rule after test case execution

Reviewed-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: fix MDS counter chart is not displayed

Unit of timetamp for performance counters is changed to nanoseconds [1].
Adapt this for CephFS MDS performance counters in backend.

[1]: https://github.com/ceph/ceph/pull/28882

Fixes: https://tracker.ceph.com/issues/40971
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

test/librados: avoid residual crush rule after test case execution
Fixes: http://tracker.ceph.com/issues/40970
Signed-off-by: Bingyi Zhang <zhangbingyi@umcloud.com>

os/bluestore: add tail length when caculating reserved buffer size

Signed-off-by: Xiangyang Yu <penglaiyxy@gmail.com>

os/bluestore: record next_offs when offset is not sequential.

Signed-off-by: Xiangyang Yu <penglaiyxy@gmail.com>

Merge pull request #29348 from myoungwon/fix-wrong-class

osd: fix wrong arguments when dropping refcount

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #29345 from majianpeng/bluestore-flush-waiter

os/bluestore: avoid unnecessary notify.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #29126 from xiexingguo/wip-clamp-ratio-to-0

mgr/progress: clamp pg recovery ratio to 0

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28012 from rouming/bootstrap-after-setuid

global/global_init: do first transport connection after setuid()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27985 from ifed01/wip-ifed-slow-omaps

tools/ceph-objectstore-tool: introduce list-slow-omap command.

Reviewed-by: Sage Weil <sage@redhat.com>

qa/tasks/mgr/test_progress.py: s/ev/new_event/

as a follow-up fix for 5604ba4e

Fixes: http://tracker.ceph.com/issues/40618
Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/volumes: set uid/gid of FS client's mount as 0/0

... to allow creating subvolumes/subvolume groups within a filesystem.

Fixes: http://tracker.ceph.com/issues/40927
Signed-off-by: Ramana Raja <rraja@redhat.com>

doc/rados/operations: crush_rule is a name

like
```
ceph osd pool set <pool-name> crush_rule <rule-name>
```
where `<rule-name>` is a string instead of a number.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #29238 into master

* refs/pull/29238/head:
os/bluestore: it's better to erase spanning blob only once

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge PR #29239 into master

* refs/pull/29239/head:
qa/tasks/ceph_manager.py: always use self.logger

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #29305 into master

* refs/pull/29305/head:
mgr: Reduce logging noise when handling commands

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29307 into master

* refs/pull/29307/head:
monmaptool: added --addv option to usage description

Reviewed-by: João Eduardo Luís <joao@suse.de>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29316 into master

* refs/pull/29316/head:
common,tools: make sure the destination buffer can handle the size of the string
src/tools: initialize variables before the goto statement
src/mount: check before dereference buf
src/crush: check before dereference out2
src/test: s/strcpy/strncpy

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29358 into master

* refs/pull/29358/head:
mgr/telemetry: salt osd ids too

Reviewed-by: Dan Mick <dmick@redhat.com>

Merge pull request #29054 from dillaman/wip-40621

mgr/rbd_support: support scheduling long-running background operations

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #29361 from tchaikov/wip-qa-osd-pause

qa: fixes

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

qa: wait for kernel client death

After sending the reboot command, we need to wait briefly for it to be
rebooted so that the kernel client doesn't voluntarily give up its Fb
cap.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: use hard_reset to reboot kclient

power_off may allow the mounts to gracefully unmount. We don't want this if the
kclient is stuck or we desire the client to "disappear" and come back.

Fixes: http://tracker.ceph.com/issues/37681
Depends-on: https://github.com/ceph/teuthology/pull/1296
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common: add comment about pod memory requests/limits

Written in private correspondence by Travis Nielson.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>