ceph-volume tests.functional install new ceph-ansible dependencies

Make note that ceph-ansible's requirements.txt can't be used just yet

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #22084 from liewegas/wip-fix-fop

osd/PrimaryLogPG: fix try_flush_mark_clean write contention case

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22072 from dillaman/wip-24165

librbd: prevent the use of internal feature bits from external users

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21982 from tchaikov/mgr-brag

mgr/telemetry: Add Ceph Telemetry module to send reports back to project

Merge PR #21847 into wip-sage-testing-20180521.023125

* refs/pull/21847/head:
log: disk write coalescing

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #22063 from liewegas/wip-unit-spaces

include/types: space between number and units

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #22042 from liewegas/wip-rm-pool-race

ceph_test_rados_api_misc: remove obsolete LibRadosMiscPool.PoolCreationRace

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #22106 from liewegas/wip-rados-distros

qa/suites/rados/thrash-old-clients: only centos and 16.04

Merge PR #21997 into master

* refs/pull/21997/head:
mds: broadcast quota to relevant clients when quota is explicitly set

Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #22001 into master

* refs/pull/22001/head:
client: avoid freeing inode when it contains TX buffer heads

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22023 into master

* refs/pull/22023/head:
messages: print quota info in MClientQuota msg

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #22045 into master

* refs/pull/22045/head:
client: cleanup Cap init

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #22080 into master

* refs/pull/22080/head:
client: fix issue of revoking non-auth caps

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/suites/rados/thrash-old-clients: only centos and 16.04

We don't have old client builds for the newer distros.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #21961 from ceph/wip-cd-ovh-vol

qa/tests: fix volume size when running in ovh

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22098 from liewegas/wip-24180

mon/LogMonitor: call no_reply() on ignored log message

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #22092 from liewegas/wip-mimic-upgrades

qa/suites/upgrade/luminous-x: a few fixes

Merge pull request #21941 from ukernel/wip-24053

mds: reply session reject for open request from blacklisted client

qa/tasks/cephfs: add timeout parameter to kclient umount_wait

Just make caller happy. there is no easy way to support timeout.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/24053

mds: reply session reject for open request from blacklisted client

Kernel client and old version libcephfs do not check if themselves
are blacklisted. They can be stuck at opening session after getting
blacklisted. The session reject message can avoid this.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/24054

client: fix issue of revoking non-auth caps

when non-auth mds revokes caps, Fcb caps can still be issued by auth
auth mds. It's wrong to flush buffer or invalidate cache when non-auth
mds revokes other caps. This bug can cause client to not respond the
revoke.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
Fixes: https://tracker.ceph.com/issues/24172

Merge PR #22073 into master

* refs/pull/22073/head:
client: delay dentry trimming until after cap traversal
qa: test for trim_caps segfault for trimmed dentries

Reviewed-by: Zheng Yan <zyan@redhat.com>

client: delay dentry trimming until after cap traversal

Fixes: http://tracker.ceph.com/issues/24137
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: test for trim_caps segfault for trimmed dentries

Test case by Yan Zheng [1].

[1] http://tracker.ceph.com/issues/24137#note-6

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21548 into master

* refs/pull/21548/head:
osdc/Objecter: apply throttle on enumerate_objects.

Reviewed-by: Gregory Farnum <gfarnum@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge PR #21963 into master

* refs/pull/21963/head:
mds: properly setup client_need_snapflush for snap inode
Revert "mds: properly setup need_snapflush for snapped inode"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #21972 into master

* refs/pull/21972/head:
client: increase verbosity level for log messages in helper methods

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22066 from liewegas/wip-pgmap-digest-decode

mon/PGMap: fix PGMapDigest decode

Reviewed-by: Kefu Chai <kchai@redhat.com>

mon/LogMonitor: call no_reply() on ignored log message

If we're dropping it on the floor, we need to tell the mon that, so that
it can tell the forwarding mon to discard its state.

Fixes: https://tracker.ceph.com/issues/24180
Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/thrashosds-health: whitelist REQUEST_SLOW

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/luminous-x: whitelist SLOW_OPS

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/luminous-x: more cinder volumes

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/cephtool/test.sh: update unit formatting

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22040 from alfredodeza/wip-rm24147

common/options: add description of osd objectstore backends

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Nathan Cutler <cutler@suse.cz>

cls/rbd: prevent creating image with internal feature bits enabled

Fixes: http://tracker.ceph.com/issues/24165
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: fsx should use the default image features

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: prevent creation of images with internal feature bits

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

osd/PrimaryLogPG: fix try_flush_mark_clean write contention case

Release ctx *after* we dereference it to queue up the other flush waiters.
This was a dumb mistake in the recent fix from 41609ba1b24e5384494c7959494e19302b5a2429

Fixes: http://tracker.ceph.com/issues/24174
Signed-off-by: Sage Weil <sage@redhat.com>

rbd-mirror: mask off implicitly enabled feature bits for new images

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd-mirror: re-use image create option logic for clones

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22078 from tchaikov/wip-cls-rbd-fix-FTBFS

cls/rbd: fix build

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: fix build

3cbe0ced was merged after d03ff14, so update accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21915 from dillaman/wip-rbd-mirror-policy

rbd-mirror: optionally support active/active replication

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #21651 from ceph/wip-da-dmclock-submodule

move dmclock subtree into submodule

Reviewed-by: Eric Ivancich <ivancich@redhat.com>

Merge pull request #22015 from tchaikov/wip-denc

buffer,denc: use ptr::const_iterator for decode

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22059 from LenzGr/mgr-dashboard-documentation

doc: Updated dashboard documentation (features, SSL config)

Reviewed-by: Wido den Hollander <wido@42on.com>
Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa/workunits/rbd/import_export: update unit formatting

Signed-off-by: Sage Weil <sage@redhat.com>

qa/workunits/rbd/cli_generic: update unit formatting

Signed-off-by: Sage Weil <sage@redhat.com>

test/cli-integration/rbd: update unit formatting

Signed-off-by: Sage Weil <sage@redhat.com>

include/types: space between number and units

not

    client:   504B/s rd, 10.6MiB/s wr, 0op/s rd, 30op/s wr

but

    client:   504 B/s rd, 10.6 MiB/s wr, 0 op/s rd, 30 op/s wr

This was changed as part of d3cecebacdcebcf475808a6204de22dfa94d729d.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/PGMap: fix PGMapDigest decode

The compat path wasn't clearing the map before filling in new entries.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22061 from tspmelo/wip-npmrc

mgr/dashboard: Disable package-lock.json creation

Reviewed-by: Ricardo Marques <rimarques@suse.com>

doc: Updated dashboard documentation (features, SSL config)

Updated Ceph manager dashboard documentation: added SSL to the
feature overview, added new section about configuring SSL certificates.
Rearranged installation instructions for setting host name and port.

Signed-off-by: Lenz Grimmer <lgrimmer@suse.com>

Merge pull request #22058 from ricardoasmarques/wip-ng-serve-remote-access

mgr/dashboard: ng serve bind to 0.0.0.0

Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Disable package-lock.json creation

Since we are not keeping track of this file, it is better to not generate it.
Otherwise we might face some package version conflicts,
which break the dashboard build.

Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22057 from tchaikov/qa-valgrind-without-random-distro

qa/suites/rados/verify: remove random-distro$

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume util.prepare remove undocumented comments

Signed-off-by: Alfredo Deza <adeza@redhat.com>

common/options add description of osd objectstore backends

Signed-off-by: Alfredo Deza <adeza@redhat.com>

qa/suites/rados/verify: remove random-distro$

the distro specified by random-distro$ will be overwrited by the one
specfied by valgrind.yaml. and teuthology-suite will give

KeyError: '16.04 not a centos version or codename'

when scheduling a suite involving the facets above. also, i think it's
of not much value to run valgrind/lockdep with different distros.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: ng serve bind to 0.0.0.0

This will allow access to ng serve from remote machines

Signed-off-by: Ricardo Marques <rimarques@suse.com>

client: increase verbosity level for log messages in helper methods

Fixes: http://tracker.ceph.com/issues/21014
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #21981 from tspmelo/fix-momentjs

mgr/dashboard: Fix moment.js deprecation warning

Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #22046 from tchaikov/wip-23653

cmake: do not check for aligned_alloc() anymore

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

core: use const_iterator for decode

Signed-off-by: Kefu Chai <kchai@redhat.com>

mds: use const_iterator for decode

Signed-off-by: Kefu Chai <kchai@redhat.com>

rgw: use const_iterator for decode

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd: use const_iterator for decode

Signed-off-by: Kefu Chai <kchai@redhat.com>

denc: use ptr::const_iterator for decode

as ptr::iterator is able to write to the underlying buffer.

Signed-off-by: Kefu Chai <kchai@redhat.com>

buffer: add cbegin() methods

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not check for aligned_alloc() anymore

this reverts 16f0a67d

Fixes: http://tracker.ceph.com/issues/23653
Signed-off-by: Kefu Chai <kchai@redhat.com>

client: cleanup Cap init

Use reference for inode so it's write-once and must be valid.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

ceph_test_rados_api_misc: fix LibRadosMiscPool.PoolCreationRace

This test is trying to reproduce a case where we don't have a session
for the given OSD, drop our big lock to open one, and when we retake the
osdmap has changed.  Previously that was an easy situation to reproduce
by creating an ioctx for a pool that didn't exist (yet) and then creating
it.  The missing pool would send us into the path that rechecks for an
updated osdmap and trigger the race condition.

However, as of 7037cf8f77588aba1e3f251e04f696da0eccf462 we no longer create
ioctxs for pools that don't exist.  And there isn't an easy way that I can
think of to trigger the missing osd session that races with an osdmap
update.  So, use the debug option that injects the delay to drop us into
the same path.  This is a bit artificial in that we do actually have an
osdmap session, but the block handles that case (since the old no pool
case worked that way), so we cover the same code.

This fix ignores the fact that we still have all this "pool doesn't exist"
handling code in Objecter that deals with teh fact that you could create
an ioctx for a missing pool...that is no longer possible.  Cleanup for
another day!

Fixes: http://tracker.ceph.com/issues/24150
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #22028 from tchaikov/wip-wheelhouse

install-deps: nuke wheelhouse if it's stale

Merge pull request #21751 from shangfufei/wip-async-needless-code

msg/async: the ceph_abort is needless in handle_connect_msg

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #21854 from ifed01/wip-ifed-bluestore-alloc-cleanup

os/bluestore: allocator pruning

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21926 from dragonylffly/wip-fix-throttle-configurations

os/filestore: fix throttle configurations

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21937 from yanghonggang/balancer-list

mgr/balancer: add cmd to list all plans

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #21957 from tchaikov/wip-test-rados-bench

test/pybind/test_rados.py: collect output in stdout for "bench" cmd

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #21984 from liewegas/wip-24118

mon/MonClient: set configs via finisher

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #21892 from dzafman/wip-24045

osd: Don't evict even when preemption has restarted with smaller chunk

Reviewed-by: Sage Weil <sage@redhat.com>

cls/rbd: fixed last read key for interrupted mirror_image_map_list

Previously, the image map would only return a maximum of 64 mappings.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #22027 from trociny/wip-24141

rbd-mirror: potential deadlock when running asok 'flush' command

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #21624 from trociny/wip-22787

librbd: deep copy optionally support flattening cloned image

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd: Clear part of cleaned_meta_map in case of a restarted smaller chunk

This can not happen at the primary because scrub_compare_maps() is only
called once per chunk start.

Preemption causes a smaller chunk from start to be processed again at
replicas.  We clear any of the previous chunk's information.

Signed-off-by: David Zafman <dzafman@redhat.com>

mgr/dashboard: Fix moment.js deprecation warning

Fixes: https://tracker.ceph.com/issues/24115
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge pull request #22003 from votdev/fix_deletion_dialog

mgr/dashboard: Deletion dialog falsely executes deletion when pressing 'Cancel'

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #21928 from ricardoasmarques/wip-remove-useless-unsubscribe

mgr/dashboard: Remove useless observable unsubscriptions

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

rbd: deep copy option to copy clone parent data

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: deep copy optionally support flattening cloned image

Fixes: http://tracker.ceph.com/issues/22787
Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd/deep_copy: adjust in-memory overlap if set_size updated it on disk

so set_parent won't be called when overlap is changed due to image
size change.

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: hide parent object when deep copying empty child

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: fix deep copy clone to destination with larger object size

When assembling a destination object from source objects we need to
read from parent if some (but not all) of assembles do not exist.

Signed-off-by: Mykola Golub <mgolub@suse.com>

test/librbd: add deep copy clone tests

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #22002 from votdev/fix_deletion

mgr/dashboard: Handle errors during deletion

Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #22006 from liewegas/wip-mimic

doc/releases/schedule: add mimic column

doc/releases/mimic: edit/clean up upgrade notes

Signed-off-by: Sage Weil <sage@redhat.com>

install-deps: nuke wheelhouse if it's stale

otherwise, if we update *requirements.txt, and the wheelhouse fails to
offer the newly added dependency, the setup-virtualenv.sh will fail the
build.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd-mirror: make 'flush' asok command asynchronous

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd-mirror: don't reregister asok hook in process entry thread

It might get stuck if the asok is busy executing a command.
Instead kick status update, which will call reregister on finish.

Fixes: http://tracker.ceph.com/issues/24141
Signed-off-by: Mykola Golub <mgolub@suse.com>

messages: print quota info in MClientQuota msg

Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>