Merge pull request #19928 from jdurgin/wip-luminous-omap-recovery

config: lower default omap entries recovered at once

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #19938 from liewegas/wip-list-size-luminous

luminous: common: compute SimpleLRU's size with contents.size() instead of lru.size()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19947 from tchaikov/wip-pr-18191-luminous

qa/standalone/osd/osd-mark-down: create pool to get updated osdmap faster

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19969 from smithfarm/wip-22691-luminous

luminous: ceph-base symbols not stripped in debs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19976 from pdvian/wip-22690-luminous

luminous: tests: increase osd count for ec testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19977 from pdvian/wip-22692-luminous

luminous: common: compute SimpleLRU's size with contents.size() instead of lru.…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19995 from pdvian/wip-22698-luminous

luminous: common/throttle: start using 64-bit values

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #20244 from ceph/backport-19276

luminous: ceph-volume: do not use --key during mkfs

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20247 from ceph/revert-19552-wip-22452-luminous

Revert " luminous: msg/async: unregister connection failed when racing happened"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert " luminous: msg/async: unregister connection failed when racing happened"

Merge pull request #19187 from liewegas/wip-trim-mgrmap-luminous

mon/MgrMonitor: limit mgrmap history

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19269 from tchaikov/wip-pr-19202-luminous

luminous: cmake,common/RWLock: check for libpthread extensions

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19499 from shinobu-x/wip-22402-luminous

luminous: osd: replica read can trigger cache promotion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #19552 from shinobu-x/wip-22452-luminous

 luminous: msg/async: unregister connection failed when racing happened

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19628 from shinobu-x/wip-22501-luminous

 luminous: tests: do not configure ec data pool with memstore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19630 from shinobu-x/wip-22507-luminous

 luminous: bluestore: do not crash on over-large objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19742 from tchaikov/wip-luminous-pr-19649-and-19650

luminous: common/dns, erasure-code: fix mem leaks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20049 from liewegas/wip-zero-length-luminous

os: fix 0-length zero semantics, test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20068 from smithfarm/wip-22761-luminous

luminous: osd: do not check out-of-date osdmap for DESTROYED flag on start

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20069 from smithfarm/wip-22770-luminous

luminous: tools: ceph-objectstore-tool set-size should clear data-digest

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20150 from pdvian/wip-22807-luminous

luminous: mon: do not use per_pool_sum_delta to show recovery summary

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: process: disable stdin param of run()

we cannot use process.communicate() to feed the Popen with input,
because, upon return of process.communicate() the stdout,stderr are
closed. see https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6df444cf33a251f5dcb536606b44af696cdb4ed5)

Conflicts:
src/ceph-volume/ceph_volume/process.py

ceph-volume: use --keyfile instead of --key

We do not want the key to show up on the command line (it may appear in
the process list or sudo log file).

Fixes: http://tracker.ceph.com/issues/22283
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 46abd50f8bcbd347d577581ae42156b718556158)

osd: accept 'keyfile' as well as 'key' during mkfs

Also, don't print the raw key to the log.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Kefu Chai <kefu@redhat.com>
(cherry picked from commit 18aff37ee3a2c2d3016f08c98161a5acc41ff8e5)

Merge pull request #19370 from shinobu-x/wip-22339-luminous

luminous: client: quit on failed remount during dentry invalidate test #19370

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19585 from ukernel/luminous-22460

luminous: mds: handle client session messages when mds is stopping

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19626 from shinobu-x/wip-22499-luminous

 luminous: cephfs-journal-tool: tool would miss to report some invalid range

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19627 from shinobu-x/wip-22500-luminous

 luminous: cephfs: potential adjust failure in lru_expire

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19776 from ukernel/luminous-22492

luminous: mds: respect mds_client_writeable_range_max_inc_objs config

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19829 from pdvian/wip-22573-luminous

luminous: vstart_runner: fixes for recent cephfs changes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19830 from shinobu-x/wip-22579-luminous

luminous: mds: check for CEPH_OSDMAP_FULL is now wrong; cluster full flag is obsolete

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19871 from smithfarm/wip-21948-luminous

luminous: mon: MDSMonitor: reject misconfigured mds_blacklist_interval

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19959 from smithfarm/wip-22694-luminous

luminous: mds: fix dump last_sent

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19962 from smithfarm/wip-22580-luminous

luminous: tests: full flag not set on osdmap for tasks.cephfs.test_full

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19968 from ukernel/luminous-22699

luminous: ceph-fuse: ::rmdir() uses a deleted memory structure of dentry leads …

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19982 from ukernel/luminous-22719

luminous: mds: handle 'inode gets queued for recovery multiple times'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20042 from tchaikov/wip-pr-19998-luminous

luminous: fix broken use of streamstream::rdbuf()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20106 from smithfarm/wip-22502-luminous

luminous: osd: Pool Compression type option doesn't apply to new OSDs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20240 from ceph/backport-20010

luminous: ceph-volume: adds a --destroy flag to ceph-volume lvm zap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: clean up docstrings for lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 57471fe7e74c94351e713ff99cbcbaa7dc6a0f52)

ceph-volume: print a message to terminal if --destroy is skipped

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit d2529ee5dd9a4dc6a81eeb2d5e54c0bdd6beeb2f)

ceph-volume: no need to set PVolume.vg_name explicitly

This will be set automatically in __init__ by looping
through kw.items and creating properties on PVolume
with them.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 129182645c4a7e5e42500d3ef2083cfdfb0137e4)

ceph-volume: switch remove_vg and remove_pv to process.run

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5a1be9a73919e13fe603861588bcc20872a4c133)

ceph-volume: allow passing a fail_msg param to process.run

This will allow the user to provide a more detailed error message
on why the command failed.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 7204851042fa42799e1003ef57d57a512d4a7f28)

Conflicts:
src/ceph-volume/ceph_volume/process.py

Merge pull request #20239 from ceph/backport-19949

luminous: ceph-volume: adds --crush-device-class flag for lvm prepare and create #19949

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: tests for lvm zap --destroy

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit fafa183d8f13fa769eaab5f865f6592b89225901)

ceph-volume: also remove pvs with lvm zap --destroy

Leaving the pv around doesn't keep us from redeploying on
that device because we also wipefs, but explicitly destroying
it makes the output read better.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit cb748ccb2264aea93dd4717a46d7f833c5b1eccd)

ceph-volume: adds an api.lvm.remove_pv method

Used to remove physical volumes

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 0811c02b118c270b2bec87cd26a877b26bc780b1)

ceph-volume: remove the -a flag from api.lvm.get_pvs

When we run `pvs -a` it returns all pvs, even ones that
have been recently deleted. Because of this pvs that have just
been zapped with `lvm zap` show up and can cause issues if a device
is zapped more than once.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 10a349742cae909d1ebf0a0b3b49c88774929f53)

ceph-volume: docs for the --destroy flag of lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 708cb43c503e962304211041570d6ca660bca1fb)

ceph-volume: adds the --destroy flag to ceph-volume lvm zap

If you use the --destroy flag and are zapping a raw device
or parition then zap will destroy any vgs or lvs it finds on that
device.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit aea326b772395c076e0d75f7c2f591a46aef7f57)

ceph-volume: adds an api.lvm.remove_vg method

This method can be used to remove volume groups

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 50a6a543a956d133b428cf36ad629c2131be5490)

ceph-volume: expose vg_name in api.lvm.get_api_pvs

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b7f24ec451e167d81a83c1e21597a9fece4f0684)

Merge pull request #20238 from ceph/backport-19875

luminous: ceph-volume: adds success messages for lvm prepare/activate/create

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: print the lvm activate success message correctly

Move the terminal.success to the methods that have access to the
correct osd_id so that they do not print None for the osd_id
in the log message.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit c9f7689a7a71a6b2321301b84599f319e8ae8eba)

ceph-volume: docs for --crush-device-class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 21f459437bd1208812504a20e5a50f7a47bfa979)

ceph-volume: adds crush_device_class to json metadata for 'osd new'

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 3404d8bba503f3f74f411b11108d39cb626d3bed)

ceph-volume: adds crush_device_class to the functional tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 36a388d72acb049b5ee25ceea6a8d341aa58e31f)

ceph-volume: add crush_device_class to lvm metadata

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 11da2185635ce14103471d851f1e8c84199d3745)

ceph-volume: adds a --crush-device-class flag to lvm create and prepare

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit bf468bc3737c15a8f15ec19ebf81c42672e28aaf)

ceph-volume: adds success message to ceph-volume simple activate

This used to only show the message if systemd was not being used, now
it will show this always if the command is successful.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 87c6ed14e7e0b528a1fe107e2e531908cdea0f47)

ceph-volume: add success message for ceph-volume lvm create

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 0185d93cc1be1e7a6d3a803631ff92df9787754a)

ceph-volume: add success message for ceph-volume lvm activate

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit d30928d7c7d7cfb7e465ec2e7e46dc48407436f0)

ceph-volume: add success message for ceph-volume lvm prepare

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 75c460f6e1864f15e1d23146c5740ce43c000e42)

Merge pull request #20237 from ceph/backport-wip-rm22281

luminous: ceph-volume rollback on failed OSD prepare/create

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume lvm.create rollback osd creation when prepare or activate fails

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 07be6fe0ab7981ebdc34779a8ccd6220d640a549)

ceph-volume lvm.create fallback to bluestore when objectstore is not specified

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6781d6498e01bfc89189c235540e5c2a2e6bc8ca)

ceph-volume lvm.prepare fallback to bluestore when objectstore is not specified

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b17ad2f8fe617d219fdb2f46dd2675669f32f08)

ceph-volume lvm.prepare rollback osd when prepare fails and an osd id was generated

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 579d12806b2ffc693a4e70ba3eaf663fa281d35e)

ceph-volume lvm.common create a rollback_osd utility to cleanup failed osd prepare/create calls

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e533792e0dbafc160238c0f5783f4430e2705571)

Merge pull request #19483 from smithfarm/wip-22375-luminous

luminous: build/ops: luminous build fails with --without-radosgw

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20211 from dillaman/wip-22198-luminous

luminous: librbd: compare and write against a clone can result in failure

Reviewed-by: Mykola Golub <mgolub@suse.com>

test/librbd: disable compare and write if skipping partial discard

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4e8b0b60a72fe1e1ac10a2beb46b77d4b58c91c4)

librbd: refactor object write request state machines

Fixes: http://tracker.ceph.com/issues/20789
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 0e643fb926f0484bb4f860740c7f85b692de6737)

librbd: consolidate all object discard-related logic into single state machine

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 318797f59adcedcc386e01f1975011e0086434ac)

librbd: simplify interface between object IO and copyup state machines

The initial copyup was not receiving a write hint and the
code for hints was duplicated multiple times. Additionally,
the object map state should match the last executed write op.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7117aa4ecaf0b619b527b2783fa1d09b11f3dd55)

Conflicts:
src/librbd/io/ObjectRequest.h: trivial resolution

librbd: copyup state machine needs to handle empty write ops

The compare-and-write object operation cannot be executed
concurrently within a copyup operation since the object might not
exist yet (if not performing a deep-copy).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit f6db9b8027b6978b4d28fcf9f0389c88f9109e75)

librbd: object cacher should re-use read state machine

This adds support for sparse-reads and ensures all object reads
utilize a single, tested code path.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit e79c0cf2009c69681ed4c70f52ade2f5fd570567)

librbd: refactor io::ObjectReadRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 62c3282b875f7345d7b71228d1e35e0cc87d44fa)

librbd: default template parameter for C_SparseReadRequest

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ab0e558961ed9b63c00ad1d09ee0f9f79fd62295)

librbd: reduce lock scope when assembling read results

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 284feb6a32795c64b1c5ec3b3bf0d959e72dd940)

librbd: ObjectReadRequest does not require buffer extents

The ReadResult wrapper requires the buffer extents to know
how to properly reconstruct the out buffer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 35ce1c2d39302011adbf494ada6248495681a436)

librbd: templatize IO object request state machines

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 654a78fa54520370f5466f9d282d4e9f8365ad48)

test/librados_test_stub: mocked remainder of IO ops used by librbd

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c4e5613e4a74316ba9667654e887a835b240609a)

test/librados_test_stub: simulate set_alloc_hint creating a non-existent file

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 4b851ce0fa534ca043aca835057b9d68660971a4)

test/librados_test_stub: pass snap context to zero op

Signed-off-by: Mykola Golub <mgolub@mirantis.com>
(cherry picked from commit 78ca0fde22076e8b156feb8dfd60aaee316a1895)

Merge pull request #20209 from dillaman/wip-22033-luminous

luminous: mgr/dashboard: added iSCSI IOPS/throughput metrics

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20210 from dillaman/wip-22169-luminous

luminous: librbd: set deleted parent pointer to null

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20208 from dillaman/wip-21920-luminous

luminous: librbd: default to sparse-reads for any IO operation over 64K

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20207 from dillaman/wip-21793-luminous

luminous: rbd-mirror: primary image should register in remote, non-primary image's journal

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20206 from dillaman/wip-21694-luminous

luminous: librbd: journal should ignore -EILSEQ errors from compare-and-write

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20205 from dillaman/wip-22577-luminous

luminous: qa/workunits/rbd: simplify split-brain test to avoid potential race

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #20153 from pdvian/wip-22809-luminous

luminous: librbd: fix snap create/rm may taking long time

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #20135 from dillaman/wip-22806-luminous

luminous: librbd: force removal of a snapshot cannot ignore dependent children

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #19123 from pdvian/wip-22174-luminous

luminous: possible deadlock in various maintenance operations

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #17875 from smithfarm/wip-21359-luminous

luminous: cephfs: racy is_mounted() checks in libcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

client: quit on failed remount during dentry invalidate test

Fixes: http://tracker.ceph.com/issues/22269
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5b2b1d14468c290c56ee6c95ea557c99464e0098)

Conflicts:
  PendingReleaseNotes: trivial conflict
src/ceph_fuse.cc
src/client/Client.cc: i chosed to pick code from 5b2b1d1 because, to fixthe issue, we need to call _do_remount which was introduced in 5b2b1d1.

Merge pull request #20121 from batrick/i21252p2

luminous: mds: fix return value of MDCache::dump_cache

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mon: s/mds_blacklist_interval/mon_mds_blacklist_interval/

We can rename LEVEL_DEV options at will.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit ed7fc5370c45deb4c80a36e9166e6dd84d4a2b99)

mon: tidy up mds_blacklist_interval

Remove the misleading reference to this from the client
eviction page, it was never the right option to mention
there (my mistake).

Demote the option from LEVEL_ADVANCED to LEVEL_DEV as it
is hard to imagine a good reason for the user to change it.

Set a hard minimum of one hour, to make it harder to
corrupt` a system by setting it close to zero.

Remove the legacy definition of the field while we're at it.

Fixes: http://tracker.ceph.com/issues/21821
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 0d68d3513e03cc2400facf421e08c8a92cb2886a)

Conflicts:
    doc/cephfs/eviction.rst (trivial resolution of conflict resulting from
        0e43f0f01cd800fee4cd800f1545405b449fa55b being merged to luminous)