os/bluestore: change bdev parse error to ENOENT

If there is not a valid label, then the label is not found.  This is a
more reasonable error code than "Invalid argumnet".

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 5cd98b0dfdc7758761ff789b12faca2266a7e128)

os/bluestore: make bdev label parse error less noisy

This happens during the normal initialization of a new bluestore osd and it
is confusing for users.  Make it less noisy.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 14c498dd22d1e89d30a0896e89fe83e8adf6ac76)

Merge pull request #20241 from ceph/backport-wip-rm22619

luminous ceph-volume: dmcrypt support for lvm

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #18930 into luminous

* refs/pull/18930/head:
mds: additional damage handling case in EImportStart

Merge pull request #20098 from ovh/bp-luminous-snaptrimq-visibility

luminous: osd, pg, mgr: make snap trim queue problems visible

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19267 from tchaikov/wip-ceph-disk-fsid-luminous

luminous: ceph_disk: allow "no fsid" on activate

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #19387 from ceph/wip-cd-luminous-upgrade

luminous: tests: ceph-deploy: use wip branch for final upgrade step

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19610 from ukernel/luminous-22306

luminous: osdc/Journaler: add 'stopping' check to various finish callbacks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19967 from smithfarm/wip-22707-luminous

luminous: ceph_objectstore_tool: no flush before collection_empty() calls; ObjectStore/StoreTest.SimpleAttrTest/2 fails

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19047 from shinobu-x/wip-22193-luminous

luminous: bluestore: OSD crash on boot with assert caused by Bluefs on flush write

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19388 from ceph/wip-cv-luminous-bp

luminous: qa: ceph-volume updates

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19487 from smithfarm/wip-22389-luminous

luminous: ceph-objectstore-tool: Add option dump-import to examine an export

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #19501 from shinobu-x/wip-22399-luminous

luminous: Manager daemon x is unresponsive. No standby daemons available

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19865 from smithfarm/wip-22634-luminous

luminous: build/ops: ceph-mgr dashboard has dependency on python-jinja2

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19966 from smithfarm/wip-22706-luminous

luminous: tests: force backfill test can conflict with pool removal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20116 from smithfarm/wip-22266-luminous

tools/ceph_monstore_tool: include mgrmap in initial paxos epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20215 from smithfarm/wip-22701-luminous

luminous: build/ops: ceph-volume fails when centos7 image doesn't have lvm2 installed

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20219 from tchaikov/wip-pr-20128-luminous

luminous: common/pick_address: wrong prefix_len in pick_iface()

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19849 from liewegas/wip-monlog-luminous

mon: allow cluster and debug logs to both go to stderr with different prefixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #19888 from pdvian/wip-22633-luminous

luminous: OSD crushes with FAILED assert(used_blocks.size() > count) during the first start after upgrade 12.2.1 -> 12.2.2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19928 from jdurgin/wip-luminous-omap-recovery

config: lower default omap entries recovered at once

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #19938 from liewegas/wip-list-size-luminous

luminous: common: compute SimpleLRU's size with contents.size() instead of lru.size()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19947 from tchaikov/wip-pr-18191-luminous

qa/standalone/osd/osd-mark-down: create pool to get updated osdmap faster

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19969 from smithfarm/wip-22691-luminous

luminous: ceph-base symbols not stripped in debs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19976 from pdvian/wip-22690-luminous

luminous: tests: increase osd count for ec testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19977 from pdvian/wip-22692-luminous

luminous: common: compute SimpleLRU's size with contents.size() instead of lru.…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19995 from pdvian/wip-22698-luminous

luminous: common/throttle: start using 64-bit values

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc/ceph-volume add lvm/encryption to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 163ee6a61235e97c341b2817114d98b576697096)

doc/man/ceph-volume add dmcrypt support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit babcdf2fa59baab6bc2cd8a1ac190b8d02fa3fce)

doc/man/ceph-volume add docs for zap sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ad73be2c2a6ee273d37b706626974d6900742256)

doc/man/ceph-volume add docs for list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 498d7a06a1f17c769c11da7ba9f83bcf2c0e433d)

doc/dev/ceph-volume update internal LVM metadata information

Includes dmcrypt information, and updated key names to reflect current
code

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 46babe80b638eb607e2af6f0c8bb77e675d4d63b)

ceph-volume tests.functional add xenial filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f0f0c1f8c4dc433d8978de134c120328842a5d7f)

ceph-volume tests.functional add xenial bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit be0dfe76535fa135168763275513d7fafba6fe4c)

ceph-volume tests.functional add dmcrypt support in tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d1ef5c487ba1ea868fd85b8c425986a2cf435519)

ceph-volume tests.functional add centos7 filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2477ac805e172a10ae3214151a78b4601cf88021)

ceph-volume tests.functional add centos7 bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 18972322567e158a56d0e71ca4f2afbed413c855)

ceph-volume tests add util.prepare unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a774eae43a089798f895507f18b0c5d68a49fbb4)

ceph-volume tests add fixture helpers for call and run

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d51973a64d9f18e0dfde7601b3ffbc57571ea653)

ceph-volume util allow prepare module to work with encryption keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a36dcdf9cfbc3ca012c5cb022e0d7dbcd95984f4)

ceph-volume lvm.activate allow encryption setups for luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3476e87cec353b100cf8c117d74708f51dd6de2c)

ceph-volume lvm.prepare allow encryption setups for luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit be3300455e479b03ac072d28bb05ccc7eaddb2e9)

ceph-volume process allow silencing output on file logging for sensitive data

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0e2190c7a22b61aeecdb60b6c8bfe5ba6aa873a4)

ceph-volume util create a helper module for encryption

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dc34a32675a870da98931adffacb2d3db4f74c6c)

ceph-volume lvm.common add dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 88ff873e9a60bd811356f7e15c8ef7f826d09337)

doc/ceph-volume create an encryption doc explaining dmcrypt workflows

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 477670a0912c9e01fe59155bf135eaf92bdc3318)

Merge pull request #20244 from ceph/backport-19276

luminous: ceph-volume: do not use --key during mkfs

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20247 from ceph/revert-19552-wip-22452-luminous

Revert " luminous: msg/async: unregister connection failed when racing happened"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert " luminous: msg/async: unregister connection failed when racing happened"

Merge pull request #19187 from liewegas/wip-trim-mgrmap-luminous

mon/MgrMonitor: limit mgrmap history

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19269 from tchaikov/wip-pr-19202-luminous

luminous: cmake,common/RWLock: check for libpthread extensions

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19499 from shinobu-x/wip-22402-luminous

luminous: osd: replica read can trigger cache promotion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #19552 from shinobu-x/wip-22452-luminous

 luminous: msg/async: unregister connection failed when racing happened

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19628 from shinobu-x/wip-22501-luminous

 luminous: tests: do not configure ec data pool with memstore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19630 from shinobu-x/wip-22507-luminous

 luminous: bluestore: do not crash on over-large objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19742 from tchaikov/wip-luminous-pr-19649-and-19650

luminous: common/dns, erasure-code: fix mem leaks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20049 from liewegas/wip-zero-length-luminous

os: fix 0-length zero semantics, test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20068 from smithfarm/wip-22761-luminous

luminous: osd: do not check out-of-date osdmap for DESTROYED flag on start

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20069 from smithfarm/wip-22770-luminous

luminous: tools: ceph-objectstore-tool set-size should clear data-digest

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20150 from pdvian/wip-22807-luminous

luminous: mon: do not use per_pool_sum_delta to show recovery summary

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: process: disable stdin param of run()

we cannot use process.communicate() to feed the Popen with input,
because, upon return of process.communicate() the stdout,stderr are
closed. see https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6df444cf33a251f5dcb536606b44af696cdb4ed5)

Conflicts:
src/ceph-volume/ceph_volume/process.py

ceph-volume: use --keyfile instead of --key

We do not want the key to show up on the command line (it may appear in
the process list or sudo log file).

Fixes: http://tracker.ceph.com/issues/22283
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 46abd50f8bcbd347d577581ae42156b718556158)

osd: accept 'keyfile' as well as 'key' during mkfs

Also, don't print the raw key to the log.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Kefu Chai <kefu@redhat.com>
(cherry picked from commit 18aff37ee3a2c2d3016f08c98161a5acc41ff8e5)

Merge pull request #19370 from shinobu-x/wip-22339-luminous

luminous: client: quit on failed remount during dentry invalidate test #19370

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19585 from ukernel/luminous-22460

luminous: mds: handle client session messages when mds is stopping

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19626 from shinobu-x/wip-22499-luminous

 luminous: cephfs-journal-tool: tool would miss to report some invalid range

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19627 from shinobu-x/wip-22500-luminous

 luminous: cephfs: potential adjust failure in lru_expire

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19776 from ukernel/luminous-22492

luminous: mds: respect mds_client_writeable_range_max_inc_objs config

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19829 from pdvian/wip-22573-luminous

luminous: vstart_runner: fixes for recent cephfs changes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19830 from shinobu-x/wip-22579-luminous

luminous: mds: check for CEPH_OSDMAP_FULL is now wrong; cluster full flag is obsolete

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19871 from smithfarm/wip-21948-luminous

luminous: mon: MDSMonitor: reject misconfigured mds_blacklist_interval

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19959 from smithfarm/wip-22694-luminous

luminous: mds: fix dump last_sent

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19962 from smithfarm/wip-22580-luminous

luminous: tests: full flag not set on osdmap for tasks.cephfs.test_full

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19968 from ukernel/luminous-22699

luminous: ceph-fuse: ::rmdir() uses a deleted memory structure of dentry leads …

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19982 from ukernel/luminous-22719

luminous: mds: handle 'inode gets queued for recovery multiple times'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20042 from tchaikov/wip-pr-19998-luminous

luminous: fix broken use of streamstream::rdbuf()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20106 from smithfarm/wip-22502-luminous

luminous: osd: Pool Compression type option doesn't apply to new OSDs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20240 from ceph/backport-20010

luminous: ceph-volume: adds a --destroy flag to ceph-volume lvm zap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: clean up docstrings for lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 57471fe7e74c94351e713ff99cbcbaa7dc6a0f52)

ceph-volume: print a message to terminal if --destroy is skipped

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit d2529ee5dd9a4dc6a81eeb2d5e54c0bdd6beeb2f)

ceph-volume: no need to set PVolume.vg_name explicitly

This will be set automatically in __init__ by looping
through kw.items and creating properties on PVolume
with them.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 129182645c4a7e5e42500d3ef2083cfdfb0137e4)

ceph-volume: switch remove_vg and remove_pv to process.run

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5a1be9a73919e13fe603861588bcc20872a4c133)

ceph-volume: allow passing a fail_msg param to process.run

This will allow the user to provide a more detailed error message
on why the command failed.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 7204851042fa42799e1003ef57d57a512d4a7f28)

Conflicts:
src/ceph-volume/ceph_volume/process.py

Merge pull request #20239 from ceph/backport-19949

luminous: ceph-volume: adds --crush-device-class flag for lvm prepare and create #19949

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: tests for lvm zap --destroy

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit fafa183d8f13fa769eaab5f865f6592b89225901)

ceph-volume: also remove pvs with lvm zap --destroy

Leaving the pv around doesn't keep us from redeploying on
that device because we also wipefs, but explicitly destroying
it makes the output read better.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit cb748ccb2264aea93dd4717a46d7f833c5b1eccd)

ceph-volume: adds an api.lvm.remove_pv method

Used to remove physical volumes

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 0811c02b118c270b2bec87cd26a877b26bc780b1)

ceph-volume: remove the -a flag from api.lvm.get_pvs

When we run `pvs -a` it returns all pvs, even ones that
have been recently deleted. Because of this pvs that have just
been zapped with `lvm zap` show up and can cause issues if a device
is zapped more than once.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 10a349742cae909d1ebf0a0b3b49c88774929f53)

ceph-volume: docs for the --destroy flag of lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 708cb43c503e962304211041570d6ca660bca1fb)

ceph-volume: adds the --destroy flag to ceph-volume lvm zap

If you use the --destroy flag and are zapping a raw device
or parition then zap will destroy any vgs or lvs it finds on that
device.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit aea326b772395c076e0d75f7c2f591a46aef7f57)

ceph-volume: adds an api.lvm.remove_vg method

This method can be used to remove volume groups

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 50a6a543a956d133b428cf36ad629c2131be5490)

ceph-volume: expose vg_name in api.lvm.get_api_pvs

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b7f24ec451e167d81a83c1e21597a9fece4f0684)

Merge pull request #20238 from ceph/backport-19875

luminous: ceph-volume: adds success messages for lvm prepare/activate/create

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: print the lvm activate success message correctly

Move the terminal.success to the methods that have access to the
correct osd_id so that they do not print None for the osd_id
in the log message.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit c9f7689a7a71a6b2321301b84599f319e8ae8eba)

ceph-volume: docs for --crush-device-class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 21f459437bd1208812504a20e5a50f7a47bfa979)

ceph-volume: adds crush_device_class to json metadata for 'osd new'

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 3404d8bba503f3f74f411b11108d39cb626d3bed)

ceph-volume: adds crush_device_class to the functional tests

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 36a388d72acb049b5ee25ceea6a8d341aa58e31f)

ceph-volume: add crush_device_class to lvm metadata

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 11da2185635ce14103471d851f1e8c84199d3745)

ceph-volume: adds a --crush-device-class flag to lvm create and prepare

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit bf468bc3737c15a8f15ec19ebf81c42672e28aaf)

ceph-volume: adds success message to ceph-volume simple activate

This used to only show the message if systemd was not being used, now
it will show this always if the command is successful.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 87c6ed14e7e0b528a1fe107e2e531908cdea0f47)