ceph-volume: when reusing an osd ID you must give that id to 'osd new'

If you do not then auth will not be created for the new OSD and the
daemon will not be able to start.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 43f699fcbc9f30875f70295e086a4e423008b266)

ceph-volume: prepare_* methods do not need to recreate fsid or osd_id

The prepare_filestore and prepare_bluestore methods do not need to
attempt to recreate osd_id or fsid because this happens in prepare
already.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 7c809166aed82298b1fe1581f0e28174a535574f)

ceph_volume: modify util.prepare.check_id to handle stdout as a list

process.call returns stdout as a list, not a string

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a08fc639613b3d9bcb3a09875e535287dd58870a)

ceph-volume: tests for util.prepare.check_id

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 0d27fff319f1c4705648fdaff18ee47441f26a24)

ceph-volume: return osd_id from util.prepare.check_id if it exists

This also changes this so the osd_id is returned as a string so
an ID of 0 would evaluate to True.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 699aa65878ca18f2e2cd70d7444cad18b4a3fd98)

ceph-volume: check to see if an OSD id exists before reusing it

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 548eadeec6fb42a18c3e84dbbd8164862f8836e6)

ceph-volume: adds a prepare util for checking OSD ID existance

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 4ac99de6d98b646bf160230584f6532775cdc9cb)

Merge pull request #20241 from ceph/backport-wip-rm22619

luminous ceph-volume: dmcrypt support for lvm

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge PR #18930 into luminous

* refs/pull/18930/head:
mds: additional damage handling case in EImportStart

Merge pull request #20098 from ovh/bp-luminous-snaptrimq-visibility

luminous: osd, pg, mgr: make snap trim queue problems visible

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19267 from tchaikov/wip-ceph-disk-fsid-luminous

luminous: ceph_disk: allow "no fsid" on activate

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #19387 from ceph/wip-cd-luminous-upgrade

luminous: tests: ceph-deploy: use wip branch for final upgrade step

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19610 from ukernel/luminous-22306

luminous: osdc/Journaler: add 'stopping' check to various finish callbacks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19967 from smithfarm/wip-22707-luminous

luminous: ceph_objectstore_tool: no flush before collection_empty() calls; ObjectStore/StoreTest.SimpleAttrTest/2 fails

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19047 from shinobu-x/wip-22193-luminous

luminous: bluestore: OSD crash on boot with assert caused by Bluefs on flush write

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19388 from ceph/wip-cv-luminous-bp

luminous: qa: ceph-volume updates

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19487 from smithfarm/wip-22389-luminous

luminous: ceph-objectstore-tool: Add option dump-import to examine an export

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #19501 from shinobu-x/wip-22399-luminous

luminous: Manager daemon x is unresponsive. No standby daemons available

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19865 from smithfarm/wip-22634-luminous

luminous: build/ops: ceph-mgr dashboard has dependency on python-jinja2

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #19966 from smithfarm/wip-22706-luminous

luminous: tests: force backfill test can conflict with pool removal

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20116 from smithfarm/wip-22266-luminous

tools/ceph_monstore_tool: include mgrmap in initial paxos epoch

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20215 from smithfarm/wip-22701-luminous

luminous: build/ops: ceph-volume fails when centos7 image doesn't have lvm2 installed

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #20219 from tchaikov/wip-pr-20128-luminous

luminous: common/pick_address: wrong prefix_len in pick_iface()

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #19849 from liewegas/wip-monlog-luminous

mon: allow cluster and debug logs to both go to stderr with different prefixes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Joao Eduardo Luis <joao@suse.de>

Merge pull request #19888 from pdvian/wip-22633-luminous

luminous: OSD crushes with FAILED assert(used_blocks.size() > count) during the first start after upgrade 12.2.1 -> 12.2.2

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19928 from jdurgin/wip-luminous-omap-recovery

config: lower default omap entries recovered at once

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

Merge pull request #19938 from liewegas/wip-list-size-luminous

luminous: common: compute SimpleLRU's size with contents.size() instead of lru.size()

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19947 from tchaikov/wip-pr-18191-luminous

qa/standalone/osd/osd-mark-down: create pool to get updated osdmap faster

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19969 from smithfarm/wip-22691-luminous

luminous: ceph-base symbols not stripped in debs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19976 from pdvian/wip-22690-luminous

luminous: tests: increase osd count for ec testing

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19977 from pdvian/wip-22692-luminous

luminous: common: compute SimpleLRU's size with contents.size() instead of lru.…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19995 from pdvian/wip-22698-luminous

luminous: common/throttle: start using 64-bit values

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc/ceph-volume add lvm/encryption to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 163ee6a61235e97c341b2817114d98b576697096)

doc/man/ceph-volume add dmcrypt support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit babcdf2fa59baab6bc2cd8a1ac190b8d02fa3fce)

doc/man/ceph-volume add docs for zap sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ad73be2c2a6ee273d37b706626974d6900742256)

doc/man/ceph-volume add docs for list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 498d7a06a1f17c769c11da7ba9f83bcf2c0e433d)

doc/dev/ceph-volume update internal LVM metadata information

Includes dmcrypt information, and updated key names to reflect current
code

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 46babe80b638eb607e2af6f0c8bb77e675d4d63b)

ceph-volume tests.functional add xenial filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f0f0c1f8c4dc433d8978de134c120328842a5d7f)

ceph-volume tests.functional add xenial bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit be0dfe76535fa135168763275513d7fafba6fe4c)

ceph-volume tests.functional add dmcrypt support in tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d1ef5c487ba1ea868fd85b8c425986a2cf435519)

ceph-volume tests.functional add centos7 filestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2477ac805e172a10ae3214151a78b4601cf88021)

ceph-volume tests.functional add centos7 bluestore tests for dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 18972322567e158a56d0e71ca4f2afbed413c855)

ceph-volume tests add util.prepare unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a774eae43a089798f895507f18b0c5d68a49fbb4)

ceph-volume tests add fixture helpers for call and run

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d51973a64d9f18e0dfde7601b3ffbc57571ea653)

ceph-volume util allow prepare module to work with encryption keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a36dcdf9cfbc3ca012c5cb022e0d7dbcd95984f4)

ceph-volume lvm.activate allow encryption setups for luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3476e87cec353b100cf8c117d74708f51dd6de2c)

ceph-volume lvm.prepare allow encryption setups for luks

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit be3300455e479b03ac072d28bb05ccc7eaddb2e9)

ceph-volume process allow silencing output on file logging for sensitive data

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0e2190c7a22b61aeecdb60b6c8bfe5ba6aa873a4)

ceph-volume util create a helper module for encryption

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dc34a32675a870da98931adffacb2d3db4f74c6c)

ceph-volume lvm.common add dmcrypt flag

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 88ff873e9a60bd811356f7e15c8ef7f826d09337)

doc/ceph-volume create an encryption doc explaining dmcrypt workflows

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 477670a0912c9e01fe59155bf135eaf92bdc3318)

Merge pull request #20244 from ceph/backport-19276

luminous: ceph-volume: do not use --key during mkfs

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20247 from ceph/revert-19552-wip-22452-luminous

Revert " luminous: msg/async: unregister connection failed when racing happened"

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Revert " luminous: msg/async: unregister connection failed when racing happened"

Merge pull request #19187 from liewegas/wip-trim-mgrmap-luminous

mon/MgrMonitor: limit mgrmap history

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #19269 from tchaikov/wip-pr-19202-luminous

luminous: cmake,common/RWLock: check for libpthread extensions

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #19499 from shinobu-x/wip-22402-luminous

luminous: osd: replica read can trigger cache promotion

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Amit Kumar <amitkuma@redhat.com>

Merge pull request #19552 from shinobu-x/wip-22452-luminous

 luminous: msg/async: unregister connection failed when racing happened

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19628 from shinobu-x/wip-22501-luminous

 luminous: tests: do not configure ec data pool with memstore

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19630 from shinobu-x/wip-22507-luminous

 luminous: bluestore: do not crash on over-large objects

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #19742 from tchaikov/wip-luminous-pr-19649-and-19650

luminous: common/dns, erasure-code: fix mem leaks

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20049 from liewegas/wip-zero-length-luminous

os: fix 0-length zero semantics, test

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20068 from smithfarm/wip-22761-luminous

luminous: osd: do not check out-of-date osdmap for DESTROYED flag on start

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20069 from smithfarm/wip-22770-luminous

luminous: tools: ceph-objectstore-tool set-size should clear data-digest

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #20150 from pdvian/wip-22807-luminous

luminous: mon: do not use per_pool_sum_delta to show recovery summary

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume: process: disable stdin param of run()

we cannot use process.communicate() to feed the Popen with input,
because, upon return of process.communicate() the stdout,stderr are
closed. see https://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate .

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 6df444cf33a251f5dcb536606b44af696cdb4ed5)

Conflicts:
src/ceph-volume/ceph_volume/process.py

ceph-volume: use --keyfile instead of --key

We do not want the key to show up on the command line (it may appear in
the process list or sudo log file).

Fixes: http://tracker.ceph.com/issues/22283
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 46abd50f8bcbd347d577581ae42156b718556158)

osd: accept 'keyfile' as well as 'key' during mkfs

Also, don't print the raw key to the log.

Signed-off-by: Sage Weil <sage@redhat.com>
Signed-off-by: Kefu Chai <kefu@redhat.com>
(cherry picked from commit 18aff37ee3a2c2d3016f08c98161a5acc41ff8e5)

Merge pull request #19370 from shinobu-x/wip-22339-luminous

luminous: client: quit on failed remount during dentry invalidate test #19370

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19585 from ukernel/luminous-22460

luminous: mds: handle client session messages when mds is stopping

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19626 from shinobu-x/wip-22499-luminous

 luminous: cephfs-journal-tool: tool would miss to report some invalid range

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19627 from shinobu-x/wip-22500-luminous

 luminous: cephfs: potential adjust failure in lru_expire

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19776 from ukernel/luminous-22492

luminous: mds: respect mds_client_writeable_range_max_inc_objs config

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19829 from pdvian/wip-22573-luminous

luminous: vstart_runner: fixes for recent cephfs changes

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19830 from shinobu-x/wip-22579-luminous

luminous: mds: check for CEPH_OSDMAP_FULL is now wrong; cluster full flag is obsolete

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19871 from smithfarm/wip-21948-luminous

luminous: mon: MDSMonitor: reject misconfigured mds_blacklist_interval

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19959 from smithfarm/wip-22694-luminous

luminous: mds: fix dump last_sent

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19962 from smithfarm/wip-22580-luminous

luminous: tests: full flag not set on osdmap for tasks.cephfs.test_full

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19968 from ukernel/luminous-22699

luminous: ceph-fuse: ::rmdir() uses a deleted memory structure of dentry leads …

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #19982 from ukernel/luminous-22719

luminous: mds: handle 'inode gets queued for recovery multiple times'

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #20042 from tchaikov/wip-pr-19998-luminous

luminous: fix broken use of streamstream::rdbuf()

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #20106 from smithfarm/wip-22502-luminous

luminous: osd: Pool Compression type option doesn't apply to new OSDs

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #20240 from ceph/backport-20010

luminous: ceph-volume: adds a --destroy flag to ceph-volume lvm zap

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: clean up docstrings for lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 57471fe7e74c94351e713ff99cbcbaa7dc6a0f52)

ceph-volume: print a message to terminal if --destroy is skipped

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit d2529ee5dd9a4dc6a81eeb2d5e54c0bdd6beeb2f)

ceph-volume: no need to set PVolume.vg_name explicitly

This will be set automatically in __init__ by looping
through kw.items and creating properties on PVolume
with them.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 129182645c4a7e5e42500d3ef2083cfdfb0137e4)

ceph-volume: switch remove_vg and remove_pv to process.run

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 5a1be9a73919e13fe603861588bcc20872a4c133)

ceph-volume: allow passing a fail_msg param to process.run

This will allow the user to provide a more detailed error message
on why the command failed.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 7204851042fa42799e1003ef57d57a512d4a7f28)

Conflicts:
src/ceph-volume/ceph_volume/process.py

Merge pull request #20239 from ceph/backport-19949

luminous: ceph-volume: adds --crush-device-class flag for lvm prepare and create #19949

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: tests for lvm zap --destroy

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit fafa183d8f13fa769eaab5f865f6592b89225901)

ceph-volume: also remove pvs with lvm zap --destroy

Leaving the pv around doesn't keep us from redeploying on
that device because we also wipefs, but explicitly destroying
it makes the output read better.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit cb748ccb2264aea93dd4717a46d7f833c5b1eccd)

ceph-volume: adds an api.lvm.remove_pv method

Used to remove physical volumes

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 0811c02b118c270b2bec87cd26a877b26bc780b1)

ceph-volume: remove the -a flag from api.lvm.get_pvs

When we run `pvs -a` it returns all pvs, even ones that
have been recently deleted. Because of this pvs that have just
been zapped with `lvm zap` show up and can cause issues if a device
is zapped more than once.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 10a349742cae909d1ebf0a0b3b49c88774929f53)

ceph-volume: docs for the --destroy flag of lvm zap

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 708cb43c503e962304211041570d6ca660bca1fb)

ceph-volume: adds the --destroy flag to ceph-volume lvm zap

If you use the --destroy flag and are zapping a raw device
or parition then zap will destroy any vgs or lvs it finds on that
device.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit aea326b772395c076e0d75f7c2f591a46aef7f57)

ceph-volume: adds an api.lvm.remove_vg method

This method can be used to remove volume groups

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 50a6a543a956d133b428cf36ad629c2131be5490)

ceph-volume: expose vg_name in api.lvm.get_api_pvs

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b7f24ec451e167d81a83c1e21597a9fece4f0684)

Merge pull request #20238 from ceph/backport-19875

luminous: ceph-volume: adds success messages for lvm prepare/activate/create

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: print the lvm activate success message correctly

Move the terminal.success to the methods that have access to the
correct osd_id so that they do not print None for the osd_id
in the log message.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit c9f7689a7a71a6b2321301b84599f319e8ae8eba)

ceph-volume: docs for --crush-device-class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 21f459437bd1208812504a20e5a50f7a47bfa979)