tools/ceph_monstore_tool: rebuild initial mgrmap also

Fixes: http://tracker.ceph.com/issues/22266
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit f63d1da4afa3463730ecbc0be29df6375b79fa8f)

Merge pull request #19216 from liewegas/wip-p2p-app-warning-luminous

qa/suites/upgrade/jewel-x/point-to-point: fix suppression of pool app warning

qa/suites/upgrade/jewel-x/point-to-point: fix suppression of pool app warning

It's generated on the mgr.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #19205 from tchaikov/wip-22136-luminous

luminous: qa/ceph-disk: enlarge the simulated SCSI disk

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #19152 from smithfarm/wip-22235-luminous

luminous: ceph-disk flake8 test fails on very old, and very new, versions of flake8

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/ceph-disk: enlarge the simulated SCSI disk

100MB will be allocated for journal, and the remaining 100MB is for data
device. taking the inode into consideration, there will be approximately
87988 kB available for the activated OSD. and it will complain with a
"nearfull" state.

Fixes: http://tracker.ceph.com/issues/22136
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit b3c159e9fb8c31d0acd75f0702080f18959f672e)

Merge pull request #19173 from smithfarm/wip-22251-luminous

luminous: build/ops: macros expanding in spec file comment

Reviewed-by: David Disseldorp <ddiss@suse.com>
Reviewed-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #19179 from ceph/wip-yuriw-BP-19175-luminous

qa/suites/rados/rest: move rest_test from qa/suites/rest/

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/suites/rados/rest: move rest_test from qa/suites/rest/

...and add more health whitelists.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit ddb274669ca16b95c452b256e58a2830af10c73d)
Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #19177 from liewegas/wip-jewel-x-p2p

qa/suites/upgrade/jewel-x/point-to-point: skip ec tests when mons may be old

Reviewed-by: Yuri Weinstein <yweins@redhat.com>

qa/suites/upgrade/jewel-x/point-to-point: skip ec tests when mons may be old

Early point release mons don't handle legacy ruleset-* ec profiles, new
ones do.  Skip the ec tests that may trigger this when we are doing a
workload that races with mon upgrades.

Signed-off-by: Sage Weil <sage@redhat.com>

rpm: rm macros in comments

rpm expands all macros in a .spec file, even those in comments. Drop the
percent signs so rpm will not expand these.

This change silences rpmlint's warning about macros in comments.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>
(cherry picked from commit 5c1e9f55d3a144c93c9c18b031db3cdc320561ef)

tests: ceph-disk: ignore E722 in flake8 test

Very old, and very new, versions of flake8 treat E722 as an error:

flake8 runtests: commands[0] | flake8 --ignore=H105,H405,E127 ceph_disk tests
ceph_disk/main.py:1575:9: E722 do not use bare except'
ceph_disk/main.py:1582:9: E722 do not use bare except'
ceph_disk/main.py:3252:5: E722 do not use bare except'
ceph_disk/main.py:3288:21: E722 do not use bare except'
ceph_disk/main.py:3296:17: E722 do not use bare except'
ceph_disk/main.py:4358:5: E722 do not use bare except'
tests/test_main.py:26:1: E722 do not use bare except'
ERROR: InvocationError: '/opt/j/ws/mkck/src/ceph-disk/.tox/flake8/bin/flake8 --ignore=H105,H405,E127 ceph_disk tests'

Fixes: http://tracker.ceph.com/issues/22207
References: https://gitlab.com/pycqa/flake8/issues/361

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit 3600cd7bfdca76485d2998c5da1a0dd25816d1ce)

Merge pull request #19080 from smithfarm/wip-22212-luminous

luminous: tests: ceph-disk: silence deprecation warnings

Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/workunits: silence py warnings for ceph-disk tests

ceph-disk now prints "depreacted" warning message when it starts. but
the tests parses its stdout and stderr for a json string. so we need to
silence the warnings for the tests.

Fixes: http://tracker.ceph.com/issues/22154
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit d44334f31704487ec3574738e75145872d9932cf)

Merge pull request #18989 from ceph/wip-luminous-rm22154

luminous: ceph-disk create deprecation warnings

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #19025 from tchaikov/wip-pr-19024-luminous

luminous: qa/tasks: prolong revive_osd() timeout to 6 min

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18945 from liewegas/wip-22128

mon/OSDMonitor: add option to fix up ruleset-* to crush-* for ec profiles

Reviewed-by: Joao Eduardo Luis <joao@suse.de>

qa/tasks: prolong revive_osd() timeout to 6 min

see also #17902

Fixes: http://tracker.ceph.com/issues/21474
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 749bbda07522114b99b17a4a01cfcf019520285d)

Merge pull request #18983 from pdvian/wip-22150-luminous

luminous: os/bluestore: fix SharedBlob unregistration

mon/OSDMonitor: add option to fix up ruleset-* to crush-* for ec profiles

The jewel->luminous upgrade test will fail if we finish the upgrade while
a workload setting old-style ec profiles is running.  Add option to
automatically fix them up.  Warn to the cluster log when this happens.

For now, enable this option to ease upgrades and whitelist the warning.

Only include this option in luminous so that we implicitly sunset this
compatibility kludge immediately.

Fixes: http://tracker.ceph.com/issues/22128
Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #18751 from shinobu-x/wip-21701-luminous

luminous: ceph-kvstore-tool does not call bluestore's umount when exit

Merge pull request #18750 from shinobu-x/wip-21702-luminous

luminous: BlueStore::umount will crash when the BlueStore is opened by start_kv_only()

Merge pull request #18860 from liewegas/wip-filestore-rocksdb-compression-luminous

luminous: os/filestore: disable rocksdb compression

Reviewed-by: Douglas Fuller <dfuller@redhat.com>

doc/ceph-volume create a migration section to help users coming from ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e433efe51481ec63bc9753a0f446fe2b12b5bb6a)

ceph-disk add deprecation warnings in favor of ceph-volume

Signed-off-by: Alfredo Deza <adeza@redhat.com>
Fixes: http://tracker.ceph.com/issues/22154
(cherry picked from commit c9da92e1ec8d60dc40d86630dab95e4fec4d0d30)

Merge pull request #18973 from ceph/backport-18924

luminous: ceph-volume: allow using a device or partition for `lvm --data`

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.devices.lvm prepare isn't bluestore specific anymore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2f40dbac386be631ac97e2450117c218d9bf02f9)

os/bluestore: fix SharedBlob unregistration

We use the SharedBlobSet remove() in three cases:

- from SharedBlob::put(), we try to remove ourselves from the set, but
  have to deal with a racing lookup, so the removal is conditional on
  nref still being 0.
- from split_cache(), we move the SharedBlob to another collection
- from make_blob_unshared(), we remove the entry when we clear the sbid.

The problem is that the condtiional remove() (for the first case) was being
used for all three cases, and in the second two cases nref is always != 0,
so it doesn't actually happen.  This can lead to a crash during cache
shutdown.

Fix by making two variants: remove() that is unconditional, and
try_remove() that is conditional.

Set the sb->coll pointer after because remove() asserts the parent matches
where we are unregistering.

Fixes: http://tracker.ceph.com/issues/22039
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 78a465b90b1c5187cfec9f25eb58b3bf617ca39b)

Merge pull request #18957 from tchaikov/wip-pr-18808-luminous

luminous: qa: do not wait for down/out osd for pg convergence

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume tests.functional create a common playbook directory for setups

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit def89a411f2f4bdb23c4bbd3b71c1310a564f07d)

doc/ceph-volume lvm prepare can accept devices for --data

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9bfab348b2ff59c0b939a19c22d4a06bc5236b96)

Conflicts:
doc/ceph-volume/lvm/prepare.rst

ceph-volume tests.functional add setup playbook for xenial filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 210c1e1d4770dde47f0008cadaaec35deb729272)

ceph-volume tests.functional add data device and journal to xenial filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d0754e25177b9bf3bce695b7991e99f49312d91c)

ceph-volume tests.functional add setup playbook for xenial bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d0f5623963f58fc4d5727a79d8377b2191b0f9dc)

ceph-volume tests.functional add raw data device to xenial bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fbcfb522981cd4039cfe617609c5f160f0b05c52)

ceph-volume tests.functional tox.ini addition of setup playbook for partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d074b534643dde37d732b34c36c5089e41848d6d)

ceph-volume tests.functional add raw data device to centos7 bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2d416a849ad7b33dde1386da09c81249e6c2299d)

ceph-volume tests.functional add setup playbook for centos7 bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 161fd373d2f11f64466927e5d615d039822e2ab5)

ceph-volume tests.functional add setup playbook for centos7 filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b3bb147298f8816651f231f77bc91a3a1b50d96f)

ceph-volume tests.functional add raw device to centos7 filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c5b8d72570b24dfd0d2f8c1cc8f13812c6c11a69)

ceph-volume lvm.common update --data flag to reflect device acceptance

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4df7ea8d83cfa02c8864d043b5d5fb2f5409dc96)

ceph-volume lvm.prepare add example of raw device or partition in help menu

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6fb8d566aac0e0fa081e6500852855ad177f784f)

ceph-volume lvm.prepare allow partition or raw device as input for --data in filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 89236ee3f81e50b9b059c10f6d6e7ce048c06474)

qa: do not wait for down/out osd for pg convergence

that osd is not invovlved in the PG state changes.

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 7f549af459b40c5bb51a24a3a443ed7a7116915e)

Merge pull request #18908 from smithfarm/wip-luminous-fixup

qa/suites/fs/basic_functional/clusters: more osds

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

Merge pull request #18910 from ceph/backport-18882

luminous: ceph-volume: add functional tests for simple, rearrange lvm tests

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume tests.functional update lvm/tox.ini paths after moving it

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7d787849556788961155534039886aedfcdb2a88)

ceph-volume tests.functional move top level tox.ini to lvm/tox.ini

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9b14b96d3bf6c8326cfc91e657161ffab3b8c2b4)

ceph-volume tests.functional move xenial to lvm/xenial

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 00d576d266b63080129e58482bd6fba62534ee97)

ceph-volume tests.functional lvm move xenial/bluestore to xenial/bluestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d5ec7d1c58141f4593ed9d1e4501c9f3be6d9755)

ceph-volume tests.functional lvm move xenial/create to xenial/filestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c504b8781b3a31475280a032cd75c9e1eaa6d8c9)

ceph-volume tests.functional move centos7 to lvm/centos7

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dc1da3fd0aeafe59a82343fb16fe1fcea66ae3eb)

ceph-volume tests.functional lvm move bluestore to bluestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit aec1a80a3e1e61720f494980711808bfee783a03)

ceph-volume tests.functional lvm move create to filestore/create

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d6905f3f620d4c733835e21a275d4127cdcb769b)

ceph-volume tests.functional simple xenial bluestore scenario

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8f0fdc975951a2a8af31bbaa3abd2b5d807a0add)

ceph-volume tests.functional simple xenial filestore scenario

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7452e1b6f9b82c2ef62bae19c632e81039638ba5)

ceph-volume tests.functional simple tox.ini update for xenial scnearios

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 76ed6a8f2ecab7c58def0453ffed9d8a997e0d5b)

ceph-volume tests.functional simple tox.ini addition for bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 02154e445b87a8fdcabbd699a2532833764140b5)

ceph-volume tests.functional add initial files for simple bluestore activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3320f28e799e1d77d1ea69be9ced603bb69cc5fa)

ceph-volume tests.functional add initial files for simple filestore activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9d18bae52ad77be0e64ef105c33a019446b6f1cf)

ceph-volume tests.functional create a specific test.yml for simple activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 37bde0d3b37a87c456438b0edc0cb9e2dcd39120)

ceph-volume tests.functional create a separate tox.ini environ for
`simple`

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 89757ad12e6938fe2bf997e0f21a5ffac082e30b)

Merge pull request #18907 from ceph/luminous-wip-bz1498200

luminous: ceph-volume: support GPT and other deployed OSDs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

qa/suites/fs/basic_functional/clusters: more osds

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 998122c324047c96c16d60d7fbb3a21f4dd63c56)

doc/ceph-volume include the top level systemd.rst in the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e2f281c115c15e820588b478214c3600ff97470f)

ceph-volume simple.activate add --file in help docstring as an example

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ff60592897b63581ebce372b3f6ffa0f10d2fe5a)

doc/ceph-volume activate has a --file option for json files

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 29913951b71d68b3f3f9b9fa076f309c0252da36)

ceph-volume simple.activate enable the osd when activating

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 366fb8c43af54e9c92f4e95d9a0582c1fc378dbc)

ceph-volume systemd.systemctl always force symlink when masking ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cb470622fd4f13fafe9cb2f38fb25fd1fbd8d89c)

ceph-volume simple.activate use the contents of the JSON file, fallback to CLI args

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c954c913395557bcd89f25b0496c4428e6109494)

ceph-volume simple.activate only check for missing id and uuid if file is not passed in

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 826c2709dd5ab0fbf6eb419a735d15e718a7712c)

ceph-volume simple.activate allow to pass a file to activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bd3498f3eabf07637cb936e6c036d1d8d75415b0)

ceph-volume tests.util add tests for OSDPath validator

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d7fc26ab4e2cebe485647510119ea300d47db184)

ceph-volume systemd.systemctl masking ceph-disk needs to be done by directly symlinking to dev/null

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cda47c247b929401a7dce3f1ef70371ea80eeceb)

ceph-volume systemd.systemctl masking needs to use systemctl mask, not disable

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bad2f62b34243addc854abf691854b886aecdc19)

ceph-volume simple.scan if mounted device cannot be detected, raise an error

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6b2fc4c0ba63e4bd51774ea09cc6542738128621)

ceph-volume simple.activate remove str formatting from logging

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 230faf04a185c0b95c8ad497ddd5501dc70c1409)

ceph-volume util.arg_validators always convert the path to an absolute path

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b5acf4d720b5395ae4ccbf007c94c99610e40089)

ceph-volume util.arg_validators require superuser privs to check paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 376dad74159532f901d0044f95330fc02f93bd25)

ceph-volume simple.scan skip directories to scan

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6bc1e9ee0b8d7bc475f218ae3dd8c6c7fe5a983a)

ceph-volume tests.devices.simple add activate CLI unit tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cde0d517563b537319d06c758b93e421cdabe59d)

ceph-volume tests.devices add help menu unit test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dcc7c72196a327f9501045f025761b228b5379df)

doc/ceph-volume systemd describe the interaction with 'simple' activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7071eeaeb5d77c6ab6e7a24f1b630cb0cf8a8e7c)

doc/ceph-volume index initial description of simple

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f9ee51f9feadb6d0f91b76a2b2ba8a24f1523a77)

doc/ceph-volume lvm add back a system.rst with just the lvm portion of it

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f872cd29b93e02b77172f7b382c4746af9ba845d)

doc/ceph-volume make systemd.rst generic enough to be related to other sub-commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cf4e32a0dc3ab4f55fe7857b51afbc63a638ec66)

doc/ceph-volume move lvm/systemd.rst to top level, making it generic

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0672dd608d4601a936d703ea5ba6edc14a161d8e)

doc/ceph-volume simple.activate initial take on documenting activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 80c796bd1c43125f5678710b494b03c486b3fce6)

doc/ceph-volume lvm update systemd anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 53420a4756a5e4fa6b0f2aa74d6ad1e2919f1954)

doc/ceph-volume lvm.activate update systemd anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7fc0e7ebcae06f28f6ed08c0db14075ada34e8a7)

doc/ceph-volume lvm.systemd update anchor to reflect lvm parent

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 01a8b7e6a45f6fd85bf53d3cab15acc81d0bae1b)

doc/ceph-volume document the scan functionality

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b30a3eaf17a20a1273724b2cfcaf0a7b36eb08f6)

doc/ceph-volume update the index to include the simple sub command and files

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d7f85c64e4b409ea95e30ccb5a1388278843c478)

ceph-volume tests.simple add tests for trigger parsing from systemd

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0e2620234e82abf85274b3c08b39cdf76430ce48)

ceph-volume tests.simple create etc_path scan tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 90ebc4e1ffa7b3273787d708bb2c05180530b5d1)

ceph-volume tests.simple add checks for get_contents in scan

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 647bfe0883d2bb05b4ff7f99894075df3e73d2d5)

ceph-volume tests.util add binary verification tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 827aa57557561355dd4068e26801b5f1ed62f295)

ceph-volume tests create a fixture for temporary files

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 23da3445718c01f71e2a5748fb4fc1c5fdc85ca5)

ceph-volume systemd.systemctl helpers to mask ceph-disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3da23fa4caea8c9dc91da3eee5b5acc1860bf242)