ceph-volume lvm.prepare use wal and db for mkfs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4970ab5c804111ce4b2b57817f1e29d58b18a893)

ceph-volume lvm.prepare ensure tags are set for ceph.type correctly for block and data

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 58dbefd3be7bad51e6c56c50fbfb450e205b3036)

ceph-volume lvm.activate check if lv device is mounted at path

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1ead135bef801280e7cec21283c95c7ba440d84b)

ceph-volume lvm.prepare default to using bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 39dd0d1d10aba08c3e8299eb1702260274a756a7)

ceph-volume lvm.create default to using bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d3145da6e643dedb2fbed72e7d4d0cddf397c5e3)

ceph-volume lvm.common remove boolean defaults for CLI flags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a7d5965ab532a621f402bad55d45dfed61d90cc5)

ceph-volume lvm.prepare update to use create_osd_path

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bafb72144666f891773bed95e9a0f757673af2e0)

ceph-volume lvm.activate only prime-osd-dir when directory is empty

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9bb6cfd867c2d8e972093d023622f8c2e5b440d4)

ceph-volume lvm.activate safeguard against auto detect objectstore flag not being set

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c9b5f352a8f9e2743e5dca3a0479c8099a59fd12)

ceph-volume util.system use strings for comparison in py3

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1b671a8602b036cb8d53bc40eb70198c13143051)

f ceph-volume tests update changes from the help menu in lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3ddcd3c2f19cc848a11a2882f150462ad2f26c65)

ceph-volume lvm.common --journal-size doesn't accept a volume group

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a8282e9a580bcfa74abdd33032e2ce2851b5cf29)

ceph-volume lvm.prepare filestore does not require a volume group

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3653f105927d455b8df143797af0cfe72f8d6834)

ceph-volume lvm.common volume groups are no longer required

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 526c0aa81ac8eed95c2b4664a00ded2118372fff)

doc/ceph-volume activate workflow mention tmpfs in osd dirs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8788fe3721034f390240c01493bc1d27d65a73a5)

doc/ceph-volume update activation to include bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 288a3124793d11d71b540555fd03c2f61a79dd16)

doc/ceph-volume update create to indicate bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 61ce8b4b844fcb64fc95373c5515de902a52fc8c)

doc/ceph-volume update prepare with bluestore workflow

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9f1a5627399e2589fe706b158a32a9fb8642ac23)

doc/ceph-volume update filestore to indicate it is not the only/default objectstore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c93603e5764e021305cfeb1c297ee16318126fe1)

doc/ceph-volume elaborate on env vars for interval and tries in activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1dc24d1a7065b4e23b1bfa0e7c6d263b5ebc6906)

ceph-volume lvm.common update cli flags help for bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c9693fb3f8386016aba5d7c8f53f5f71f948c88d)

ceph-volume util.disk st_mode is needed for stat()

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e3b5663f7968e44ec0cd6db2bfc43acdb233e314)

ceph-volume decorators always log to the file with exceptions

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit cb3533a72571251d9f1b6171f783246462726857)

ceph-volume tests verify behavior of get_lv_from_argument

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit adabbd462bcb19a6da65211e24dfb2b9b5043ef4)

ceph-volume api.lvm return result of get_lv when using arguments

If the argument is an absolute path it is fine to just return whatever
get_lv finds since it is a "safe" call, it will return a None if nothing
is found

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f209c9d4d50bd7abe0f7f6b9fb29359cb7196718)

ceph-volume lvm.listing support bluestore, not only filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6a5b89a475c2a36022ca09dbb7f6b94cc3b44881)

ceph-volume api.lvm allow full paths to detect lv from arguments

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8d257101b1556ec5ce805622e9b7b6b74f6b000d)

ceph-volume tests add support for bluestore list verification

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 4674da9c07f871bf1f25bb782c136fe082095a14)

ceph-volume tests activate with fsid works for bluestore and filestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5b9f0853f780db3b0af6b4b7c1ad1d59184069c9)

ceph-volume tests arg validator can accept absolute paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 15586715f506505a57ec5122416541479df74bc8)

ceph-volume tests update changes from the help menu in lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 39bdc2e3f621a32bcbb3dfe64f156bbfd7e4f5ee)

ceph-volume tests update to get_lv from get_journal_lv

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9be1012072f36e21a3f3cef42a42f7f1eb83c64e)

ceph-volume tests.util new path/device mount interface

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d5984bad97347e58318793fc05da84832d323113)

ceph-volume tests update api.lvm tests

The create_lv signature changed to require full size description and
tags need to be an actual dictionary (vs. keyword args)

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e8a18637531cde59d73a97d106958489c1d12117)

ceph-volume lvm.activate ceph-bluestore-tool should populate the osd dir

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f3bb616bcf8a4737a3701ebf812de831785d6341)

ceph-volume lvm.activate auto detect objectstore type

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 472aa0b1997a0b1301b699b7ce30dce728aaea0c)

ceph-volume lvm.trigger enable objectstore auto detection for activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f579ef07026564f1a94b2923a2e4640ade2da503)

ceph-volume lvm.activate add bluestore support for activation

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5696fed09b65044142b34f89492a6d91d3243e14)

ceph-volume util.system create mappings of mounts

This makes it easier for consumers to detect a path, or a device that
might be mounted or might have more than one mount, including tmpfs and
devtmpfs support, which was just not possible before.

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f61007cc650f6964ef2ff3ea98a1debf02713198)

ceph-volume util.prepare separate filestore vs. bluestore prepare utils

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 85cdc783008ff0e0566c34aaab7cd5f12db1cbc7)

ceph-volume process allow obfuscating a key or index when logging commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bbcca70d962855d991f0aa8e86a19f1855345668)

ceph-volume util.prepare add other ln helpers for bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 555b664882db2cd247087f907bdf4d426fb652f8)

ceph-volume util.arg_validators allow to pass a group as --data for lvm

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 211c38f591945dee53196068533784c385bae102)

ceph-volume util.disk add utilities for is_device and is_partition

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6b23fc72240ced158cf5335f6e815dbfd804dfab)

ceph-volume api.lvm udpate create_lv for bluestore

Require full size notation for LVM (e.g. 50G or 100M) and do not
"translate" tags by pre-fixing them with 'ceph'. That is already how the
rest of the API works, no need to make that translation in one place

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ac0d1555d3d9caeef9d4fc3e46bc11b7e437413b)

ceph-volume lvm.prepare update filestore workflow to use new helpers

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1e8a79b8f4b9e6360e8e3ace580e46caeedca8c7)

ceph-volume lvm.prepare initial take on bluestore support

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e4fc3464af472a8dbdf049917eed73519ff82c3b)

ceph-volume lvm.common update help flags for bluestore usage

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 17028f3a3c1f9394fa82dc6f2bbd596d96acb037)

ceph-volume api.lvm allow to create a vg

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 16d7feb2b5d5eb5fa0bf23a95b60623ae2e1676f)

ceph-volume lvm add block CLI flags for bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 63d2afbc605ac1531bc9f00a1891017bfd42fbce)

ceph-volume lvm make bluestore the default option

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dc3ced30747312dedd0e895a75ec15b1925ef1ea)

Merge pull request #18433 from theanalyst/wip-21545-luminous

luminous: rgw file write error

Reviewed-By: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #18587 from ceph/luminous-wip-volume-api-change

luminous ceph-volume lvm api refactor/move

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #18568 from liewegas/wip-fix-rop-leak-luminous-2

luminous: osd/PG: fix recovery op leak due to recovery preemption

Reviewed-by: David Zafman <dzafman@redhat.com>

ceph-volume create an api module to hold common api modules

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f72368641d7f38b6395afb7ed70a0a9d0794a03a)

ceph-volume tests move lvm api tests into its new test module

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit eec0c8a5c27adc9b3fc5e999cbee8165bdb45736)

ceph-volume tests refactor lvm api import for test_listing

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 66fd41fd4b35e275196bfb94794f3fda9fd77f76)

ceph-volume tests refactor lvm api import for api tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2a9a36bd24bddb35a7618127a0ec57f661a71566)

ceph-volume tests refactor lvm api import for activate tests

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d81aa8c112312f2fb49d94ce3512b3356b0b8814)

ceph-volume tests refactor lvm api import in conftest

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit a00aff7066fe6a679498ac8a02ec22ef29feced3)

ceph-volume devices.lvm refactor importing api to come from api/lvm.py

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f4bee6cb42a017abce269ed3926deff04038a47e)

ceph-volume move lvm/api.py to api/lvm.py so disk can consume it

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit bb72480d2bf44ff04ea93c98f4a4e59032a28896)

Merge pull request #18498 from liewegas/wip-recovery-fixes-luminous

luminous: osd: fix recovery priority and pg state on recovery->backfill transition

Reviewed-by: David Zafman <dzafman@redhat.com.

Merge pull request #18567 from ceph/backport-wip-bz1491250

luminous: ceph-volume lvm list

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

osd/PG: fix recovery op leak due to recovery preemption

This was fixed in master in a different patch, but are not yet ready to
backport the bits there that came before this.  For now, fix it
specifically for luminous.  We can either sort out the conflicts later
or revert this and backport the master parts conflict-free.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #17805 from tchaikov/wip-luminous-pr-17508

luminous: mon: more aggressively convert crush rulesets -> distinct rules

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18364 from kmroz/wip-21544-luminous

luminous: mon: osd feature checks with 0 up osds

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18413 from kmroz/wip-21693-luminous

luminous: osd: additional protection for out-of-bounds EC reads

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #18529 from smithfarm/wip-21918-luminous

luminous: Disable messenger logging (debug ms = 0/0) for clients unless overridden.

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #18540 from liewegas/wip-pr-17894-luminous

luminous: ceph.spec.in,debian/rules: change aio-max-nr to 1048576

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Boris Ranto <branto@redhat.com>

ceph-volume lvm add listing module to main

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ff72fd0ed52ef2d166007c7769c0140424982dc7)

doc/ceph-volume add lvm/list to the toctree

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d11c80c917f44b33077715b19564aeea30c78e42)

doc/ceph-volume create the reference for the list sub-command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9a52939ef8fc4abe720d2525c59997f4f9760902)

doc/ceph-volume add create and list to the index

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ed5fa519aca1335039bfc32bc060af5181265517)

ceph-volume tests create a unit test module for listing features

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2c2cf547b07f9510146b86d0746190b6d3fead3d)

ceph-volume tests create a small factory fixture

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5a0b17de81770ef3b851a1ff98232d93c78f49f9)

ceph-volume tests ensure Volume.as_dict gets populated correctly

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c5b933a96d067fb8c7f753a67123e5662398f964)

ceph-volume lvm.listing initial take on listing command

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 2d80190a039798a637383513f292e6aab62cd3f8)

ceph-volume lvm.api be able to return a dictionary repr of a Volume object

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0cae22750f396d1c1664875edb2742c50a41bffd)

ceph-volume process allows to fully mute terminal output on commands

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 233f64a4fd90a19c50d57b849ffaa6a25c835f34)

ceph-volume lvm.api use a helper to get an lv from an arg

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 3c5bbbc5822f02a038f093647c5de58bef2fe8f5)

Merge pull request #18456 from liewegas/wip-21882-luminous

luminous: messages/MOSDMap: do compat reencode of crush map, too

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/suites/upgrade/jewel-x: make sure min compat client is hammer for straw2

Signed-off-by: Sage Weil <sage@redhat.com>

ceph.spec.in,debian/rules: change aio-max-nr to 1048576

when osd is more than 14 in one host,it report error that aio is not enough.
As the default aio-max-nr is 65536, one OSD needs 4096 aios and other programs may use aios.
This patch change aio-max-nr to 1048576 when install ceph-osd rpm package and debian package

Signed-off-by: chenliuzhong <liuzhong.chen@easystack.cn>
(cherry picked from commit 36326dc7104fc2f20f19d51b6f618a029ba072d7)

common/common_init: disable ms subsystem log gathering for clients

The log gathering causes large performance degradation to clients
with high message throughputs. This is hopefully a short-term
workaround until per-message logging can be replaced with an
efficient data recording system for post-incident analysis
use-cases.

Fixes: http://tracker.ceph.com/issues/21860
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit a3a40413f7908b08c40dec4020034cca4a0c4798)

Merge pull request #18503 from liewegas/wip-21878-luminous

luminous: os/bluestore/BlueFS: fix race with log flush during async log compaction

Reviewed-by: Varada Kari <varada.kari@gmail.com>

Merge pull request #18501 from liewegas/wip-21766-luminous

luminous: os/bluestore: handle compressed extents in blob unsharing checks

Merge pull request #18459 from liewegas/wip-pr-18457-luminous

luminous: qa/suites/rados/rest/mgr-restful: whitelist more health

osd/PrimaryLogPG: clear DEGRADED at recovery completion even if more backfill

We may have log recovery *and* backfill to do, but cease to be degraded
as soon as the log recovery portion is done.  If that's the case, clear
the DEGRADED bit so that the PG state is not misleading.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3e91fda79d3be771f46394312a6b72ab75660012)

Merge pull request #18491 from liewegas/wip-21573-luminous

luminous: buffer: fix ABI breakage by removing list _mempool member

Reviewed-by: Kefu Chai <kchai@redhat.com>

os/bluestore/BlueFS: fix race with log flush during async log compaction

During async log compaction we rely on _flush-and_sync_log to update the
log_writer to jump_to.  However, if racing threads are also trying to flush
the log and manage to flush our new log events for us, then our flush will
turn into a no-op, and we won't update jump_to correctly at all.  This
results in a corrupted log size a bit later one.

Fix by ensuring that there are no in-progress flushes before we add our
log entries.  Also, add asserts to _flush_and_sync_log to make sure we
never bail out early if jump_to is set (which would indicate this or
another similar bug is still present).

Fixes: http://tracker.ceph.com/issues/21878
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 4324c8bc7e66633035c15995e3f82ef91d3a5e8c)

os/bluestore: handle compressed extents in blob unsharing checks

If the blob is compressed, we aren't mapping to a range within
the allocated extents, but rather referencing the entire blob.

Fixes: http://tracker.ceph.com/issues/21766
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit eb26cfbfbb2b9b9fe84dfe9c999e2d67fd2661c0)

osd/PG: on recovery done, requeue for backfill

We were keeping our existing recovery reservation slot (with a high
priority) and going straight to waiting for backfill reservations on
the peers.  This is a problem because the reserver thinks we're doing
high priority work when we're actually doing lower-priority backfill.

Fix by closing out our recovery reservation and going to the
WaitLocalBackfillReserved state, where we'll re-request backfill at the
appropriate priority.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 09c50e2fe4fc3e8951a483e4b2693f0ffc008dcf)

osd/PG: move local_reserver recovery cancel to Recovering state transition

This is easier to follow than canceling the reservation in the next state.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit dde007749bb960122f399b4cb8608a8c1ea4eb9c)

qa/suites/upgrade/jewel-x/parallel: run some jewel after completed upgrade

Signed-off-by: Sage Weil <sage@redhat.com>

qa/suites/upgrade/jewel-x/: set up compat weight-set after cluster upgrade

...but before we run some client workloads.  This will hopefully ensure
that older clients will behave.

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: add 'osd crush set-all-straw-buckets-to-straw2'

This is a common and recommended step that users should take once they
have clients that support straw2 (hammer!).  Let's make it easy.

This may result in a small amount of data movement, but usually it is
very little.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 928770513cbe3a244095e36dcb41c83c0724cfe9)

crush/CrushWrapper: add bucket_set_alg()

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit edc411ee4cb7b88a970ccfc9d29b3a3ba15d066c)

buffer: remove list _mempool member

This broke the C++ ABI by changing the list structure size.  Also, it's
not necessary as we can infer the mempool by looking at the other list
contents.  We don't (currently) have a need to map an empty list to a
particular mempool and have that state stick.

Fixes: http://tracker.ceph.com/issues/21573
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 9b92d87d37e3892b0096cd728b46154aed1e2d86)

buffer: allow mempool to be passed into raw* ctors and create methods

This is more convenient, and also faster than initializing it in
buffer_anon and the immediately moving it elsewhere.

Drop the optionality of the alignment argument.

No users yet.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 481277b8c92368b8cac8fdafe1848720ec1035c2)