doc: make note of pg log hard limit upgrade bug in PendingReleaseNotes

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd: make calc_trim_to() independent of min_last_complete_ondisk"

This reverts commit 3ecbf2a41fef3591ff247a212338b9a2d02a3468.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd: print pg log length and trim_to"

This reverts commit 21a72d2a52f8241ebe0a5455f030d2095e413025.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd: handle trim() during backfill"

This reverts commit bdfc2a3b4fa8038ade31ccab5f5479508e9afff3.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd: allow trim() to proceed when there are missing items"

This reverts commit d08f0aec86ecd9b23310f1c41f9cb9060f10bc6b.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd: reset complete_to when trimming the log past it"

This reverts commit 715d1fdbfc407de81686784d99fe28b51c6d97ef.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd/PGLog: allow pg log trim when complete_to is less than trim_to"

This reverts commit be8b406d9162a1f46b3eac2fca07b0cf20697b90.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd/PGLog.cc: use lgeneric_subdout instead of generic_dout"

This reverts commit f2ea4f8b67ac972cd428e8ea8f2ff062b1624677.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "src/osd/PG.cc: remove redundant call to trim_log()"

This reverts commit 31616627dcfb2b6ab7db7da025e191dc34094bb8.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd/PGLog.cc: check if complete_to points to log.end()"

This reverts commit ac772c90dfc3749527246242a3a84e5e48ff52e1.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd/PrimaryLogPG.cc: limit trimming at can_rollback_to"

This reverts commit 0e57c101c9dd51ee3a84123c96e626799ab6d21e.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd/PrimaryLogPG: avoid dereferencing invalid complete_to"

This reverts commit eb3bdf0714b9c78c5b3634d4efe3b603824fea09.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Revert "osd/PrimaryLogPG: fix potential pg-log overtrimming"

This reverts commit 85a029a7242a1c6eef4e90ae246541e08207302a.

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24854 from cbodley/wip-36536

luminous: rgw: user stats account for resharded buckets

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24855 from cbodley/wip-36534

luminous: cls/user: cls_user_remove_bucket writes modified header

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24862 from jonsger/luminous-backport-pr#23242

luminous: rgw: set default objecter_inflight_ops = 24576

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24454 from croit/luminous-36041

luminous: rgw/beast: drop privileges after binding ports

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24507 from pdvian/wip-36212-luminous

luminous: rgw: RGWAsyncGetBucketInstanceInfo does not access coroutine memory

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24519 from pdvian/wip-36214-luminous

luminous: rgw: fix leak of curl handle on shutdown

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24570 from joke-lee/luminous-list-RGWPostObj-when-enable-versioning

luminous: rgw: list bucket can not show the object uploaded by RGWPostObj when enable bucket versioning

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24857 from jonsger/luminous-backport-pr#23534

luminous: rgw: multisite: enforce spawn_window for data full sync

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: set default objecter_inflight_ops = 24576

Fixes: http://tracker.ceph.com/issues/25109
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 102c5bd)
Signed-off-by: Jonathan Brielmaier <jbrielmaier@suse.de>
Conflicts:
src/rgw/rgw_main.cc: use the existing vector, there is no map

cls/user: cls_user_remove_bucket writes modified header

if cls_user_remove_bucket() modifies the header with dec_header_stats(),
it should write the updated header back

Fixes: http://tracker.ceph.com/issues/36496
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit dac26de1fa7b56ce0ec2fd43d26ab9a9b6a32e3d)

rgw: cls_user_set_buckets_info can update bucket id

if cls_user_set_bucket_info() finds an existing bucket entry, it does
not update its bucket id

Fixes: https://tracker.ceph.com/issues/24505
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit e1b97fc2e804d040e8aa6fc8a94e336222d1175e)

rgw: rgw_user_sync_all_stats uses bucket entrypoint

when bucket reshard completes, rgw_link_bucket() passes the new bucket
instance id down to cls_user, but cls_user_set_buckets_info() does not
change the instance id when it's updating an existing bucket. so when
rgw_user_sync_all_stats() looks up each of the user's buckets, it uses
the original bucket instance id instead of the resharded one and
calculates user stats that may not match the current bucket stats

as a workaround, rgw_user_sync_all_stats() no longer relies on the
bucket instance id it gets from rgw_read_user_buckets(), and instead
calls get_bucket_info() to look up the current instance in the bucket
entrypoint

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit b33328ff9bacd11d8449cb3b5c52f94d9dffec6e)

rgw: data sync checks whether the lease failed to renew

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry pick from commit 3754871)
Signed-off-by: Jonathan Brielmaier <jbrielmaier@suse.de>
Conflicts:
src/rgw/rgw_data_sync.cc: trivial

rgw: data sync holds lease over transition from full to incremental

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked commit from 3e9ac0f)
Signed-off-by: Jonathan Brielmaier <jbrielmaier@suse.de>
Conflicts:
src/rgw/rgw_data_sync.cc: use ldout instead of tn->log, reflect
        state of luminous in multiple places

Merge pull request #24808 from ceph/wip-luminous-mclock-test

test/common: unittest_mclock_priority_queue builds with "make" command

rgw: enforce spawn_window for data full sync

limit the number of concurrent bucket sync crs spawned by data full sync

Fixes: http://tracker.ceph.com/issues/26897
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 895c28a)
Signed-off-by: Jonathan Brielmaier <jbrielmaier@suse.de>
Conflicts:
src/rgw/rgw_data_sync.cc: parameters in yield spawn

Merge pull request #24783 from alfredodeza/luminous-evelu-ceph-volume-choose_disk

luminous Additional work on ceph-volume to add some choose_disk capabilities

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

test/common: unittest_mclock_priority_queue builds with "make" command

Although it would be built with "make tests",
unittest_mclock_priority_queue was not built with a simple "make".

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 494b2f8f5ae11a689bb2032e4527968ff09bcdeb)

ceph_volume: Checking device validity at init time

When initializing the Device structure, it have to run is_valid() to
ensure the data structures (_is_valid & rejected_reasons) to be
populated accordingly to the device state.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit ef0ceef7c717ef615dc214299cb4a6b32c1edbe6)

ceph_volume: Rejecting locked devices

If we cannot open a block device in O_RDWR in exclusive mode, it means
someone is actually using it like a raw database or similar.

In that case, the device should be considered as unusable as OSDs will
not be in a position to use it.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit d5de9583ee6c4d9ccaab2b1146606a0ee316cc27)

ceph_volume: Reporting nr_requests

We are already reporting the rotational & scheduler of a disk device.
Reporting the nr_requests could be useful to get how many concurrent IOs
the device supports/reports.

That could help detecting badly detected/configured devices.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit e0ea3d475a85844afdcdf92cf638b8128c8e2d9e)

ceph_volume: Reporting firmware revision

We are already reporting model & vendor of a given disk, let's also
report the revision of the firmware. That is useful to filter-out some
known broken revisions.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit 59720794079bef0ba7f8bc2d9dfc8aaddd210005)

ceph_volume: Rejecting Read-only devices

If a devices is said to be read-only, there is no chance we can actually
use it. So let's report it as unusable.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit f1a94350064263f27f67d2c715c5ca5922481a01)

ceph_volume: Adding Device.is_valid()

A block device can be filtered-out/ignored because it have features that
doesn't match Ceph's expectations.

As of today, the current code was rejected removable devices but it was
pretty hidden from the user, and implicit in the get_devices() function.

This patch is creating a new is_valid() function to perform all the
rejection tests and returns if this device can be used in the Ceph
context or not.

If is_valid() is returning False, the 'rejected_reasons' list reports all
the reasons why that devices got rejected.

Signed-off-by: Erwan Velu <erwan@redhat.com>
(cherry picked from commit d8fdf0b7532cdffcae56511c377679e51841caec)

Merge pull request #24759 from ceph/backport-luminous-24587

luminous: ceph-volume: adds a --prepare flag to `lvm batch`

Merge pull request #24482 from pdvian/wip-36149-luminous

luminous: crush/CrushWrapper: fix crush tree json dumper

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #24532 from dzafman/wip-test-luminous

luminous: backport and other test fixes for osd-scrub-repair.sh

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #24538 from pdvian/wip-36229-luminous

luminous: qa: add test that builds example librados programs

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24574 from pdvian/wip-36295-luminous

luminous: osdc/Objecter: possible race condition with connection reset

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24582 from smithfarm/wip-36437-luminous

luminous: ec: src/common/interval_map.h: 161: FAILED assert(len > 0)

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24593 from dzafman/wip-36419

luminous: osd: get loadavg per cpu for scrub load threshold check

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24602 from pdvian/wip-36297-luminous

luminous: osd: add creating to pg_string_state

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #24659 from tchaikov/luminous-26890

luminous: osd: cast 'whoami' to unsigned so it can be used as the seed for RNG

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge commit 'f0d49da1be98291f26f70b524f182eba4fe78cee' into luminous

ceph-volume: update man page for batch --prepare

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit faa3aed4ad4df2dbdd853235e364d6e3b9b176b6)

ceph-volume: docs for --prepare flag of lvm batch

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b4825e1bd7b75a0a53ee5678230b262286293d75)

ceph-volume: when --prepare is added to batch the OSDs are only prepared

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit abd238dc1f33252a43adc8b86efb3d3b4a076964)

ceph-volume: adds a --prepare flag to ceph-volume lvm batch

This flag will only prepare the OSDs, not activate them. This is useful
in our containerized ceph solution.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 4368ca7d48f4ce35f67854e8b0ca2be9babc5371)

Merge pull request #24754 from alfredodeza/luminous-rm36386

luminous ceph-volume remove version reporting from help menu

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #23493 from VictorDenisov/backport_24333

luminous: PG: add custom_reaction Backfilled and release reservations after bac…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

ceph-volume remove version reporting from help menu

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 20db7bf585f6433c6705396264005a4227cbf2e3)

12.2.9

Merge pull request #24741 from alfredodeza/luminous-rm36492

luminous ceph-volume: do not send (lvm) stderr/stdout to the terminal, use the logfile

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

ceph-volume tests.api update monkeypatching to use **kw

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6d6bcda75f846bc7c44c7b16b3687a65fed7cafc)

ceph-volume tests allow **kw when monkeypatching

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 04e51c956eadcbd38f39ec3dd4b78009d2cd4a70)

ceph-volume api.lvm: do not spit out errors on the terminal from LVM

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 581eed36ea0e7b5c5e41f8b83350403bcf73a31a)

Merge pull request #24620 from dillaman/wip-36431-luminous

luminous: qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24650 from yuriw/wip-yuriw-fix-jewel-x-luminous

Excluded 'python34-cephfs' from the install tasks

Reviewed-by: David Galloway <dgallowa@redhat.com>

Excluded 'python34-cephfs','python34-rados','python34-rbd','python34-rgw','python34-ceph-argparse','python3-cephfs','python3-rados' from the install tasks

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #24663 from alfredodeza/luminous-rm23650

luminous: specs: require of e2fsprogs

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rpm: require 2fsprogs needed for ceph-disk when using dmcrypt

Signed-off-by: Alfredo Deza <adeza@redhat.com>

specs: require of e2fsprogs

in ceph/ceph-container we've realized that `e2fsprogs` isn't installed in
centos container image because ceph hasn't a dependency for it.
It has for consequence to fail when deploying a containerized cluster
with dmcrypt when using centos image.

Typical error encountered:

typical error:
```
......
get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
get_dm_uuid: get_dm_uuid /dev/sda5 uuid path is /sys/dev/block/8:5/dm/uuid
populate: Creating lockbox fs on %s: mkfs -t ext4 /dev/sda5
command_check_call: Running command: /usr/sbin/mkfs -t ext4 /dev/sda5
mkfs.ext4: No such file or directory
Traceback (most recent call last):
  File "/usr/sbin/ceph-disk", line 9, in <module>
    load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
......
```

Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>
(cherry picked from commit a99177834120e7a2c4592054f6a8b8736e0ffb92)

osd: cast `whoami` to unsigned so it can be used as the seed for RNG

default_random_engine's result_type is `unsigned int`, so we need to
pass an `unsigned int` as its seed.

Fixes: http://tracker.ceph.com/issues/26890
Signed-off-by: Kefu Chai <kchai@redhat.com>
Conflicts:
src/osd/OSD.cc: this breaks the build with clang. and in master
we are not using std::default_random_engine for setting the scrub
interval. so this change is not cherry-picked from master.

Merge pull request #24589 from ceph/backport-luminous-24404

luminous: ceph-volume: make `lvm batch` idempotent

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24451 from alfredodeza/luminous-wip-rm24795

luminous: ceph-volume lvm.prepare update help to indicate partitions are needed, not devices

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

qa/tasks/workunit: use suite branch/SHA1 when cloning workunits

Right now it's using the Ceph branch/SHA1 but it's using the suite
Git URL.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 1e3dc02604cf7c0d3991dfd9fe2596ea34e80bad)

qa/tasks: qemu task now uses a relative path in suite repo for test

This makes it easier to re-run tests against a suite branch without
requiring a full ceph-ci build and repo.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c1f950236b2f1ba3b588fc638ebf8cb785c36e08)

qa/workunits: replace 'realpath' with 'readlink -f' in fsstress.sh

Fixes: http://tracker.ceph.com/issues/36409
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit ddb7e5eb80ebb78beff6dfb25042f0016a474491)

qa/tasks/cram: tasks now must live in the repository

Commit 0d8887652d53 ("qa/tasks/cram: use suite_repo repository for all
cram jobs") removed hardcoded git.ceph.com links, but as it turned out
it is still used for nightlies.  There is no good way to accommodate
the different URL schemes, so let's get rid of URLs altogether.

Fixes: https://tracker.ceph.com/issues/27211
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 592f566b4e270d3833ba896dabb193d5241dd5de)

Conflicts:
qa/suites/krbd/basic/tasks/krbd_blkroset.yaml
qa/suites/krbd/basic/tasks/krbd_huge_image.yaml
qa/suites/krbd/basic/tasks/krbd_msgr_segments.yaml
qa/suites/krbd/basic/tasks/krbd_parent_overlap.yaml
qa/suites/krbd/basic/tasks/krbd_whole_object_discard.yaml
- in master, the cram task is referred to in these additional yaml
files, but in luminous it's only referred to in
qa/suites/krbd/unmap/tasks/unmap.yaml

qa/tasks/workunit: factor out overrides and refspec logic

Allow for reuse in the cram task.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit e1c89b51c80407fadbfee82d5d396cfcbd791aae)

qa/tasks/cram: use suite_repo repository for all cram jobs

Currently git.ceph.com is hardcoded for all cram jobs.  Testing
modifications is a pain: one needs to push to either ceph/ceph.git or
ceph/ceph-ci.git (depending on where the ceph branch is at, triggering
unnecessary builds in the latter case) and wait for the mirror to sync.
Runs scheduled against branches in developer's forks fail.

Move away from git.ceph.com to allow mixing branches and repositories,
similar to workunits.

Fixes: https://tracker.ceph.com/issues/27211
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 0d8887652d5312f7059ab2bdb52c948aa01680b0)

Conflicts:
qa/suites/krbd/basic/tasks/krbd_blkroset.yaml
qa/suites/krbd/basic/tasks/krbd_huge_image.yaml
qa/suites/krbd/basic/tasks/krbd_msgr_segments.yaml
qa/suites/krbd/basic/tasks/krbd_parent_overlap.yaml
qa/suites/krbd/basic/tasks/krbd_whole_object_discard.yaml
- in master, the cram task is referred to in these additional yaml
files, but in luminous it's only referred to in
qa/suites/krbd/unmap/tasks/unmap.yaml

osd: add creating to pg_string_state

Fixes: http://tracker.ceph.com/issues/36174
Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>
(cherry picked from commit d38f6a11701ec788e4d384aa5b0ae65b8e57da64)

Conflicts:
src/osd/osd_types.cc : Resolved in pg_string_state

ceph-volume: extracts batch.filter_devices from Batch._get_strategy

This allows us to easily provide tests for that method.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit df7ef5383bb476ee020b898cd2e8fbce044fc07e)

ceph-volume: failing to get block db size from conf logs an exception

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 172d4af9b97d8d64ce35dc95efb72d9f190bc170)

ceph-volume: when all devices are filtered exit gracefully

Even if all devices are filtered we want to return a 0 exit code and
make sure the json reporting still works.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit aa4fcd602f8b1a1b17bc12e59334b19508f97d6d)

ceph-volume: filter devices used by journals/block.db

If after filterering of data/block devices there are only
one device left it can not be used if it is an SSD and
has been used previously as a journal or block.db

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit fc9a10e54813a402ecd1d823ea5a33f85e8eb963)

ceph-volume: add rotational property to Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 7f6bfaaf0cf30a56481d6d67f2fa2a7e785cc344)

ceph-volume: add info about filtered devices to batch pretty reports

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 62426db799c54436c50b5751c8e72b3dbd710a2a)

ceph-volume: remove the used_by_ceph key in the json output

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 3b6d82afe7ee83283b25cefda00c7a59cf5144af)

ceph-volume: fix idempotency checks for lvm batch tests

The mixed type tests will change strategy after the idempotency test so
we need to handle that in test playbook.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1143482061da50910f4f883aafdbe0a0e4269f39)

ceph-volume: fix bluestore strategy json reporting and type

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b3cf90604d31709996538e612b1767d6fd4da8b5)

ceph-volume: ignore failure to load ceph configuration for block.db size

If we fail to load a ceph configureation file when trying to get the
block.db size then just use defaults instead of throwing an error.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a7ee36ca92b6592b6b3e218252c6a4c30416591b)

ceph-volume: fix strategy comparison in 'lvm batch'

This also fixes some small json reporting issues with the
filestore MixedType strategy

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1dd15025bfd52af9c31cd281f92d743b9ca0eeb8)

ceph-volume: raise a non zero exit code if strategy changes with batch

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 9dfc00f8e10e10fbdc52c44259596c6a96a90edd)

ceph-volume: add functional tests to ensure lvm batch is idempotent

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 9752d03cc16664cddd9ac7741284a2fb5b31f0e7)

ceph-volume: add tests for util.device.Device.used_by_ceph

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit a28e6531e47d60d9eb7f62b67f122578daa2a683)

ceph-volume: update tests to account for filtered_devices in batch

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 36396229d3b8d20862565b35d431ff22fa92cd1c)

ceph-volume: consider block and data devices used_by_ceph

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 67512530116c16d072c91867a259aeb429d32ff6)

ceph-volume: add filtered_devices and used_by_ceph to all batch reports

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 0718d2e2663c8e807dbb5171a58759c76ed92c08)

ceph-volume: pick strategy for batch with only the unused devices

This will pick a strategy, filter out any devices already been used by
ceph and then pick a strategy again. If the strategy has changed the
call should error, if the strategy is the same proceed. If there are no
unused devices then the command is a noop.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 4529f2d6053b0b07583a3d7501f4e05e08cad385)

ceph-volume: adds a 'changed' key to lvm batch --report

This will indicate if the command would result in any OSDs being created
or not. Other tooling can use that key for idempotency checks.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 10f1d577d4c4e66c77046fc3b274d6653af99586)

ceph-volume: adds used_by_ceph to filestore singletype batch report

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 48d10c9ccf13ba7f42750b37d26aeb28f6b1c606)

ceph-volume: adds a used_by_ceph property to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 9d49a3708e34bccd183d5c32d022c36d8b118b42)

ceph-volume: adds a lvs property to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 3dae3247adc1b96ca688ec81cec03180d5943823)

ceph-volume: add vg_name to the Device class

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 619810c0ef48c1f19db8821a33f9c48cc11a0161)

Merge pull request #24527 from theanalyst/wip-luminous-36382

luminous: rgw: resharding produces invalid values of bucket stats

Reviewed-By: Casey Bodley <cbodley@redhat.com>

osd/PrimaryLogPG.cc: reassign size only when object size > truncate_size

Before setting size equal to op.extent.truncate_size, we need to check
if the size of the object is greater than the truncate_size. We do not
need to set size to op.extent.truncate_size, in the case where the size of
the object is less than op.extent.truncate_size.

Without this change, we were always setting size =
op.extent.truncate_size, when (seq < op.extent.truncate_seq) and
(op.extent.offset + op.extent.length > op.extent.truncate_size), were both
true. This ended up in:

1. overestimating the size of the object
2. not considering the correct size of the object, for
   the later checks, which calculate op.extent.length for the read ops
3. causing crashes when trying to read more data than what was present

Fixes: http://tracker.ceph.com/issues/21931
Fixes: http://tracker.ceph.com/issues/22330
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit 76c57810ee2346c392834206331aacb0faaa5b54)