ceph-volume tests.functional set root:root in OSDs in filestore

To make sure that ceph-volume will be able to set the right permissions
when trying to activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 503396f685f6a93e0c63e97477677b7145f96c26)

ceph-volume tests.functional set root:root in OSDs in bluestore

To make sure that ceph-volume will be able to set the right permissions
when trying to activate

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 98631fdeeb6dbe765d3eb80c5c1800946735f2a2)

ceph-volume lvm.activate always chown osd path before osd-prime-dir in bluestore

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0af31ec24533091aa4991b2343fe80da94777b4d)

Merge pull request #25352 from alfredodeza/luminous-rm37449

luminous: ceph-volume: zap: improve zapping to remove all partitions and all LVs, encrypted or not

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #24468 from vshankar/wip-36281

luminous: mds: add "drop cache" command

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #24728 from vshankar/wip-36217

luminous: cephfs-journal-tool: make --rank argument mandatory

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #24924 from vshankar/wip-24759

luminous: qa/suites/fs: use bigger cinder volumes

Reviewed-by:  Venky Shankar <vshankar@redhat.com>

Merge pull request #24324 from smithfarm/wip-32091-luminous

luminous: mds: migrate strays part by part when shutdown mds

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #24440 from pdvian/wip-36200-luminous

luminous: mds: fix mds damaged due to unexpected journal length

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #24465 from pdvian/wip-36206-luminous

luminous: libcephfs: expose CEPH_SETATTR_MTIME_NOW and CEPH_SETATTR_ATIME_NOW

Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #24552 from pdvian/wip-36279-luminous

luminous : qa: tolerate failed rank while waiting for state

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #24872 from trociny/wip-36642-luminous

luminous: osdc: reduce ObjectCacher's memory fragments

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25569 from batrick/i24929

luminous: qa: test_recovery_pool tries asok on wrong node

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #25752 from alfredodeza/luminous-better-devid

luminous common/blkdev, ceph-volume: improve get_device_id

Reviewed-by: Sage Weil <sage@redhat.com>

ceph-volume: Micron SSDs don't include vendor name in ID_SERIAL

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3e4a638c931a5198aca74dc4037d0b03d5e563ba)

ceph-volume: update get_device_id to match in-tree implementation

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit c967dbf81f85602d2cd044ee5a91ba38cd76bdb9)

Merge pull request #25532 from smithfarm/wip-37643-luminous

luminous: ceph-create-keys: fix octal notation for Python 3 without losing compatibility with Python 2

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #25609 from joscollin/wip-37685-luminous

luminous: doc/user-management: Remove obsolete reset caps command

Reviewed-by: Nathan Cutler <ncutler@suse.com>

doc/user-management: Remove obsolete reset caps command

This command has not worked since hammer and makes no sense in an
environment where we do not allow empty caps

Fixes: http://tracker.ceph.com/issues/37663
Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
(cherry picked from commit e4e7f3ea6af8316cd46bb74c3f6d6819b99cea53)

 Conflicts:
doc/rados/operations/user-management.rst
Dropped the allow empty caps paragraph, as done in master.

qa: add status field to method

From: ab4f33bd248bf192c1a81fea05a81c7f85ede52d
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25552 from jan--f/25203-luminous

luminous ceph-volume: fix Batch object in py3 environments

Reviewed-by: Alfredo Deza <adeza@redhat.com>

qa: add status option

From: 333444e8aac065058c0a2353beff8005044659f7
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: remove dead code

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit a2ff87d4e25bc891fbfce4b0bdeb1a308ec22cb5)

qa: run asok command on correct machine

The MDS may not be on the same machine where the cluster command is run.

Fixes: http://tracker.ceph.com/issues/24858
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit fa25d6c8d1890bed2082b5c14f1c62a3db9ec27e)

ceph-volume: add Batch smoketest

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 936ee680f7633620c8a1127db9630a22326b9e2c)

ceph-volume: fix Batch object in py3 environments

Fixes: https://tracker.ceph.com/issues/37356
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 851c7693daa315b7972702bedf09dd66a37667d8)

Merge pull request #25500 from joscollin/wip-37600-luminous

luminous: doc: broken link on troubleshooting-mon page

Reviewed-by: Nathan Cutler <ncutler@suse.com>

ceph-create-keys: Misc Python 3 fixes

Use octal notation for file permissions.

Switch file() calls to open().

Signed-off-by: James Page <james.page@ubuntu.com>
(cherry picked from commit 107579196eb4b52b1965fec6dcc53eba9d314b97)

Merge pull request #25418 from xiexingguo/wip-luminous-upmap-fixes

luminous: osd: backport recent upmap fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25286 from smithfarm/wip-37397-luminous

luminous: tests: /usr/bin/ld: cannot find -lradospp in rados mimic

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #25041 from smithfarm/wip-36750-luminous

luminous: mgr: [restful] deep_scrub is not a valid OSD command

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #25047 from swinds24/fix_36764

luminous: mgr: hold lock while accessing the request list and submitting request

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #25216 from k0ste/luminous_backports

luminous: mgr: prometheus: added bluestore db and wal devices to ceph_disk_occupation metric.//

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #25259 from k0ste/luminous_backports4

luminous: mgr/balancer: add cmd to list all plans

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25326 from theanalyst/wip-luminous-reshard-fixes-1

luminous: bucket resharding fixes

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #24547 from linuxbox2/luminous-24595

luminous: rgw: apply quota config to users created via external auth

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24621 from cbodley/wip-24358

luminous: rgw: add ssl support to beast frontend

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24909 from smithfarm/wip-36414-luminous

luminous: rgw: librgw: crashes in multisite configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #24944 from jonsger/luminous-backport-pr#24700

luminous: rgw: SSE encryption does not detect ssl termination in proxy

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25088 from jonsger/wip-36757-luminous

luminous: rgw: rgw-admin: reshard add can add a non-existent bucket

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25195 from pdvian/wip-37284-luminous

luminous: rgw: radosgw-admin: translate reshard status codes (trivial)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25320 from theanalyst/wip-luminous-zone-deletion

luminous: rgw: fixes for zone deletion

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc: broken link on troubleshooting-mon page
Fixes: https://tracker.ceph.com/issues/37451
This fix involves adding :ref: labels to the add-or-rm-mons.rst
page. I also added :ref: labels for other headings within
add-or-rm-mons.rst (for future reference).

Signed-off-by: James McClune <jmcclune@mcclunetechnologies.net>
(cherry picked from commit ca05e456969a721631f5e9882c896cab76a4a3dc)

Conflicts:
doc/rados/troubleshooting/troubleshooting-mon.rst
Fixed the :ref: sentence and removed the _replace line.

Merge pull request #25284 from jdurgin/wip-osd-memory-target-docs-luminous

luminous: doc/rados: update bluestore provisioning and autotuning docs

Reviewed-by: Mark Nelson <mnelson@redhat.com>

rgw: fix typo

This fixes a typo in a log message. It's a separate commit so
downstream commits point to the right upstream commits via
cherry-pick.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 7d1768f2a332c5ac389cbb4bbeda28752e4f17bc)

rgw: perf -- remove bucket shards asynchronously rather than synchronously

We can now take advantage of the new asynchronous bucket shard removal
code and where we used to remove each shard synchronously now remove
them asynchronously. This would be a huge win when we have tens of
thousands of shards.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit cb0da45c5d6e4dff6514919b3b894df0eea7c1e4)

 Conflicts:
src/rgw/rgw_rados.cc
conflicts with placement set and rgw rados refactor

rgw: clean up unused bucket index shards

Clean up old bucket index shards when a resharding is complete. Also,
when a resharding fails, clean up unfinished bucket index shards. Do
both clean-ups asynchronously.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit f84f70d451036ee99011b8e2c7b974f15d3a005a)

 Conflicts:
src/rgw/rgw_rados.h
merge conflict as bucket_placement functions were moved after the rgw rados
refactor

rgw: a little code clean-up by adding const

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 48e22fb76feffb551074f274986bc4125a02003f)

rgw: remove never-called RGWRados member function

Because RGWRados::cls_rgw_init_index is never called, remove it.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 4593778c6d830a1a0d33528507f1795aee019ce4)

rgw: rename function for clarity

The function cls_rgw_bucket_init was renamed to
cls_rgw_bucket_init_index in order to better describe its
functionality.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 20868bd3b893ba2e17d2bfdc63d6544dbe677a9b)

rgw: fix bug in error codes returned

Fixes two bugs where positive error code was returned rather than
negative error code, which is the RGW norm.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit dc0732c524c2d8fb5b423f8cce8c6cc9aa7ca44a)

osd/OSDMap: fix upmap mis-killing for erasure-coded PGs

The up-set of erasure-coded PGs may include CRUSH_ITEM_NONE,
which as a result causes mis-killing of valid upmap items.

Fixes: https://tracker.ceph.com/issues/37493
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
Signed-off-by: ningtao <ningtao@sangfor.com.cn>
(cherry picked from commit f043dcc6d8663ed960a354b2441fd3f8dd66c62d)

Conflicts:
        - *maybe_remove_pg_upmaps* input changed, in master
          we now have a passed in *nextmap* parameter

mon: auto clean up stale upmap items

See https://www.spinics.net/lists/ceph-users/msg49518.html

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 4181aa86906886423af80e478e45554606fc7cf6)

Conflicts:
        - *nextmap* has been renamed to *tmpmap*
        - add the *have_pg_upmaps* wrapper which is necessary for unittest

osd/OSDMap: add pg-existence sanity check

The reason why __get_pg_pool_size(pg)__ or __get_pg_pool_crush_rule(pg)__ fails is
that the pg does not exist anymore. So it generally makes sense to check __pg_exists(pg)__
before moving further.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 79ef3bea7550559b7ea4e2b68dc407c7eb25defa)

Conflicts:
- *nextmap* has been renamed to *tmpmap*

osd/OSDMap.cc: move sanity checks to the proper place

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 3dcd4eaf867369437720a21daf3c344cc4b9844e)

osd/OSDMap.cc: remove pg_upmap/pg_upmap_items too if osd is gone

If an osd is gone or moved out from the specific crush rule,
we should cancel any pg_upmap/pg_upmap_items still bound to
that osd too.

The original code does not work for the above case because
get_parent_of_type() will fail if that osd does not belong
to the crush_rule passed in and hence hits the assert below:

```
src/osd/OSDMap.cc: 4078: FAILED assert(target > 0)
```

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 8a11ed479d7ea95f76170e74123ca2c48e3a3963)

Merge pull request #25350 from alfredodeza/luminous-c-v-inventory-add-device_id

luminous ceph-volume add device_id to inventory listing

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

PendingReleaseNotes: add release notes for cephfs-journal-tool changes

Fixes: https://tracker.ceph.com/issues/36217
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: make rank argument mandatory when running journal_tool

Also, fix a bunch of quirky journal_tool invocations that pass
"--rank" argument as the command argument rather than passing it
as function argument.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit f65193d95534dd34b593c92e1848b181737b0561)

cephfs-journal-tool: make "--rank" argument mandatory

... and do not silenty act on the default filesystem.
Force users to specify the filesystem name and rank.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit 5e4702e303bd70b4ff1de973375bef952c0d8b6a)

 Conflicts:
src/tools/cephfs/JournalTool.cc
src/tools/cephfs/JournalTool.h

cephfs-journal-tool: pass local arg vector for Journal actions

Some helper functions clobber the passed in arg vector. This
causes operations on all but the first rank to fail as the
operations for the first rank tampers with the arg vector.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit d5e482a6c2782993e6e9903eb6f11f5ebe260925)

cephfs-journal-tool: dump to per rank output file wherever necessary

cephfs-journal-tool supports operations on all ranks. Operations
such as dump/export do not write to distinct filenames hence
overwriting the data dumped or exported for the previous rank.

With this change (and further commits), for operations on all
ranks, dump/export would write to distinct filenames suffixed
by the mds rank (.0, .1, etc..). For operations on a single rank
or if there exist a single rank, the passed in filename is used
as it is.

Signed-off-by: Venky Shankar <vshankar@redhat.com>
(cherry picked from commit cb19db75c7f23278d2598b58526d1bf8bf00b679)

 Conflicts:
src/tools/cephfs/JournalTool.h

ceph-volume tests patch udevadm so it doesn't actually call udevadm in a test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9f440fb96694e38f69e174c01f284ab9a2066bc8)

ceph-volume util.disk use device.path for error reporting

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1670585ebed7c685b49a13c1066eedc1aa37d29b)

ceph-volume tests.functional add back sdc1 partitions with parted to redeploy after zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b0bc162b601c2dbc9827df0ab472ba2a1146076c)

ceph-volume tests.functional add back partitions with parted to redeploy after zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 945e8f56ce21411f664a58147afb343f0671b591)

ceph-volume tests.functional recreate partitions previously removed with --destroy

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e1d136ee3bcb2a7bbe0790b87e1d54a39b522e01)

ceph-volume tests.util ensure that both disk and device report is_device

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit df8ba476d68a62f6e0449aa27c81f3effa1e0536)

ceph-volume tests.util ensure is_encrypted works for lv, partition, and mapper devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ff0166f9f445d229216113623f18244af188b57f)

ceph-volume tests.util ensure partitions are listed as top-level keys

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6dc01772245e557b73551111f7c48cc60aec0893)

ceph-volume tests add mapper test for dm paths

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 9ebf7d7631be362e6a26af19225bfc07ec48138d)

ceph-volume tests update is_mapper unit test when zapping

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 830cd7d638b70e97723eb41f0d33f6a60e3ece32)

ceph-volume util.device check both dm and mapper paths for is_mapper property

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ba83c1f37e6ba6fa695ff97a028479e88e2b762c)

ceph-volume lvm skip empty vg_name removals

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 14c74ede9a83975a3464c219cab0be3c06238ff9)

ceph-volume util.arg_validators optionally skip gpt header check

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 58be4d0e18c4b2f3560361345f91c5a1c7b6c5c4)

ceph-volume util.device if no vgs, set it to an empty list

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 822ca670eef57e009e356fc58d6732cdeecdecdc)

ceph-volume util.device set vgs to Device object if available

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 91c01e88ecc8f94f78b30463f13a51affe405d31)

ceph-volume util update is_device when reported as device or disk

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit f74635d16d7ceefb8e0de02ded85a6e7be53457f)

ceph-volume util add method to detect encryption on devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 1ee7435aef2f6475256e8482194d55e16b77eee8)

ceph-volume util.disk add a helper to remove partitions

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 870f9e5b48a5fe13e213be86ee8947ca7207c3bf)

ceph-volume util.disk include partitions as top level devices in get_devices

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 7f3c3598491e556ec1b93ea0a54f52144202fab6)

ceph-volume util.disk fix issue when capturing partition information

Before, the `if` condition meant that it would only work when the output
was '1', which is incorrect as that would only happen if a partition was
the first one, ignoring any other partition. The contents of that file
is the partition number, not a boolean to tell if it is a partition or
not.

It now includes the `holders` file contents which is needed for
dm-mapper work

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 28cb13a970909dc8efe508aee18f47e0fb3fdde3)

ceph-volume lvm.zap split zapping in methods per device type

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 0d2a9a40d2c301039d571101d1e92a82058a51e0)

ceph-volume: add test for disk.udevadm_property

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 58316e3f826f8bb91b375589aa3676b664ce3984)

ceph-volume: add device_id field to inventory output

This intends to mimic the C++ implementation in src/common/blkdev.cc.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 41da5e9a4334628db09e3166eaceff79b9b6c7e0)

Merge pull request #24914 from smithfarm/wip-36429-luminous

luminous: tests: rbd: move OpenStack devstack test to rocky release

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24917 from smithfarm/wip-36554-luminous

luminous: rbd: [rbd-mirror] periodic mirror status timer might fail to be scheduled

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #24975 from dillaman/wip-36568-luminous

luminous: qa/tasks/qemu: use unique clone directory to avoid race with workunit

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #25006 from dillaman/wip-36407-luminous

luminous: rbd: pybind: added missing RBD_FLAG_FAST_DIFF_INVALID constant

Reviewed-by: Mykola Golub <mgolub@mirantis.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #25253 from pdvian/wip-37364-luminous

luminous:librbd: fix missing unblock_writes if shrink is not allowed

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rgw: batch and process bucket instances

Sort through and batch bucket instances so that multiple calls to reading
current bucket info and locking can be avoided. For the most trivial case when
the bucket is already deleted we exit early with all the stale instances. When
the bucket reshard is in progress we only process the stale entries with status
done, if the bucket is available for locking then we lock down and mark the
other instances as well.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit fb9c0497621fce34283094cfe260480f9122573f)

 Conflicts:
src/rgw/rgw_bucket.cc
Get rid of the following c++17isms:
- split_tenant auto return type -> trailing return type
- tuple destructuring bind for split tenant with std::tie

cli: use rgw-admin reshard stale-instances list/delete

Add a delete command as well that clears the resharded instances. We print out
the json status to indicate success or error state

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit bf8f885016a588130f05c25fdef3d3b6e1216c40)

 Conflicts:
src/rgw/rgw_admin.cc:
  ceph::util::match_str and the related rm/delete normalizations introduced in
  mimic are not backported

rgw: implement listing stale instances from a reshard

Dynamic resharding used to leave behind stale bucket instances; walk through the
metadata pool and identify these instances by comparing the reshard status. If
the reshard status is done, these instances are ok to be cleared. For reshard
status of none we compare against the bucket entry point to ensure that we don't
match the current entry point.

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 0c35a6fab250a817b2752f319b5c9f1da13bb3df)

rgw: recover from incomplete reshard attempt

In case a reshard attempt is left in an incomplete state, i.e., flags
still show resharding even though the bucket reshard lock isn't being
held, try to recover by taking the bucket reshard lock and clearing
flags associated with resharding.

This change requires access to an RGWBucketInfo object. So call stack
into this function should provide that to prevent unnecessary
work. Changes were made to provide this object.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 4891ae59314041802da0f6dc249ccbeb761616dc)

rgw: move RGWReshardBucket lock to its own separate class

There are other processes beyond resharding that would need to take a
bucket reshard lock (e.g., correcting bucet resharding flags in event
of crash, tools to remove bucket shard information from earlier
versions of ceph). Pulling this logic outside of RGWReshardBucket
allows this code to be re-used.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 18ab99cd542cc253d62719a18a9d13e7a312eb15)

rgw: failed resharding clears resharding status from shard heads

Previously, when resharding failed, we restored the shard status on
the bucket info object. However the status on each of the shards was
left indicating a reshard was underway. This prevented some write
operations from taking place, as they would wait for resharding to
complete. This adds the missing functionality. It also makes the
functionality available to other classes via static functions in
RGWBucketReshard.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 4577801271454b147cb3b1f3169d268d1c14948b)

rgw: change the bucket reshard lock to exclusive-ephemeral

The bucket reshard lock was simply an exclusive lock that existed on
an object solely for the purpose of representing the lock. This is now
changed to exclusvie-ephemeral lock, so as not to leave these objects
behind.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit bc0a5ff95247b41b5a8336a8c4ed83c4315c1966)

cls: add exclusive ephemeral locks that auto-clean

Add a new type of cls lock -- exclusive ephemeral for which the
object only exists to represent the lock and for which the object
should be deleted at unlock. This is to prevent the accumulation of
unneeded objects in the cluster by automatically cleaning them up.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit a289f2d8654cf4b865430465b87299d3618b41c8)

rgw: renew resharding locks to prevent expiration

Fix lock expiration problem with resharding. The resharding process
will renew its bucket lock (and logshard lock if necessary) when half
the remaining time is left on the lock. If the lock is expired and
cannot renew the process fails and errors out appropriately.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 8cebffa1d8ad4df6fdae4e10e782aad0753545ce)

cls: add semantics for cls locks to require renewal without expiring

Add ability to *require* renewal of an existing lock in addition
toexisting ability to *allow* renewal of an existing lock. The key
difference is that a MUST_RENEW will fail if the lock has expired
(where a MAY_RENEW) will succeed. This provides calling code with the
ability to verify that a lock is held continually and that it was
never lost/expired.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 479c90993cc80b140af010a72ff9a34ecb7b3b32)