rbd-mirror: ignore empty snapshot sequence mapping

This invalid condition will be gracefully detected and handled
when the snapshot mappings are computed.

Fixes: http://tracker.ceph.com/issues/16329
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 58ed8a18e91401333bc3f3f957ce5d715b6687b6)

qa/workunits/rbd: remove temporary image exports

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 45498d04990861a034e2aad69a93658e018c991a)

rbd-mirror: fix potential image replayer state transition race

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 3e224c6c03a6a5de0e179bd788387cfa3ff49e9d)

rbd-mirror: cancel image deletion callback on shut down

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c3f1cb3a34e1b4d258877d519e683e25bf65c65a)

rbd-mirror: fixed potential leaking image deletion context callback

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 9d1cd90c220f95d23d2694b4f1fb6b2fed073ced)

rbd-mirror: ImageReplayer doesn't need image deleted callback

The Replayer will detect that the ImageReplayer is stopped and will
schedule the "wait for deleted" callback before attempting to
restart the ImageReplayer.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 7e5afc71dfd3ec3e7b0ddaca96f92ac6e0414006)

rbd-mirror: use async callback when deletion not in-progress

Fixes: http://tracker.ceph.com/issues/16491
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit c136f443366fe804057a9c1f8c01c3f0a33a4d63)

rbd-mirror: tests: ImageSyncThrottler unit tests

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 8ca9a84f36efa73ad17fd27f545dff3716cd798c)

rbd-mirror: Usage of image-sync throttler in BootstrapRequest

Fixes: http://tracker.ceph.com/issues/15239
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 6a91146255d84229688bd8b378732be5975a778b)

rbd-mirror: Implementation of image-sync throttler

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit c4f926d6980d1efd95771885a11d8cc4ebd2e4c3)

rbd-mirror: tests: Support for inflight image sync point update

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit cee543bd96bc99f219024aedbfafc5dcd99abb5a)

rbd-mirror: image-sync: Periodically update sync point object number

Fixes: http://tracker.ceph.com/issues/15108
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit c2eedf4d27b73d7eaf3fda54f9b0ee74e455bc1d)

rbd-mirror: image-replayer: Fix bug in resync listener remotion

Fixes: http://tracker.ceph.com/issues/16488
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 11a5851e90bca2c1813156bf12dda0192965db37)

rbd-mirror: resync: Added unit tests

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 8953825956f2a560d17c7b651d1724a95f2f7d1d)

rbd-mirror: image-replayer: Implementation of resync operation

Fixes: http://tracker.ceph.com/issues/15670
Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit f2b114238103315197a9eefc8490786c484c8520)

rbd: journal: Support for listening updates on client metadata

Currently we only support listening for image resync requests.

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit 0dd85739e9034912b86250ced2834dd7fb3d92fd)

journal: Support for registering metadata listeners in the Journaler

Signed-off-by: Ricardo Dias <rdias@suse.com>
(cherry picked from commit c535eb0399cd649371db233ee1ed01cf5441a4b3)

Merge pull request #9790 from SUSE/wip-16381-jewel

jewel: comparing return code to ERR_NOT_MODIFIED in rgw_rest_s3.cc (needs minus sign)
Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Merge pull request #10026 from SUSE/wip-16392-jewel

jewel: master: build failures with boost > 1.58

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10625 from dachary/wip-16969-jewel

jewel: src/script/subman fails with KeyError: 'nband'

Reviewed-by: Kefu Chai <kchai@redhat.com>

subman: use replace instead of format

Otherwise all {} are assumed to be substituted. Add a test.

Fixes: http://tracker.ceph.com/issues/16961
Signed-off-by: Loic Dachary <ldachary@redhat.com>
(cherry picked from commit b006c136761746e12704e349a672325b8b6e51f5)

cmake: script that sets env vars for unit tests

Signed-off-by: Ali Maredia <amaredia@redhat.com>
(cherry picked from commit 15a669435aaa78562746f6728bb185b1e99a4274)

Merge pull request #10054 from dachary/wip-16484-jewel

jewel: ExclusiveLock object leaked when switching to snapshot

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10041 from dachary/wip-16315-jewel

jewel: When journaling is enabled, a flush request shouldn't flush the cache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #10055 from dachary/wip-16485-jewel

jewel: Whitelist EBUSY error from snap unprotect for journal replay

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10053 from dachary/wip-16483-jewel

jewel: Close journal and object map before flagging exclusive lock as released

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10052 from dachary/wip-16482-jewel

jewel: Timeout sending mirroring notification shouldn't result in failure

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10051 from dachary/wip-16460-jewel

jewel: Crash when utilizing advisory locking API functions

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10050 from dachary/wip-16459-jewel

jewel: rbd-mirror should disable proxied maintenance ops for non-primary image

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10047 from dachary/wip-16426-jewel

jewel: Possible race condition during journal transition from replay to ready

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10046 from dachary/wip-16425-jewel

jewel: rbd-mirror: potential race condition accessing local image journal

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10045 from dachary/wip-16424-jewel

jewel: Journal needs to handle duplicate maintenance op tids

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10044 from dachary/wip-16423-jewel

jewel: Journal duplicate op detection can cause lockdep error

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10043 from dachary/wip-16371-jewel

jewel: rbd-mirror: ensure replay status formatter has completed before stopping replay

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10042 from dachary/wip-16372-jewel

jewel: Unable to disable journaling feature if in unexpected mirror state

Merge pull request #10010 from dachary/wip-16486-jewel

jewel: Object map/fast-diff invalidated if journal replays the same snap remove event

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #10009 from dachary/wip-16514-jewel

jewel: Image removal doesn't necessarily clean up all rbd_mirroring entries

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #9952 from dillaman/wip-fix-use-after-free-jewel

jewel: librbd: potential use after free on refresh error

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9752 from dillaman/wip-fix-task-finisher-jewel

jewel: librbd: cancel all tasks should wait until finisher is done

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #10561 from dzafman/wip-16672-jewel

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9562 from Abhishekvrshny/wip-16152-jewel

jewel: client: fstat cap release

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9561 from Abhishekvrshny/wip-16136-jewel

jewel: MDSMonitor fixes

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9557 from Abhishekvrshny/wip-15898-jewel

jewel: Confusing MDS log message when shut down with stalled journaler reads

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9560 from Abhishekvrshny/wip-16135-jewel

jewel: MDS: fix getattr starve setattr

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9559 from Abhishekvrshny/wip-16041-jewel

jewel: mds/StrayManager.cc: 520: FAILED assert(dnl->is_primary())

Reviewed-by: John Spray <john.spray@redhat.com>

Merge pull request #9996 from dachary/wip-16437-jewel

jewel: async messenger mon crash

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #9997 from dachary/wip-16431-jewel

jewel: librados,osd: bad flags can crash the osd

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9998 from dachary/wip-16429-jewel

jewel: OSDMonitor: drop pg temps from not the current primary

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #10001 from dachary/wip-16427-jewel

jewel: prepare_pgtemp needs to only update up_thru if newer than the existing one

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #9743 from vumrao/wip-vumrao-16339

jewel : rgw: support size suffixes for --max-size in radosgw-admin command

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #10007 from dachary/wip-15806-jewel

jewel: New pools have bogus stuck inactive/unclean HEALTH_ERR messages until they are first active and clean

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #10006 from dachary/wip-16249-jewel

jewel: sparse_read on ec pool should return extends with correct offset

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #9740 from vumrao/wip-vumrao-16338

jewel : rados: Add cleanup message with time to rados bench output

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #10004 from dachary/wip-16374-jewel

jewel: AsyncConnection::lockmsg/async lockdep cycle: AsyncMessenger::lock, MDSDaemon::mds_lock, AsyncConnection::lock

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10003 from dachary/wip-16380-jewel

jewel: msg/async: connection race hang

Reviewed-by: Haomai Wang <haomai@xsky.com>

Merge pull request #10036 from liewegas/wip-16297-jewel

mon: Monitor: validate prefix on handle_command()

Reviewed-by: Loic Dachary <ldachary@redhat.com>

Merge pull request #9547 from jcsp/wip-jewel-15705

jewel backport: mds: fix mdsmap print_summary with standby replays

Reviewed-by: Loic Dachary <ldachary@redhat.com>

os: Fix HashIndex::recursive_remove() to remove everything but original path

Fixes: http://tracker.ceph.com/issues/16672
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ba88a3aac0d3b620986e32ed718237513d6592f6)

filestore: Clear objects before calling collection_list() again

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 892303cd02b64b622669183c2acffcdd7a8f3547)

filestore: Improve logging

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit b78c457785553e3a6e4f901bfebb41fc9c4d9bd5)

Merge pull request #10293 from theanalyst/wip-16589-jewel

jewel: multisite sync races with deletes

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

Merge pull request #10453 from ajarr/wip-manila-backports-jewel

jewel: essential backports for OpenStack Manila

Reviewed-by: John Spray <john.spray@redhat.com>

ceph_volume_client: version on-disk metadata

Version on-disk metadata with two attributes,
'compat version', the minimum CephFSVolume Client
version that can decode the metadata, and
'version', the version that encoded the metadata.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 1c1d65a45f4574ca5f33cc9d949089a5c956e363)

ceph_volume_client: add versioning

Add class attributes to CephFSVolumeClient to version
its capabilities.

'version' attribute stores the current version number
of CephFSVolumeClient.

'compat_version' attribute stores the earliest version
number of CephFSVolumeClient that the current version is
compatible with.

Fixes: http://tracker.ceph.com/issues/15406
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 46876fb2ceb22082c0a1703fe77ad1694b508ad8)

ceph_volume_client: disallow tenants to share auth IDs

Restrict an auth ID to a single OpenStack tenant to enforce
strong tenant isolation of shares.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 82445a20a258a4c8800f273dc5f2484aace0e413)

ceph_volume_client: cleanup auth meta files

Remove auth meta files on last rule for an auth ID deletion

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit ec2e6e37d01c961d269d83661d0b95ada6a8449e)

ceph_volume_client: fix log messages

Log the path of the volume during creation and deletion of volumes.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 7731287761f91e1fdee0e6306d9ecf9b04ad363c)

ceph_volume_client: create/delete VMeta for create/delete volume

Create and delete volume meta files during creation and deletion of
volumes.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 37fbfc7aa8f10d37f5202603a906425507271ff5)

ceph_volume_client: modify locking of meta files

File locks are applied on meta files before updating the meta
file contents. These meta files would need to be cleaned up
sometime, which could lead to locks being held on unlinked meta
files. Prevent this by checking whether the file had been deleted
after lock was acquired on it.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit f7c037229bcf3f5a3d06897ec7fe0c5419dd7143)

cephfs.pyx: implement python bindings for fstat

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit f58403f3d19e22edeb8f91b6f87a0b7947b0ff21)

ceph_volume_client: restrict volume group names

Prevent craftily-named volume groups from colliding with meta files.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 7f7d2a76ae9b556c1de418f0eab8461c538f91d9)

ceph_volume_client: use fsync instead of syncfs

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 27eb51baab4cda6b385aef53fc7c3962a9debae5)

pybind: expose fsync in cephfs binding.

So we don't necessary to syncfs when want to persistent
some file.

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
(cherry picked from commit 1c952fbaf0fd393ef2dcb83a3db721a077b4274e)

ceph_volume_client: recover from dirty auth and auth meta updates

Check dirty flag after locking something and call recover() if we are
opening something dirty (racing with another instance of the driver
restarting after failure) -- only required if someone running multiple
manila-share instances with Ceph loaded.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 647a2447f0c4354dc21d1083043591d2b6f6f94f)

ceph_volume_client: modify data layout in meta files

Notable changes to data layout in auth meta and volume meta files:

In the auth meta files, add a 'dirty' flag to track the status of auth
updates to a single volume.

In the volume meta file, make the 'dirty' flag track the status of
auth updates for a single ID.

Optimize the recovery of partial auth update changes to auth meta,
volume meta, and the Ceph backend, facilitated by changes in the
data layout in the meta files.

Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 7c8a28a7e817d030a4d419b0cc627c30c1064270)

pybind: ceph_volume_client authentication metadata

Store a two-way mapping between auth IDs and volumes.

Enables us to record some metadata on auth ids (which
openstack tenant created it) so that we can avoid exposing
keys to other tenants who try to use the same ceph
auth id.

Enables us to expose the list of which auth ids have access
to a volume, so that Manila's update_access() can be
implemented efficiently.

DNM: see TODOs inline.

Fixes: http://tracker.ceph.com/issues/15615
Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit d2e9eb55ca6ed5daa094cf323faf143615b9380b)

pybind: enable integer flags to libcephfs open

The 'rw+' style flags are handy and convenient, but
they don't capture all possibilities.  Change to
optionally accept an integer here for advance users
who want to specify arbitrary combinations of
flags.

Signed-off-by: John Spray <john.spray@redhat.com>
(cherry picked from commit 5678584f4176d07301acd7f57acc4efd7fb20e43)

ceph_volume_client: allow read-only authorization for volumes

Allow clients to be restricted to read-only mount of the volume
by restricting their ceph auth ID's MDS and OSD caps to read-only.

Fixes: http://tracker.ceph.com/issues/15614
Signed-off-by: Ramana Raja <rraja@redhat.com>
(cherry picked from commit 011ea5e7fb35ee07848e0c3abac24702a778ad63)

Merge pull request #10373 from ceph/jewel-mds-snap-failover

Jewel mds snap failover

Reviewed-by: John Spray <john.spray@redhat.com>

client: fix MetaRequest::set_other_inode()

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 265f96bda7e106883063f074133450ce39ee262c)

client: close directory's snapdir when deleting directory

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 3099cabd11a16d22c6707631861bef0aa176ed02)

client: invalidate snap inodes after removing snapshot

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit f180ad149ab510626fcd6cbd8221f550f9858126)

mds: fix incorrect "unconnected snaprealm xxx" warning

If a snaprealm has no child/parent snaprelam, and the snaprealm inode
is not in the cache while client reconnects. The snaprealm does not
get properly removed from MDCache::reconnected_snaplrealm. This causes
incorrect "unconnected snaprealm xxx" warning

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 85e687de876ca7e0174734ba81130949c4ab6a40)

qa/workunits/fs: fix expect_failure function in test scripts

The origin expect_failure function return 0 regardness of command's
return value.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 16f7d7c71e65cd68cecde3e5b89d189ab5950548)

client: make sure snapflush is sent before normal cap message

MDS does null snapflush when it receives normal cap message. So client
must send snapflush first.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit d3916717e2edc8000400f678fa7134ca1406a074)

client: unify cap flush and snapcap flush

This patch includes following changes
- assign flush tid to snapcap flush
- remove session's flushing_capsnaps list. add inode with snapcap
  flushes to session's flushing_caps list instead.
- when reconnecting to MDS, re-send one inode's snapcap flushes and
  cap flushes at the same time.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit a05e996b2a2c36496abd8538829ac4897392f6eb)

mds: handle partly purged directory

For a snapshoted direcotry whose snaprealm parents are being opened,
MDS does not know if the directory is purgeable. So MDS can't skip
committing dirfrags of the directory. But if the direcotry is purgeale,
some dirfrags could have already been deleted during MDS failover.
Committing them could return -ENOENT.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit bc50e0309280c08c3ca79dfa5514ac3a15f81a23)

mds: do files recovery after processing cap flushes

File recovery may update inode and trigger inode COW. MDS relies on
client caps to setup CInode::client_need_snapflush. But for a given
client, the reconnected caps may not include the flushing caps.
(Before MDS failover, client released and flushed some caps at the
same time. When MDS recovers, client re-send the cap flush and send
cap reconnect to the MDS.) This may cause later snapflush to get
dropped.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit dd98448d3d0bbb7d338f03f7ad1e40f217eebe0d)

mds: combine MDCache::{reconnected_caps,cap_imports_dirty}

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 57067e032e84bbdb69c499aa7ea93ca68082569b)

mds: remove CEPH_LOCK_IFLOCKL from cinode_lock_info

Currently we don't support dirty CEPH_CAP_FLOCK_EXCL

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit cfc3ec17b6f245e6d8a0be4fdf6cfa64d2fb725f)

mds: rebuild the internal states that tracking pending snapflush

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 1b7d198f633a8608b704f82f01a3f4a4a3a4892b)

mds: using cap_reconnect_t to track cap recconect

Previous commit extended cap_reconnect_t to include snapflush
related information. This information are needed in various
places

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 25b6011dcf07c1fc663cbe29ffd119b66545a0ac)

mds: add 'follows' of first pending snapflush to cap reconnect message

This helps the recovering MDS to reconstruct the internal states that
tracking pending snapflush.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit dd3963a878d4bb2fb5992278ccbc9f7633fb8786)

mds: journal snap inodes that need flush when expiring log segment

Treat snap inodes that need flush in the same way as open files.
When MDS recovers, this make sure that journal replay bring snap
inodes that need flush into the cache

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 306153b3d012832bdfa20402077fa60a9a5d626c)

mds: properly update client_snap_caps when splitting snap inode

update the new snap inode's client_snap_caps according to the old
snap inode.

Signed-off-by: Yan, Zheng <zyan@redhat.com>
(cherry picked from commit 4883779935219817c7e391940a025be1679daeb5)

install-deps: Get the latest virtualenv in a temporary folder
to work around a bug in old virtualenv

Signed-off-by: Oleh Prypin <oleh@pryp.in>
(cherry picked from commit 2699b6d89b8c1c193fd86b5233d1ea86458753a0)

Merge remote-tracking branch 'upstream/pull/10298/head' into jewel

* upstream/pull/10298/head:
  doc: fix standby replay config

doc: fix standby replay config

I tried using these settings in tests without success. The correct config names
are prefixed with "mds".

Fixes: http://tracker.ceph.com/issues/16664
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

rgw: add pg_ver to tombstone_cache

a tombstone cache was added to remember the mtime of deleted objects for
use with the HTTP_IF_MODIFIED_SINCE header, but the comparison was still
failing because of a missing pg_ver. added pg_ver to the tombstone cache
so it can be passed with HTTP_DEST_PG_VER

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit adb529f2fb983df671a1db58a0b17862a29762f0)

rgw: add obj tombstone cache

The obj tombstone cache is used in multi-zone environmet to keep
track of removed objects' mtime. This is then used to fetch remote
object only if its newer than the object that was removed, otherwise
we're just fetching ghost of the past.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit eb10214920c23b24edd94ca53d0f36c85404644d)