Merge pull request #17711 from batrick/bp20594

luminous: mds: support limiting cache by memory

Reviewed-By: Yan Zheng <zyan@redhat.com>

Merge pull request #17714 from batrick/bp21222

luminous: MDS: standby-replay mds should avoid initiating subtree export

Reviewed-By: Yan, Zheng <zyan@redhat.com>

Merge pull request #17570 from theanalyst/wip-luminous-21233

luminous: rgw: memory leak in MetadataHandlers

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17642 from theanalyst/wip-21350-luminous

luminous: rgw: data encryption sometimes fails to follow AWS settings

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17674 from theanalyst/luminous-rgw-reshard-enable

luminous: rgw: reshard enable

Reviewed-By: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17479 from smithfarm/wip-21139-luminous

luminous: rgw: put lifecycle configuration fails if Prefix is not set

Reviewed-By: Daniel Gryniewicz <dang@redhat.com>

rgw: fix resource leak in rgw_bucket.cc and rgw_user.cc

Fixes: http://tracker.ceph.com/issues/21214
Signed-off-by: Luo Kexue <luo.kexue@zte.com.cn>
(cherry picked from commit 1f23976adab74e25ce06941692a881521885c6ee)

rgw: add missing override in list_keys_init()

Added missing override in list_keys_init().

Signed-off-by: Jos Collin <jcollin@redhat.com>
(cherry picked from commit bfc999f1db96293e6a2188922074031fa0b40ecc)

Merge pull request #17612 from smithfarm/wip-qemu-cleanup

build/ops: rpm: do not create librbd.so.1 symlink in /usr/lib64/qemu

Reviewed-By: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #17478 from smithfarm/wip-21138-luminous

luminous: rgw: object copied from remote src acl permission become full-control issue

Reviewed-By: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17445 from smithfarm/wip-21110-luminous

luminous: rgw: send data-log list infinitely

Reviewed-By: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17488 from smithfarm/wip-21241-luminous

luminous: rgw: usage of --inconsistent-index should require user confirmation and print a warning

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17484 from smithfarm/wip-21184-luminous

luminous: tests: NameError: global name 'name' is not defined

Reviewed-By: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17474 from smithfarm/wip-21114-luminous

luminous: cephfs: tests: FS_DEGRADED spurious health warnings in some sub-suites

Reviewed-By: Patrick Donelly <pdonelly@redhat.com>

Merge pull request #17471 from smithfarm/wip-21107-luminous

luminous: cephfs: client/mds has wrong check to clear S_ISGID on chown

Reviewed-By: Patrick Donelly <pdonelly@redhat.com>

ceph: fix "stopping mds can't export subtree"

caused by commit
790c34138f "Make standby-replay mds avoid initiating subtree export"

Fixes: http://tracker.ceph.com/issues/21378
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit c294051987de1ff79540060a0563fa02c8f03f91)

Make standby-replay mds not migrate empty subtree when trimming cache

Signed-off-by: Jianyu Li <joannyli@tencent.com>
(cherry picked from commit 178ca58fec2d820f61724d223addac1c4fa67086)

Make standby-replay mds avoid initiating subtree export

Signed-off-by: Jianyu Li <joannyli@tencent.com>
(cherry picked from commit 790c34138fbdf995c72e558057b9e64c882b522b)

mds: optimize MDCache::rejoin_scour_survivor_replicas()

avoid iterating dentries if dirfrag is non-auth

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit d32a2376047bc1363b9f7a2ed09060ec84c78b97)

mds: fix MDSCacheObject::clear_replica_map

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit fd447405bef347da80ed5611331b211021b1628b)

mds: support limiting cache by memory

This introduces two config parameters:

    mds_cache_memory_limit: Sets the soft maximum of the cache to the given
    byte count. (Like mds_cache_size, this doesn't actually limit the maximum
    size of the cache. It just dictates the steady-state size.)

    mds_cache_reservation: This replaces mds_health_cache_threshold everywhere
    except the Beacon heartbeat sent to the mons. The idea here is to specify a
    reservation of memory (5% by default) for operations and the MDS tries to
    always maintain that reservation. So, the MDS will recall caps from clients
    when it begins dipping into its reservation of memory.

mds_cache_size still limits the cache by Inode count but is now by-default 0
(i.e. unlimited). The new preferred way of specifying cache limits is by memory
size. The default is 1GB.

Fixes: http://tracker.ceph.com/issues/20594
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1464976
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 06c94de584e6cd7d347bcdfb79d9fef4fed0d277)

 Conflicts:
PendingReleaseNotes
src/mds/MDCache.cc

common: refactor of lru

Avoids an unnecessary "max" size of the LRU which was used to calculate the
midpoint. Instead, just dynamically move the LRUObjects between top and bottom
on-the-fly.

This change is necessary for a cache which which does not limit by the number
of objects but by some other metric. (In this case, memory.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 12d615b3c546211b48ee75921f0e06371dd62dae)

mds: resolve unsigned coercion compiler warning

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0c2032c2876c3e2cd0d23ab883fc72375731cadf)

common: use safer uint64_t for list size

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0ddd260a32e781ff6bfa49c1a01d6b2d80e5dcd3)

common: add bytes2str pretty print function

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 7fff24e10e9cacc94ec320b87e60c7735ff028b6)

mds: check if waiting is allocated before use

This prevents accidental allocation of the map.

Also, privatize the variable to protect from this in child classes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 055020ce80b1f08d258b4c023bf5465ed7a46034)

mds: go back to compact_map for replicas

Zheng observed that an alloc_ptr doesn't really work in this case since any
call to get_replicas() will cause the map to be allocated, nullifying the
benefit. Use a compact_map until a better solution can be written. (This means
that the map will be allocated outside the mempool.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5d67b5cc57ac803e60c2921f554cd0507ed4848e)

mds: use mempool for cache objects

The purpose of this is to allow us to track memory usage by cached objects so
we can limit cache size based on memory available/allocated to the MDS.

This commit is a first step: it adds CInode, CDir, and CDentry to the mempool
but not all of the containers in these classes (e.g. std::map). However,
MDSCacheObject has been changed to allocate its containers through the mempool
by converting compact_* containers to the std versions offered through mempool
via the new alloc_ptr.

(A compact_* class simply wraps a pointer to the std:: version to reduce memory
usage of an object when the container is only occasionally used. The alloc_ptr
allows us to achieve the same thing explicitly with only a little handholding:
when all entries in the wrapped container are deleted, the caller must call
alloc_ptr.release().)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit e035b64fcb0482c3318656e1680d683814f494fe)

mds: cleanup replica_map access

The gymnastics protecting the map failed as the code evolved. Just expose it
normally with a getter.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit d1b6cadd6c1514aeacfbacec253305e1a715c387)

common: add alloc_ptr smart pointer

This ptr is like a unique_ptr except it allocates the underlying object on
access. The idea being that we can save memory if the object is only needed
sometimes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5fa557d2713558038af7579de8a3ca56d58b8d90)

common: add warning on base class use of mempool

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit c0d0fa804ee0fa775ae18f9b7bde2317bb6af52d)

common: use atomic uin64_t for counter

Making this interface thread-safe...

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 59b5931a2f47f90fced779fb89e06868c739c688)

Merge pull request #17646 from theanalyst/wip-luminous-multifixes

luminous: rgw fixes for multisite tests in luminous

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17647 from dillaman/wip-21345

luminous: qa/workunits/rbd: relax greps to support upgrade formatting change

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

rgw: LCFilter_S3::to_xml the second "prefix" missed "/"

Signed-off-by: Shasha Lu <lu.shasha@eisoo.com>
(cherry picked from commit dcef8555e1bf2656aeb5316c4b78f1facf2ae172)

test/rgw: fix test_encrypted_object_sync for 3+ zones

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8762c4b29051b2b49bfd8c1e0ceaa1aee6417a74)

luminous: rgw: re enable dynamic resharding

Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #17653 from ceph/backport-wip-bz1478598

luminous: ceph-volume allow filtering by `uuid`, do not require osd id

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #17485 from smithfarm/wip-21185-luminous

luminous: rgw: rgw_file:  incorrect lane lock behavior in evict_block()

Reviewed-By: Orit Wasserman <oritwas@redhat.com>

Merge pull request #17476 from smithfarm/wip-21135-luminous

luminous: rgw: bucket index sporadically reshards to 65521 shards

Reviewed-By: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17470 from smithfarm/wip-21104-luminous

luminous: build/ops: ceph-fuse RPM should require fusermount

Reviewed-By: Ken Dreyer <ktdreyer@redhat.com>
Reviewed-By: Kefu Chai <kchai@redhat.com>

Merge pull request #17477 from smithfarm/wip-21137-luminous

luminous: mgr: 500 error when attempting to view filesystem data

Reviewed-By: John Spray <jcsp@redhat.com>

Merge pull request #17495 from smithfarm/wip-21235-luminous

luminous: tests: thrashosds read error injection doesn't take live_osds into account

Merge pull request #17483 from smithfarm/wip-21183-luminous

luminous: mgr: Crash in MonCommandCompletion

Merge pull request #17486 from smithfarm/wip-21188-luminous

luminous: pybind: dashboard usage graph getting bigger and bigger

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17350 from liewegas/wip-pr-17334-luminous

os/bluestore: reshard-fix wrong shard length

Reviewed-by: Amit Kumar <amitkuma@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17473 from smithfarm/wip-21112-luminous

luminous: cephfs: get_quota_root sends lookupname op for every buffered write

Reviewed-By: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17465 from smithfarm/wip-21045-luminous

luminous: rbd: TestMirroringWatcher.ModeUpdated: periodic failure due to injected message failures

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

Merge pull request #17565 from theanalyst/wip-21278-luminous

luminous: the standbys are not updated via ceph tell mds.* command

Reviewed-By: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17566 from theanalyst/wip-21267-luminous

luminous: Incorrect grammar in FS message 1 filesystem is have a failed mds daemon

Reviewed-By: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #17557 from theanalyst/wip-21269-luminous

luminous: some generic options can not be passed by rbd-nbd

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

Merge pull request #17448 from smithfarm/wip-21118-luminous

luminous: rgw: need to stream metadata full sync init

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17464 from smithfarm/wip-20964-luminous

luminous: rbd: switch to new config option getter methods

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

ceph-volume lvm.trigger just split the osd_id once

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit ecd5af88da2815603283742c661d27e2a48f9f7e)

ceph-volume tests catch improper parsing of osd uuid

When the osd ID existed in the UUID in the same format (e.g. '1-') the
parsing would break returning a bogus UUID that is impossible to find

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 5392697121ef150f0edac05df14d116cf0148593)

tests: remove 'ceph-' prefix from test node hostnames

This updates our Vagrantfile to match the changes in ceph-ansible
introduced by https://github.com/ceph/ceph-ansible/commit/298a63c43

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit f41d359ebaec8762ed1a7b8cfd85c23f351a5070)

ceph-volume lvm.api PVolumes should filter by all matching tags, not just one

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit b8613c1d7bf56c49d1b7f1c41a37073b8739bbc3)

ceph-volume tests pv API should also match all pv_tags

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 120fd6a196045c8511c3eaafec37de43003bf450)

ceph-volume tests prove all volume tags have to match in vgs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit fe2a7106decb586e83f5b60d6c18c454afe4883e)

ceph-volume lvm.api all vg tags should match, not just one

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 6d679855aa077e2de3f20316625d478a3697c78f)

ceph-volume tests prove all tags have to match in lvs

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 80e401ca970066541ae7255be1caf42883048df4)

ceph-volume lvm.api all lv tags should match, not just one

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dbaf30c3e4d921df2a92812383c8805a77eb8f5b)

ceph-volume lvm.activate allow not using osd_id

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 99a8a719695591558da29b3d16a347d5bff8bf3d)

ceph-volume tests prove missing args.osd_id is fine

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit 8d1a1a885fd65073308c205949bc70b41805f92f)

ceph-volume tests add is_root and lv fixtures to conftest

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit d18d5e139be82de36e1a60340525adbf79fdedd9)

Merge pull request #17446 from smithfarm/wip-21115-luminous

luminous: rgw multisite: objects encrypted with SSE-KMS are stored unencrypted in target zone

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17468 from smithfarm/wip-21052-luminous

luminous: build/ops: RHEL 7.3 Selinux denials at OSD start

Reviewed-By: Ken Dreyer <ktdreyer@redhat.com>

Merge pull request #17444 from smithfarm/wip-20968-luminous

luminous: rgw: bytes_send and bytes_recv in the msg of usage show returning is 0 in master branch

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #17555 from theanalyst/wip-21270-luminous

luminous: rgw: shadow objects are sometimes not removed

Reviewed-By: Orit Wasserman <owasserm@redhat.com>

Merge pull request #17558 from theanalyst/wip-21277-luminous

luminous: [cls] metadata_list API function does not honor `max_return` parameter.

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

Merge pull request #17556 from theanalyst/wip-21265-luminous

luminous: [cli] rename of non-existent image results in seg fault

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

Merge pull request #17560 from theanalyst/wip-21288-luminous

luminous: [test] various teuthology errors

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

Merge pull request #17561 from theanalyst/wip-21289-luminous

luminous: [rbd] image-meta list does not return all entries

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

qa/workunits/rbd: relax greps to support upgrade formatting change

Fixes: http://tracker.ceph.com/issues/21181
Signed-off-by: Jason Dillaman <dillaman@redhat.com>
(cherry picked from commit 273c84578b06905229a7c6eae7b1a12bbe60de95)

Conflicts:
qa/workunits/rbd/import_export.sh: trivial resolution

Merge pull request #17336 from tchaikov/wip-luminous-pr-16774

luminous: mgr/dashboard: fix rbd's pool sub menu

Reviewed-By: Jason Dillaman <jdillaman@redhat.com>

rgw: Data encryption is not follow the AWS agreement

Encryption request headers should not be sent for GET requests and HEAD
requests if your object uses SSE-KMS/SSE-S3 or you’ll get an HTTP 400
BadRequest error.

Signed-off-by: hechuang <hechuang@xsky.com>
(cherry picked from commit dfe1a7c1e6c470d623c70d83c32cc5b7a921bbc5)

Merge pull request #17622 from liewegas/wip-more-lum-pi

osd: update info only if new_interval

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17564 from theanalyst/wip-21283-luminous

qa/objectstore/bluestore*: less debug output

osd: update info only if new_interval

refactor OSD::build_initial_pg_history() so we update the info only if a
new interval is identified. also, this is consistent with
OSD::build_past_intervals_parallel().

Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit 4ae053439e526fab0451d1502576dce89eb6b44f)

Merge pull request #17496 from smithfarm/wip-21236-luminous

luminous: osd: build_initial_pg_history doesn't update up/acting/etc

Merge pull request #17601 from liewegas/wip-21171-luminous

os/bluestore: fix deferred write deadlock, aio short return handling

build/ops: rpm: do not create librbd.so.1 symlink in /usr/lib64/qemu

As reported by Ademar de Souza Reis Jr <areis@redhat.com>:

This symlink should not be necessary anymore. QEMU is properly linked to
ceph/librbd these days (look at the qemu-block-rbd sub-package in Fedora and
the respective package in RHEL).

The symlink was a hack from a time when librbd was distributed and supported by
ceph and the rbd driver would be enabled at runtime only when/if the symlink
was present.

So the right fix is to actually get rid of the symlink altogether and never
touch (much less own) /usr/lib64/qemu.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(cherry picked from commit be4d6b1a74a0612621c3ad2cedebaa1fa40ffbab)

Merge pull request #17551 from dillaman/wip-21280

luminous: test/cls_rbd: mask newer feature bits to support upgrade tests

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #17467 from smithfarm/wip-21046-luminous

luminous: common: adding line break at end of some cli results

Merge pull request #17472 from smithfarm/wip-21108-luminous

luminous: mon: OSDMonitor: osd pool application get support

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17475 from smithfarm/wip-21133-luminous

luminous: osd: PrimaryLogPG: sparse read won't trigger repair correctly

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17481 from smithfarm/wip-21182-luminous

luminous: core: osd crush rule rename not idempotent

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17494 from smithfarm/wip-21234-luminous

luminous: bluestore: asyn cdeferred_try_submit deadlock

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17469 from smithfarm/wip-21103-luminous

luminous: cephfs: client: missing space in some client debug log messages

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #17562 from theanalyst/wip-21276-luminous

luminous: os/bluestore/BlueFS.cc: 1255: FAILED assert(!log_file->fnode.extents.empty())

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17345 from liewegas/wip-pr-17311-luminous

mon: use ceph_clock_now if message is self-generated

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #17344 from liewegas/wip-pr-17330-luminous

mon/OSDMonitor: make 'osd crush class rename' idempotent

Merge pull request #17527 from liewegas/wip-pr-17510-luminous

crush: fix fast rule lookup when uniform

Merge pull request #17511 from liewegas/wip-stupid-luminous

os/bluestore: default to stupid allocator

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #17501 from smithfarm/wip-21242-luminous

luminous: osd: osd/PG: discard msgs from down peers

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17500 from smithfarm/wip-21240-luminous

luminous: mon: rate limit on health check update logging

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17498 from smithfarm/wip-21238-luminous

luminous: osd: subscribe to new osdmap while waiting_for_healthy

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17497 from smithfarm/wip-21237-luminous

luminous: bluestore: osd crash when change option bluestore_csum_type from none to CRC32

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #17378 from jan--f/wip-21187-luminous

luminous: core: osd/PGLog: write only changed dup entries

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

os/bluestore/aio: handle short return from io_submit

io_submit may return a value less than nr, indicating that only some of
the provided iocbs were queued.  If that happens we should loop, not
return and silently drop those aios on the floor.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit dc17dfd9ff05b5676488c2b1bca53026b2ca6244)