test: Add option to wait_for_clean() to execute at every sleep

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 70b51362084d2ae5f2a6e9e5856f17452c53e3f3)

osd/PG: fix misused FORCE_RECOVERY[BACKFILL] flags

__set_force_backfill__ should set PG_STATE_FORCED_BACKFILL
instead of PG_STATE_FORCED_RECOVERY.

Fixes: http://tracker.ceph.com/issues/27985
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 60ef742a84d5201b2a6e6c1780741da0b31c49a8)

Conflicts:
src/osd/PG.cc (trivial)

osd: print summary for forced backfill/recovery to debug

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7816ab5a8eb0ea754a0809eb9da915a16ad39e83)

Conflicts:
src/osd/PG.h (pg_id public like master for OSDService::adjust_pg_priorities())

osd/PG: clean up set_force_{recovery,backfill} interface

- update state under pg lock
- clean up PG interface
- log only when we adjust the state

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2482b4cc91579b4ef5748097f1adacaad91510b1)

Conflicts:
src/osd/PG.h (trivial)

Merge pull request #26686 from dzafman/wip-test-fix

luminous: qa/workunits/cephtool/test.sh: Use :- to handle TMPDIR=""

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yuri Weinstein <yweinstei@redhat.com>

Merge pull request #26546 from cbodley/wip-38410-luminous

luminous: rgw: fix cls_bucket_head result order consistency

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #26548 from cbodley/wip-38414-luminous

luminous: rgw: bug in versioning concurrent, list and get have consistency issue

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25825 from pdvian/wip-36434-luminous

luminous: mon: monstore tool rebuild does not generate creating_pgs

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25855 from ashishkumsingh/wip-37825-luminous

luminous: os/bluestore: KernelDevice::read() does the EIO mapping now.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #25956 from xiexingguo/wip-liminous-async-fixes

luminous: msg/async: backport recent messenger fixes

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26042 from trociny/wip-37972-luminous

luminous: msg/msg_types: fix the dencoder of entity_addr_t

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26284 from smithfarm/wip-38140-luminous

luminous: tests: Add hashinfo testing for dump command of ceph-objectstore-tool

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #26501 from k0ste/luminous_backports2

luminous: mgr/balancer: restrict automatic balancing to specific weekdays

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26656 from jan--f/c-v-simple-activate-all-luminous

luminous: ceph-volume: add --all flag to simple activate

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

Merge pull request #26065 from pdvian/wip-37989-luminous

luminous: MDSMonitor: missing osdmon writeable check

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25858 from ashishkumsingh/wip-37823-luminous

luminous: mds: log evicted clients to clog/dbg

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #25979 from pdvian/wip-37908-luminous

luminous: mds: wait shorter intervals if beacon not sent

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26215 from pdvian/wip-38102-luminous

luminous: mds: trim cache after journal flush

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26293 from pdvian/wip-38190-luminous

luminous: mds: broadcast quota message to client when disable quota

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26473 from pdvian/wip-38336-luminous

luminous: mds: fix potential re-evaluate stray dentry in _unlink_local_finish

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

ceph-volume: add --all flag to simple activate

This is intended to behave similarly to the lvm activate --all argument.
When passed, c-v will scan /etc/ceph/osd/ (or the location specified by
CEPH_VOLUME_SIMPLE_JSON_DIR) for json files (glob *.json) and call activate
for each file.  This should greatly ease the take-over of ceph-disk OSDs
with manual commands and deployment tools like DeepSea and ceph-ansible.
Also adds a simple unit test and modifies the
simple/centos7/filestore/activate functional test to use --all.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 05af94c6fca66083107e96e9aa1db87619e77e66)

 Conflicts:
src/ceph-volume/ceph_volume/tests/functional/simple/centos7/filestore/activate/test.yml
        resolved by deleting everything after conflict marker

Merge pull request #25611 from pdvian/wip-37561-luminous

luminous: rgw: rgwgc: process coredump in some special case

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25986 from ashishkumsingh/wip-37937-luminous

luminous: librgw: export multitenancy support

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26310 from pdvian/wip-38193-luminous

luminous: Object can still be deleted even if s3:DeleteObject policy is set

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25738 from pdvian/wip-37648-luminous

luminous: rgw: feature -- log successful bucket resharding events

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #25901 from ashishkumsingh/wip-37559-luminous

luminous: rgw: handle S3 version 2 pre-signed urls with meta-data

Reviewed-by: Matt Benjamin <mbenjami@redhat.com>

Merge pull request #26131 from pdvian/wip-38038-luminous

luminous: rgw: fails to start on Fedora 28 from default configuration

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26192 from cbodley/wip-38079-luminous

luminous: rgw: bucket full sync handles delete markers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26263 from pdvian/wip-38148-luminous

luminous: rgw: "radosgw-admin bucket rm ... --purge-objects" can hang

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #26332 from smithfarm/wip-38232-luminous

luminous: rgw: fix radosgw linkage with WITH_RADOSGW_BEAST_FRONTEND=OFF

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

mgr/balancer: make weekday be compatible with C

The __desc__ field of **begin[end]_weekend** suggests
sunday is the prevailing start of the week, whereas the
Python implementation says no.

See https://stackoverflow.com/questions/21045569/why-is-pythons-weekday-different-from-tm-wday-in-c

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit b9da0925490d4e9e964249c54e7775d9c76b1719)

mgr/balancer: restrict automatic balancing to specific weekdays

We have implemented similar semantics for the scrub behaviour.
Let's try to keep pace with that.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 0412a51ac6dfecc1a72b654196ecb1a599dcc94d)

Conflicts:
- path: src/pybind/mgr/balancer/module.py
  comment: 'get_module_option()' -> 'get_config()' for mgr config-key
    'begin_weekday' and 'begin_weekday' should be int, but
    luminous/mimic config-key always return string.

Merge pull request #26587 from smithfarm/wip-doc-fix-luminous

luminous: doc: Fix default value of 'mon_osd_max_split_count'

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

doc: Fix default value of 'mon_osd_max_split_count'

This commit is not cherry-picked from master because the config
parameter mon_osd_max_split_count has been dropped in master

Signed-off-by: Kai Wagner <kwagner@suse.com>
(cherry picked from commit 17d503f5adfbfd96502877bcb55c42ba1eb34a5c)

Merge pull request #24906 from tchaikov/wip-23673-luminous

luminous: auth: ceph auth add does not sanity-check caps

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25711 from ashishkumsingh/wip-37756-luminous

luminous: osd/PrimaryLogPG: fix the extent length error of the sync read

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #25722 from ashishkumsingh/wip-37690-luminous

luminous: tools: ceph-objectstore-tool: Dump hashinfo

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #25851 from ashishkumsingh/wip-37815-luminous

luminous: qa/workunits/rados/test_health_warnings: prevent out osds

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #25853 from ashishkumsingh/wip-37833-luminous

luminous: objecter: avoid race when reset down osd's session

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #26305 from dzafman/wip-38207

luminous: mon: A PG with PG_STATE_REPAIR doesn't mean damaged data, PG_STATE_IN…

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #26078 from ashishkumsingh/wip-37993-luminous

luminous: osd: keep using cache even if op will invalid cache

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26148 from ashishkumsingh/wip-38046-luminous

luminous: tests: qa/overrides/short_pg_log.yaml: reduce osd_{min,max}_pg_log_entries

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26236 from pdvian/wip-38105-luminous

luminous: osd/PG.cc: account for missing set irrespective of last_complete

Reviewed-by: David Zafman <dzafman@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26261 from pdvian/wip-38142-luminous

luminous: os/bluestore: fixup access a destroy cond cause deadlock or undefine

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

rgw: bug in versioning concurrent, list and get have consistency issue

Fixes: https://tracker.ceph.com/issues/38060
Signed-off-by: Wang Hao <wanghao72@baidu.com>
(cherry picked from commit cc2a96a47fd71dc8399e31f1a9e87d8a13ea3419)

rgw: fix cls_bucket_head result order consistency

headers's key is string but bucket_instance_ids's key is int
looked around and found no one use header's key, so change
header from map to vector

Signed-off-by: Tianshan Qu <tianshan@xsky.com>
(cherry picked from commit 46657a0f72530f50f1c94374a27edb57deb70de0)

Conflicts:
src/rgw/rgw_rados.h: unordered bucket listing

Merge pull request #25720 from ashishkumsingh/wip-36692-luminous

luminous: rbd-mirror: update mirror status when stopping

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #25768 from ashishkumsingh/wip-26913-luminous

luminous: mgr: "balancer execute" only requires read permissions

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #25955 from pdvian/wip-37901-luminous

luminous: [journal] max journal order is incorrectly set at 64

Reviewed-by: Mykola Golub <mgolub@mirantis.com>

Merge pull request #26000 from pdvian/wip-37926-luminous

luminous: common: fix for broken rbdmap parameter parsing

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #26286 from smithfarm/wip-38186-luminous

luminous: tests: krbd discard no longer guarantees zeroing

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>

mds: fix potential re-evaluate stray dentry in _unlink_local_finish

If MDS handles cap release before _unlink_local_finish, then when MDS
processes _unlink_local_finish, eval_stray might be re-entered twice,
so this will cause crashing.

Fixes: http://tracker.ceph.com/issues/38263
Signed-off-by: Zhi Zhang <zhangz.david@outlook.com>
(cherry picked from commit 72c93baac64211e9f02fa01e743451bf5d63d7a6)

Conflicts:
src/mds/MDCache.h : Resolved in notify_stray

test: Add "slow request" to whitelist for Luminous
Redo backport 0e987665fe480530a0d1664392604713828bfa5c

Replaces: 77258ed181dbe04e56da249a71edd3220baf954a

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 0e987665fe480530a0d1664392604713828bfa5c)

Conflicts:
qa/suites/rados/singleton/all/osd-recovery.yaml (Use "slow request" for luminous)
qa/suites/rados/singleton/all/thrash-eio.yaml (Use "slow request" for luminous, keep OBJECT_MISPLACED)

Revert "qa/suites/rados/singleton: more whitelist"

This reverts commit 77258ed181dbe04e56da249a71edd3220baf954a.

Signed-off-by: David Zafman <dzafman@redhat.com>
Conflicts:
qa/suites/rados/singleton/all/osd-recovery.yaml (trivial)
qa/suites/rados/singleton/all/thrash-eio.yaml (trivial)

qa/workunits/cephtool/test.sh: Use :- to handle TMPDIR=""

Only Luminous has seen TMPDIR="" coming into this script.

Signed-off-by: David Zafman <dzafman@redhat.com>

msg/async: do not force updating rotating keys inline

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

msg/async: fix mark_down vs accept race

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #25990 from ukernel/luminous-37760

luminous: mds: handle fragment notify race

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26005 from ukernel/luminous-37481

luminous: mds: handle state change race

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26257 from batrick/i38084

luminous: mds: log new client sessions with various metadata

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26271 from batrick/i38104

luminous: client: session flush does not cause cap release message flush

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #26278 from ukernel/luminous-stale-caps

luminous: mds: optimize revoking stale caps

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26326 from alfredodeza/luminous-rm38229

luminous: doc: pin the version for "breathe" to 4.1.11

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rgw: fix radosgw linkage with WITH_RADOSGW_BEAST_FRONTEND=OFF

The master commit 5c040d991510cb4ff0d74305889130e2d84fedc1 fixing issue
http://tracker.ceph.com/issues/23680 was backported to luminous for v12.2.11 by
a47e714e7f5ce803ba7d8986c5d954123b85fc8e which was included in
https://github.com/ceph/ceph/pull/24621, where it came as the first of a series
of five cherry-picks.

This, it turns out, was the wrong order - it should have come last since it
was a follow-up fix.

Signed-off-by: Nathan Cutler <ncutler@suse.com>
(partial manual backport of 5c040d991510cb4ff0d74305889130e2d84fedc1)

doc: pin the version for `breathe` to 4.1.11

It has the compatibility needed for Sphinx version 1.6.3

Signed-off-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #26295 from jan--f/25390-luminous

[luminous] ceph-volume: revert partition as disk

Reviewed-by: Andrew Schoen <aschoen@redhat.com>

rgw: cleanup for RGWDeleteObj::verify_permission()

- if the 'Effect' field of user policy and bucket policy
  is set to 'Allow', RGWDeleteObj::verify_permission()
  should return 0 instead of true;

- as the same with bucket policy, if the 'Effect' field
  of user policy is set to 'Deny', RGWDeleteObj::verify_permission()
  should return -EACCES instead of false;

Signed-off-by: Enming.Zhang <zvampirem77@gmail.com>
(cherry picked from commit 86334645cf96c03515d30804e2df0bdf22916fb5)

Conflicts:
src/rgw/rgw_op.cc : Resolved in verify_permission

rgw: fix obj can still be deleted even if deleteobject policy is set

Fixes: http://tracker.ceph.com/issues/37403
Signed-off-by: Enming.Zhang <zvampirem77@gmail.com>
(cherry picked from commit 84879e915eb3f01b453aa2138cd44c09925e7cd1)

Conflicts:
src/rgw/rgw_op.cc : Resolved in verify_permission

ceph-volume: add inventory call to functional tests

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit a183a28c5e423c5658a37f7a1d3add9d64c6d683)

ceph-volume: add partition support to Device

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 05ac680cb37fd15fdeadaea86780bf1b4f2d566a)

Revert "ceph-volume tests.util ensure partitions are listed as top-level keys"

This reverts commit 6dc01772245e557b73551111f7c48cc60aec0893.

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 3c1c32cbc2736d72295f1bd7921c4a3cd2f4727e)

Revert "ceph-volume util.disk include partitions as top level devices in get_devices"

This reverts commit 7f3c3598491e556ec1b93ea0a54f52144202fab6.
Fixes: http://tracker.ceph.com/issues/37506
Signed-off-by: Jan Fajerski <jfajerski@suse.com>
(cherry picked from commit 50bc44f1c2327ad38504950fe62820a44ead9b14)

mds: broadcast quota message to client when disable quota

When disable quota by setting quota.max_files or quota.max_bytes
to zero, client does not receive the quota broadcast message.

Signed-off-by: Junhui Tang <tangjunhui@sangfor.com.cn>
Fixes: https://tracker.ceph.com/issues/38054
(cherry picked from commit 061ace2eb7c08d271db4d5d08ffa91ae832715f5)

Conflicts:
src/mds/MDCache.cc : Resolved in broadcast_quota_to_client

qa: update krbd tests for zeroout

Discard no longer guarantees zeroing, use BLKZEROOUT and "fallocate -z"
instead (blkdiscard(8) in xenial doesn't support -z).

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>
(cherry picked from commit 04f5b343f9987f0cf67509c2a76ddf7ef46288e5)

Conflicts:
qa/rbd/krbd_whole_object_zeroout.t
qa/suites/krbd/basic/tasks/krbd_whole_object_zeroout.yaml
- the modifications to these files are not relevant for luminous

test: ceph-objectstore-tool cut down on large run

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 388e54d906512106c38088d4ce18842985dd000c)

test: ceph-objectstore-tool: Add test for EC object dump to check hinfo section

Fixes: http://tracker.ceph.com/issues/38053
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 07e4273c6a0d48adf0dd2f3370e07b99f43bd013)

test: ceph-objectstore-tool: Fix EC code handling so it doesn't skip EC objects

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 786b39f18f6df21991582c178595ce87d28633d1)

mds: optimize resuming stale caps

If client doesn't want any cap, there is no need to re-issue stale
caps.

A special case is that client wants some caps, but skipped updating
'wanted'. For this case, client needs to update 'wanted' when stale
session get renewed.

Fixes: http://tracker.ceph.com/issues/38043
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit e824b3d2024db36789fdf579f0af9bf3bbe55d51)

 Conflicts:
src/client/Client.cc
src/mds/cephfs_features.h

client: avoid unnecessary wakeup when handling RENEWCAPS

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit c744bc1673ca0b4e50f41516fdf49c8560db073a)

client: set cap->wanted when adding new cap

This avoids unnecessary cap message if cap is added by open/create
request reply.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 314660a46a8a1af97f70b2bac05b2f6fa5d23bc4)

 Conflicts:
src/client/Client.cc
src/client/Client.h

client: don't wakeup cap waiters twice when mds recovered

Both kick_maxsize_requests() and wake_inode_waiters() wake up cap
waiters

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 5993a93e606d23ef311831863444294d7c64cc04)

 Conflicts:
src/client/Client.cc
src/client/Client.h

mds: optimize revoking stale caps

For caps that are not being revoked and don't have writeable range
and don't want exclusive caps or file read/write. there is no need
to call Locker::revoke_stale_caps(Capability*). Because these caps
don't need recover and don't affect eval_gather()/try_eval().

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit cb62030f0468fc04735c1b4cff73da779cb11ad8)

 Conflicts:
src/mds/CInode.cc
src/mds/Capability.h
src/mds/Locker.cc
src/mds/Migrator.cc

mds: put notable caps at the front of session's caps list

Notable Capabilities are ones that are being revoked, ones that
have writeable ranges and ones that want exclusive caps or want
file read/write.

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit cb6e7184458e64720ad83c266e5e393a80c32697)

 Conflicts:
src/mds/Capability.cc

mds: track if client has writeable range in Capability

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 370ae1cb3e5dc07867d80e998082bc514e8fccfd)

 Conflicts:
src/mds/Capability.h
src/mds/Locker.cc
src/mds/MDCache.h
src/mds/Server.cc

mds: add session pointer to Capability

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 7c628472a86c6acebb20f0a2504744b10f250587)

 Conflicts:
src/mds/CInode.cc
src/mds/Capability.cc
src/mds/Capability.h
src/mds/Locker.cc

client: skip updating 'wanted' caps if caps are already issued

When reading cached inode that already has Fscr caps, this can avoid
two cap messages (one updats 'wanted' caps, one clears 'wanted' caps).

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit d20b260ecf3f323c87ad1e6865f87e2381444546)

 Conflicts:
src/client/Client.cc

client: sync 'retain caps' logical from kernel client

The main change is keeping CEPH_CAP_ANY_RD for unused file inodes

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 29034396398b1706db625e1abf2e6682be7130a5)

Merge pull request #26207 from vumrao/wip-vumrao-38108

luminous: mgr/DaemonServer: log pgmap usage to cluster log

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26186 from tchaikov/luminous-36737

luminous: build/ops: Allow multi instances of "make tests" on the same machine

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

client: flush cap release on session flush

MDS expects the completed cap release prior to responding to the session flush
for cache drop.

Fixes: http://tracker.ceph.com/issues/38009
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 59eca7a17816317d5064840745ea5e28d3349d1a)

Conflicts:
src/client/Client.cc

rgw: `radosgw-admin bucket rm ... --purge-objects` can hang...

This command can hang (i.e., enter an infinite loop) due to
problematic bucket index entries left as a result of bug
https://tracker.ceph.com/issues/38007 .

The fix is to ignore the false bucket index entries -- since they do
not represent actual objects -- and remove all actual objects in the
bucket, so that bucket itself can be removed.

This fixes the both code paths whether `--bypass-gc` is specified or
not.

Furthermore, to made these operations more efficient, the internal
listing of the bucket is done unordered. This would improve behavior
when removing buckets with a large number of objects.

Fixes: http://tracker.ceph.com/issues/38134
Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>
(cherry picked from commit 5b3dc38c628728651b26415ab3d2190cf06b9e39)

mon: A PG with PG_STATE_REPAIR doesn't mean damaged data, PG_STATE_INCONSISTENT does

Fixes: http://tracker.ceph.com/issues/38070
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit a4b1574532ee4f789b30634f0eca03cc30ddcf8d)

os/bluestore: fixup access a destroy cond cause deadlock or undefine
              behavior.

Fixes: https://tracker.ceph.com/issues/37733
Signed-off-by: linbing <linbing@t2cloud.net>
(cherry picked from commit dbbde306484276251cbd7a7b7a6fde0093f98ba2)

Conflicts:
src/os/bluestore/BlockDevice.h : Resolved in try_aio_wake

mds: do not log internal journal error

MDS can't reach this code due to assert in C_MDS_session_finish.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 0992ac1fa62edb6143efc80a6fea7b4704107705)

Conflicts:
src/mds/Server.cc

mds: clean up log level 2 messages

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit a225c266052d09ffcb066aead83a18dec918aa35)

Conflicts:
src/mds/MDSRank.cc

mds: log session open stats

This is to assist with tracking what clients are connecting and how long it
takes for them to open a session (a rough estimate of how long it takes to
mount).

Example line:

    2018-12-17 17:21:31.682 7f822703c700  2 mds.0.server New client session: addr="127.0.0.1:60552/2307908923",elapsed=0.013384,throttled=0.000071,status="ACCEPTED",root="/"

Fixes: http://tracker.ceph.com/issues/37678
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit c89cd8cac16789ed98dc1487ec17f45adf03dd26)

Conflicts:
src/mds/Server.cc
src/mds/SessionMap.cc
src/mds/SessionMap.h
src/mds/events/ESession.h

osd/PG.cc: account for missing set irrespective of last_complete

When adding source info from another OSD, check if an object that needs
recovery is present in its missing set. If yes, do not include the OSD
as a missing loc.

Fixes: http://tracker.ceph.com/issues/37919
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit e6b1637e00a3e98e588e4b83a13db5703fb96fce)

cmake: use $CMAKE_BINARY_DIR for default $CEPH_BUILD_VIRTUALENV

to avoid conflict with other `make check` runs in the same host

See-also: http://tracker.ceph.com/issues/36737
Signed-off-by: Kefu Chai <kchai@redhat.com>
(cherry picked from commit a8d6b2170dfa0838dad0d1b7b03fc8cf19e4eeff)

Conflicts:
- drop the change to mgr/*/run-tox.sh, as they don't exist in
luminous.
        - apply the change to ceph-disk/run-tox.sh
- apply the change to ceph-helpers.sh: as it set $PATH using
          $TMPDIR, which is set to /tmp if $TMPDIR is empty.

mds: trim cache after journal flush

Dropping cache should trim the cache only after journal flush otherwise dirty
inodes will remain pinned in cache.

Fixes: http://tracker.ceph.com/issues/38010
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit d7f6c1583b4162fade434e376176f3068472f53f)

mgr/DaemonServer: log pgmap usage to cluster log

- change default of mon_cluster_log_file_level to debug
- log to clog->debug()

Fixes: https://tracker.ceph.com/issues/37886
Signed-off-by: Neha Ojha <nojha@redhat.com>
(cherry picked from commit f64a69e5054fb319e9f717bbcb7ddd8541e01c6f)

 Conflicts:
src/common/options.cc
        dropped master .add_service, .add_see_also