rgw: data sync checks empty next_marker for datalog

RGWReadRemoteDataLogShardCR tracks the marker and next_marker
separately, because next_marker will be empty when it reaches the end

this allows RGWDataSyncShardCR to avoid clearing its sync_marker and
restarting datalog listing from the beginning

Fixes: http://tracker.ceph.com/issues/39033
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2ff63778342282ab5c676529f6e7d90f30e6da1a)

Conflicts:
src/rgw/rgw_data_sync.cc: no sync tracing

Merge pull request #26642 from pdvian/wip-38449-luminous

luminous: MDSMonitor: do not assign standby-replay when degraded

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26776 from pdvian/wip-38541-luminous

luminous : qa: bump fsstress timeout to 6h

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26805 from pdvian/wip-38543-luminous

luminous: qa: unmount clients before deleting fs

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26820 from smithfarm/wip-38545-luminous

luminous: qa: Loading libcephfs-jni: Failure!

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #26964 from ashishkumsingh/wip-38735-luminous

luminous: qa: extend MDS heartbeat grace for valgrind

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #27024 from pdvian/wip-38669-luminous

luminous: mds: wait for client to release shared cap when re-acquiring xlock

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Yan, Zheng <zyan@redhat.com>

Merge pull request #27207 from pdvian/wip-38854-luminous

luminous: mon/MgrStatMonitor: ensure only one copy of initial service map

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #27224 from xiexingguo/wip-luminous-upmap-backports

luminous: osd: backport recent upmap fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27226 from pdvian/wip-38857-luminous

luminous: should set EPOLLET flag on del_event()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26358 from smithfarm/wip-37557-luminous

luminous: multisite: es sync null versioned object failed because of olh info

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26530 from pdvian/wip-38354-luminous

luminous: rgw: check for non-existent bucket in RGWGetACLs

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #26619 from pdvian/wip-38446-luminous

luminous: rgw: only update last_trim marker on ENODATA

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26761 from pdvian/wip-38529-luminous

luminous: rgw: data sync drains lease stack on lease failure

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26946 from ashishkumsingh/wip-38727-luminous

luminous: rgw: bucket limit check misbehaves for > max-entries buckets (usually 1000)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #27030 from pdvian/wip-38671-luminous

luminous: rgw: sync module: avoid printing attrs of objects in log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26516 from yehudasa/wip-rgw-es-fixes-luminous

 luminous: rgw: multiple es related fixes and improvements

Reviewed-by: Abhishek Lekshmanan <abhishek.lekshmanan@gmail.com>

osd/OSDMap: add log for better debugging

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a89281ffbb50a4dfc700398e864138b5faaf00f5)

osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only

The current implementation will try to cancel any pg_upmaps that
would otherwise re-map a PG out from an underfull osd, which is wrong,
e.g., because it could reliably fire the following assert:

src/osd/OSDMap.cc: 4405: FAILED assert(osd_weight.count(i.first))

Also it would not match the expectation if automatic balancing
has been strictly restricted to some specific pools by admin.

Fix by excluding any wild PG that does not belong to the origin pools
passed in when trying to do upmap/unmap.

Fixes: http://tracker.ceph.com/issues/38897
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 01e8e9482ce7194d347e02ef41acfa6d8d14f614)

osd/OSDMap: drop local pool filter in calc_pg_upmaps

The local pre-loaded pool filter is completely redundant since
the below check:

if (!only_pools.empty() && !only_pools.count(i.first))

could reliably catch both cases - either optimization should be
restricted to specific pools feeded, or all existing pools.

Let's clean it up.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 3e6bad9821b5fb3e780d970666fbdfbf217d905e)

crush: add root_bucket to identify underfull buckets

All underfull buckets under root_buckets will be taken as target

For the crule rule:
    step take datacenter0
    step chooseleaf firstn 2 type host
    step emit
    step take datacenter1
    step chooseleaf firstn 2 type host
    step emit

If one host contains overfull osd but no underfull osd,
it will use other underfull buckets as target, which
maybe not in the same datacenter, that will
broke the rule.

Fixes: http://tracker.ceph.com/issues/38826
Signed-off-by: huangjun <huangjun@xsky.com>
(cherry picked from commit 3d5678d3561d90a10d9de3cb6e7e0405dbe8fdfe)

Merge pull request #27104 from rzarzynski/wip-nss-importsymkey-in-fips-luminous

luminous: crypto: don't use PK11_ImportSymKey() in FIPS mode

Reviewed-by: Neha Ojha <nojha@redhat.com>

EventEpoll: refactor del_event() a bit

The main purpose of the patch is to set 'ee' members only on EPOLL_CTL_MOD
path, deletion path does not requie any of the members set.

This will make code a bit better.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
(cherry picked from commit 2292120ffd2730a20fd2750a605952982f031c83)

EventEpoll: set EPOLLET flag on del_event()

When new fd is being added to epoll fd EPOLLET (edge triggered) flag
is set, i.e. we assume that in and out works in edge triggered mode.

But on del_event(), when eventually epoll_ctl(EPOLL_CTL_MOD) is
invoked, EPOLLET flag is cleared, i.e. level triggered mode is
enabled.

Do not lose EPOLLET.

Signed-off-by: Roman Penyaev <rpenyaev@suse.de>
(cherry picked from commit c415b9fed4fbecd2fb0709fa0c0c783102f60e01)

auth, rgw, common: switch to PK11_ImportSymKey_FIPS wrapper.

PK11_ImportSymKey() is a part of NSS API that becomes unavailable
in the FIPS mode. Apparently NSS targets stricter restrictions
than those coming from Level 1 of FIPS 140-2. In the consequence,
loading a symmetric key from plain keyring or key db (which Ceph
needs to do due to architectural reasons) fails.

The same issue affected corosync and this patchset deals with it
the same way like already followed by corosync [1]:
a raw crypto key is in-memory wrapped with fresh, random wrapping
key just before being imported via PK11_UnwrapSymKey(). Of course,
this effectively lowers to FIPS level 1. Still, this would be no
different from what OpenSSL (to which we are currently migrating
in master) gives in the matter.

The patch can be *roughly* verified in following steps:
  1. mkdir ./nssdb
  2. certutil -N -d ./nssdb --empty-password
  3. modutil -dbdir ./nssdb -fips true
  4. ../src/vstart.sh -l -n -b -o "nss_db_path=/work/ceph-3/build/nssdb"

This fix is dedicated to Luminous. In master we're switching to OpenSSL.

[1] 5dadebd21862074deaeb9a337fc9e49f5e9f692a in corosync's public repo.

Fixes: http://tracker.ceph.com/issues/38843
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crypto: bring PK11_ImportSymKey_FIPS doing PK11_UnwrapSymKey.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mon/MgrStatMonitor: ensure only one copy of initial service map

It's possible for create_pending to be called multiple times.

Fixes: http://tracker.ceph.com/issues/38839
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 204a8deacd2068989bd52eb130f4aac434b45f35)

Conflicts:
src/mon/MgrStatMonitor.cc : Resolved in create_initial

Merge pull request #26758 from pdvian/wip-38510-luminous

luminous: CLI: ability to change file ownership

Merge pull request #26830 from pdvian/wip-38562-luminous

luminous: mgr/BaseMgrModule: drop GIL for ceph_send_command

Reviewed-by: Tim Serong <tserong@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26900 from pdvian/wip-38586-luminous

luminous: OSD crashes in get_str_map while creating with ceph-volume

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26962 from pdvian/wip-38665-luminous

luminous: qa: ignore slow metadata io wrn during osd thrash

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #27033 from cbodley/wip-38796

luminous: doc/rgw: document placement target configuration

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #26557 from dzafman/wip-38244

luminous: scrub warning check incorrectly uses mon scrub interval

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #26793 from dzafman/wip-38274

luminous: Fix recovery and backfill priority handling

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26544 from ifed01/wip-ifed-fix-compress-luminous

luminous: os/bluestore: do not assert on non-zero err codes from compress() call

Reviewed-by: Sage Weil <sage@redhat.com>

doc/rgw: document placement target configuration

Fixes: http://tracker.ceph.com/issues/24508
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit c95585b1984e60708addee436a774d0c92ce57a6)

rgw: sync module: avoid printing attrs of objects in log

Since attrs are verbose and not necessarily needed as a debug info (as they can
be obtained from the required object itself) avoid logging them. Also
additionally removed another place where such a log is done by the ES sync
module

Fixes: http://tracker.ceph.com/issues/37646
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>
(cherry picked from commit 8f37b70630ceb62aff4019e9ad2508977590a873)

Conflicts:
src/rgw/rgw_sync_module.cc : Resolved in RGWCallStatRemoteObjCR::operate

mds: wait for client to release shared cap when re-acquiring xlock

MDS may revoke shared cap from xlocker client during 'LOCK_XLOCKDONE ->
LOCK_LOCK_XLOCK' state transition.

Fixes: http://tracker.ceph.com/issues/38491
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>
(cherry picked from commit 218e784f2eb1eec6bbb7910b5432f7f36263dbe7)

Conflicts:
src/mds/SimpleLock.h : Resovled in put_xlock

Merge pull request #26597 from smithfarm/wip-release-notes-12_2_12-luminous

luminous: doc: move keyring caps RN to correct point release

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #26861 from ashishkumsingh/wip-38647-luminous

luminous: doc: Fix incorrect mention of 'osd_deep_mon_scrub_interval'

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26372 from pdvian/wip-38095-luminous

luminous: doc: rados/configuration: refresh osdmap section

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #26969 from pdvian/wip-38667-luminous

luminous: doc/radosgw: Document mappings of S3 Operations to ACL grants

Reviewed-by: Nathan Cutler <ncutler@suse.com>

qa: extend MDS heartbeat grace for valgrind

Valgrind makes the MDS slowwwww. The newish mds_heartbeat_grace config allows
us to keep sending beacons to the mons even if the internal heartbeat is slow.
This avoids the laggy messages which are useful to grep for unrelated messaging
issues.

Fixes: http://tracker.ceph.com/issues/38723
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 7b520755ce62461060b6dc75fb7a39c464d2bdcb)

doc/radosgw: Document mappings of S3 Operations to ACL grants

Since the ACL documentation is a bit sparse, let people know exactly
what operations they can expect to enable with a given ACL grant.

Fixes: https://tracker.ceph.com/issues/38523
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>
(cherry picked from commit bd94c93fada952537c7015743db55bff7619afa8)

qa: ignore slow metadata io wrn during osd thrash

Fixes: http://tracker.ceph.com/issues/38651
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit 5abcc32ff642b03c4b53e565e995ebb94802579a)

Merge pull request #26798 from dzafman/wip-slow-luminous

luminous: qa: Add "slow request" where missing based on REQUEST_SLOW

Reviewed-by: Neha Ojha <nojha@redhat.com>

rgw: bucket limit check misbehaves for > max-entries buckets (usually 1000)

RGWBucketAdminOp::limit_check does not correctly handle
truncation/continuation markers across calls to rgw_read_user_buckets(...).
Also, it should, but does not, flush its formatter periodically.

Fixes: https://tracker.ceph.com/issues/35973
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>
(cherry picked from commit 675b775ea52e0d7b81efb32c6cffa6987190184b)

Merge pull request #26574 from badone/wip-luminous-test-dup_index-trim

luminous: test: Verify a log trim trims the dup_index

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26613 from badone/wip-luminous-drop-GIL-in-get_config

luminous: mgr: drop GIL in get_config

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26427 from smithfarm/wip-38318-luminous

luminous: mgr: deadlock: _check_auth_rotating possible clock skew, rotating keys expired way too early

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #26706 from pdvian/wip-38506-luminous

luminous: osd/PrimaryLogPG: handle object !exists in handle_watch_timeout

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #26871 from neha-ojha/wip-38316-luminous

luminous: core: os/filestore: ceph_abort() on fsync(2) or fdatasync(2) failure

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26677 from smithfarm/wip-38501-luminous

luminous: rgw: rgw_file: only first subuser can be exported to nfs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26791 from cbodley/wip-38608

luminous: qa/rgw: reduce number of multisite log shards

Reviewed-by: Casey Bodley <cbodley@redhat.com>

common/str_map: better trim() impl using boost

Thanks Kefu!

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 6154874aa9ae6dd66a2a0decb289f2a718f65048)

Conflicts:
src/common/str_map.cc : Resolved in trim

common/str_map: fix trim() on empty string

If was pass an empty string (e.g., with something like
get_str_map("M= T= P=")) we end up with a (size_t)-1 for end.

Fixes: http://tracker.ceph.com/issues/38329
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 16ffca6f0a23a96284a7d17c890047121d32058e)

Merge pull request #25810 from pdvian/wip-37741-luminous

luminous: build/ops: rpm: require ceph-base instead of ceph-common

Reviewed-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #26127 from xiexingguo/wip-luminous-upmap-fixes

luminous: osd: backport recent upmap fixes

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #26199 from b-ranto/wip-restful-luminous

luminous: mgr/restful: fix py got exception when get osd info

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26355 from pdvian/wip-38240-luminous

luminous: qa/tasks/radosbench: default to 64k writes

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #26540 from ifed01/wip-ifed-38395

luminous: os/bluestore: fix lack of onode ref during removal

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #26579 from pdvian/wip-38400-luminous

luminous: common: ceph_timer: stop timer's thread when it is suspended

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #26866 from neha-ojha/wip-38574-luminous

luminous: os/bluestore: Fix problem with bluefs's freespace not being balanced when kv_sync_thread is sleeping

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

os/filestore: ceph_abort() on fsync(2) or fdatasync(2) failure

If we get an error from f[data]sync that is always a fatal error.

Fixes: http://tracker.ceph.com/issues/38258
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 3892b81303bcc35056ca06371651e13b7a4c4f2a)

 Conflicts:
src/os/filestore/FileStore.cc - encode trivial resolution
src/os/filestore/WBThrottle.cc - trivial resolution

os/bluestore: Fix problem with bluefs's freespace not being balanced when kv_sync_thread is sleeping.

Fixes: https://tracker.ceph.com/issues/38574
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>
(cherry picked from commit cbaa17c1fca858ce25c1e3de193151a8b7b10ef7)

Conflicts:
src/os/bluestore/BlueStore.cc: trivial resolution

doc: Fix incorrect mention of 'osd_deep_mon_scrub_interval'

Fixed the incorrect mention of 'osd_deep_mon_scrub_interval' in health-checks.rst.
Changed it to 'osd_deep_scrub_interval'.

Fixes: https://tracker.ceph.com/issues/38310
Signed-off-by: Ashish Singh <assingh@redhat.com>
(cherry picked from commit 7108e6a3c7ec1f3ae3c39682f43db99efb2b7a6c)

Conflicts:
doc/rados/operations/health-checks.rst
    - Fixed conflicts to keep relevant changes.

Merge pull request #26857 from ceph/backport-luminous-26826

luminous: ceph-volume: `simple scan` will now scan all running ceph-disk OSDs

Reviewed-by: Alfredo Deza <adeza@redhat.com>

ceph-volume: make systemctl.get_running_osds resilient to garbage output

Makes systemd.systemctl.get_running_osds more resilient to output
that doesn't match what we're expecting to parse. This also handles
the case of a nonzero exit status by retuning an empty list.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit f3707f1ad43513c2a6df17e9dce756533c3b4a1d)

ceph-volume: remove obsoleted simple unit test

This test is no longer valid because `simple scan`
without any arguments will now attempt to scan all
running OSDs.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit b59dcd01c2dd386ca1b63eb66ed243afffb80211)

ceph-volume: tox tests should ignore the tests/functional directory

Without this the functional testinfra tests try to run when we
are intending to only run the unit tests.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1881f386d8207ee8c10f0863c4a1abf18cf3c5d9)

ceph-volume: use `simple scan` and `simple activate --all` in tests

In a couple of the functional tests for simple we should run `simple
scan` without any arguments and `simple activate --all` to increase
the test coverage. We want to test pointing at a diretory and scanning
all running OSDs.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 87c33f149a83bb1175bfb473b226e8c46e4e2a6e)

ceph-volume: update docs and help menu for `simple scan`

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit f7943ef6007fb05d83602bc22cbacf3bb67d8625)

ceph-volume: adds unit tests for systemd.systemctl.get_running_osd_ids

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 7cb098b6899542c875d911d28771a4c450d0bb3a)

ceph-volume: update `simple scan` to scan all running OSDs

If no argument is passed to `ceph-volume simple scan` it will
inspect any running osds and scan them if they were created by
ceph-disk.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 28e454ec30f867ae1229c66d263bcd608654553f)

ceph-volume: add systemd.systemctl.get_running_osd_ids

This is used to retrieve a list of osd ids from the currently
running ceph-osd systemd units.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 15d20e29a2f2d74caf05dfec559e30941c7a46ab)

mgr/BaseMgrModule: drop GIL for ceph_send_command

Otherwise, we can easily deadlock, since other bits of the code hold the
objecter lock and then take the GIL.

Fixes: http://tracker.ceph.com/issues/38537
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 94486a9fb6827157bee2d2483f20222126083b22)

Conflicts:
src/mgr/BaseMgrModule.cc : Resolved in ceph_send_command

Merge pull request #26824 from ceph/backport-luminous-26739

luminous: ceph-volume: replace testinfra command with py.test

Reviewed-by Alfredo Deza <adeza@redhat.com>

ceph-volume tests.simple replace testinfra command for py.test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit e90ebc3264004cab3286895c843db43f20a4840b)

ceph-volume tests.lvm replace testinfra command for py.test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit c4597fc63195c654b960f747f9d50a558fa719c0)

ceph-volume tests.batch replace testinfra command for py.test

Signed-off-by: Alfredo Deza <adeza@redhat.com>
(cherry picked from commit dc7501af69ccbdabb5e6a77a91696454855201e9)

qa: remove libcephfs-java tests

These have bit-rotted and no longer work. No cycles from interested parties
available to fix.

Fixes: https://tracker.ceph.com/issues/38487
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit df62f2d91dbae19df71eb2be7998e2737e407e3c)

Conflicts:
qa/suites/fs/basic_functional/tasks/libcephfs_java.yaml
- file removed
qa/suites/rados/upgrade/jewel-x-singleton/2-partial-upgrade/.qa
- symlink retained (not sure why "git cherry-pick" sees a conflict in this file...)

qa: unmount clients before deleting fs

Client unmount during test cleanup will hang if the file system was deleted.

Fixes: http://tracker.ceph.com/issues/38518
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit a20a61f4764d38c0076f53560b1e2a577d2f07c8)

qa/rgw: reduce number of multisite log shards

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2629231ca1782b6a02e8279182c707d56d95290a)

Conflicts:
qa/suites/rgw/multisite/overrides.yaml curl timeout

Merge pull request #26232 from batrick/i38132-i38130

luminous: mds: stopping MDS with a large cache (40+GB) causes it to miss heartbeats

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #26661 from batrick/i38488

luminous: mds: message invalid access

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa: Add "slow request" for luminous only where missing based on REQUEST_SLOW

Signed-off-by: David Zafman <dzafman@redhat.com>

osd: Add OSD_BACKFILL_PRIORITY_FORCED with lower priority
than OSD_RECOVERY_PRIORITY_FORCED

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit d088ffb09f46c6207cfedd2cdab9f074197af4bb)

test: Add scripts to test backfill/recovery priority handling

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit ca5cf14fa83ee89260688d467257bb3d81efdd36)

osd AsyncReserver: Update priority in AsyncReserver for force and cancel-force

Fixes: http://tracker.ceph.com/issues/38041
Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 11097f1235265a3eb753adc7e6d2818e5902e39e)

Conflicts:
src/osd/PG.cc (trivial)
        src/common/AsyncReserver.h (update_priority() assert, lock for luminous)

test: Add option to wait_for_clean() to execute at every sleep

Signed-off-by: David Zafman <dzafman@redhat.com>
(cherry picked from commit 70b51362084d2ae5f2a6e9e5856f17452c53e3f3)

osd/PG: fix misused FORCE_RECOVERY[BACKFILL] flags

__set_force_backfill__ should set PG_STATE_FORCED_BACKFILL
instead of PG_STATE_FORCED_RECOVERY.

Fixes: http://tracker.ceph.com/issues/27985
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 60ef742a84d5201b2a6e6c1780741da0b31c49a8)

Conflicts:
src/osd/PG.cc (trivial)

osd: print summary for forced backfill/recovery to debug

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 7816ab5a8eb0ea754a0809eb9da915a16ad39e83)

Conflicts:
src/osd/PG.h (pg_id public like master for OSDService::adjust_pg_priorities())

osd/PG: clean up set_force_{recovery,backfill} interface

- update state under pg lock
- clean up PG interface
- log only when we adjust the state

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 2482b4cc91579b4ef5748097f1adacaad91510b1)

Conflicts:
src/osd/PG.h (trivial)

qa: bump fsstress timeout to 6h

When run with valgrind, it takes a significant amount of time to complete.

Fixes: http://tracker.ceph.com/issues/38520
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>
(cherry picked from commit f9146f2accb6eba2e10da578a8a605ab1aa88893)

Merge pull request #26703 from ceph/backport-luminous-26685

luminous: ceph-volume: use our own testinfra suite for functional testing

Reviewed-by: Alfredo Deza <adeza@redhat.com>

rgw: data sync drains lease stack on lease failure

on lease failure, call drain_all() to collect the spawned lease stack.
otherwise, the parent coroutine will inherit this spawned stack and may
not drain or collect it manually. RGWDataSyncShardControlCR, for
example, does not drain/collect because it only uses call() to run a
single RGWDataSyncShardCR, so all lease stacks from RGWDataSyncShardCR
just pile up in the RGWDataSyncShardControlCR

Fixes: http://tracker.ceph.com/issues/38479
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 11ff8653dcaf7663a0553e3175dec75843b3ccea)

CLI: ability to change file ownership

When creating/fetching key it's nice to have the ability to change the
ownership of the created file.

This commit adds the '--setuser' and 'setgroup' which respectively apply
the desired owner and group to a file user when '--output' is passed.

Closes: https://tracker.ceph.com/issues/38370
Signed-off-by: Sébastien Han <seb@redhat.com>
(cherry picked from commit 0e26090960213ef911cfc7bfcdf2925ddf1b268f)

Conflicts:
src/ceph.in : Resolved in  new_style_command

Merge pull request #26686 from dzafman/wip-test-fix

luminous: qa/workunits/cephtool/test.sh: Use :- to handle TMPDIR=""

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Yuri Weinstein <yweinstei@redhat.com>

ceph-volume: set number of osd ports in the tests

The number of ports the OSDs listen on depend on the version of ceph
being used, so we need to test for that number accordingly.

Signed-off-by: Andrew Schoen <aschoen@redhat.com>
(cherry picked from commit 1132cb3c25fef9f853488cc8aab1327aff6bcf96)

mds: handle negative decay counter

Problem only exists in Luminous/Mimic.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>