Merge pull request #28704 from joke-lee/sts_assume_policy

rgw: Policy should be url_decode when assume_role

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #28709 from tchaikov/wip-seastar

seasetar: pick up the latest seastar

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

seasetar: pick up the latest seastar

couple noticeable changes

* use dpdk v19.05, please note the older version of dpdk might not
  work, for instance dpdk v18.11 does not compile with the latest
  seastar master HEAD
* future cleanups

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28707 from majianpeng/perf-async-msgr-remove-unsued-header

test/crimson/perf_async_msgr: remove unsued header file.

Reviewed-by: Kefu Chai <kchai@redhat.com>

test/crimson/perf_async_msgr: remove unsued header file.

Signed-off-by: Jianpeng Ma <jianpeng.ma@intel.com>

Merge pull request #28684 from tchaikov/wip-silence-warnings

src/: silence GCC warnings

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #28677 from liewegas/wip-recovery-nvme

osd: add hdd and ssd variants for osd_recovery_max_active

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

rgw, Policy should be url_decode when assume_role

Signed-off-by: yuliyang <yuliyang@cmss.chinamobile.com>

Merge pull request #17619 from liuchang0812/wip-ec-below-min-size

osd: allow EC PGs to do recovery below min_size

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28697 from cbodley/wip-qa-rgw-swift-skip-client

qa/rgw: fix swift warning message

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

qa/rgw: fix swift warning message

  File "/home/teuthworker/src/github.com_ceph_ceph_master/qa/tasks/swift.py", line 245, in task
    log.warning('Swift tests cannot run on rhel 7.6+, skipping client {client}'.format(client))
KeyError: 'client'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28657 from dillaman/wip-40435

rbd: use the ordered throttle for the export action

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge PR #28567 into master

* refs/pull/28567/head:
Revert "mon/OSDMonitor: report pg[pgp]_num_target instead of pg[pgp]_num"

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28674 from tchaikov/wip-doc/crimson

doc/dev: add crimson.rst

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge pull request #28462 from myoungwon/wip-doc-dedup

doc: add a deduplication document

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #26251 from majianpeng/add-new-op-create

os/objectstore: add new op OP_CREATE for create a new object.

Reviewed-by: Sage Weil <sage@redhat.com>

doc: add a deduplication document

Signed-off-by: Myoungwon Oh <omwmw@sk.com>

doc/dev: add crimson.rst

Signed-off-by: Kefu Chai <kchai@redhat.com>

test: Replace deprecated TYPED_TEST_CASE with TYPED_TEST_SUITE

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: Fix pessimizing move

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

client,rbd,mon: Fix signed compare warnings

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

test/simple_spin: Explicitly wait for futures from async

Rather than count on implicitly blocking destruct.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

OSD: Initialize pool and recovery_state in the order they are defined

Currently we call get_pool on the PeeringState before the PeeringState
has been initialized.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rbd: Remove unused capture

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

immutable_object_cache: Remove unused capture

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

immutable_object_cache: Remove pessimizing moves

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

osd: add hdd and ssd variants for osd_recovery_max_active

Semi-arbitrarily set the SSD max to 10 (instead of 3).  This should be
tuned based on some real data.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28676 from tchaikov/wip-stop.sh-with-crimson

stop.sh: add --crimson option

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28204 from dzafman/wip-39555

mon: Improve health status for backfill_toofull and recovery_toofull

Reviewed-by: Joao Eduardo Luis <joao@suse.de>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #28663 into master

* refs/pull/28663/head:
cephfs-shell: use consistent naming for paths
cephfs-shell: remove unnecessary instances of to_bytes
cephfs-shell: refactor list_items
pybind/cephfs: improve user-friendliness of DirResult
pybind/cephfs: avoid unicode check

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge PR #28561 into master

* refs/pull/28561/head:
vstart_runner: upgrade the check for commands to be run as another user
vstart_runner: split unicode arguments into lists

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28049 from theanalyst/rgw-lc-bl-fix

rgw_lc: use a new bl while encoding RGW_ATTR_LC

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #28276 from liuchang0812/multi-site-request-forward

rgw: forwarded some requests to master zone

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26167 from yaozongyou/rgw-lc-add-more-check

rgw: check lc objs not empty after fetching

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #26538 from zhangsw/feature-rgw-object-lock

rgw: add S3 object lock feature to support object worm

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #26351 from linuxbox2/wip-rgw-gcretire

rgw: perfcounters: add gc retire counter

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>
Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #28527 from tianshan/fix_ret

rgw: fix miss get ret in STSService::storeARN

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #28327 from linuxbox2/wip-rgw-email

rgw: conditionally allow non-unique email addresses

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28308 from joke-lee/usage_dump

rgw: usage dump_unsigned instead dump_int

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #28172 from joke-lee/rgw-sts-bad-token

rgw: fix rgw crash and set correct error code

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

vstart_runner: upgrade the check for commands to be run as another user

Rectify the condition that checks if command to be issued as another
user using sudo is issued as a single argument after "-c".

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28539 from dillaman/wip-40072

librbd: improve journal performance to match expected degradation

Reviewed-by: Mykola Golub <mgolub@suse.com>

stop.sh: add --crimson option

so we can pkill crimson-osd if this option is specified

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28655 from dengchj/multisite_doc

doc: fixed --read-only argument value in multisite doc

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge PR #28194 into master

* refs/pull/28194/head:
test_volume_client: declare only one default for python version
test_volume_client: don't shadow class variable

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: use consistent naming for paths

And some small changes to make some commands take variable arguments and fix
defaults.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: remove unnecessary instances of to_bytes

cephfs binding already automatically converts paths and some arguments to
bytes.

Fixes: http://tracker.ceph.com/issues/40455
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: refactor list_items

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/cephfs: improve user-friendliness of DirResult

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/cephfs: avoid unicode check

py3 does not have the unicode built-in. Instead, simply try to do the encoding
and catch failures appropriately. If it quacks like a duck...

(Note that cython apparently cheats and allows the unicode check but this is
simpler.)

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28638 into master

* refs/pull/28638/head:
qa: elide python version config
cephfs-shell: assert python3

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28498 from kamoltat/wip-mgr-progress-add-event-osd-marked-in

mgr/progress: Add recovery event when OSD marked in

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #28564 from ceph/store_statfs-log

log: fix store_statfs log line

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28621 from Aran85/fix-ec-rf

osd: accident of rollforward may need to mark pglog dirty

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #28656 from dvanders/zstd-upgrade

zstd: upgrade to v1.4.0

Reviewed-by: Kefu Chai <kchai@redhat.com>

test_volume_client: declare only one default for python version

Fixes: http://tracker.ceph.com/issues/40460
Signed-off-by: Rishabh Dave <ridave@redhat.com>

vstart_runner: split unicode arguments into lists

Split not just string arguments but also unicode arguments into lists.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28570 from xiexingguo/wip-objecter-direct-spgid

osdc/Objecter: _calc_target - inline spgid

Reviewed-by: Sage Weil <sage@redhat.com>

doc: fixed --read-only argument value in multisite doc

Fixes: https://tracker.ceph.com/issues/40458
Signed-off-by: Chenjiong Deng <dengchenjiong@umcloud.com>

Merge pull request #28568 from xiexingguo/wip-off-by-one-max-osd

mon/OSDMonitor: fix off-by-one when updating new_last_in_change

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28629 from xiexingguo/wip-osd-df-by-pool-name

mgr, osd: 'osd df' by pool

Reviewed-by: Sage Weil <sage@redhat.com>

mon: Improve health status for backfill_toofull and recovery_toofull

Treat backfull_toofull as a warning condition because it can resolve itself.
Includes test case for PG_BACKFILL_FULL
Includes test case for recovery_toofull / PG_RECOVERY_FULL

Fixes: https://tracker.ceph.com/issues/39555
Signed-off-by: David Zafman <dzafman@redhat.com>

qa: elide python version config

This test doesn't actually use the config and cephfs-shell is py3 only.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

cephfs-shell: assert python3

There's little reason to support this new tool with python2.

Fixes: http://tracker.ceph.com/issues/40418
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28553 from trociny/wip-40071

librbd: mirror image enable/disable should enable/disable journaling

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28627 from trociny/wip-40409

journal: properly advance read offset after skipping invalid range

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #28460 from tchaikov/wip-cmake-detect-libnl

cmake: rewrite Findgenl to support components argument

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

rbd: use the ordered throttle for the export action

This allows exports to STDOUT to use multiple concurrent operations
and also fixes a potential race condition with concurrent callbacks
and file seeking.

Fixes: http://tracker.ceph.com/issues/40435
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

zstd: compat with v1.4.0

In zstd d8e215cbee03b038fffe74aebad63b625c42f23c
ZSTD_compress_generic() is renamed to ZSTD_compressStream2().

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

librbd: tweaks to improve throughput for journaled IO

Increase the per-object append flush bytes to 1MiB to mimic the 16MiB
in-memory writeback cache. Finally, default enable the new journal
writethrough until flush option.

Performance results comparing no journal, pre-change, and post-change:

no journal original modified
4K 40310.84 7223.73 27427.27
32K 21530.37 3256.39 10284.05
256K 3159.67 564.73 1627.16
4M 223.03 48.03 91.91

Fixes: http://tracker.ceph.com/issues/40072
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new rbd_journal_object_writethrough_until_flush option

When set to true, the journal will not attempt to batch appends until
after it receives the the first flush request from the user.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: support dynamically updating recorder flush options

Default to disabling writeback-style append flushes unless overridden
by a call to 'set_append_batch_options'.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

journal: fix broken append batching implementation

The original flush_interval/bytes/age batching options no longer
actually do anything. Integrate these better with the max in-flight
AIO handling.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

zstd: upgrade to v1.4.0

Signed-off-by: Dan van der Ster <daniel.vanderster@cern.ch>

Merge pull request #28631 from cbodley/wip-17882

qa/rgw: extra s3tests tasks use rgw endpoint configuration

Reviewed-by: Kefu Chai <kchai@redhat.com>

debian/control: remove "libnl-3-dev" from build dependencies

what we need is "libnl-genl-3-dev", libnl-3-dev should be a dependency
of "libnl-genl-3-dev". the only reason we would have added it, is if
the package maintainer of "libnl-genl-3-dev" fails to do so. and
the packages of "libnl-genl-3-dev" on both debian and ubuntu just look
correct in this perspective.

so let's remove this.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rewrite Findgenl to support components argument

* rename genl to nl:
  "genl" is not very specific without more context. and libnl is actually
  a library suite. so it would be better if we can rename the library to
  be found to "libnl", and by following the naming convention of CMake,
  in this change, Findgenl.cmake is renamed to Findnl.cmake
* Findnl.cmake: support the component argument, in our specific case,
  what we want is "libnl-genl" library, which is one of the libraries
  offered by libnl. so let's just make it a component.
* Findnl.cmake: should pass the names of the required variables instead of
  their values to `find_package_handle_standard_args()`. before this
  change, we pass `GENL_LIBRARIES` to this function. it was correct at the
  first glance. but it is not able to handle the case where case where
  libnl-genl is not installed. so the fix is to pass all the names of
  required library paths to this function. in this change, their name
  are concatenated to a single variable -- `nl_LIBRARIES`, and the
  value of this variable is passed to
  `find_package_handle_standard_args()`. and the error message would
  be more specific this way if libnl-genl is not found:
    Could NOT find nl (missing: nl_genl_LIBRARY)
* Findnl.cmake: add nl::<component> as imported library, it helps the
  consumer of these libraries to import them -- no need to
  specify the `target_include_directories()` separately anymore.
* move the find_package() code to where it is used. it helps to improve
  the readability.
* remove `HAVE_GENL` variable: it's not used anywhere.
* drop the messages of "Not using foobar", they do not help.

Signed-off-by: Kefu Chai <kchai@redhat.com>

journal: improve logging on recorder append path

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/dashboard: Ensure E2E tests can be run independently (#28592)

mgr/dashboard: Ensure E2E tests can be run independently

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Nathan Weinberg <nweinber@redhat.com>
Reviewed-by: Rafael Quintero <rquinter@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #28453 into master

* refs/pull/28453/head:
qa/valgrind.supp: be slightly less specific on suppression
msg/async, v2: make the reset_recv_state() unconditional.

Reviewed-by: Sage Weil <sage@redhat.com>

test_volume_client: don't shadow class variable

Name class and instance variables differently so that the latter doesn't
shadow the former.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #28566 from liewegas/wip-40377

osd: copy (don't move) pg list when sending beacon

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Kefu Chai <kchai@redhat.com>

qa/rgw: update default port in perl workunits

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: extra s3tests tasks use rgw endpoint configuration

Signed-off-by: Casey Bodley <cbodley@redhat.com>

journal: properly advance read offset after skipping invalid range

Fixes: https://tracker.ceph.com/issues/40409
Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge PR #28212 into master

* refs/pull/28212/head:
mds: trim cache during standby replay

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28551 into master

* refs/pull/28551/head:
mds: don't mark cap NEEDSNAPFLUSH if client has no pending capsnap

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28545 into master

* refs/pull/28545/head:
client: set snapdir's link count to 1

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

Merge pull request #28496 from ifed01/wip-ifed-fix-40080

os/bluestore: fix duplicate allocations in bmap allocator

Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28509 from tchaikov/wip-mon_clean_pg_upmaps_per_chunk

common/options: make mon_clean_pg_upmaps_per_chunk unsigned

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #28418 from xiexingguo/wip-kick-recovery-priority

osd: give recovery ops initialized by client op a higher priority

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge pull request #25395 from rzarzynski/wip-build-compiletest_cxx11_client

tests: introduce compiletest_cxx11_client for C++11 conformity.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #28229 into master

* refs/pull/28229/head:
os/bluestore: do garbage collection if blob count is too high.
common/perf_conters: make dump_formatted_xxx funcs as const.
os/bluestore: store extents for GC within WriteContext.
os/bluestore: GC class, make some members local.
os/bluestore: vector -> interval set in GC to track extents to collect.
tests/store_test: many-many spanning blobs test case

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28630 into master

* refs/pull/28630/head:
os/bluestore: fix duplicative and misleading debug in KernelDevice::open().

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

mgr/dashboard: Ensure E2E tests can be run independently

Introduces the PageHelper.

- Reads class attributes from Helpers to reduce boilerplace code for
  navigation.
- The PageHelper is supposed to be the new class for code that's reused
  across all Helpers for page related code.
- The Helper class is by now meant to be used for non-page specific
  helper code, like used in the `checkConsole` method.
- The Helper class will act as central registry to enable all other
  tests to use helper functions of all derived PageHelper tests.

  Example:

  `Helper.pools.create('foobar', ...);`

Fixes: http://tracker.ceph.com/issues/40397
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

os/bluestore: fix duplicative and misleading debug in KernelDevice::open().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr, osd: 'ceph osd df' by pool

Our test admin has been asking for this for the past few years:-)
Besides, this is also useful for operating on large Ceph clusters with
mutliple storage pools possibly spanning over all osds.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>

mgr/dashboard: Update translations (#28059)

mgr/dashboard: Update translations

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

qa/workunits/rbd: tweak mirror tests

(after "image mirror disable" started to disable journaling)

Signed-off-by: Mykola Golub <mgolub@suse.com>

librbd: mirror image enable/disable should enable/disable journaling

Fixes: https://tracker.ceph.com/issues/40071
Signed-off-by: Mykola Golub <mgolub@suse.com>