rgw: pass null_yield to ctl->user->read_stats()

this change fixes the FTBFS introduced by
4758e50c96afb2e14700e3b91161313ec9567692

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38167 from CloudFerro/wip-disable-prefetch-in-rgw-file

rgw: disable prefetch in rgw_file to fix 3x read amplification

Merge pull request #37595 from pritha-srivastava/wip-rgw-sts-old-users

rgw/sts: adding code for creating oidc users in a backward

Merge pull request #38412 from badone/wip-build-uring-alternate-git-repo

cmake: Alter GIT_REPOSITORY for liburing

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Alter GIT_REPOSITORY for liburing

https://git.kernel.dk/liburing appears, at least currently, to be
unavailable. This could be reverted once it comes back.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38409 from rhcs-dashboard/fix-48441-master

mgr/dashboard: fix wrong import in iscsi unit test

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #38289 from smithfarm/wip-rn-hotfix-octo

doc: add release notes for Octopus 15.2.7

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: fix wrong import in iscsi unit test

This commit bdf866e436b8772a4080ade7298192eee9adcc97 happened after the
last run of https://github.com/ceph/ceph/pull/37198 make check. That's
why that latter PR passed the tests then but failed when merged in
master.

Fixes: https://tracker.ceph.com/issues/48441
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38385 from rzarzynski/wip-rbd-memset_s

librbd: use the ceph_memzero_s() wrapper instead of explicit_bzero()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38395 from trociny/wip-rbd-test-migration-verbose

test/librbd: print difference if deep-copy or migration test fails

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38386 from petrutlucian94/duplicate_get_size

Revert "rbd: add missing FileStream::get_size"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #38364 into master

* refs/pull/38364/head:
mds/CDir: make the data length as long as possible for each op
mds/CDir: correct the write_size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38326 into master

* refs/pull/38326/head:
client: cleanup the handle_mds_map() code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38112 into master

* refs/pull/38112/head:
Revert "client: optimize rename operation under different quota root"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38033 into master

* refs/pull/38033/head:
client: do not use g_conf().get_val<>() in libcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #37708 into master

* refs/pull/37708/head:
qa/suites/fs: enable thrashing in multifs environment
qa/workunits/fs/snaps: allow tests to be run
qa/tasks/{kclient,ceph_fuse}: allow mounting
qa/tasks: allow per file system config setting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38369 from linuxbox2/wip-rgwfile-cporder

rgw_file: return common_prefixes in lexical order

Merge pull request #38383 from shellwedance/doc_fix_typos

doc/rbd: fix doc typos

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38320 from tchaikov/wip-cmake-std-fs

cmake: do not always require extra link flags for std::filesystem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38390 from xxhdx1985126/wip-obc-related-issues

crimson/osd: fix several obc related issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38397 from tchaikov/wip-crimson-tests

test/crimson: adapt to the changes of messenger

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/releases: add recent releases to timeline

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #38136 from yuvalif/wip-yuval-fix-47904

rgw/notification: trigger notifications on changes from any user

Merge pull request #38171 from yuvalif/wip-yuval-fix-46296

rgw/notification: support GetTopicAttributes API

crimson/osd: drop redundant locking operations in ObjectContext::wait_recovery_read()

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

cmake: pass REQUIRED to find_package(StdFilesystem)

otherwise there is chance that we could link to
StdFilesystem::filesystem which is not defined anywhere.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not always require extra link flags for std::filesystem

since GCC-9

> Using the types and functions in <filesystem> does not require linking
> with -lstdc++fs now.

see https://gcc.gnu.org/gcc-9/changes.html#libstdcxx

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38300 from rhcs-dashboard/fix-saml2-endpoint

mgr/dashboard: SSO not working after REST API versioning

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38154 from votdev/issue_48181_pipe_function

mgr/dashboard: Use pipe instead of calling function within template wherever possible

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38019 from bk201/wip-48176

mgr/dashboard: allow selecting all daemons for Orchestrator NFS clusters

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37198 from tspmelo/wip-iscsi-fix-frontend

mgr/dashboard: Allow managing iSCSI Initiator after removing from group

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

test/crimson: adapt to the changes of messenger

the crimson::net::Messenger interface was changed in
44585adc78bded751b8b50d6304068a8c5186fa1 and
ff2c3b597de4c5707e18529dfa6bed162026014c, so need to change the tests
accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd: print difference if deep-copy or migration test fails

It may appear to be useful to track the sporadic test failures
observed on jenkins, not reproducible locally.

Previously it was disabled because the output could be too
large. But after the hexdump was improved to skip repeating bytes
the output will hopefully be much smaller.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38360 from cyx1231st/wip-seastar-msgr-fix-gate-exception

crimson/net: misc fixes and cleanups to messenger

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: call ObjectContext::with_promoted_lock() with the currently demanding lock type

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #38327 from lnsyyj/wip-make-error

debian/control: add libffi-dev build dep for aarch64

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38380 from lixiaoy1/pkg_fix

cmake: fix unknown pkg_search_module

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: fix unknown pkg_search_module

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Revert "rbd: add missing FileStream::get_size"

This function was missing initially and was added by two separate
commits [1][2]. We have to remove one of the two definitions.

This reverts commit c17c53f460254c29e961227e19484c527fd08fc7.

[1] c17c53f460254c29e961227e19484c527fd08fc7
[2] 074dd33c4f0112839a7c51ab97c805ee288d84c1

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

librbd: use the ceph_memzero_s() wrapper instead of explicit_bzero().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/rbd: fix doc typos

fix typos in Ceph iSCSI manual

Signed-off-by: Taeuk Kim <taeuk_kim@tmax.co.kr>

debian/control: building wheel for cffi(no dependency)

This commit fixes Building wheel for cffi (setup.py) ... error:
   c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory

some python packages do not offer precompiled binary packages for aarch64 on PyPI.
only happens on non-amd64 arches, so add [!amd64].

Signed-off-by: JiangYu <lnsyyj@hotmail.com>

crimson/net: make ms_dispatch() return optional<future<>>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38378 from xxhdx1985126/wip-obc-with-lock-issue

crimson/osd: add _with_lock method to obc

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: use boost::container::small_vector in ChainedDispatchers

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: add _with_lock method to obc

this is modeling the seastar::with_lock() method with the exception
that obc is also captured in the finally block.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #34462 from inspur-wyq/wip-44987

rgw: belong the anonymous object to the bucket owner

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35511 from yehudasa/wip-rgw-cr-drain

rgw: refactor boilerplate coroutine drain callers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37733 from orztt/master

cls/rgw/cls_rgw.cc: fix multiple lastest version problem

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31393 from jiahuizeng/int-range

rgw: avoid expiration early triggering caused by overflow

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #35059 from matthewoliver/s3_put_bucket_policy_204

rgw: S3 Put Bucket Policy should return 204 on success

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37933 from markhoughton-microfocus/issue_47586_fix

rgw: check object locks in multi-object delete

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

rgw/notification: trigger notifications on changes from any user

any user authorized to make changes to a bucket may trigger
notifications defined on that bucket.
manual test procedure of the fix is described here:
https://gist.github.com/yuvalif/39c183aa0f74d286ecef7844268817df

fixes: https://tracker.ceph.com/issues/47904

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #38311 from anthonyeleven/anthonyeleven/fix-typo

doc/rados/operations: typo in stretch-mode.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38366 from tchaikov/wip-templated-doc

doc: drop edit-on-github extension and templatize page.html further

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #38034 into master

* refs/pull/38034/head:
lvm/create.py: fix a typo in the help message

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

doc: check "release" context variable for #dev-warning

if $release is "dev", then the release being built is not a stable
version.

drop ceph.js, as the web page is generated at build time.

since the last user of "releases.json" is gone, now we can
drop the hooks preparing "releases.json" in conf.py as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: disable eol-warning based on "is_release_eol"

enable/disable #eol-warning based on "is_release_eol" context variable.
this allows us to generate a more static web page. and pave the road to
a releases.json-free doc build process.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add docubetter link back

check if the release being built is EOL when building the document, so
no need to use javascript to read releases.json anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38219 from rzarzynski/wip-crimson-restore-obs

crimson: conditionally reload ObjectState if execution fails

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38370 from rzarzynski/wip-kv-memdb-uninitialized_in_init

kv/memdb: fix uninitialized return of MemDB::_init().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36118 from rzarzynski/wip-crimson-backfill-part1

crimson: add support for backfill, part 1 -- unit testing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38365 into master

* refs/pull/38365/head:
doc/mds: fix the default value for mds_dir_max_commit_size

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #38218 from aaugustinas/patch-1

doc: Remove duplicate sentence in sync module documentation

Reviewed-by: Zac Dover <zac.dover@gmail.com>

kv/memdb: fix uninitialized return of MemDB::_init().

Without this fix GCC complains:

```
src/kv/MemDB.cc:179:10: warning: ‘r’ may be used uninitialized in this function [-Wmaybe-uninitialized]
  179 |   return r;
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: conditionally reload ObjectState if execution fails.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: unify error handling paths in PG::do_osd_ops().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw_file: return common_prefixes in lexical order

Since inception RGWReaddirRequest has sent all leaf objects first
(in lexical order), then common_prefixes (in lexical order). In
hindsight, an overall listing could trivially be returned out of
lexical order, which can cause continued listing of large, mixed
directories to fail.

RCA by Dan Gryniewicz.

Fixes: https://tracker.ceph.com/issues/48410
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #38008 from trociny/wip-47388

librbd: serialize maintenance operations by type

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson: fix the condition in the loop of BackfillState::Enqueuing.

The bug here was that if a backfill target has an object-to-delete
placed after the very last object of primary, it will be missed.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/tests: bring two_pseudorandomized_replicas.

It discovered an issue with occasional missing deletion
of the very last object on replica if it doesn't exist
on primary.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: fix mismatch between backfill's enqueue_push() and recovery.

The unittesting discovered an issue between `BackfillState` and
the low-layer recovery code: `RecoveryBackend::recover_object()`.
`BackfillState::enqueue_push()` was assuming it controlls also
to which backfill target the push is sent while `recover_object()`
calculated the set on its own.
Fixing this is the reason why `enqueue_push()` loses the `const
pg_shard_t& target` parameter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/tests: add simple multi-replica test for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: adhere to aliasing of a variable in BackfillState::Enqueuing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/tests: implement randomized testing for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tests: modernize all occurrences of rand_choose().

We should consider deduplication as well.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/tests: add unittesting for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop unnecessary includes from backfill_state.{cc,h}.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: do not refence PGLog in BackfillState.

This commit makes the backfill code unware about pg log
implementation details. The benefit is easier unit testing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop the depedency on pg_info_t from PGFacade.

`get_info()` gets replaced with `get_last_update() and
`get_log_tail()`. This entirely removes the awareness of
`pg_info_t` in the backfill code.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: replace get_peer_info() in BackfillState::PGFacade.

`get_peer_info()` gets replaced with `get_peer_last_backfill()`.
The benefit is no need to mock peer's `pg_info_t` in unit test.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/conf: find the latest stable release in releases.yml

instead of hardwiring the "|stable-release|" to "octopus" use the latest
release name found in 'doc/releases/releases.yml'

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/conf: read version and release from src/ceph_release

instead of hardwiring them to 'dev' and 'dev', use the src/ceph_release
as the source of truth.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: remove edit_on_github sphinx extension

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: SSO not working after REST API versioning

Fixes: https://tracker.ceph.com/issues/48362
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
saml2: Fix versioning parameter for saml2 endpoints, including minor fix for
endpoints with xml=true

doc/mds: fix the default value for mds_dir_max_commit_size

The default value has changed to 10MB.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

mds/CDir: make the data length as long as possible for each op

The max_write_size is 10M as default, and every time when the len
of the op data exceed it with a very small size it will be split
into two ops, one of which will always be very small.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

mds/CDir: correct the write_size

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge PR #38093 into master

* refs/pull/38093/head:
qa: add debug information for client address for kclient
qa/kernel_mount.py: rename _read_debug_file to read_debug_file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38083 into master

* refs/pull/38083/head:
mds: don't access mds->mdsmap without holding mds_lock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38315 from badone/wip-mon-scrub-testing

Test mon scrub and exercise mon scrub error injectors

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/net: ms_dispatch() use ConnectionRef

The future returned by ms_dispatch() is only for throttling, not for
Connection lifecycle management. And Messenger may not hold the
connection reference once it is closed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/test: implement get_server_addr()

So we can distinguish which messenger test is still pending when we have
bind failure.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: avoid using finally

Messenger will only throw unexpected exceptions when there is a bug. In
that case, we should not do any further operations which may result in
segmentation fault and hide the original exception.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: make bind()/try_bind() return errorated future

Also fixed callers to handle the error: abort immediately upon bind
failure and report error. Previously, these callers didn't handle bind
failures correctly and would result in misleading undefined behaviors.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: fix incorrect header

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: cleanup interfaces to start and stop a messenger

There is no on-going iterations in the new
ChainedDispatchers::ms_dispatch() implementation, so we no longer need
to worry about removing dispatchers when stopping the messenger. So the
""boost::intrusive::slist" is not needed, and we can use cleaner
interfaces to start and stop the messenger.

Also fixed an regression issue in perf_crimson_msgr caused by
ChainedDispatchers.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/rados/operations: typo in stretch-mode.rst

Merge pull request #38127 from tchaikov/wip-crimson-client-request-wait

crimson/osd: do not wait for sent epoch

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38203 from tchaikov/wip-doc-build-releases.yaml

doc/conf: generate releases.json in conf.py and cleanups

Reviewed-by: Josh Durgin <jdurgin@redhat.com>