rbd: current pc value should be greater than last_pc

If rbd image enabled striping feature, stderr will
print repeat rate of progress when execu rbd export-diff.

Fixes: https://tracker.ceph.com/issues/48431
Signed-off-by: Yite Gu <yite.gu@easystack.cn>

Merge pull request #38360 from cyx1231st/wip-seastar-msgr-fix-gate-exception

crimson/net: misc fixes and cleanups to messenger

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38327 from lnsyyj/wip-make-error

debian/control: add libffi-dev build dep for aarch64

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38380 from lixiaoy1/pkg_fix

cmake: fix unknown pkg_search_module

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: fix unknown pkg_search_module

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

debian/control: building wheel for cffi(no dependency)

This commit fixes Building wheel for cffi (setup.py) ... error:
   c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory

some python packages do not offer precompiled binary packages for aarch64 on PyPI.
only happens on non-amd64 arches, so add [!amd64].

Signed-off-by: JiangYu <lnsyyj@hotmail.com>

crimson/net: make ms_dispatch() return optional<future<>>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38378 from xxhdx1985126/wip-obc-with-lock-issue

crimson/osd: add _with_lock method to obc

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: use boost::container::small_vector in ChainedDispatchers

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: add _with_lock method to obc

this is modeling the seastar::with_lock() method with the exception
that obc is also captured in the finally block.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #34462 from inspur-wyq/wip-44987

rgw: belong the anonymous object to the bucket owner

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35511 from yehudasa/wip-rgw-cr-drain

rgw: refactor boilerplate coroutine drain callers

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #37733 from orztt/master

cls/rgw/cls_rgw.cc: fix multiple lastest version problem

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #31393 from jiahuizeng/int-range

rgw: avoid expiration early triggering caused by overflow

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #35059 from matthewoliver/s3_put_bucket_policy_204

rgw: S3 Put Bucket Policy should return 204 on success

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37933 from markhoughton-microfocus/issue_47586_fix

rgw: check object locks in multi-object delete

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #38311 from anthonyeleven/anthonyeleven/fix-typo

doc/rados/operations: typo in stretch-mode.rst

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #38366 from tchaikov/wip-templated-doc

doc: drop edit-on-github extension and templatize page.html further

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #38034 into master

* refs/pull/38034/head:
lvm/create.py: fix a typo in the help message

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Dimitri Savineau <dsavinea@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

doc: check "release" context variable for #dev-warning

if $release is "dev", then the release being built is not a stable
version.

drop ceph.js, as the web page is generated at build time.

since the last user of "releases.json" is gone, now we can
drop the hooks preparing "releases.json" in conf.py as well.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: disable eol-warning based on "is_release_eol"

enable/disable #eol-warning based on "is_release_eol" context variable.
this allows us to generate a more static web page. and pave the road to
a releases.json-free doc build process.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: add docubetter link back

check if the release being built is EOL when building the document, so
no need to use javascript to read releases.json anymore.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38219 from rzarzynski/wip-crimson-restore-obs

crimson: conditionally reload ObjectState if execution fails

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38370 from rzarzynski/wip-kv-memdb-uninitialized_in_init

kv/memdb: fix uninitialized return of MemDB::_init().

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #36118 from rzarzynski/wip-crimson-backfill-part1

crimson: add support for backfill, part 1 -- unit testing

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38365 into master

* refs/pull/38365/head:
doc/mds: fix the default value for mds_dir_max_commit_size

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #38218 from aaugustinas/patch-1

doc: Remove duplicate sentence in sync module documentation

Reviewed-by: Zac Dover <zac.dover@gmail.com>

kv/memdb: fix uninitialized return of MemDB::_init().

Without this fix GCC complains:

```
src/kv/MemDB.cc:179:10: warning: ‘r’ may be used uninitialized in this function [-Wmaybe-uninitialized]
  179 |   return r;
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: conditionally reload ObjectState if execution fails.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: unify error handling paths in PG::do_osd_ops().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #38008 from trociny/wip-47388

librbd: serialize maintenance operations by type

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

crimson: fix the condition in the loop of BackfillState::Enqueuing.

The bug here was that if a backfill target has an object-to-delete
placed after the very last object of primary, it will be missed.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/tests: bring two_pseudorandomized_replicas.

It discovered an issue with occasional missing deletion
of the very last object on replica if it doesn't exist
on primary.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: fix mismatch between backfill's enqueue_push() and recovery.

The unittesting discovered an issue between `BackfillState` and
the low-layer recovery code: `RecoveryBackend::recover_object()`.
`BackfillState::enqueue_push()` was assuming it controlls also
to which backfill target the push is sent while `recover_object()`
calculated the set on its own.
Fixing this is the reason why `enqueue_push()` loses the `const
pg_shard_t& target` parameter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/tests: add simple multi-replica test for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: adhere to aliasing of a variable in BackfillState::Enqueuing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/tests: implement randomized testing for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

tests: modernize all occurrences of rand_choose().

We should consider deduplication as well.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/tests: add unittesting for backfill.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop unnecessary includes from backfill_state.{cc,h}.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: do not refence PGLog in BackfillState.

This commit makes the backfill code unware about pg log
implementation details. The benefit is easier unit testing.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop the depedency on pg_info_t from PGFacade.

`get_info()` gets replaced with `get_last_update() and
`get_log_tail()`. This entirely removes the awareness of
`pg_info_t` in the backfill code.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: replace get_peer_info() in BackfillState::PGFacade.

`get_peer_info()` gets replaced with `get_peer_last_backfill()`.
The benefit is no need to mock peer's `pg_info_t` in unit test.
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/conf: find the latest stable release in releases.yml

instead of hardwiring the "|stable-release|" to "octopus" use the latest
release name found in 'doc/releases/releases.yml'

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/conf: read version and release from src/ceph_release

instead of hardwiring them to 'dev' and 'dev', use the src/ceph_release
as the source of truth.

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc: remove edit_on_github sphinx extension

Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/mds: fix the default value for mds_dir_max_commit_size

The default value has changed to 10MB.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge PR #38093 into master

* refs/pull/38093/head:
qa: add debug information for client address for kclient
qa/kernel_mount.py: rename _read_debug_file to read_debug_file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38083 into master

* refs/pull/38083/head:
mds: don't access mds->mdsmap without holding mds_lock

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38315 from badone/wip-mon-scrub-testing

Test mon scrub and exercise mon scrub error injectors

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/net: ms_dispatch() use ConnectionRef

The future returned by ms_dispatch() is only for throttling, not for
Connection lifecycle management. And Messenger may not hold the
connection reference once it is closed.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/test: implement get_server_addr()

So we can distinguish which messenger test is still pending when we have
bind failure.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: avoid using finally

Messenger will only throw unexpected exceptions when there is a bug. In
that case, we should not do any further operations which may result in
segmentation fault and hide the original exception.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: make bind()/try_bind() return errorated future

Also fixed callers to handle the error: abort immediately upon bind
failure and report error. Previously, these callers didn't handle bind
failures correctly and would result in misleading undefined behaviors.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: fix incorrect header

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: cleanup interfaces to start and stop a messenger

There is no on-going iterations in the new
ChainedDispatchers::ms_dispatch() implementation, so we no longer need
to worry about removing dispatchers when stopping the messenger. So the
""boost::intrusive::slist" is not needed, and we can use cleaner
interfaces to start and stop the messenger.

Also fixed an regression issue in perf_crimson_msgr caused by
ChainedDispatchers.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

doc/rados/operations: typo in stretch-mode.rst

Merge pull request #38127 from tchaikov/wip-crimson-client-request-wait

crimson/osd: do not wait for sent epoch

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #38203 from tchaikov/wip-doc-build-releases.yaml

doc/conf: generate releases.json in conf.py and cleanups

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38239 from tchaikov/wip-48320

mon/OSDMonitor: drop osd_epoch of manually marked out osd

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38281 from tchaikov/wip-mgr-no-kv-options

mgr: do not migrate conf from config-key store to new-style conf

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38305 from tchaikov/wip-crimson-finally

crimson: fix finally and erroratorize PG::with_*_obc()

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge PR #38248 into master

* refs/pull/38248/head:
doc: Adding missing Ceph Foundation board members

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38313 from wjwithagen/wjw-fix-src-HttpClient.h

librbd: fix Clang warnings

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #38113 from rhcs-dashboard/login-name-cleanup

mgr/dashboard: Change the text 'Login' to 'Log in'

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38209 from votdev/issue_44404_placement_column

mgr/dashboard: display placement column in service table

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38163 from bk201/wip-48245

mgr/dashboard: fix exception in get_smart_data_by_daemon

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: p-se <pseidensal@suse.com>

Merge pull request #38330 from tchaikov/wip-librados-no-reuse

include/rados/librados.h: note that {read,write}_op cannot be reused

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #37994 from s0nea/wip-dashboard-button-panel

mgr/dashboard: refactor the order of buttons shown in forms

Reviewed-by: Courtney Caldwell <ccaldwel@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

include/rados/librados.h: note that {read,write}_op cannot be reused

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/net: make Dispatcher::ms_dispatch() a pure virtual interface

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: link the throttling future in both v1 and v2

The current implementations are correct and will not link the
throttleing future with unrelated futures.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #37503 from TRYTOBE8TME/wip-rgw-sts-web-identity

Wip rgw sts web identity

qa: STS Web Identity task files

This commit comprises of 3 things:
1) Keycloak task (to bring up keycloak in teuthology inorder to execute Assume_Role_With_Web_Identity tests).
2) The required changes for this in s3tests.py which consists of writing some parameters to configuration file.
3) The yaml files which executes these tasks.

Signed-off-by: Kalpesh Pandya <kapandya@redhat.com>

Merge pull request #38277 from b-ranto/wip-prom-fixes

mgr/prometheus: Make module more stable

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38286 from votdev/issue_48360_disable_tlsv1

mgr/dashboard: Disable TLS 1.0 and 1.1

Reviewed-by: Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38321 from rhcs-dashboard/add-dashboard-master

mailmap: add dashboard contributors

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #38221 into master

* refs/pull/38221/head:
.github: add mon and mgr labelers

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

mailmap: add dashboard contributors

Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

mgr/dashboard: Change the text 'Login' to 'Log in'

Fixes: https://tracker.ceph.com/issues/48253
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38304 from sebastian-philipp/cephadm-inventory-cache-docstring

mgr/cephadm: add docstring describing the HostCache

Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

Merge pull request #38078 from zdover23/wip-doc-docs-style-guide-prompts-nov-2020

doc/start: update prompts in documenting-ceph.rst

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38303 from aclamk/wip-bs-types-test-leak

os/bluestore: Fixed memory leak in test

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38058 from lxbsz/rbd_expand11

global: reexpand the config meta just after the fork() is done

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38240 from ifed01/wip-ifed-is-valid-io-log

blk: log is_valid_io() parameters when unsuccessful.

Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38262 from tchaikov/wip-librbd-gcc-10

neorados: use lambda instead of std::mem_fn()

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38278 from votdev/fix_cephadm_typo

pybind/mgr/cephadm: Fix various typos

Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #38211 from ricardoasmarques/sort-host-device-ls

mgr/orchestrator: Sort 'ceph orch device ls' by host

Reviewed-by: Thomas Bechtold <tbechtold@suse.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #38138 from batrick/centos-8.2

qa: update centos release to 8.2

Reviewed-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #38224 from Rethan/fix_pg_dump_json

mgr: fix "ceph pg dump_json/dump_pools_json" format error

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38206 from dvanders/dvanders_upmap

mgr/balancer: fix available pgs sent to calc_pg_upmaps

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge pull request #38264 from tchaikov/wip-mgr-prometheus-cleanups

pybind/mgr/prometheus: cleanups

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #38241 from Rethan/update_url_from_master_to_lastest

doc/dev: use http://docs.ceph.com/en/latest/ instead of /docs/master/ for docs

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #38306 from wjwithagen/wjw-fix-uring-functype

blk: fix parameters for non native uring

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/net: return tuple<bool, future<>> from ms_dispatch()

Return a boolean to notify whether the Dispatcher has claimed the
message. And fix all the ms_dispatch() to return future only for
throttling purposes, which is currently always seastar::now().

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/net: isolate exceptions from dispatcher implementations

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson: add a missing header to admin_socket.cc

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

qa/suites/rados/monthrash: Exercise mon scrub error injectors

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

qa/config/rados.yaml: Test mon scrub

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

librbd: remove unwanted std::move

Clang argues:
```
/home/jenkins/workspace/ceph-master-compile/src/librbd/migration/HttpClient.cc:463:16: warning: moving a temporary object prevents copy elision [-Wpessimizing-move]
    response = std::move(m_parser->release());
               ^
/home/jenkins/workspace/ceph-master-compile/src/librbd/migration/HttpClient.cc:463:16: note: remove std::move call here
    response = std::move(m_parser->release());
               ^~~~~~~~~~                   ~
1 warning generated.

```

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>