pybind/rados: extract non-shared interface definitions into c_rados.pxd

so it'd be simpler if we want to parse and generate the stub
for these declarations when building document where where the annotations
are used, but importable python bindgs of rados,ceph,rbd,cephfs are required.

also move the time.h declarations into ctime.pxd, are they can be shared
by the stub implementation and the non-stub one.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38314 from badone/wip-monitor-reset-abort-if-is_shutdown

mon: paxos: Delete logger in destructor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38410 from badone/wip-monitor-logger-deelete-in-destructor

mon: Delete logger in destructor

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37314 from ifed01/wip-ifed-faster-rm-p1

osd: optimize PG removal (part1)

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Adam Kupczyk <akupczyk@redhat.com>

Merge pull request #38438 from ifed01/wip-ifed-fix-avl

os/bluestore: fix inappropriate ENOSPC from avl/hybrid allocator

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38436 from ktdreyer/prometheus-err

mgr/prometheus: don't store exception as e

Reviewed-by: Boris Ranto <branto@redhat.com>

Merge pull request #38400 from tchaikov/wip-pybind-use-non-deprecated-functions

pybind/rados: use rados_set_pool_full_try instead of the deprecated one

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #36384 from linuxbox2/wip-doc-ippolicy

rgw:doc: document support for (Not)IpAddress in S3 policy

Merge pull request #37546 from myoungwon/wip-revamp-tier-flush

osd: refactoring tier_flush()

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/osd-config-ref: adjust default value for osd_delete_sleep_hybrid

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd: adjust osd_delete_sleep_ssd[_hybrid]

1) Do not differentiate all-flash and hybrid(ssd+hdd) deployments since
they both bound to SSD drive backing DB volume.
2) Drop osd_delete_sleep_hybrid from 2s to 1s
3) Raise osd_delete_sleep_ssd to 1s since 0 is a way too low - having
it that low tend to cause a significant negative imact on OSD's regular
operation.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: introduce removal perf counter

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: track kv utilization in kv_sync_thread

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

os/bluestore: perf counters for omap_get_keys[_get_values]

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

osd/pg: use next when calling collection_list for pg removal

Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38449 from trociny/wip-rbd-flush-finisher

librbd: flush task finisher queue on unregistering watcher

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38399 from es-gyt/master

rbd: current pc value should be greater than last_pc

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38403 from zdover23/wip-doc-CLT-update-02-Dec-2020-1

doc/governance.rst - updating CLT list

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #38440 from bk201/wip-48454

mgr/dashboard: fix trusted_ip_list in iSCSI service form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #38398 from tchaikov/wip-48210

pybind/mgr: unpin urllib3

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38298 from bk201/wip-43165

mgr/dashboard: refactor /api/orchestrator/* endpoints

Reviewed-by: Courtney Caldwell <ccaldwel@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #38432 from rhcs-dashboard/orch-not-available-incorrect-status

mgr/dashboard: Fix for misleading "Orchestrator is not available" error

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

test/librbd: fix race in TestLibRBD.ConcurentOperations

It was possible that before image1 was closed, both quiesce2 and
quiesce3 requests were received and the test got stuck on
create_snap1.join() waiting for the image1 to be closed, while
the close was waiting for quiesce3 to be completed.

Signed-off-by: Mykola Golub <mgolub@suse.com>

rbd: current pc value should be greater than last_pc

If rbd image enabled striping feature, stderr will
print repeat rate of progress when execu rbd export-diff.

Fixes: https://tracker.ceph.com/issues/48431
Signed-off-by: Yite Gu <yite.gu@easystack.cn>

Merge pull request #38379 from dillaman/wip-librbd-migration-5

librbd/migration: add thick-provisioned snapshot support to the raw format

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38285 from jmolmo/include_hw_info_in_HostCache

mgr/cephadm: Get host facts data in host cache

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

doc/governance.rst - updating CLT list

This PR removes the following people from the
CLT. They've left the CLT:
* Lenz Grimmer
* Alfredo Deza
* Zack Cerza

This commit also adds:
* Ernesto Puerta

Fixes: https://tracker.ceph.com/issues/48436
Signed-off-by: Zac Dover <zac.dover@gmail.com>

mgr/cephadm: Get host facts data in host cache

Include cephadm gather-facts information in host cache

Signed-off-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>

os/bluestore: fix inappropriate ENOSPC from avl/hybrid allocator

Under high fragmentation circumstances (in best-fit mode) these allocators might fail to
return continuous(longer than a single alloc unit) extents. Occurs when
requested size is longer than the maximum available extents and these
extents are unaligned with alloc unit. The latter results in the actual lengths of these extens are shorter
than the requested length and they do not match to the lookup. Which
finally causes no allocation at all.

Fixes: https://tracker.ceph.com/issues/47883
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

librbd: flush task finisher queue on unregistering watcher

To make sure all inflight requests like scheduled
cancel_async_requests are complete on shutdown.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37631 from zjcmszh/aio_setxattr

pybind/rados: add Ioctx::aio_setxattr() and remove_omap_range2()

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38168 from agayev/cleaner-plumbing

os/bluestore: Introduce plumbing necessary for garbage collecting zones.

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38323 from wjwithagen/wjw-fix-nasm-wrapper

nasm-wrapper: Make nasm-wrapper OS independent

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38325 from aclamk/bs-onode-put

os/bluestore: Only pass that decremented nref to 0 deletes object

Reviewed-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #38387 from hualongfeng/vstart_add_path

vstart.sh: add PATH in vstart_environment

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38358 from batrick/i48030

librados: add RadosClient to conf observers

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: fix trusted_ip_list in iSCSI service form

The value should be a comma-separated string rather than a list of strings.

Fixes: https://tracker.ceph.com/issues/48454
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

mgr/prometheus: don't store exception as e

Python's logging module's exception() method will log the full exception
and stack trace for us, so we do not need to store the exception in the
"e" variable here.

Signed-off-by: Ken Dreyer <kdreyer@redhat.com>

Merge pull request #37548 from ofriedma/wip-ofriedma-quota-hdrs

rgw: send headers of quota settings

mgr/dashboard: Fix for misleading "Orchestrator is not available" error

When switching between options, in the Services and Inventory page sometimes it shows "Orchestrator not available" error which appears for 2-3 second and then disappears and shows the normal page. This commit fixes that.

Fixes: https://tracker.ceph.com/issues/48448
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38433 from tchaikov/wip-rgw

rgw: pass null_yield to ctl->user->read_stats()

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: pass null_yield to ctl->user->read_stats()

this change fixes the FTBFS introduced by
4758e50c96afb2e14700e3b91161313ec9567692

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/rbd: new basic migration CLI test case

Pending a larger suite of tests for instant-restore image migration,
this test provides a basic sanity check for both the native and
raw image formats -- including basic snapshot tests.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38167 from CloudFerro/wip-disable-prefetch-in-rgw-file

rgw: disable prefetch in rgw_file to fix 3x read amplification

Merge pull request #37595 from pritha-srivastava/wip-rgw-sts-old-users

rgw/sts: adding code for creating oidc users in a backward

Merge pull request #38412 from badone/wip-build-uring-alternate-git-repo

cmake: Alter GIT_REPOSITORY for liburing

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: Alter GIT_REPOSITORY for liburing

https://git.kernel.dk/liburing appears, at least currently, to be
unavailable. This could be reverted once it comes back.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

Merge pull request #38409 from rhcs-dashboard/fix-48441-master

mgr/dashboard: fix wrong import in iscsi unit test

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>

Merge pull request #38289 from smithfarm/wip-rn-hotfix-octo

doc: add release notes for Octopus 15.2.7

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mon: Delete logger in destructor

In theory we could race on the 'logger' object in a similar fashion to
https://tracker.ceph.com/issues/48386 where the race is on the
Paxos::logger object.

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>

mgr/dashboard: fix wrong import in iscsi unit test

This commit bdf866e436b8772a4080ade7298192eee9adcc97 happened after the
last run of https://github.com/ceph/ceph/pull/37198 make check. That's
why that latter PR passed the tests then but failed when merged in
master.

Fixes: https://tracker.ceph.com/issues/48441
Signed-off-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38385 from rzarzynski/wip-rbd-memset_s

librbd: use the ceph_memzero_s() wrapper instead of explicit_bzero()

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38395 from trociny/wip-rbd-test-migration-verbose

test/librbd: print difference if deep-copy or migration test fails

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38386 from petrutlucian94/duplicate_get_size

Revert "rbd: add missing FileStream::get_size"

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge PR #38364 into master

* refs/pull/38364/head:
mds/CDir: make the data length as long as possible for each op
mds/CDir: correct the write_size

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38326 into master

* refs/pull/38326/head:
client: cleanup the handle_mds_map() code

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38112 into master

* refs/pull/38112/head:
Revert "client: optimize rename operation under different quota root"

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #38033 into master

* refs/pull/38033/head:
client: do not use g_conf().get_val<>() in libcephfs

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #37708 into master

* refs/pull/37708/head:
qa/suites/fs: enable thrashing in multifs environment
qa/workunits/fs/snaps: allow tests to be run
qa/tasks/{kclient,ceph_fuse}: allow mounting
qa/tasks: allow per file system config setting

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #38369 from linuxbox2/wip-rgwfile-cporder

rgw_file: return common_prefixes in lexical order

Merge pull request #38383 from shellwedance/doc_fix_typos

doc/rbd: fix doc typos

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #38320 from tchaikov/wip-cmake-std-fs

cmake: do not always require extra link flags for std::filesystem

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #38390 from xxhdx1985126/wip-obc-related-issues

crimson/osd: fix several obc related issues

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38397 from tchaikov/wip-crimson-tests

test/crimson: adapt to the changes of messenger

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/releases: add recent releases to timeline

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #38136 from yuvalif/wip-yuval-fix-47904

rgw/notification: trigger notifications on changes from any user

Merge pull request #38171 from yuvalif/wip-yuval-fix-46296

rgw/notification: support GetTopicAttributes API

crimson/osd: drop redundant locking operations in ObjectContext::wait_recovery_read()

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

pybind/rados: use rados_set_pool_full_try instead of the deprecated one

this change silences the warning when compiling the cythonized python
binding:

build/src/pybind/rados/pyrex/rados.c:64435:9: warning: 'rados_set_osdmap_full_try' is deprecated [-Wdeprecated-declarations]
         rados_set_osdmap_full_try(__pyx_v_self->io);
         ^~~~~~~~~~~~~~~~~~~~~~~~~

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: pass REQUIRED to find_package(StdFilesystem)

otherwise there is chance that we could link to
StdFilesystem::filesystem which is not defined anywhere.

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not always require extra link flags for std::filesystem

since GCC-9

> Using the types and functions in <filesystem> does not require linking
> with -lstdc++fs now.

see https://gcc.gnu.org/gcc-9/changes.html#libstdcxx

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #38300 from rhcs-dashboard/fix-saml2-endpoint

mgr/dashboard: SSO not working after REST API versioning

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #38154 from votdev/issue_48181_pipe_function

mgr/dashboard: Use pipe instead of calling function within template wherever possible

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #38019 from bk201/wip-48176

mgr/dashboard: allow selecting all daemons for Orchestrator NFS clusters

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37198 from tspmelo/wip-iscsi-fix-frontend

mgr/dashboard: Allow managing iSCSI Initiator after removing from group

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

pybind/mgr: unpin urllib3

now that requests is able to support urllib v1.26, there is no need to
pin it anymore.

see also
https://github.com/psf/requests/blob/03957eb1c2b9a1e5e6d61f5e930d7c5ed7cfe853/HISTORY.md#2250-2020-11-11

Fixes: https://tracker.ceph.com/issues/48210
Signed-off-by: Kefu Chai <kchai@redhat.com>

test/crimson: adapt to the changes of messenger

the crimson::net::Messenger interface was changed in
44585adc78bded751b8b50d6304068a8c5186fa1 and
ff2c3b597de4c5707e18529dfa6bed162026014c, so need to change the tests
accordingly.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test/librbd: print difference if deep-copy or migration test fails

It may appear to be useful to track the sporadic test failures
observed on jenkins, not reproducible locally.

Previously it was disabled because the output could be too
large. But after the hexdump was improved to skip repeating bytes
the output will hopefully be much smaller.

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #38360 from cyx1231st/wip-seastar-msgr-fix-gate-exception

crimson/net: misc fixes and cleanups to messenger

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: call ObjectContext::with_promoted_lock() with the currently demanding lock type

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #38327 from lnsyyj/wip-make-error

debian/control: add libffi-dev build dep for aarch64

Reviewed-by: Kefu Chai <kchai@redhat.com>

vstart.sh: add PATH in vstart_environment

Signed-off-by: Hualong Feng <hualong.feng@intel.com>

Merge pull request #38380 from lixiaoy1/pkg_fix

cmake: fix unknown pkg_search_module

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: fix unknown pkg_search_module

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Revert "rbd: add missing FileStream::get_size"

This function was missing initially and was added by two separate
commits [1][2]. We have to remove one of the two definitions.

This reverts commit c17c53f460254c29e961227e19484c527fd08fc7.

[1] c17c53f460254c29e961227e19484c527fd08fc7
[2] 074dd33c4f0112839a7c51ab97c805ee288d84c1

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

librbd: use the ceph_memzero_s() wrapper instead of explicit_bzero().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

doc/rbd: fix doc typos

fix typos in Ceph iSCSI manual

Signed-off-by: Taeuk Kim <taeuk_kim@tmax.co.kr>

debian/control: building wheel for cffi(no dependency)

This commit fixes Building wheel for cffi (setup.py) ... error:
   c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory

some python packages do not offer precompiled binary packages for aarch64 on PyPI.
only happens on non-amd64 arches, so add [!amd64].

Signed-off-by: JiangYu <lnsyyj@hotmail.com>

librbd/migration: minor debug message tweaks

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: generic method for applying source image metadata

Collect the image size and snapshot details from the constructed format
and apply it to the source image ctx.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd/migration: avoid async callback on image ctx open request

If the ImageState::open call fails, we cannot have the callback be
asynchronously dispatched via the now deleted ImageCtx object.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

crimson/net: make ms_dispatch() return optional<future<>>

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #38378 from xxhdx1985126/wip-obc-with-lock-issue

crimson/osd: add _with_lock method to obc

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

osd: return proper error

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: remove chunk_map.clear() becuase new chunk_map
     will be overwritten entirely

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: do not reply EAGAIN if get_lock_type() is false

This will be retried later

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

osd: correct the if case to handle exceptional case

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>

crimson/net: use boost::container::small_vector in ChainedDispatchers

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/osd: add _with_lock method to obc

this is modeling the seastar::with_lock() method with the exception
that obc is also captured in the finally block.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

osd: move setting callback and manifest_ops to caller

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsumg.com>