test/store_test: use 'threadsafe' style for death tests.

Hopefully Fixes: https://tracker.ceph.com/issues/47328
Signed-off-by: Igor Fedotov <ifedotov@suse.com>

Merge pull request #37660 from adamemerson/wip-datalog-fix

cls/fifo: Switch use CLS_ERR for errors
rgw/fifo: Fix a few missed return value assignments
rgw/fifo: Add some error logging
rgw/fifo: Catch two instances journaling a new part
rgw/fifo: Use unique_ptr and explicit release for callbacks

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37575 from bk201/wip-47742

qa/suites/rados/cephadm/dashboard: fix creating OSD failure

Merge pull request #37669 from lixiaoy1/fix_cache_state_ut_1015

rbd/cache: fix UTs related to image cache state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

rbd/cache: fix UTs related to image cache state

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37633 from tchaikov/wip-pybind-rados-cleanup

pybind/rados: add more type hintings and cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37624 from lxbsz/cram

qa/tasks: add a 'parallel' option support for the cram task

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37621 from dsavineau/cephadm_ntpsec

cephadm: allow ntpsec service

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #37601 from tspmelo/wip-rm-wait-for-expected-get-result

qa/mgr: Remove _wait_for_expected_get_result

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37609 from lixiaoy1/remove_writelog

rbd/cache: remove unnecessary WriteLogCache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #37625 from lixiaoy1/rwl_path

rbd/cache: store full cache path to image metadata

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

rgw/fifo: Use unique_ptr and explicit release for callbacks

To simplify and clarify memory allocation in AIO cases.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Catch two instances journaling a new part

If we see another part journaled, just run the journal ourselves.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Add some error logging

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Fix a few missed return value assignments

Slipped in when I converted from call-back based to sequential.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Switch use CLS_ERR for errors

So things going wrong show up in the log by default. Also use
__PRETTY_FUNCTION__ instead of __func__ and print some more context.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #34702 into master

* refs/pull/34702/head:
doc: Make time-related osd default values in documentation big-endian.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37587 from anthonyeleven/yet-more-doc-polishing

doc: clarity, detail, modernization, capitalization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37168 from ifed01/wip-ifed-fix-compress-csum

os/bluestore: attach csum for compressed blobs

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #37611 from tchaikov/wip-cmake-tox

cmake: do not always add py3 to TOX_ENVS

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37594 from trociny/wip-rbd-quiesce-cancel

librbd: fix race on watcher unregister

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37650 from jschmid1/gs

mgr/cephadm: adapt <placement> usage string

mgr/dashboard: add an empty line to appease flake8

silence the warnings like

./controllers/mgr_modules.py:38:5: E301 expected 1 blank line, found 0'
./controllers/user.py:69:5: E301 expected 1 blank line, found 0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37394 from rosinL/cleanup-queue-option

common/options: remove unavailable values of osd_op_queue

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37627 from changchengx/msg

msg: remove stale comment and unnecessary code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37626 from tchaikov/wip-drop-std-iterator

include/interval_set: do not inherit from std::iterator

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #37455 from smithfarm/wip-spec-file-cleanup

rpm: three spec file cleanups

Reviewed-by: Tim Serong <tserong@suse.com>

pybind/rados: correct the declarations of C APIs

so they are consistent with the ones declared in include/rados/librados.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: add more type hintings

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not always add py3 to TOX_ENVS

before this change add_tox_test() always add "py3" to testenv, even the
caller specifies TOX_ENVS explicitly.

after this change, py3 is added only if the caller does not specify any
TOX_ENVS.

this change helps with the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #37583 into master

* refs/pull/37583/head:
mgr/volumes/nfs: Fix wrong error message for pseudo path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37585 into master

* refs/pull/37585/head:
doc/cephfs/nfs: Remove outdated doc related to rook

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37610 from anthonyeleven/doc-rados-troubleshooting

doc/rados/troubleshooting: clarity and modernization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #36537 into master

* refs/pull/36537/head:
qa/cephfs: update ephemeral pin tests
mds: distribute dirfrags for ephemeral distributed directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: update ephemeral pin tests

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: distribute dirfrags for ephemeral distributed directory

Instead of distribute individual dir inodes inside the ephemeral
distributed dir. Distributing dirfrags can limit number of subtrees
created by the ephemeral dist pin.

This patch also unifies codes that handle export pin and ephemeral pin.

Fixes: https://tracker.ceph.com/issues/46696
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mgr/cephadm: adapt <placement> usage string

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #35477 from yuvalif/add_lua_to_rgw

rgw/lua: embedding lua in rgw in s3 request context

Merge pull request #37581 from dillaman/wip-librbd-exclusive-lock-shutdown

librbd: avoid failing IO with -ESHUTDOWN when disabling exclusive-lock

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37613 from tchaikov/wip-cls-osd-cleanup

cls, osd: cleanups

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #37537 from yuvalif/missing_brackets_in_rgw_file

rgw/file: missing brackets around if statement

Merge pull request #37538 from yuvalif/potential_crash_in_kafka

fix potential crash in rgw_kafka

Merge pull request #37599 from yuvalif/remove_doxyfile

cmake: remove Doxyfile as doxygen run via make

Merge pull request #37534 from yuvalif/fix_ownership_in_role_applier

 rgw/auth: fix copy&paste bug in RoleApplier::is_owner_of

Merge PR #29951 into master

* refs/pull/29951/head:
test: add tests for validating MDS metrics via `perf stats` module
test: Filesystem class helpers to grow and shrink MDS cluster
mgr/stats: mds performance stats module
mds: support sending empty perf metrics to ceph-manager

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: misc clarity and capitalization

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37541 from anthonyeleven/osd-internals-tweaks

doc/dev: doc/dev/osd_internals capitalization, formatting, clarity

Reviewed-by: Zac Dover <zac.dover@gmail.com>

rgw/lua: run lua scripts in s3 requests context

for more details on design and
remaining work see:
https://gist.github.com/yuvalif/60d5984c28af89ba17443ce947540c1f

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

doc/rados/troubleshooting: clarity and modernization

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge PR #37593 into master

* refs/pull/37593/head:
qa/tasks/cephfs: Refactor test_volumes

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks: add a 'parallel' option support for the cram task

For the ceph-iscsi test case we need to run the tests sequentially,
because the client test will depend on the gateway ones.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #37154 from bk201/wip-47397

mgr/dashboard: fix the error when exporting CephFS path "/" in NFS exports

Reviewed-by: Laura Paduano lpaduano@suse.com
Reviewed-by: Stephan Müller smueller@suse.com
Reviewed-by: Varsha Rao varao@redhat.com

test: add tests for validating MDS metrics via `perf stats` module

Fixes: http://tracker.ceph.com/issues/24285
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: Filesystem class helpers to grow and shrink MDS cluster

These routines were test specific. Make them a part of Filesystem
class.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/stats: mds performance stats module

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: support sending empty perf metrics to ceph-manager

Right now, there are no per-mds metrics that are tracked and
sent by mds. However, such metrics will get added soon. So,
send empty performance metrics to ceph-manager for now.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/tasks/cephfs: Refactor test_volumes

... into smaller test classes. This enables breaking up the volumes yaml
into smaller yaml fragments.

Fixes: https://tracker.ceph.com/issues/47160
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #37628 from changchengx/mail_update

mailmap: update Intel employee mail/org

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg: remove unnecessary assignment

memset(&header, 0, sizeof(header)) clear the variable to
be 0.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

msg: remove outdated comments

Parameter "features" has been removed in commit 0dbe8fd3987d.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

mailmap: update Intel employee mail/org

1. "changcheng.liu@aliyun.com" need be classified into intel until now.
   This reverts part of commit: df07e9f3
2. add "Yuan Lu <yuan.y.lu@intel.com>" in mailmap

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #36942 from rosinL/wip-fix-librgw

rgw/rgw_file: Fix the incorrect lru object eviction

rgw/file: add missing brackets around if statement

also some small style issues
issues were detected by pvs-studio static analyzer

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rbd/cache: save full cach path instead of folder

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

rbd/cache: remove unsupported poolset

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

include/interval_set: do not inherit from std::iterator

std::iterator is deprecated in C++17, so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd/cache: remove unnecessary WriteLogCache

Remove the class librbd::cache::WriteLogCache.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37580 from aclamk/wip-bluestore-fix-2q-cache

os/bluestore: fix memory accounting in TwoQBufferCacheShard

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37325 from badone/wip-angular-cli-analytics-dont-prompt

mgr/dashboard: Don't prompt for analytics during @angular/cli install

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37542 from SUSE/wip-fix-47745

cephadm: allow uid/gid == 0 in copy_tree, copy_files, move_files

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37559 from wjwithagen/wjw-fix-iterator

include: explicitly define all types needed for libc++ iterator

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37104 from sebastian-philipp/cephadm-serve.py-2

mgr/cephadm: move functions that do remote calls to serve.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #37441 into master

* refs/pull/37441/head:
qa: fix proc exit status check

Reviewed-by: Jos Collin <jcollin@redhat.com>

common/options: remove unavailable values of osd_op_queue

Signed-off-by: luo rixin <luorixin@huawei.com>

cephadm: allow ntpsec service

ntpsec is available starting Ubuntu Focal (20.04) and Debian Buster (10)
as an alternate NTP daemon.

Fixes: https://tracker.ceph.com/issues/47525
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #36979 from tspmelo/wip-npm-update-20-8

mgr/dashboard: Update npm packages

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

rgw/rgw_file: Fix the incorrect lru object eviction

In func lookup_fh, when RGWFileHandle not be found in fh_cache, it
need to recycle an object and create an new RGWFileHandle. When there
are multi threads use lookup_fh to find and create RGWFileHandle concurrently,
it must to make sure evict lru object from the partiton of fh_cache which new
RGWFileHandle will be inserted to.

Fixes: https://tracker.ceph.com/issues/47235
Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #37449 from ceph/47676-zone-empty-realm-id

mgr/dashboard: get rgw daemon zonegroup name from mgr

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37375 from ceph/47615-fix-endpoint-responses

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37064 from bk201/wip-47372

mgr/dashboard: fix error when typing existing paths in the Ganesha form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #37596 from rhcs-dashboard/telemetry_link_cleanup

mgr/dashboard: Improve HTML formatting of the telemetry report preview

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc/cephfs/nfs: Remove outdated doc related to rook

The doc is outdated because the mentioned orch commands have changed[1] and
using dashboard backend script to create exports is buggy[2].

[1] https://docs.ceph.com/en/latest/mgr/orchestrator/#current-implementation-status
[2] https://github.com/rook/rook/issues/6147

Fixes: https://tracker.ceph.com/issues/47784
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/dashboard: Update npm packages

Angular 10.1 includes extraction of TS translations,
so @locl/cli is no longer needed.

Updating Angular automatically removed the usage of base tsconfig.

Downgraded css-loader to be able to use css modules.

angular-tree-component has been deprecated and
replaced by @circlon/angular-tree-component

Fixes: https://tracker.ceph.com/issues/47289
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #32778 into master

* refs/pull/32778/head:
msg: Use sockets instead of pipes for wakeup events on win32
common: Convert remote errors on win32
common,msg: Initialize Windows WSA and TLS
common,os,kv: Define aligned_free
common: avoid CLOCK_*_COARSE warnings on win32
global: Windows support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/OSD: remove unused #include

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/fifo: cast ceph_le64 to uint64_t before printing it

silences warning like:

../src/cls/fifo/cls_fifo.cc: In member function ‘int rados::cls::fifo::{anonymous}::EntryReader::get_next_entry(ceph::buffer::v15_2_0::list*, uint64_t*, ceph::real_time*)’:
../src/include/rados/objclass.h:33:18: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 7 has type ‘ceph_le64’ {aka ‘ceph_le<long long unsigned int>’} [-Wformat=]
   33 |   cls_log(level, "<cls> %s:%d: " fmt, __FILE__, __LINE__, ##__VA_ARGS__)
../src/include/rados/objclass.h:33:18: note: in definition of macro ‘CLS_LOG’
   33 |   cls_log(level, "<cls> %s:%d: " fmt, __FILE__, __LINE__, ##__VA_ARGS__)
      |                  ^~~~~~~~~~~~~~~
../src/cls/fifo/cls_fifo.cc:703:48: note: format string is defined here
  703 |   CLS_LOG(10, "%s():%d: pre_header.pre_size=%llu", __func__, __LINE__,
      |                                             ~~~^
      |                                                |
      |                                                long long unsigned int

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #36554 into master

* refs/pull/36554/head:
mgr/volumes: Make number of cloner threads configurable

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Shyamsundar R <srangana@redhat.com>

Merge PR #36457 into master

* refs/pull/36457/head:
qa/tasks/cephfs: fix fs authorize cmd args
mds: add root_squash mode in MDS auth caps
vstart_runner: add write_file and sudo_write_file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36556 into master

* refs/pull/36556/head:
doc: cephfs mirroring design document

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37483 from dzafman/wip-46405

osd/osd-rep-recov-eio.sh: TEST_rados_repair_warning:  return 1

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37500 from ricardoasmarques/fix-bootstrap-container-init

cephadm: Fix error setting 'mgr/cephadm/container_init' config

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

librbd: fix race on watcher unregister

It was possible that "unregister_watch" would get stuck forever
in "m_async_op_tracker.wait_for_ops", waiting for unqueiesce
notifications to complete, which had been already canceled
when "unregister" called "TaskFinisher::cancel_all".

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37582 from ivancich/wip-fix-bucket-list-namespace

rgw: fix setting of namespace in ordered and unordered bucket listing

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37572 from ivancich/wip-orphan-list-namespace-locator

rgw: allow rgw-orphan-list to note when rados objects are in namespace

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37511 from ivancich/wip-paginate-rgw-bucket-list

rgw: radosgw-admin should paginate internally when listing bucket

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

qa/mgr: Remove _wait_for_expected_get_result

Fixes: https://tracker.ceph.com/issues/47793
Signed-off-by: Tiago Melo <tmelo@suse.com>

build/doc: remove Doxyfile as doxygen run via make

also update document doxygen behavior
this is to complete: https://github.com/ceph/ceph/pull/35027

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/dashboard: Improve HTML formatting of the telemetry report preview

Convert the string "Community Data License Agreement - Sharing - Version 1.0" to a clickable link

Fixes: https://tracker.ceph.com/issues/47790
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37584 from rhcs-dashboard/report-preview-helper

mgr/dashboard: Add short descriptions to the telemetry report preview

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #37591 into master

* refs/pull/37591/head:
ceph-volume: don't exit before empty report can be printed

Reviewed-by: Joshua Schmid <jschmid@suse.de>