Merge pull request #37624 from lxbsz/cram

qa/tasks: add a 'parallel' option support for the cram task

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37621 from dsavineau/cephadm_ntpsec

cephadm: allow ntpsec service

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #37601 from tspmelo/wip-rm-wait-for-expected-get-result

qa/mgr: Remove _wait_for_expected_get_result

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37609 from lixiaoy1/remove_writelog

rbd/cache: remove unnecessary WriteLogCache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #37625 from lixiaoy1/rwl_path

rbd/cache: store full cache path to image metadata

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge PR #34702 into master

* refs/pull/34702/head:
doc: Make time-related osd default values in documentation big-endian.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37587 from anthonyeleven/yet-more-doc-polishing

doc: clarity, detail, modernization, capitalization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37168 from ifed01/wip-ifed-fix-compress-csum

os/bluestore: attach csum for compressed blobs

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #37611 from tchaikov/wip-cmake-tox

cmake: do not always add py3 to TOX_ENVS

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37594 from trociny/wip-rbd-quiesce-cancel

librbd: fix race on watcher unregister

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37650 from jschmid1/gs

mgr/cephadm: adapt <placement> usage string

mgr/dashboard: add an empty line to appease flake8

silence the warnings like

./controllers/mgr_modules.py:38:5: E301 expected 1 blank line, found 0'
./controllers/user.py:69:5: E301 expected 1 blank line, found 0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37394 from rosinL/cleanup-queue-option

common/options: remove unavailable values of osd_op_queue

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37627 from changchengx/msg

msg: remove stale comment and unnecessary code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37626 from tchaikov/wip-drop-std-iterator

include/interval_set: do not inherit from std::iterator

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #37455 from smithfarm/wip-spec-file-cleanup

rpm: three spec file cleanups

Reviewed-by: Tim Serong <tserong@suse.com>

cmake: do not always add py3 to TOX_ENVS

before this change add_tox_test() always add "py3" to testenv, even the
caller specifies TOX_ENVS explicitly.

after this change, py3 is added only if the caller does not specify any
TOX_ENVS.

this change helps with the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #37583 into master

* refs/pull/37583/head:
mgr/volumes/nfs: Fix wrong error message for pseudo path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37585 into master

* refs/pull/37585/head:
doc/cephfs/nfs: Remove outdated doc related to rook

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37610 from anthonyeleven/doc-rados-troubleshooting

doc/rados/troubleshooting: clarity and modernization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #36537 into master

* refs/pull/36537/head:
qa/cephfs: update ephemeral pin tests
mds: distribute dirfrags for ephemeral distributed directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: update ephemeral pin tests

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: distribute dirfrags for ephemeral distributed directory

Instead of distribute individual dir inodes inside the ephemeral
distributed dir. Distributing dirfrags can limit number of subtrees
created by the ephemeral dist pin.

This patch also unifies codes that handle export pin and ephemeral pin.

Fixes: https://tracker.ceph.com/issues/46696
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mgr/cephadm: adapt <placement> usage string

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #35477 from yuvalif/add_lua_to_rgw

rgw/lua: embedding lua in rgw in s3 request context

Merge pull request #37581 from dillaman/wip-librbd-exclusive-lock-shutdown

librbd: avoid failing IO with -ESHUTDOWN when disabling exclusive-lock

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37613 from tchaikov/wip-cls-osd-cleanup

cls, osd: cleanups

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #37537 from yuvalif/missing_brackets_in_rgw_file

rgw/file: missing brackets around if statement

Merge pull request #37538 from yuvalif/potential_crash_in_kafka

fix potential crash in rgw_kafka

Merge pull request #37599 from yuvalif/remove_doxyfile

cmake: remove Doxyfile as doxygen run via make

Merge pull request #37534 from yuvalif/fix_ownership_in_role_applier

 rgw/auth: fix copy&paste bug in RoleApplier::is_owner_of

Merge PR #29951 into master

* refs/pull/29951/head:
test: add tests for validating MDS metrics via `perf stats` module
test: Filesystem class helpers to grow and shrink MDS cluster
mgr/stats: mds performance stats module
mds: support sending empty perf metrics to ceph-manager

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: misc clarity and capitalization

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37541 from anthonyeleven/osd-internals-tweaks

doc/dev: doc/dev/osd_internals capitalization, formatting, clarity

Reviewed-by: Zac Dover <zac.dover@gmail.com>

rgw/lua: run lua scripts in s3 requests context

for more details on design and
remaining work see:
https://gist.github.com/yuvalif/60d5984c28af89ba17443ce947540c1f

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

doc/rados/troubleshooting: clarity and modernization

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge PR #37593 into master

* refs/pull/37593/head:
qa/tasks/cephfs: Refactor test_volumes

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks: add a 'parallel' option support for the cram task

For the ceph-iscsi test case we need to run the tests sequentially,
because the client test will depend on the gateway ones.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #37154 from bk201/wip-47397

mgr/dashboard: fix the error when exporting CephFS path "/" in NFS exports

Reviewed-by: Laura Paduano lpaduano@suse.com
Reviewed-by: Stephan Müller smueller@suse.com
Reviewed-by: Varsha Rao varao@redhat.com

test: add tests for validating MDS metrics via `perf stats` module

Fixes: http://tracker.ceph.com/issues/24285
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: Filesystem class helpers to grow and shrink MDS cluster

These routines were test specific. Make them a part of Filesystem
class.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/stats: mds performance stats module

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: support sending empty perf metrics to ceph-manager

Right now, there are no per-mds metrics that are tracked and
sent by mds. However, such metrics will get added soon. So,
send empty performance metrics to ceph-manager for now.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/tasks/cephfs: Refactor test_volumes

... into smaller test classes. This enables breaking up the volumes yaml
into smaller yaml fragments.

Fixes: https://tracker.ceph.com/issues/47160
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #37628 from changchengx/mail_update

mailmap: update Intel employee mail/org

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg: remove unnecessary assignment

memset(&header, 0, sizeof(header)) clear the variable to
be 0.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

msg: remove outdated comments

Parameter "features" has been removed in commit 0dbe8fd3987d.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

mailmap: update Intel employee mail/org

1. "changcheng.liu@aliyun.com" need be classified into intel until now.
   This reverts part of commit: df07e9f3
2. add "Yuan Lu <yuan.y.lu@intel.com>" in mailmap

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #36942 from rosinL/wip-fix-librgw

rgw/rgw_file: Fix the incorrect lru object eviction

rgw/file: add missing brackets around if statement

also some small style issues
issues were detected by pvs-studio static analyzer

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rbd/cache: save full cach path instead of folder

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

rbd/cache: remove unsupported poolset

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

include/interval_set: do not inherit from std::iterator

std::iterator is deprecated in C++17, so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd/cache: remove unnecessary WriteLogCache

Remove the class librbd::cache::WriteLogCache.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37580 from aclamk/wip-bluestore-fix-2q-cache

os/bluestore: fix memory accounting in TwoQBufferCacheShard

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37325 from badone/wip-angular-cli-analytics-dont-prompt

mgr/dashboard: Don't prompt for analytics during @angular/cli install

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37542 from SUSE/wip-fix-47745

cephadm: allow uid/gid == 0 in copy_tree, copy_files, move_files

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37559 from wjwithagen/wjw-fix-iterator

include: explicitly define all types needed for libc++ iterator

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37104 from sebastian-philipp/cephadm-serve.py-2

mgr/cephadm: move functions that do remote calls to serve.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #37441 into master

* refs/pull/37441/head:
qa: fix proc exit status check

Reviewed-by: Jos Collin <jcollin@redhat.com>

common/options: remove unavailable values of osd_op_queue

Signed-off-by: luo rixin <luorixin@huawei.com>

cephadm: allow ntpsec service

ntpsec is available starting Ubuntu Focal (20.04) and Debian Buster (10)
as an alternate NTP daemon.

Fixes: https://tracker.ceph.com/issues/47525
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #36979 from tspmelo/wip-npm-update-20-8

mgr/dashboard: Update npm packages

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

rgw/rgw_file: Fix the incorrect lru object eviction

In func lookup_fh, when RGWFileHandle not be found in fh_cache, it
need to recycle an object and create an new RGWFileHandle. When there
are multi threads use lookup_fh to find and create RGWFileHandle concurrently,
it must to make sure evict lru object from the partiton of fh_cache which new
RGWFileHandle will be inserted to.

Fixes: https://tracker.ceph.com/issues/47235
Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #37449 from ceph/47676-zone-empty-realm-id

mgr/dashboard: get rgw daemon zonegroup name from mgr

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37375 from ceph/47615-fix-endpoint-responses

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37064 from bk201/wip-47372

mgr/dashboard: fix error when typing existing paths in the Ganesha form

Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>

Merge pull request #37596 from rhcs-dashboard/telemetry_link_cleanup

mgr/dashboard: Improve HTML formatting of the telemetry report preview

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

doc/cephfs/nfs: Remove outdated doc related to rook

The doc is outdated because the mentioned orch commands have changed[1] and
using dashboard backend script to create exports is buggy[2].

[1] https://docs.ceph.com/en/latest/mgr/orchestrator/#current-implementation-status
[2] https://github.com/rook/rook/issues/6147

Fixes: https://tracker.ceph.com/issues/47784
Signed-off-by: Varsha Rao <varao@redhat.com>

mgr/dashboard: Update npm packages

Angular 10.1 includes extraction of TS translations,
so @locl/cli is no longer needed.

Updating Angular automatically removed the usage of base tsconfig.

Downgraded css-loader to be able to use css modules.

angular-tree-component has been deprecated and
replaced by @circlon/angular-tree-component

Fixes: https://tracker.ceph.com/issues/47289
Signed-off-by: Tiago Melo <tmelo@suse.com>

Merge PR #32778 into master

* refs/pull/32778/head:
msg: Use sockets instead of pipes for wakeup events on win32
common: Convert remote errors on win32
common,msg: Initialize Windows WSA and TLS
common,os,kv: Define aligned_free
common: avoid CLOCK_*_COARSE warnings on win32
global: Windows support

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

osd/OSD: remove unused #include

Signed-off-by: Kefu Chai <kchai@redhat.com>

cls/fifo: cast ceph_le64 to uint64_t before printing it

silences warning like:

../src/cls/fifo/cls_fifo.cc: In member function ‘int rados::cls::fifo::{anonymous}::EntryReader::get_next_entry(ceph::buffer::v15_2_0::list*, uint64_t*, ceph::real_time*)’:
../src/include/rados/objclass.h:33:18: warning: format ‘%llu’ expects argument of type ‘long long unsigned int’, but argument 7 has type ‘ceph_le64’ {aka ‘ceph_le<long long unsigned int>’} [-Wformat=]
   33 |   cls_log(level, "<cls> %s:%d: " fmt, __FILE__, __LINE__, ##__VA_ARGS__)
../src/include/rados/objclass.h:33:18: note: in definition of macro ‘CLS_LOG’
   33 |   cls_log(level, "<cls> %s:%d: " fmt, __FILE__, __LINE__, ##__VA_ARGS__)
      |                  ^~~~~~~~~~~~~~~
../src/cls/fifo/cls_fifo.cc:703:48: note: format string is defined here
  703 |   CLS_LOG(10, "%s():%d: pre_header.pre_size=%llu", __func__, __LINE__,
      |                                             ~~~^
      |                                                |
      |                                                long long unsigned int

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #36554 into master

* refs/pull/36554/head:
mgr/volumes: Make number of cloner threads configurable

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Shyamsundar R <srangana@redhat.com>

Merge PR #36457 into master

* refs/pull/36457/head:
qa/tasks/cephfs: fix fs authorize cmd args
mds: add root_squash mode in MDS auth caps
vstart_runner: add write_file and sudo_write_file

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #36556 into master

* refs/pull/36556/head:
doc: cephfs mirroring design document

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37483 from dzafman/wip-46405

osd/osd-rep-recov-eio.sh: TEST_rados_repair_warning:  return 1

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37500 from ricardoasmarques/fix-bootstrap-container-init

cephadm: Fix error setting 'mgr/cephadm/container_init' config

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Joshua Schmid <jschmid@suse.de>
Reviewed-by: Nathan Cutler <ncutler@suse.com>

librbd: fix race on watcher unregister

It was possible that "unregister_watch" would get stuck forever
in "m_async_op_tracker.wait_for_ops", waiting for unqueiesce
notifications to complete, which had been already canceled
when "unregister" called "TaskFinisher::cancel_all".

Signed-off-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37582 from ivancich/wip-fix-bucket-list-namespace

rgw: fix setting of namespace in ordered and unordered bucket listing

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37572 from ivancich/wip-orphan-list-namespace-locator

rgw: allow rgw-orphan-list to note when rados objects are in namespace

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #37511 from ivancich/wip-paginate-rgw-bucket-list

rgw: radosgw-admin should paginate internally when listing bucket

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

qa/mgr: Remove _wait_for_expected_get_result

Fixes: https://tracker.ceph.com/issues/47793
Signed-off-by: Tiago Melo <tmelo@suse.com>

build/doc: remove Doxyfile as doxygen run via make

also update document doxygen behavior
this is to complete: https://github.com/ceph/ceph/pull/35027

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

mgr/dashboard: Improve HTML formatting of the telemetry report preview

Convert the string "Community Data License Agreement - Sharing - Version 1.0" to a clickable link

Fixes: https://tracker.ceph.com/issues/47790
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37584 from rhcs-dashboard/report-preview-helper

mgr/dashboard: Add short descriptions to the telemetry report preview

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge PR #37591 into master

* refs/pull/37591/head:
ceph-volume: don't exit before empty report can be printed

Reviewed-by: Joshua Schmid <jschmid@suse.de>

doc: cephfs mirroring design document

Signed-off-by: Venky Shankar <vshankar@redhat.com>

ceph-volume: don't exit before empty report can be printed

get_plan() called exit in case of an empty plan. This prevented a report
being printed under these circumstances. Avoid exit in this case. Also
adds tests to ensure an empty report is printed.

Fixes: https://tracker.ceph.com/issues/47760
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #37561 from tchaikov/wip-cmake-node-mirror

cmake: support Node.js and npm registry mirror

Reviewed-by: Brad Hubbard <bhubbard@redhat.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #37570 from ceph/wip-install-ceph-deploy

doc: add a link to ceph-deploy document

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #37446 into master

* refs/pull/37446/head:
mds: remove dead mdsmap broadcast code

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge PR #37497 into master

* refs/pull/37497/head:
client,mds: Drop execute permission bits

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37571 into master

* refs/pull/37571/head:
qa/tasks/cephfs/nfs: Check ganesha daemon status after restarting mgr

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37579 into master

* refs/pull/37579/head:
SubmittingPatches: use "doc" prefix in title of doc-only commits

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mgr/dashboard: Add short descriptions to the telemetry report preview

A small info icon with a popover on the telemetry report preview form step2 for
Report ID and Report Preview.

Report ID: "A randomized UUID to identify a particular cluster over the course of several telemetry reports."
Report preview: "The actual telemetry data that will be submitted."

Fixes: https://tracker.ceph.com/issues/47610
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/volumes/nfs: Fix wrong error message for pseudo path

Fixes: https://tracker.ceph.com/issues/47783
Signed-off-by: Varsha Rao <varao@redhat.com>

librbd: avoid failing IO with -ESHUTDOWN when disabling exclusive-lock

When dynamically disabling the exclusive-lock feature with in-flight IO,
it was previously possible for IO to fail with -ESHUTDOWN when it
attempts to acquire the lock due to the PreReleaseRequest setting the
lock required flag. There is also the possibility for a race with the
first IO racing with exclusive-lock shutdown when the lock was not
already acquired.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: fix setting of namespace in ordered and unordered bucket listing

The namespace is not always set correctly during bucket listing. This
can, for example, cause the listing of incomplete multipart uploads,
which are in the _multipart_ namespace, to not paginate correctly, and
cause entries to be re-listed.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge PR #37577 into master

* refs/pull/37577/head:
PendingReleaseNotes: add note about batch refactor

Reviewed-by: Nathan Cutler <ncutler@suse.com>