mgr/dashboard: Created a Download and Copy-to-Clipboard option for the logs

Created Download Button Component.
Added a Download and Copy-to-Clipboard button for the Cluster and Audit Logs in the Log section. Also added a flag in the current Copy2ClipboardButton directive to indicate the incoming string is actually a log.

Fixes: https://tracker.ceph.com/issues/47498
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #37698 from tchaikov/wip-crimson-do-until

crimson/common: schedule action only if the future is not available

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #37695 from tchaikov/wip-dashboard-pep8

mgr/dashboard: do not use "l" for variable name

Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37645 from dillaman/wip-librbd-aio-completion-race

librbd: update AioCompletion return value before evaluating pending count

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37643 from dillaman/wip-47840

librbd: ignore -ENOENT error when disabling object-map

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37444 from dillaman/wip-librbd-copyup-api

librbd: support the ability to process parent data prior to copyup

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Or Ozeri <oro@il.ibm.com>

crimson/common: schedule action only if the future is not available

otherwise we could call do_until() recursively if we have other tasks
which need to prempt the reactor and current future's state is actually
always available.

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/dashboard: do not use "l" for variable name

see also https://www.flake8rules.com/rules/E741.html

also silences flake8 warnings like:

2: {tty:'./controllers/home.py:90:26: E741 ambiguous variable name 'l'':'./controllers/home.py:90:26: E741 ambiguous variable name 'l''}
2: {tty:'./controllers/logs.py:52:13: E741 ambiguous variable name 'l'':'./controllers/logs.py:52:13: E741 ambiguous variable name 'l''}
2: {tty:'./services/ganesha.py:36:40: E741 ambiguous variable name 'l'':'./services/ganesha.py:36:40: E741 ambiguous variable name 'l''}
2: 3     E741 ambiguous variable name 'l'

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37687 from tchaikov/wip-crimson-errorator

crimson/common: do not take from a future twice

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #37602 from tspmelo/wip-flake8-ignore

mgr/dashboard: Remove some Flake8 ignore rules

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

crimson/common: do not take from a future twice

before this change, in our specialization of seastar::do_until(),
we access `f` after calling `f.get()`, this is not correct. as `f.get()`
actually moves `f._state` away and detaches the associated promise if any.
so we cannot call `f._then()` anymore after calling `f.get()`. as
`f._then()` schedules `f` by detaching the future from promise and
attaching the scheduled task to the promise. but `future_base::detach_promise()`
does not check `_promise` before accessing it, hence the segfault.

after this change, the order of the checks is rearranged so that
`f.get()` is called at the end. and also use `f.get0()` to be more
explicit, as we are accessing the only element of the returned
value.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37684 from ivancich/rgw-yummy-orphan-testing

rgw: use yum rather than dnf for teuthology testing of rgw-orphan-list

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

rgw: use yum rather than dnf for testing rgw-orphan-list

The teuthology testing for rgw-orphan-list needs to install
`s3cmd`. Switch from using dnf to yum to work on a wider variety of
platforms.

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37660 from adamemerson/wip-datalog-fix

cls/fifo: Switch use CLS_ERR for errors
rgw/fifo: Fix a few missed return value assignments
rgw/fifo: Add some error logging
rgw/fifo: Catch two instances journaling a new part
rgw/fifo: Use unique_ptr and explicit release for callbacks

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #37575 from bk201/wip-47742

qa/suites/rados/cephadm/dashboard: fix creating OSD failure

Merge pull request #37669 from lixiaoy1/fix_cache_state_ut_1015

rbd/cache: fix UTs related to image cache state

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

rbd/cache: fix UTs related to image cache state

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37633 from tchaikov/wip-pybind-rados-cleanup

pybind/rados: add more type hintings and cleanup

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #37624 from lxbsz/cram

qa/tasks: add a 'parallel' option support for the cram task

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37621 from dsavineau/cephadm_ntpsec

cephadm: allow ntpsec service

Reviewed-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #37601 from tspmelo/wip-rm-wait-for-expected-get-result

qa/mgr: Remove _wait_for_expected_get_result

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #37609 from lixiaoy1/remove_writelog

rbd/cache: remove unnecessary WriteLogCache

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

Merge pull request #37625 from lixiaoy1/rwl_path

rbd/cache: store full cache path to image metadata

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Mahati Chamarthy <mahati.chamarthy@intel.com>

rgw/fifo: Use unique_ptr and explicit release for callbacks

To simplify and clarify memory allocation in AIO cases.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Catch two instances journaling a new part

If we see another part journaled, just run the journal ourselves.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Add some error logging

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/fifo: Fix a few missed return value assignments

Slipped in when I converted from call-back based to sequential.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

cls/fifo: Switch use CLS_ERR for errors

So things going wrong show up in the log by default. Also use
__PRETTY_FUNCTION__ instead of __func__ and print some more context.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

Merge PR #34702 into master

* refs/pull/34702/head:
doc: Make time-related osd default values in documentation big-endian.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37587 from anthonyeleven/yet-more-doc-polishing

doc: clarity, detail, modernization, capitalization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #37168 from ifed01/wip-ifed-fix-compress-csum

os/bluestore: attach csum for compressed blobs

Reviewed-by: Adam Kupczyk <akucpzyk@redhat.com>

Merge pull request #37611 from tchaikov/wip-cmake-tox

cmake: do not always add py3 to TOX_ENVS

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #37594 from trociny/wip-rbd-quiesce-cancel

librbd: fix race on watcher unregister

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #37650 from jschmid1/gs

mgr/cephadm: adapt <placement> usage string

mgr/dashboard: add an empty line to appease flake8

silence the warnings like

./controllers/mgr_modules.py:38:5: E301 expected 1 blank line, found 0'
./controllers/user.py:69:5: E301 expected 1 blank line, found 0

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37394 from rosinL/cleanup-queue-option

common/options: remove unavailable values of osd_op_queue

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37627 from changchengx/msg

msg: remove stale comment and unnecessary code

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37626 from tchaikov/wip-drop-std-iterator

include/interval_set: do not inherit from std::iterator

Reviewed-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #37455 from smithfarm/wip-spec-file-cleanup

rpm: three spec file cleanups

Reviewed-by: Tim Serong <tserong@suse.com>

pybind/rados: correct the declarations of C APIs

so they are consistent with the ones declared in include/rados/librados.h

Signed-off-by: Kefu Chai <kchai@redhat.com>

pybind/rados: add more type hintings

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: do not always add py3 to TOX_ENVS

before this change add_tox_test() always add "py3" to testenv, even the
caller specifies TOX_ENVS explicitly.

after this change, py3 is added only if the caller does not specify any
TOX_ENVS.

this change helps with the readability.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #37583 into master

* refs/pull/37583/head:
mgr/volumes/nfs: Fix wrong error message for pseudo path

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #37585 into master

* refs/pull/37585/head:
doc/cephfs/nfs: Remove outdated doc related to rook

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #37610 from anthonyeleven/doc-rados-troubleshooting

doc/rados/troubleshooting: clarity and modernization

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge PR #36537 into master

* refs/pull/36537/head:
qa/cephfs: update ephemeral pin tests
mds: distribute dirfrags for ephemeral distributed directory

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/cephfs: update ephemeral pin tests

Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

mds: distribute dirfrags for ephemeral distributed directory

Instead of distribute individual dir inodes inside the ephemeral
distributed dir. Distributing dirfrags can limit number of subtrees
created by the ephemeral dist pin.

This patch also unifies codes that handle export pin and ephemeral pin.

Fixes: https://tracker.ceph.com/issues/46696
Signed-off-by: "Yan, Zheng" <zyan@redhat.com>

librbd: support preprocessing source object data prior to deep-copy

Let object dispatch layers potentially mutate the data read from the
source image prior to issuing the actual deep-copy operations against
the destination image.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: continue to simplify deep-copy object copy write ops types

Replace the custom WriteOp data structure and just re-use the new
io::SnapshotSparseBufferlist structure to record write and zero ops.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: simplify deep-copy object copy write ops types

The write-ops now only stores write vs zero ops and the type of
zero operation is delayed until the actual op is sent. This will
make the state machine compatible with the copyup process hook.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: support preprocessing parent data prior to copyup

Let object dispatch layers potentially mutate the copyup data read
from the parent prior to issuing the actual copyup operation. This
can allow for a layer like the crypto layer to re-encrypt the parent
image data using the child image's encryption keys, for example.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: new hook for pre-processing copyup data

This will permit the crypto layer to properly encrypt and potentially
align the sparse copyup data prior to it being written. It passes
potentially multiple sets of data in one pass to permit the deep-copy
state machine to utilize the same API and allow the crypto layer to
potentially handle layered alignment issues.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: rename SnapshotExtent to SparseExtent

The processing of copyup needs to be able to denote data extents that
are potentially zeroed or included in the associated bufferlist. By
renaming the type, it can be re-used for this second purpose.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: copyup state machine should always issue a sparse-read

When reading from the parent, always keep the data in a sparse
extent-map format. The forthcoming copyup preprocessing hook will
want to pass a set of sparse image-extent data.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: switch remaining uses of ExtentMap to Extents

The neorados API already requires the vector-based approach vs
the map-based approach. Now the remaining sparse-read functionality
has been switched to use the consistent approach.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: deep-copy should update object-map before writing to object

For the original use-case of RBD mirroring it was (maybe) more
acceptable to write to the object before updating the object map
because an interrupted sync will be retried. However, when using
the deep-copy object copy state machine as part of copyup, it's
more likely that the object-map has the potential to become
out-of-sync with reality if it's updated after the object is
written.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: pass write op object extents to copyup state machine

It will use these extents in a later commit to potentially reduce the
parent image data that is processed for copyup portion of the write
request.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: copy-on-read should skip copyup if deep-copy invoked

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

test/librbd: fix issue with mock test when copy-on-read is enabled

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: add interval_map initializer list constructor

This simplifies unit test cases by allowing interval_maps to be
trivially constructed with pre-populated expectations.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: update AioCompletion return value before evaluating pending count

If the pending count is decremented before the return value is updated,
there is a possibility of two ASIO threads concurrently decrementing the
pending count down from 2 -> 1 -> 0. In the second thread (the one that
performs the final decrement from 1 -> 0), it can finalize the completion
before the first thread has had a chance to update the return value.

Fixes: https://tracker.ceph.com/issues/47847
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

mgr/cephadm: adapt <placement> usage string

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #35477 from yuvalif/add_lua_to_rgw

rgw/lua: embedding lua in rgw in s3 request context

Merge pull request #37581 from dillaman/wip-librbd-exclusive-lock-shutdown

librbd: avoid failing IO with -ESHUTDOWN when disabling exclusive-lock

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #37613 from tchaikov/wip-cls-osd-cleanup

cls, osd: cleanups

Reviewed-by: Adam C. Emerson <aemerson@redhat.com>

Merge pull request #37537 from yuvalif/missing_brackets_in_rgw_file

rgw/file: missing brackets around if statement

Merge pull request #37538 from yuvalif/potential_crash_in_kafka

fix potential crash in rgw_kafka

Merge pull request #37599 from yuvalif/remove_doxyfile

cmake: remove Doxyfile as doxygen run via make

Merge pull request #37534 from yuvalif/fix_ownership_in_role_applier

 rgw/auth: fix copy&paste bug in RoleApplier::is_owner_of

Merge PR #29951 into master

* refs/pull/29951/head:
test: add tests for validating MDS metrics via `perf stats` module
test: Filesystem class helpers to grow and shrink MDS cluster
mgr/stats: mds performance stats module
mds: support sending empty perf metrics to ceph-manager

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc: misc clarity and capitalization

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #37541 from anthonyeleven/osd-internals-tweaks

doc/dev: doc/dev/osd_internals capitalization, formatting, clarity

Reviewed-by: Zac Dover <zac.dover@gmail.com>

librbd: ignore -ENOENT error when disabling object-map

Fixes: https://tracker.ceph.com/issues/47840
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw/lua: run lua scripts in s3 requests context

for more details on design and
remaining work see:
https://gist.github.com/yuvalif/60d5984c28af89ba17443ce947540c1f

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

doc/rados/troubleshooting: clarity and modernization

Signed-off-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge PR #37593 into master

* refs/pull/37593/head:
qa/tasks/cephfs: Refactor test_volumes

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks: add a 'parallel' option support for the cram task

For the ceph-iscsi test case we need to run the tests sequentially,
because the client test will depend on the gateway ones.

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #37154 from bk201/wip-47397

mgr/dashboard: fix the error when exporting CephFS path "/" in NFS exports

Reviewed-by: Laura Paduano lpaduano@suse.com
Reviewed-by: Stephan Müller smueller@suse.com
Reviewed-by: Varsha Rao varao@redhat.com

test: add tests for validating MDS metrics via `perf stats` module

Fixes: http://tracker.ceph.com/issues/24285
Signed-off-by: Venky Shankar <vshankar@redhat.com>

test: Filesystem class helpers to grow and shrink MDS cluster

These routines were test specific. Make them a part of Filesystem
class.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mgr/stats: mds performance stats module

Signed-off-by: Venky Shankar <vshankar@redhat.com>

mds: support sending empty perf metrics to ceph-manager

Right now, there are no per-mds metrics that are tracked and
sent by mds. However, such metrics will get added soon. So,
send empty performance metrics to ceph-manager for now.

Signed-off-by: Venky Shankar <vshankar@redhat.com>

qa/tasks/cephfs: Refactor test_volumes

... into smaller test classes. This enables breaking up the volumes yaml
into smaller yaml fragments.

Fixes: https://tracker.ceph.com/issues/47160
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #37628 from changchengx/mail_update

mailmap: update Intel employee mail/org

Reviewed-by: Kefu Chai <kchai@redhat.com>

msg: remove unnecessary assignment

memset(&header, 0, sizeof(header)) clear the variable to
be 0.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

msg: remove outdated comments

Parameter "features" has been removed in commit 0dbe8fd3987d.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

mailmap: update Intel employee mail/org

1. "changcheng.liu@aliyun.com" need be classified into intel until now.
   This reverts part of commit: df07e9f3
2. add "Yuan Lu <yuan.y.lu@intel.com>" in mailmap

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #36942 from rosinL/wip-fix-librgw

rgw/rgw_file: Fix the incorrect lru object eviction

rgw/file: add missing brackets around if statement

also some small style issues
issues were detected by pvs-studio static analyzer

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rbd/cache: save full cach path instead of folder

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

rbd/cache: remove unsupported poolset

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

include/interval_set: do not inherit from std::iterator

std::iterator is deprecated in C++17, so drop it.

Signed-off-by: Kefu Chai <kchai@redhat.com>

rbd/cache: remove unnecessary WriteLogCache

Remove the class librbd::cache::WriteLogCache.

Signed-off-by: Li, Xiaoyan <xiaoyan.li@intel.com>

Merge pull request #37580 from aclamk/wip-bluestore-fix-2q-cache

os/bluestore: fix memory accounting in TwoQBufferCacheShard

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #37325 from badone/wip-angular-cli-analytics-dont-prompt

mgr/dashboard: Don't prompt for analytics during @angular/cli install

Reviewed-by: Tiago Melo <tmelo@suse.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>

Merge pull request #37542 from SUSE/wip-fix-47745

cephadm: allow uid/gid == 0 in copy_tree, copy_files, move_files

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #37559 from wjwithagen/wjw-fix-iterator

include: explicitly define all types needed for libc++ iterator

Reviewed-by: Jason Dillaman <dillaman@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #37104 from sebastian-philipp/cephadm-serve.py-2

mgr/cephadm: move functions that do remote calls to serve.py

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #37441 into master

* refs/pull/37441/head:
qa: fix proc exit status check

Reviewed-by: Jos Collin <jcollin@redhat.com>