test_sessionmap: use sudo_write_file() from teuthology.misc

Instead of defining a method to write files within the testsuite, use
sudo_write_file() from teuthology.misc.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #29067 into master

* refs/pull/29067/head:
qa/tasks: add exception in do_thrash()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

qa/tasks: add exception in do_thrash()

Fixes the bug during DaemonWatchdog testing:

Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-daemonwatchdog-testing13/qa/tasks/daemonwatchdog.py", line 38, in _run
    self.watch()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_wip-daemonwatchdog-testing13/qa/tasks/daemonwatchdog.py", line 112, in watch
    if thrasher.e is not None:
AttributeError: MonitorThrasher instance has no attribute 'e'

Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge PR #29011 into master

* refs/pull/29011/head:
doc/releases/nautilus: move note to 14.2.2
doc/releases/nautilus: typo
doc/releases/nautilus.rst: pool stats fix was merged for 14.2.2 after all
doc: release notes for nautilus 14.2.2

Reviewed-by: Yuri Weinstein <yweins@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28692 into master

* refs/pull/28692/head:
test_volume_client: add positive test for ceph_volume_client method
test_volume_client: rename test_put_object_versioned()
test_volume_client: rewrite test_put_object_versioned
test_volume_client: use sudo_write_file() form teuthology
test_volume_client: make test_object_versioned py3 compatible

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #22788 from adamyanova/wip-s3java

A task to run S3 Java tests against RGW

Merge pull request #29048 from rjfd/wip-dashboard-progress-tasks

mgr/dashboard: integrate progress mgr module events into dashboard tasks list

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #28006 from Devp00l/wip-39482

mgr/dashboard: Watch for pool pg's increase and decrease

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #29092 into master

* refs/pull/29092/head:
osd/ECTransaction,ReplicatedBackend: create op is new in octopus

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28935 from votdev/issue_40248

mgr/dashboard: Allow users to change their password on the UI

Reviewed-by: Ricardo Dias <rdias@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>

Merge pull request #28928 from p-na/wip-pna-e2e-pools

mgr/dashboard: Write E2E tests for pool creation, deletion and verification

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Rafael Quintero <rquinter@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28882 from rjfd/wip-mgr-perf-counter-timestamp

mgr: return perf_counters data timestamps in nanosecs

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

doc/releases/nautilus: move note to 14.2.2

Oops! This change was merged in 404ed645d42471b675f11fb7d11f46c006248152,
which was not part of 14.2.1 after all.

Signed-off-by: Sage Weil <sage@redhat.com>

doc/releases/nautilus: typo

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #29069 into master

* refs/pull/29069/head:
qa/tasks/ceph_manager: make upmap thrasher behave when no pools/pgs

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29095 into master

* refs/pull/29095/head:
script/ptl-tool: update for python3

Reviewed-by: Sage Weil <sage@redhat.com>

Merge PR #28900 into master

* refs/pull/28900/head:
doc: Update troubleshooting.rst
doc: drop kernel-client-troubleshooting.rst

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

script/ptl-tool: update for python3

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #29071 into master

* refs/pull/29071/head:
selinux: Allow ceph to read udev db

Reviewed-by: Sage Weil <sage@redhat.com>

osd/ECTransaction,ReplicatedBackend: create op is new in octopus

Not nautilus!

Fixes: https://tracker.ceph.com/issues/40755
Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28987 into master

* refs/pull/28987/head:
Revert "mgr/dashboard: Add nolockdep option to e2e-script"
common: refactor handling of lockdep's group name in debug locks.
common: shared_mutex_debug doesn't use empty group name for lockdep.
common, crypto: give names to OpenSSL's mutexes.
common: tiny_vector – CPU-friendly container for mutexes & co.

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #29051 into master

* refs/pull/29051/head:
mon/OSDMonitor: tolerate duplicate MRemoveSnaps messages

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #28468 from jmolmo/ansible_orchestrator

mgr/ansible: RGW service

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

mgr: return perf_counters data timestamps in nanosecs

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: frontend: move ellipsis to before progress in task execution description

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: test_pool: fix pool unit tests

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Add rudimentary pool create/exist/delete test

Fixes: http://tracker.ceph.com/issues/38093
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge branch 'master' into ansible_orchestrator

mgr/dashboard: tests: fix tests by mocking progress service

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: frontend/task.service: format task title for progress tasks

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: add progress module tasks to list of dashboard tasks

Fixes: https://tracker.ceph.com/issues/38202
Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: progress mgr module service

This service module implements helper methods to retrieve
tasks information from the progress mgr module.

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/progress: persist started_at and finished_at timestamps

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/progress: export more event properties to json

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: Add SSO guard service.

Signed-off-by: Volker Theile <vtheile@suse.com>

mgr/dashboard: Allow users to change their password on the UI

Fixes: https://tracker.ceph.com/issues/40248
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #28959 from tchaikov/wip-crimson-op-write

crimson/osd: differentiate write from writefull

Reviewed-by: Samuel Just <sjust@redhat.com>

doc/releases/nautilus.rst: pool stats fix was merged for 14.2.2 after all

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #28861 from smanjara/wip-multi-meta

rgw/multisite: Don't allow certain radosgw-admin commands to run on non-master zone

Merge pull request #28809 from joscollin/wip-fix-rgw-warning

rgw: Silence warning: control reaches end of non-void function

Merge PR #28003 into master

* refs/pull/28003/head:
client: do not return EEXIST for mkdirs
pybind/mgr/volumes: print exceptions in purge thread
pybind/mgr/volumes: refactor trash readdir
pybind/mgr/volumes: use existing client provided recursive mkdir
pybind/mgr/volumes: cleanup fs removal
pybind/mgr/subvolumes: use bytes for paths
pybind/mgr/volumes: remove unused property
test: add basic purge queue validation test
mgr / volumes: schedule purge job for volumes on init
mgr / volumes: purge queue for async subvolume delete
mgr / volumes: maintain connection pool for fs volumes
mgr / volumes: wrap rmtree() call within try..except block
mgr / volumes: use negative error codes everywhere
test: cleanup removing all subvolumes before removing subvolume group

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

selinux: Allow ceph to read udev db

We are using libudev and reading the udev db files because of that. We
need to allow ceph to access these files in the SELinux policy.

Signed-off-by: Boris Ranto <branto@redhat.com>

Merge PR #29030 into master

* refs/pull/29030/head:
qa/tasks/ceph_deploy: assume systemd and simplify shutdown wonkiness

Reviewed-by: Alfredo Deza <adeza@redhat.com>

Merge pull request #28994 from smithfarm/wip-40645

build/ops: run "npm ci" with a one-hour timeout

Reviewed-by: Tiago Melo <tmelo@suse.com>

qa/tasks/ceph_deploy: assume systemd and simplify shutdown wonkiness

The 'service' command was succeeding but not actually stopping ceph.  But
we can safely assume systemd now so get rid of the weirdness here.

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #28839 into master

* refs/pull/28839/head:
osd: support osd_repair_during_recovery

Reviewed-by: David Zafman <dzafman@redhat.com>

Merge PR #28907 into master

* refs/pull/28907/head:
mgr/progress: added the time an event has been in progress
mgr/progress: added the time an event has been in progress
mgr/progress: added the time an event has been in progress
mgr/progress: added the time an event has been in progress

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>

Merge PR #29027 into master

* refs/pull/29027/head:
common/options: Set concurrent bluestore rocksdb compactions to 2

Reviewed-by: Neha Ojha <nojha@redhat.com>

qa/tasks/ceph_manager: make upmap thrasher behave when no pools/pgs

Fixes: https://tracker.ceph.com/issues/40635
Signed-off-by: Sage Weil <sage@redhat.com>

mgr/dashboard: pool: make _get and _wait_for_pgs class methods

Signed-off-by: Ricardo Dias <rdias@suse.com>

Merge pull request #28820 from trociny/wip-journal-fetch-corrupt-end

journal: return error after first corruption detected

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

client: do not return EEXIST for mkdirs

Behavior should be similar to `mkdir -p`.

Introduced-by: 26905ca82776e8119a347a9a2e8d65c272c7ed62
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28633 from p-na/wip-pna-refactor-unit-tests

mgr/dashboard: Reduce code duplication through TableActionComponent UnitTests

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28999 from rafaelquint/bucketstest3

mgr/dashboard: RGW bucket E2E Tests

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Laura Paduano <lpaduano@suse.com>
Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Reduce code duplication through TableActionComponent Unit Tests

Fixes: http://tracker.ceph.com/issues/40399
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: E2E Tests for Create, Edit, Delete, and Invalid Input on RGW Buckets

Fixes: https://tracker.ceph.com/issues/40372
Fixes: https://tracker.ceph.com/issues/40484
Fixes: https://tracker.ceph.com/issues/40259
Signed-off-by: Adam King <adking@redhat.com>
Signed-off-by: Rafael Quintero <rquinter@redhat.com>

Merge pull request #28879 from Devp00l/wip-40580

mgr/dashboard: Fix the table mouseenter event handling test

Reviewed-by: Patrick Nawracay <pnawracay@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #28916 from p-na/wip-pna-fix-tag

mgr/dashboard: Add linter for unclosed HTML tags

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mgr/dashboard: Add missing closing tag for container

Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

Merge pull request #27463 from rjfd/wip-dashboard-ssl-cert

mgr/dashboard: commands to set SSL certificate and key

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Volker Theile <vtheile@suse.com>

Merge pull request #28213 from sebastian-philipp/orchestrator_cache

mgr/orchestrator: Add cache for Inventory and Services

Reviewed-by: Tim Serong <tserong@suse.com>

pybind/mgr/volumes: print exceptions in purge thread

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: refactor trash readdir

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: use existing client provided recursive mkdir

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/volumes: cleanup fs removal

In Nautilus, the simplest sequence is:

    fs fail name
    fs rm name --yes-i-really-mean-it

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

pybind/mgr/subvolumes: use bytes for paths

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

Merge pull request #28700 from batrick/i39620

doc/cephfs: improve add/remove MDS section

Reviewed-by: Jos Collin <jcollin@redhat.com>

Merge pull request #29033 from neha-ojha/wip-add-yt

doc/dev/developer_guide/index.rst: add youtube references

Reviewed-by: Noah Watkins <nwatkins@redhat.com>

Merge pull request #28649 from votdev/issue_40427

mgr/dashboard: Pool form uses different loading spinner

Reviewed-by: Lenz Grimmer <lgrimmer@suse.com>
Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

mon/OSDMonitor: tolerate duplicate MRemoveSnaps messages

Fixes: http://tracker.ceph.com/issues/40774
Signed-off-by: Sage Weil <sage@redhat.com>

doc/mgr/dashboard: update SSL configuration instructions

Signed-off-by: Ricardo Dias <rdias@suse.com>

mgr/dashboard: added CLI commands to set SSL certificate and key

Fixes: https://tracker.ceph.com/issues/39123
Signed-off-by: Ricardo Dias <rdias@suse.com>

Revert "mgr/dashboard: Add nolockdep option to e2e-script"

This reverts commit d92117d8ea7018c533cc422e31c09d7e45a0793a
at the rason for it has been fixed.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: refactor handling of lockdep's group name in debug locks.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: shared_mutex_debug doesn't use empty group name for lockdep.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common, crypto: give names to OpenSSL's mutexes.

Fixes: http://tracker.ceph.com/issues/40698
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common: tiny_vector – CPU-friendly container for mutexes & co.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

mgr/dashboard: Pool form uses different loading spinner

Use the 'cd-loading-panel' element to display the loading progress.

Fixes: https://tracker.ceph.com/issues/40427
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #29043 from votdev/issue_40778

mgr/dashboard: Form fields do not show error messages/hints.

Reviewed-by: Ricardo Marques <rimarques@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge PR #28951 into master

* refs/pull/28951/head:
mds: apply configuration changes through MDSRank
common/TrackedOp: make settings atomic

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd/heartbeat: use v2 address also

as we support v2 protocol, so no need to pick v1 address.

Signed-off-by: Kefu Chai <kchai@redhat.com>

messages/MOSDRepOpReply: include necessary header

for accessing the member variables of `MOSDRepOp`, we should include its
header file for its definition.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #28964 from cbodley/wip-qa-workunit-s3tests

qa/rgw: update run-s3tests.sh

Reviewed-by: Abhishek Lekshmanan <abhishek@suse.com>

mgr/dashboard: Add linter for unclosed HTML tags

Fixes: http://tracker.ceph.com/issues/40686
Signed-off-by: Patrick Nawracay <pnawracay@suse.com>

mgr/dashboard: Form fields do not show error messages/hints.

Fixes: https://tracker.ceph.com/issues/40778
Signed-off-by: Volker Theile <vtheile@suse.com>

Merge pull request #28682 from cyx1231st/wip-seastar-msgr-socket2

crimson/net: implement preemptive shutdown/close

Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: an MOSDOp should be waiting for min_epoch

for MOSDOp, get_min_epoch() is an alias of get_map_epoch(), but
get_min_epoch() is symantically more correct.

Signed-off-by: Kefu Chai <kchai@redhat.com>

crimson/osd: add a dedicate code path for WRITE

differentiate WRITE from WRITEALL.
and handles differnt cases of `os.oi.truncate_seq` caused by
`write_trunc` and/or `trunc` ops.

Signed-off-by: Kefu Chai <kchai@redhat.com>

test_volume_client: add positive test for ceph_volume_client method

ceph_volume_client.py's put_object_versioned() has only one test. Since
this only test is a negative test it may fail to assure that
put_object_versioned() works as expected with positive inputs even when
it completes successfully. Therefore, write a positive test for better
coverage.

Also, make sure the new test is both python 2 and python3 compatible.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_volume_client: rename test_put_object_versioned()

test_put_object_versioned() is too generic for a negative test that
specifically tests if the version is verified before writing.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_volume_client: rewrite test_put_object_versioned

The test's success depends on whether CommandFailedError exception is
raised. This manner of testing is unreliable since there's no way to
check if the exception was raised by the statement (in the embedded
Python program) the test expects to.

This implies that this test's failure may go undetected. This is primary
reason why bugs like tracker #38946 occured despite of having a test.

Fixes: http://tracker.ceph.com/issues/39510
Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_volume_client: use sudo_write_file() form teuthology

Instead of defining another method to write a file as root user in
test_volume_client.py, use the method from teuthology/misc.py.

Signed-off-by: Rishabh Dave <ridave@redhat.com>

test_volume_client: make test_object_versioned py3 compatible

Convert strings to bytes in the Python program embedded inside this
test. Also, in the process don't lose Python 2 compatibility.

Fixes: http://tracker.ceph.com/issues/39405
Signed-off-by: Rishabh Dave <ridave@redhat.com>

Merge PR #26104 into master

* refs/pull/26104/head:
client/Client: change dir's size delta to current write file size delta

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28683 into master

* refs/pull/28683/head:
mon/MDSMonitor: use stringstream instead of dout for mds repaired

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28710 into master

* refs/pull/28710/head:
cephfs-shell: Remove undefined variable files in do_rm()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #28831 into master

* refs/pull/28831/head:
client: support the fallocate() when fuse version >= 2.9

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Zheng Yan <zyan@redhat.com>

Merge PR #28906 into master

* refs/pull/28906/head:
cephfs-shell: Fix TypeError in poutput()

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

mds: apply configuration changes through MDSRank

This avoids the need to acquire the mds_lock prior to responding to config
changes. The MDSRank can create a finisher context that acquires the lock
later.

Fixes: https://tracker.ceph.com/issues/40694
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

common/TrackedOp: make settings atomic

To avoid locks on configuration changes.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>