rgwlc: wait for WorkQ threads on shutdown

Fixes: https://tracker.ceph.com/issues/43841
Signed-off-by: Mark Kogan <mkogan@redhat.com>

rgw: lc: restore unordered listing

With current sharding strategy, a delete marker is always coresident
with all corresponding objects, including noncurrent, so unordered
listing -is- compatible with lookahead.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: lc:  cls_rgw_lc_list_entries_op struc_v off-by-one

The incoming version of -new- clients required to be one
higher to trigger the new result format.

Yes, this means that "radosgw-admin lc list" against an
un-upgraded OSD will display Unix epoch for lc-entry
start_time (harmless, and in general will not happen).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: make invalid OpRule, lc_op_ctx accesses impossible

* At the cost of some additional overhead from deep copies,
  makes all access to these structures from WorkQ context safe.

* Lifts saved next-key-name and effective-mtime from LCOpAction
  to lc_op_ctx (via LCOpRule), former usage was incorrect

* Use ordered entry listing for Current, NonCurrent, and DM
  expiration--this is required by lookahead strategy

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgw: nfs: disable gc, lc, quota, and sync threads by default

These can be overridden by new options, should a site decide to
run a librgw with no radosgw instances.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: properly accumulate in RGWLC::list_lc_progress()

Also track shard/index progress, clearing marker when the shard
is advanced.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: LCOpAction MUST NOT acc. OCObjLister from WorkQ

Only a few actions had this coupling, and it is easily avoidable.
Making it -impossible- is left to future cleanup.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: give radosgw-admin lc process more freedom to run

Makes two primary changes:

1. allows the admin lc process to re-run a bucket (but not
when a session is ACTIVE, to avoid overwriting session state)

2. to avoid confusion when rgw_lc_debug_interval is set and is small,
don't force lc process to exit early, allowing one run to clear all
expired objects during debug and test

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: re-hook RGWRados::process_lc()

Allow the admin command to force a (single threaded) lifecycle
pass, do not instantiate background threads to do this.

Fix an apparent shutdown hang (found in the process_lc() path,
but actually general).

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: revisions

Contains concurrency fixes, as well as improved debug prints.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: make rgwlc entries extensible and extend

Add generation/run tracking to LC entries.  Define the entry as
a versioned structure, adapt accordingly.

N.B., has extra debug prints from later commit.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: permit per-bucket thread parallelism

Introduces type-safe work queues to process lifecycle actions
in parallel.  Currently, optimized for copy avoidance, so limiting
fetch overlap--because rgw_bucket_dir_entry is b.i.g.

Oh, heck.  Open it up.

Define bsf and f in the right order.

Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

rgwlc: enable thread-parallelism in RGWLC

Allow RGWLC to spawn >1 LCWorker threads, controlled by option
rgw_lc_max_worker (default to 3).

LCWorker parallelism exactly resembles previous cross-instance
parallelism (and extends it), with no new locking.

Replace in-order index shard enumeration (w/random start) with
a number sequence to avoid convoying.

Fixes: https://tracker.ceph.com/issues/43841
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #34612 from zdover23/wip-doc-tshooting-ssh-errors-44905

doc/cephadm: Add troubleshooting guidance for ssh connection failures. [cephadm]

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34661 from wjwithagen/wjw-fix-rgw_bucket_sync_cache.h

rgw: remove noexcept from function definition

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #34614 from theanalyst/rgw/reshard-num-check

rgw: reshard: skip stale bucket id entries from reshard queue

Reviewed-by: Casey Bodley <cbodley@redhat.com>

rgw: remove noexcept from function definition

Clang complains:
src/rgw/rgw_bucket_sync_cache.h:88:3: error: exception specification of explicitly defaulted copy co
nstructor does not match the calculated one
  Handle(const Handle&) noexcept = default;
  ^
1 error generated.

And a reference that I found for this:
    https://github.com/mapnik/mapnik/issues/3274
Suggesting that the noexcept is inherited from the first definition.

Signed-off-by: Willem Jan Withagen <wjw@digiware.nl>

Merge pull request #34605 from tchaikov/wip-45113

qa/suites/*/rados_cls_all.yaml: load all classes

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34655 from ceph/wip-yuriw-remove-wip-master

qa/tests: removed tests 'client-upgrade-octopus-pacific'

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: removed tests 'client-upgrade-octopus-pacific'

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34651 from ceph/wip-yuriw-crontab-master

qa/tests: fixed typo

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/tests: fixed typo

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34649 from ceph/wip-yuriw-crontab-master

qa/tests: added new clients upgrade tests `client-upgrade-octopus-pacific`

qa/tests: added new clients upgrade tests

`client-upgrade-octopus-pacific` for future 'pacific' release

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34642 from ceph/wip-yuriw-octopus-client-upgrade-octopus-pacific

qa/tests: initial checkin for clients upgrades tests for future relea…

qa/tests: initial checkin for clients upgrades tests for future release of pacific

Signed-off-by: Yuri Weinstein <yweinste@redhat.com>

Merge pull request #34608 from trociny/wip-journal-append-stripe-width

cls/journal: use EC pool stripe width for padding appends

Reviewed-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34636 from tchaikov/wip-vstart-runner-py3

qa/tasks/vstart_runner,doc: be compatible with python3

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/fuse_mount.py: by python3 compatible

pass `StringIO()` to capture stdout whose value will be interpreted as a
string later on

Signed-off-by: Kefu Chai <kchai@redhat.com>

Add troubleshooting guidance for ssh connection failures.

For more thorough information about this commit, see:
https://tracker.ceph.com/issues/44905

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Michael Fritch <mfritch@suse.com>
ibid

Signed-off-by: Zac Dover <zac.dover@gmail.com>
Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>
Update doc/cephadm/troubleshooting.rst

Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>
Added mgfritch's suggestion regarding checking to see if the public key
is listed in the authorized_keys file.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

pybind/mgr/dashboard/run-backend-api-tests.sh: use python3 by default

since vstart_runner.py is now py3, let's use python3 by default

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/vstart_runner.py: be python3 compatible

differentiate `str` and `bytes` instances, and drop python2 support from
vstart_runner.py, as we've moved to python3 already

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/suites/*/rados_cls_all.yaml: load all classes

the intention to add the whitelist was to test "sdk" class, but if we
add new classes to the list, and add tests exercising them, the tests
fail if we fail to update these `rados_cls_all.yaml` accordingly.

so in this change, the list is now '*' which allows OSD to load all
classes found in the specified directory

Fixes: https://tracker.ceph.com/issues/45113
Signed-off-by: Kefu Chai <kchai@redhat.com>

doc/dev/developer_guide: use python3 to launch vstart_runner.py

as vstart_runner.py is compatible with python3, and so is the tests
under `qa/` directory.

and use `pip` to install teuthology, simpler this way.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34432 from optimistyzy/47_upstream

os/bluestore/NVMEDevice.cc: use slist for tracking mempool

Reviewed-by: Kefu Chai <kchai@redhat.com>

NVMEDevice: Remove the performance counter

Compared with other approach (KernelDevice, iouring), it does not
have such feature, so remove it to simplify the code and improve
the performance. Because, each thread will call the logger and calculate
the time, it is very time consuming. And the upper layer in bluestore has
the feature to track each I/O task.

And it is not readable if we add ifdef configure to enable this feature,
so in my mind that I suggest removing this latency track. And the
latency benefit demonstrated by SPDK is already approved, so we do not
need to tracking every I/O time when invoking SPDK driver.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>

NVMEDevice.cc: Remove the data_buf_mempool

Use the linked list is enough.

Signed-off-by: Ziye Yang <ziye.yang@intel.com>

Merge pull request #34615 from dillaman/wip-45072

rbd-mirror: image replayer stop might race with instance replayer shut down

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34576 from jdurgin/wip-object-corpus

ceph-object-corpus: update to octopus

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34535 from changchengx/refine_build_liburing

cmake: refactor liburing support

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #34584 from bk201/wip-45108

mgr/test_orchestrator: fix error when listing services

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #32951 from matthewoliver/cephadm_rgw_ssl

cephadm: add RGW SSL support

Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34410 from lxbsz/mds_dirty_parent

tools/cephfs: add accounted_rstat/rstat when building file dentry

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #33928 from mchangir/cephfs-journal-tool-cannot-set-dry_run-arg

cephfs-journal-tool: correctly parse --dry_run argument

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

cmake: use cmake module to integrate liburing

1. WITH_LIBURING is used to set HAVE_LIBURING to decide
   use liburing in KernelDevice or not.
2. WITH_SYSTEM_LIBURING is to choose use system installed
   liburing or build the liburing from source code.

Signed-off-by: Changcheng Liu <changcheng.liu@aliyun.com>

Merge pull request #32612 from rishabh-d-dave/fs-qa-UnicodeDecodeError

test_cephfs_shell: fix UnicodeDecodeError

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

Merge pull request #33169 from moningchao/master

client: fix blacklist removal failed

Reviewed-by: "Yan, Zheng" <zyan@redhat.com>

Merge pull request #33711 from lxbsz/qa_umount_failed

qa/tasks/cephfs: umount the mountpoints when tearDown

Reviewed-by: Rishabh Dave <ridave@redhat.com>

Merge pull request #34533 from fanjunwei/fix_fuse_bug

client: fix ceph-fuse parse command line failed

Reviewed-by: Greg Farnum <gfarnum@redhat.com>

rbd-mirror: track in-flight start/stop/restart in instance replayer

The shut down waits for in-flight ops to complete but the
start/stop/restart operations were previously not tracked. This
could cause a potential race and crash between an image replayer
operation and the instance replayer shutting down.

Fixes: https://tracker.ceph.com/issues/45072
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rgw: reshard: skip stale bucket id entries from reshard queue

If we encounter a reshard queue entry that has an older ID compared to the
bucket's current ID, it'd mean that some other process or a manual reshard has
already processed this entry, skip processing the entry this time. An
alternative is to verify the num_shards that we have in queue >= the current
shards, but this would mean that we may reshard a recently manual resharded
bucket again which might not be intended

Fixes: https://tracker.ceph.com/issues/45134
Signed-off-by: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #34601 from yuriw/wip-yuriw-crontab-master

qa/tests: added octopus-p2p upgrade tests to the mix

rbd-mirror: switch to common C_TrackedOp context class

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

common: add helper C_TrackerOp context class

This wraps the functionality of starting and finishing a tracked op
into the standard context interface.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #34604 from tchaikov/wip-cmake-fuse

cmake: check for FUSE version and cleanups

Reviewed-by: Xiubo Li <xiubli@redhat.com>

cls/journal: use EC pool stripe width for padding appends

to avoid ECSubRead ops to reencode a partially updated stripe.

Signed-off-by: Mykola Golub <mgolub@suse.com>

objclass: add method to get pool stripe width

Signed-off-by: Mykola Golub <mgolub@suse.com>

cmake: detect fuse3 as well

Fixes: https://tracker.ceph.com/issues/44891
Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: check for FUSE version

and refactor FindFUSE.cmake a little bit to conditionalize OSX support

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: link against FUSE::FUSE

instead of specifying the include directory and linkage separately

Signed-off-by: Kefu Chai <kchai@redhat.com>

cmake: rename Findfuse.cmake to FindFUSE.cmake

so its name is consistent with upstream name, and with the variables
exposed by FindFUSE.cmake.

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tests: added octopus-p2p upgrade tests to the mix

Signed-off-by: Yuri Weinstein <yweinste@gmail.com>

Merge pull request #34573 from dillaman/wip-44938-2

test: tx-only peer from heartbeat can race w/ CLI

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34346 from jschmid1/osd_id_claims

mgr/cephadm: add support for osd_id_claims

Reviewed-by: Jan Fajerski <jfajerski@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34517 from tchaikov/wip-qa-dashboard-telemetry

qa/tasks/mgr/dashboard: set last_opt_revision when setting enabled

Reviewed-by: Volker Theile <vtheile@suse.com>
Reviewed-by: Tatjana Dehler <tdehler@suse.com>

Merge pull request #34210 from kotreshhr/cephfs-subvolume-info

mgr/volumes: Add interface to get subvolume metadata

mgr/test_orchestrator: fix error when listing services

- Adapting attribute change of ServiceSpec.
- Update dummy data.

Fixes: https://tracker.ceph.com/issues/45108
Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

qa/tasks/mgr/dashboard: set last_opt_revision when setting enabled

telemetry requires user to re opt-in when opt_revision is bumped up

Signed-off-by: Kefu Chai <kchai@redhat.com>

mgr/orch: Fixes some deserialization errors

Fixes errors when calling `from_json` of these classes:
- InventoryHost: parsing labels
- ServiceDescription: `last_refresh` and `created` fields should be parsed
  to datetime type.

Signed-off-by: Kiefer Chang <kiefer.chang@suse.com>

doc: add object corpus to release checklist

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

ceph-object-corpus: pick up octopus 15.2.1-40-ga838bb1aae

Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34577 from tchaikov/wip-doc-dev-corpus

doc/dev/corpus.rst: tweak formatting

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

doc/dev/corpus.rst: tweak formatting

* use `shell` lexer, otherwise the Python one is used, and the rendered
  result does not look right
* be consistent when indenting -- either use tab or spaces, otherwise
  the indent in code block would be wrong.
* double quote the variables in text

Signed-off-by: Kefu Chai <kchai@redhat.com>

cephadm: add RGW SSL support

This patch adds SSL support to RGW when using cephadm.
If an SSL certificate is provided, inside the json supplied with:

  cpeh orchestrator rgw create -i rgw.json

Then the SSL cert and/or key will be added to pushed into the mon config-key database
using the key `rgw/cert/<rgw_realm>/<rgw_zone>.[crt|key]`.
Which will then be referenced in the config:

  rgw_frontends = beast port=80 ssl_port=443 ssl_certificate=config://rgw/cert/<rgw_realm>/<rgw_zone>.crt

And if an ssl key is also supplied this becomes something like:

  rgw_frontends = beast port=80 ssl_port=443 ssl_certificate=config://rgw/cert/<rgw_realm>/<rgw_zone>.crt ssl_key=config://rgw/cert/<rgw_realm>/<rgw_zone>.key

Of course you could also just upload the cert and key yourself to
config-key location, and ssl will be enabled as well. But this patch
let's you either supply them via `-i` or as a manual upload step.

Co-Authored-By: Michael Fritch <mfritch@suse.com>
Co-Authored-By: Sebastian Wagner <sebastian@spawnhost.de>
Signed-off-by: Matthew Oliver <moliver@suse.com>

include/denc: replace bufferlist::copy with iterator version

This version was only compiled as part of ceph-object-corpus
generation, when ENCODE_DUMP_PATH is defined, so it was missed
when bufferlist::copy() was removed.

Fixes: https://tracker.ceph.com/issues/45023
Signed-off-by: Josh Durgin <jdurgin@redhat.com>

Merge pull request #34541 from neha-ojha/wip-balancer-on

mgr: turn on balancer in upmap mode by default

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

qa/workunits/rbd: retry the addition of a mirror pool peer

We might race with the remote rbd-mirror daemon creating a
tx-only peer when adding a new peer. Therefore, delete the
tx-only peer and attempt to re-create it.

Fixes: https://tracker.ceph.com/issues/44938
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: mirror peer add error message should include site name

It previously included the pointer to string holding the generated
uuid (neither of which would mean much to an end user).

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

rbd: improved error message when adding a duplicate mirror pool peer

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #32747 from Exotelis/wip-40702-collapse

Reviewed-by: Stephan Müller <smueller@suse.com>
Reviewed-by: Tiago Melo <tmelo@suse.com>

Merge pull request #34094 from cbodley/wip-rgw-data-sync-cache

rgw multisite: data sync optimizations

Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

mgr/cephadm: add replacement flag to _create_osd

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34549 from dillaman/wip-qa-rbd-read-balance

qa/suites/rbd: tweak 'require_osd_release' to 'pacific'

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #34496 from dillaman/wip-44105

rbd-mirror: clean-up unnecessary non-primary snapshots

Reviewed-by: Mykola Golub <mgolub@suse.com>

mgr/dashboard: Add expand/collapse datatable feature

Adds expand/collapse feature to every datatable with details.

Fixes: https://tracker.ceph.com/issues/40702
Signed-off-by: Sebastian Krah <skrah@suse.com>

mgr/cephadm: use --osd-ids instead of --osd-id

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: add support for osd_id_claims

Signed-off-by: Joshua Schmid <jschmid@suse.de>

Merge pull request #34529 from maran/master

cephadm: warn when no valid version arguments are supplied to add-repo

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34490 from sebastian-philipp/cephadm-command_add_repo-no-podman

cephadm: add-repo: Don't test for podman

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34443 from sebastian-philipp/cephadm-revert-33974

cephadm: Update images used

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34556 from ahaase-c4u/patch-1

mgr/cephadm: corrected documentation for starting an upgrade

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34555 from ahaase-c4u/45065

mgr/cephadm: Fixed warn_on_stray_hosts setting

Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34415 from sebastian-philipp/orch-fix-rgw-daemon-description-id

mgr/orch: Fix DaemonDescription().daemon_id() for RGWs

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #34216 from jschmid1/preview_drivegroups

mgr/cephadm: preview osd creation for osdspecs

Reviewed-by: Kiefer Chang <kiefer.chang@suse.com>
Reviewed-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #34558 from mgfritch/cephadm-enter-container

cephadm: use CephContainer during command `enter`

Reviewed-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: enable osd previews

Signed-off-by: Joshua Schmid <jschmid@suse.de>

mgr/cephadm: corrected documentation for starting an upgrade

Signed-off-by: Andreas Haase <andreas.haase@community4you.de>

doc/rados/operations: update docs to reflect balancer being on by default

Signed-off-by: Neha Ojha <nojha@redhat.com>

Merge pull request #34562 from tchaikov/wip-jenkins-trigger

github: add "jenkins test classic perf" to jenkins commands list

Reviewed-by: Josh Durgin <jdurgin@redhat.com>

github: add "jenkins test classic perf" to jenkins commands list

it's used for triggering perf test with classic OSD

Signed-off-by: Kefu Chai <kchai@redhat.com>