doc/rados: document unfound object cache-tiering scenario

Explain how to deal with "unfound objects" when restarting OSDs in a
cache-tiered environment.

Fixes: https://tracker.ceph.com/issues/44286
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58860 from adk3798/cephadm-nvmeof-require-group

mgr/cephadm: require "group" parameter in nvmeof specs

Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #59165 from NitzanMordhai/wip-nitzan-test-rados-tools-newline-trim

test: test_rados_tools compare output without trimming newline

Merge pull request #59153 from ajarr/wip-67436

rbd: fix CLI output of `rbd group snap info` command when a group snapshot with no member images

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>
Reviewed-by: Sunil Angadi <Sunil.Angadi@ibm.com>

Merge pull request #59292 from cyx1231st/wip-seastore-revert-decouple-ool-writes

Revert "crimson/os/seastore: wait ool writes in DeviceSubmission phase"

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>
Reviewed-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #59241 from tobias-urdin/openstack-upperconstraints

qa: barbican: restrict python packages with upper-constraints

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58961 from NitzanMordhai/wip-nitzan-dencoder-test-forward-incompat-fix

workunit/dencoder: dencoder test forward incompat fix

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #58594 from jamiepryde/isa-xor-raid

erasure-code/isa: Use isa/raid's xor_gen() instead of the region_xor(…

Reviewed-by: Mark Nelson <mnelson@redhat.com>

qa: barbican: restrict python packages with upper-constraints

We install barbican by doing a pip install directly on the
cloned git repository but we don't honor the upper-constraints
from the OpenStack Requirements project that handles what
versions is supported.

This changes the pip install command that we issue when
installing barbican to honor the requirements for the
version (derived from the branch) that we use, in
this case it's the 2023.1 release upper-constraints [1].

This prevents us from pulling in untested Python packages.

This only updates Barbican because for the Keystone job
we dont directly issue pip but install using tox using the
`venv` environment which already by default sets the
constraints as you can see in [2].

[1] https://releases.openstack.org/constraints/upper/2023.1
[2] https://github.com/openstack/keystone/blob/stable/2023.1/tox.ini#L12

Fixes: https://tracker.ceph.com/issues/67444
Signed-off-by: Tobias Urdin <tobias.urdin@binero.com>

Merge pull request #59239 from yuvalif/wip-yuval-67513

Reviewed-By: Casey Bodley <cbodley@ibm.com>
test/rgw/notification: use real ip address instead of localhost

based on that comment:
https://tracker.ceph.com/issues/67206#note-6
the address used by the endpoint is taken as the real IP address of the
host where the test script is running and not localhost.

we also changed the rabbitmq-server conf to allow "guest"
user to connect over non localhost address

Fixes: https://tracker.ceph.com/issues/67206
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #59200 from ifed01/wip-ifed-fix-store-test-col-ref

test/store_test: fix assertions due to unclosed collection refs.

Reviewd-by: Pere Diaz Bou <pere-altea@hotmail.com>

Merge pull request #59256 from zdover23/wip-doc-2024-08-17-cephfs-ceph-dokan-mount-point

doc/cephfs: s/mountpoint/mount point/

Reviewed-by: Jos Collin <jcollin@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #58995 from rhcs-dashboard/fix-66844-main

qa/mgr/dashboard: fix test race condition

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59212 from cyx1231st/wip-seastore-more-reports

crimson/os/seastore/cache: report lru usage/in/out with trans and extent type

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Revert "crimson/os/seastore: wait ool writes in DeviceSubmission phase"

This reverts commit c9e423facea79d42f0496264f267adee5d911b87.

The commit starts to submit OOL writes before submitting the journal
write, true, but it cannot guarantee that OOL writes finish before the
journal write.

Thus it is possible that during SeaStore restart, a journal record
appears valid but its dependent OOL records are partial written, which
leads to corruption.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #59290 from anthonyeleven/mountpoint

doc: Harmonize 'mountpoint'

doc: Harmonize 'mountpoint'

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #59257 from zdover23/wip-doc-2024-08-17-cephfs-mount-point

doc/cephfs: s/mountpoint/mount point/

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/cephfs: s/mountpoint/mount point/

Change the string "mountpoint" to "mount point" in English-language
strings (as opposed to in commands, where the string "mountpoint"
sometimes appears and is correct).

cf. https://github.com/ceph/ceph/pull/58908#discussion_r1697715486 in
which page 345 of The IBM Style Guide is referenced to back up this
change.

This commit alters only English-language text and example commands in
which the string "{mount point}" is meant to be replaced. No commands
meant for cutting-and-pasting have been altered in this commit.

Signed-off-by: Zac Dover <zac.dover@proton.me>

doc/cephfs: s/mountpoint/mount point/

Change the string "mountpoint" to "mount point" in English-language
strings (as opposed to in commands, where the string "mountpoint"
sometimes appears and is correct).

cf. https://github.com/ceph/ceph/pull/58908#discussion_r1697715486
in which page 345 of The IBM Style Guide is referenced to back up this
change.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58355 from batrick/ceph-backport-fetchhead

script/ceph-backport: robustness adjustments for local git repo quirks

Reviewed-by: Venky Shankar <vshankar@redhat.com>

test/rgw/notification: use real ip address instead of localhost

based on that comment:
https://tracker.ceph.com/issues/67206#note-6
the address used by the endpoint is taken as the real IP address of the
host where the test script is running and not localhost.

we also changed the rabbitmq-server conf to allow "guest"
user to connect over non localhost address

Fixes: https://tracker.ceph.com/issues/67206
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #59167 from zdover23/wip-doc-2024-08-12-cephfs-file-layouts

doc/cephfs: improve "layout fields" text

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

Merge PR #58896 into main

* refs/pull/58896/head:
client: flush the caps release in filesystem sync

Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #59205 from xxhdx1985126/wip-seastore-find-pending-version

crimson/os/seastore/btree: fix minor corner case issue

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

client: flush the caps release in filesystem sync

We have hit a race between cap releases and cap revoke request
that will cause the check_caps() to miss sending a cap revoke ack
to MDS. And the client will depend on the cap release to release
that revoking caps, which could be delayed for some unknown reasons.

In Kclient we have figured out the RCA about race and we need
a way to explictly trigger this manually could help to get rid
of the caps revoke stuck issue.

Fixes: https://tracker.ceph.com/issues/67221
Signed-off-by: Xiubo Li <xiubli@redhat.com>

Merge pull request #58415 from ljflores/wip-tracker-66809

qa/suites/upgrade: ignore PG_AVAILABILITY and MON_DOWN for quincy-x and reef-x upgrade suites

Merge pull request #59220 from ivoalmeida/carbon-datatable-cleanups

mgr/dashboard: carbon datatables impr and cleanups

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #59218 from yuvalif/wip-yuval-67525

rgw/notifications: fixing radosgw-admin notification json

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #59118 from xxhdx1985126/wip-crimson-backfill-cancellation

crimson/osd/backfill_state: support backfill cancellation

Reviewed-by: Matan Breizman <mbreizma@redhat.com>

crimson/osd/backfill_state: support backfilling cancellation

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

crimson/osd/pg_recovery: reset backfill_state when backfill finished

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #57966 from xxhdx1985126/wip-crimson-concurrent-recover-missing

crimson/osd/osd_operations: make the "recover_missing" phase concurrent

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #53151 from xxhdx1985126/wip-crimson-backfill-fixes

crimson/osd/backfill_state: fixes two corner cases in backfilling

Reviewed-by: Radosław Zarzyński <rzarzyns@redhat.com>

Merge pull request #58911 from yuvalif/wip-yuval-67229

test/cls_2pc_queue: prevent list+remove race between consumers

Reviewed-By: Casey Bodley <cbodley@ibm.com>

Merge pull request #59219 from yuvalif/wip-yuval-50610

doc/rgw/notification: persistent notification queue full behavior

Reviewed-By: Anthony D'Atri <anthony.datri@gmail.com>

crimson/os/seastore/cache: report lru usage/in/out with trans and extent type

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: cleanup periodical reporting

Consolidate time into a single place per SeaStore::Shard.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache/lru: renames

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: refine lru logics

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: move counter_by_extent_t definition

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/seastore_types: unify checks to the extent types

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #59225 from zdover23/wip-doc-2024-08-15-glossary-flapping-osd

doc/glossary: add "flapping OSD"

Merge pull request #59028 from cbodley/wip-67326

rgw/notify: visit() returns copy of owner string

Reviewed-by: Adam Emerson <aemerson@redhat.com>

doc/glossary: add "flapping OSD"

Add an entry for "Flapping OSD" to the glossary.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58448 from cbodley/wip-rgw-lc-async

cls/rgw: define lc ops in terms of ObjectOperation instead of IoCtx

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

mgr/dashboard: carbon datatables impr and cleanups

Fixes: https://tracker.ceph.com/issues/67544,
https://tracker.ceph.com/issues/67538,
https://tracker.ceph.com/issues/67542, https://tracker.ceph.com/issues/67545, https://tracker.ceph.com/issues/67546

Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

Merge pull request #58965 from linuxbox2/wip-lcgt-typo

rgwlc: fix typo in getlc (ObjectSizeGreaterThan)

Reviewed-by: Casey Bodley <cbodley@redhat.com>

doc/rgw/notification: persistent notification queue full behavior

Fixes: https://tracker.ceph.com/issues/50610
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #59169 from cbodley/wip-67464

rgw: revert account-related changes to get_iam_policy_from_attr()

Reviewed-by: Pritha Srivastava <prsrivas@redhat.com>

Merge pull request #57888 from liangmingyuanneo/wip-standalone-test-pg-repair

qa/standalone: bugfix for latecy repair after scrub

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>

Merge pull request #59210 from rhcs-dashboard/multi-cluster-overview-usage-template-fix

mgr/dashboard: fix multi-cluster usage bar error after carbon changes

rgw/notifications: fixing radosgw-admin notification json

Fixes: https://tracker.ceph.com/issues/67525
Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

Merge pull request #59168 from zdover23/wip-doc-2024-08-12-cephfs-cache-configuration

doc/cephfs: improve cache-configuration.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

crimson/os/seastore/cache: pass missing src to touch_extent()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup add_extent()

Move add_to_dirty() and touch_extent() out of add_extent(), this removes
duplicated calls to touch_extent() from the on_cache callback.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cache: cleanup remove_from_dirty()

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: drop duplicated calls to touch_extent()

The extent is already PRESENT, which means it was already touched in
this transaction.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/cached_extent: rename primary_ref_list

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/dashboard: fix multi-cluster usage bar error after carbon changes

fixes: https://tracker.ceph.com/issues/67536

Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #59186 from rhcs-dashboard/replace-cluster-capacity-with-usage-bar

mgr/dashboard: replace individual cluster's capacity info with Usage bar in Multi-Cluster

Merge PR #59025 into main

* refs/pull/59025/head:
tools/rados: Fix extra NL in getxattr

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Gabriel Benhanokh <gbenhano@redhat.com>

Merge pull request #58485 from ivoalmeida/carbon-datatable

mgr/dashboard: replace ngx-datatable by carbon datatable

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

crimson/os/seastore/btree: fix minor corner case issue

Signed-off-by: Xuehan Xu <xuxuehan@qianxin.com>

qa/suites/upgrade: ignore MON_DOWN and PG_AVAILABILITY warnings in upgrade tests

Signed-off-by: Laura Flores <lflores@ibm.com>

rbd: fix CLI output of `rbd group snap info` command

... when a group snapshot has no member images.

A group snapshot can be created with no member images. For such a group
snapshot, omit the 'image snap' and 'images' fields from the
unformatted CLI output of `rbd group snap info` command so as to not
confuse the user. In the librbd C/C++ data structures representing a
group snapshot with no member images, set the 'image_snap_name' data
member to an empty string.

Fixes: https://tracker.ceph.com/issues/67436
Signed-off-by: Ramana Raja <rraja@redhat.com>

Merge pull request #58140 from guits/cv-tpm2-support

ceph-volume: add TPM2 token enrollment support for encrypted OSDs

Merge pull request #58956 from ThomasLamprecht/ceph-volume-debian-dependency

debian pkg: record python3-packaging dependency for ceph-volume

qa/s3tests: configure tenant name for 's3 tenant' section

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: replace ngx-datatable by carbon

Fixes: https://tracker.ceph.com/issues/66965
* replaced ngx-datatable by carbon datatable
* created carbon themes for content and tables
* redesigned table actions to render as kebab menu options per data row
* keep only primary actions on datatable toolbar
* implemented carbon batch actions

Signed-off-by: Ivo Almeida <ialmeida@redhat.com>

test/store_test: fix assertions due to unclosed collection refs.

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>

qa/cephadm: add group param when applying nvmeof

Since it will now be required

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: make nvme-gw adds be able to handle multiple services/groups

Before this was grabbing the service spec for the first daemon
description in the list. This meant every daemon would be added
with the pool/group of whatever that spec happened to specify.
This patch grabs the spec, and therefore also the pool/group
individually for each nvmeof daemon

Signed-off-by: Adam King <adking@redhat.com>

mgr/cephadm: migrate nvmeof specs without group field

As we have added the group field as a requirement for new
nvmeof specs and check for it in spec validation, we need
a migration to populate this field for specs we find that
don't have it.

Signed-off-by: Adam King <adking@redhat.com>

mgr/orchestrator: require "group" field for nvmeof specs

Signed-off-by: Adam King <adking@redhat.com>

Merge pull request #59112 from idryomov/wip-67353

qa: adjust expected io_opt in krbd_discard_granularity.t

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #54504 from aclamk/wip-aclamk-bs-refactor-write-path

os/bluestore: Recompression, part 2. New write path.

Merge pull request #59001 from cbodley/wip-rgw-async-inline-gc

rgw/rados: delete_objs_inline() uses rgw_rados_operate()

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

mgr/dashboard: replace individual cluster's capacity info with Usage bar in Multi-Cluster

fixes: https://tracker.ceph.com/issues/67193

Signed-off-by: Naman Munet <nmunet@redhat.com>

Merge pull request #58410 from bill-scales/erasure_coding_design_document

doc: doc/dev/osd_interals/erasure_coding/enhancements.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Sam Just <sjust@redhat.com>

Merge pull request #59177 from anthonyeleven/aad-qat-followup

doc/radosgw: improve qat-accel.rst

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/radosgw: improve qat-accel.rst

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge PR #58532 into main

* refs/pull/58532/head:
qa: improve and add commit ID to comment for context

Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>

doc/cephfs: improve cache-configuration.rst

Improve the text in the section about dealing with cache-pressure alerts
that was added in https://github.com/ceph/ceph/pull/59077. The changes
in this commit were suggested by Anthony D'Atri.

Co-authored-by: Patrick Donnelly <pdonnelly@redhat.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #55050 from rhcs-dashboard/nfs-monitoring

mgr/nfs: scrape nfs monitoring endpoint

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

rgw: revert account-related changes to get_iam_policy_from_attr()

while bucket ARNs in iam policies don't include account names, policy
evaluation does need to differentiate between buckets in different
tenant namespaces

when requests pass bucket/object ARNs into
verify_bucket/object_permission(), those do include the bucket's tenant
name. to match against those ARNs, we also need to pass the requested
bucket's tenant name into get_iam_policy_from_attr()

Fixes: https://tracker.ceph.com/issues/67464
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #54075 from aclamk/wip-aclamk-bs-nice-print

os/bluestore: Recompression, part 1. Nice debugs.

Merge pull request #59164 from toficly/patch-2

doc:update e-mail addresses governance

doc/cephfs: improve "layout fields" text

Improve "layout fields" text in doc/cephfs/file-layouts.rst, as suggesed
by Anthony D'Atri in these comments:

https://github.com/ceph/ceph/pull/59021#discussion_r1704108581
https://github.com/ceph/ceph/pull/59021#discussion_r1704112320

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

test: test_rados_tools compare output without trimming newline

Fixes: https://tracker.ceph.com/issues/67419
Signed-off-by: Nitzan Mordechai <nmordech@redhat.com>

doc:update e-mail addresses governance

with the consent of my colleagues
- Dan van der Ster
- João Eduardo Luis
- Mark Nelson

changed their e-mail addresses to the current one

Signed-off-by: Tobias Fischer <tobias.fischer@clyso.com>
Signed-off-by: Tobias Fischer <138117964+toficly@users.noreply.github.com>

ceph-volume: refactor device path handling for LVM lookups

This consolidates the conditional checks for device paths to
reduce redundancy and improve readability and adds logic to
handle both '/dev/mapper' and '/dev/dm-' paths uniformly by
introducing a utility function `get_lvm_mapper_path_from_dm()`.

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

ceph-volume: add TPM2 token enrollment support for encrypted OSDs

This adds the required changes to ceph-volume and cephadm in order to support
deploying tpm2 token enrolled encrypted OSDs.

Adding `--with-tpm` when deploying with `--dmcrypt` will enroll a tpm2
token to the corresponding LUKS2 devices.

Example of a osd service spec:

```
service_type: osd
service_id: tpm2_osds
placement:
  host_pattern: '*'
spec:
  data_devices:
    paths:
      - /dev/sdb
  encrypted: true
  tpm2: true
```

Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

Merge pull request #59098 from athanatos/sjust/wip-67415-alienstore-mkfs-crash

crimson: clear AlienStore::coll_map in umount rather than in stop

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #59077 from zdover23/wip-doc-2024-08-07-cephfs-cache-configuration-cache-pressure

doc/cephfs: add cache pressure information

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #58951 from rhcs-dashboard/ceph-users-doc-fix

mgr/dashboard: fix ceph-users api

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #58976 from athanatos/sjust/for-review/wip-nvmeofpaxosservice-cosmetic-fixes

mon: cosmetic fixes for NVMeofGwMon files

Reviewed-by: Alexander Indenbaum <aindenba@redhat.com>
Reviewed-by: Leonid Chernin <lechernin@gmail.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Aviv Caro <Aviv.Caro@ibm.com>

Merge pull request #57003 from linuxkidd/wip-pool-max-avail

PGMap: remove pool max_avail scale factor

Merge pull request #58573 from idryomov/wip-selinux-drop-ignore-agetty

qa: agetty entry in selinux allowlist is redundant now

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge PR #58784 into main

* refs/pull/58784/head:
script/backport-create-issue: add debug messages for retry

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>