mgr/cephadm: mgr orchestrator module raise exception if there is trailing tab in yaml file

Fixes: https://tracker.ceph.com/issues/69192
Signed-off-by: Shweta Bhosale <Shweta.Bhosale1@ibm.com>

Merge pull request #60928 from adk3798/fix-nvme-loop-parsing

qa/tasks/nvme_loop: update task to work with new nvme list format

Reviewed-by: Laura Flores <lflores@ibm.com>

Merge pull request #60992 from MaxKellermann/test__includes

test: add missing includes

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60773 from pritha-srivastava/wip-rgw-revert-auth-code

Revert "rgw/auth: Fix the return code returned by AuthStrategy,"

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60968 from cbodley/wip-69134

qa/rgw: bump keystone/barbican from 2023.1 to 2024.1

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #59808 from yuvalif/wip-yuval-bucket-logging3

rgw/logging: add support for GetBucketLogging and PutBucketLogging

Reviewed-By: Anthony D'Atri <anthony.datri@gmail.com>, Adam Emerson <aemerson@ibm.com>, Casey Bodley <cbodley@ibm.com>

Merge pull request #61016 from idryomov/wip-p-snapname-type

pybind/rbd: fix a type comment for p_snapshot of clone()

Reviewed-by: Vinay Bhaskar Varada <vvarada@redhat.com>

test: add missing includes

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

pybind/rbd: fix a type comment for p_snapshot of clone()

This was missed in commit d7fd66ec9944 ("librbd: add rbd_clone4() API
to take parent snapshot by ID").

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #60997 from MaxKellermann/sstring__char8_t

tools/ceph-dencoder/sstring.h: use `char8_t` instead of `unsigned char`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60831 from kamoltat/wip-ksirivad-fix-68338

doc/rados/operations/stretch-mode.rst: Added Limitations to stretch pool configurations
Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60927 from ljflores/wip-tracker-69107

qa/workunits/mon: ensure election strategy is "connectivity" for stretch mode
Reviewed-by: Kamoltat Sirivadhna <ksirivad@redhat.com>

Merge pull request #60998 from MaxKellermann/test_time__using_mono_clock

test/common/test_time: add missing `using ceph::mono_clock`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

test/common/test_time: add missing `using ceph::mono_clock`

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

tools/ceph-dencoder/sstring.h: use `char8_t` instead of `unsigned char`

This fixes a build failure with libc++ (clang/LLVM).  This build
failure is correct: there exists no specialization for
`std::char_traits<unsigned char>`.  The standards-compliant way to use
unsigned chars in strings is to use `char8_t`.

Signed-off-by: Max Kellermann <max.kellermann@ionos.com>

test/rgw/logging: add bucket logging test suite to teuthology

this is depended with this s3test PR:
https://github.com/ceph/s3-tests/pull/576

Signed-off-by: Yuval Lifshitz <ylifshit@ibm.com>

rgw/logging: add support for GetBucketLogging and PutBucketLogging

this is based on AWS server access logs:
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/enable-server-access-logging.html
- https://docs.aws.amazon.com/AmazonS3/latest/userguide/ServerLogs.html
- https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLogging.html
- https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetBucketLogging.html

however, a new mode was added called "journal" where:
- logs of PUT, COPY and MPU are guaranteed
- we have logs of DELETE and multi-DELETE operations (not guaranteed)
- log records hold only minimal amount of information

Fixes: https://tracker.ceph.com/issues/984
Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #60812 from pecastro/improve-a-few-tests

tests: Improve unittest_dbstore_tests and unittest_deferred to be suitable for reruns.

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #60829 from cbodley/wip-69042

radosgw-admin: bucket link/unlink support accounts

Reviewed-by: Adam Emerson <aemerson@redhat.com>

radosgw-admin: bucket link/unlink support accounts

buckets are owned by accounts instead of their individual users

add support for --account-id to 'bucket link' and 'bucket unlink'

reject 'bucket link' when the requested --uid belongs to an account

Fixes: https://tracker.ceph.com/issues/69042
Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60129 from cbodley/wip-68396

test/rgw/multisite: add meta checkpoint after bucket creation

Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #60304 from cbodley/wip-async-completion-recycling

common/async: Completion uses asio::recycling_allocator by default

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #60415 from cbodley/wip-68645

radosgw-admin: allow 'sync group pipe modify' with existing user

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #60830 from cbodley/wip-69043

radosgw-admin: 'user create' rejects uids matching the account id format

Reviewed-by: Adam Emerson <aemerson@redhat.com>

Merge pull request #59383 from joscollin/wip-B67360-counters-set-on-mds-failed

qa: restrict 'perf dump' on active mds only

Merge pull request #60929 from dmick/wip-release-docs

doc/dev/release-process.rst: New container build/release process

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #60947 from idryomov/wip-67759

qa/suites/upgrade/*-x: skip TestClsRbd.group_snap_{set,remove}

Reviewed-by: N Balachandran <nibalach@redhat.com>
Reviewed-by: Laura Flores <lflores@ibm.com>

qa/rgw: bump keystone/barbican from 2023.1 to 2024.1

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #60966 from gbregman/main

mgr/cephadm/nvmeof: Add encryption key file for nvmeof encryption

Merge pull request #60948 from idryomov/wip-68588

qa/suites/rados/thrash-old-clients: skip TestClsRbd.group_snap_{set,remove}

Reviewed-by: N Balachandran <nibalach@redhat.com>

mgr/cephadm/nvmeof: Add key encryption support to nvmeof configuration

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #60926 from ktdreyer/container-labeler

labeler: add container labelers

Merge pull request #60946 from nbalacha/wip-nbalacha-69112

cls/rbd: fix mirror_image_map_list object

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #60913 from myoungwon/wip-fix-null-pointer-crimson

crimson/osd/ops_executer: prevent access to null pointers caused by gcc bug

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

test/objectstore: Make test idempotent

Give it a new DB name for each test so we don't get failures on rerun.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

rgw: Make test idempotent.

Give it a new DB name for each test so we don't get failures on rerun.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

rgw: Remove unnecessary whitespace.

Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com>

Merge pull request #60944 from dalgaaf/dalgaaf-fix-doc-ec

doc: fix EC overhead table m-numbering

Merge pull request #60954 from zdover23/wip-doc-2024-12-04-dev-development-workflow-backporter-manual

doc/dev: update link to backporter manual

doc/dev: update link to backporter manual

Update the link to the backporter manual, in
doc/dev/development-workflow.rst. Laura Flores requested this change on
04 Dec 2024.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60924 from ktdreyer/containerfile-date-utc

container/build.sh: use UTC dates for tags

Merge pull request #60930 from zdover23/wip-doc-2024-12-03-rados-ops-health-checks-3

doc/rados: fix sentences in health-checks (3 of x)

Reviewed-by: Cole Mitchell <cole.mitchell.ceph@gmail.com>

qa/suites/upgrade/*-x: skip TestClsRbd.group_snap_{set,remove}

These tests assert on the number of keys on the group header object,
which changed with commits ec6224db3e55 ("cls/rbd: save group snapshot
creation order in a new key") and 3662bc7b3dc3 ("cls/rbd: save max
group snap order").

Fixes: https://tracker.ceph.com/issues/67759
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

cls/rbd: fix mirror_image_map_list object

The rbd-mirror image map is stored in the rbd_mirror_leader object.
Fixed mirror_image_map_list() to operate on the correct object.

Fixes: https://tracker.ceph.com/issues/69112
Signed-off-by: N Balachandran <nibalach@redhat.com>

qa/suites/rados/thrash-old-clients: skip TestClsRbd.group_snap_{set,remove}

These tests assert on the number of keys on the group header object,
which changed with commits ec6224db3e55 ("cls/rbd: save group snapshot
creation order in a new key") and 3662bc7b3dc3 ("cls/rbd: save max
group snap order").

Fixes: https://tracker.ceph.com/issues/68588
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

qa/suites/rados/thrash-old-clients: switch to cls/test_cls_rbd.sh workunit

For consistency with other upgrade jobs, don't invoke ceph_test_cls_rbd
directly.  While at it, drop a long outdated filter.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #60514 from rhcs-dashboard/fix-68733-main

mgr/dashboard: fix total objects/Avg object size in RGW Overview Page

Reviewed-by: Afreen Misbah <afreen@ibm.com>

doc/dev/release-process.rst: New container build/release process

Signed-off-by: Dan Mick <dan.mick@redhat.com>

crimson/osd/ops_executer: prevent access to null pointers caused by gcc bug

This commit assigns the parameter eariler to work around a coroutine related gcc bug
that causes a null pointer access.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>
Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #60904 from cyx1231st/wip-seastore-hex-log

crimson/os/seastore: convert offset/length in logs to hex format

Reviewed-by: Xuehan Xu <xuxuehan@qianxin.com>

Merge pull request #60755 from dmick/wip-release-container

Modify container/ software to support release containers and the promotion of prerelease containers

Merge pull request #60018 from smanjara/rgw-script-multiple-zg

rgw/multisite: test-rgw-multisite.sh can create multiple zonegroups

Merge pull request #60919 from zdover23/wip-doc-2024-12-03-rados-ops-health-checks-2

doc/rados: fix sentences in health-checks (2 of x)

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: fix sentences in health-checks (3 of x)

Make sentences agree at the head of each section in
doc/rados/operations/health-checks.rst. The sentences were sometimes in
the imperative mood and sometimes in the declarative mood.

This commit edits the second third of
doc/rados/operations/health-checks.rst.

Note to (I hope soon) future Zac: There are a a couple of places near
the end of this file where the sentences are ungrammatical. Update these
in a separate PR (in isolation, so that the grammar and technical
accuracy of these sentences can be the primary focus of the reviewers).

Zac: cf. 000228

Signed-off-by: Zac Dover <zac.dover@proton.me>

container/{Containerfile,build.sh}: use a secret to pass secrets

Use podman's build --secret mechanism to pass credentials for
download.ceph.com

Signed-off-by: Dan Mick <dan.mick@redhat.com>

container/Containerfile: allow failure from curl to stop build

curl doesn't actually fail unless -f is used; also, add
some sh -e flags.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

container/Containerfile: purge .repo files with secrets before commit

ceph.repo had creds in it for download.ceph.com/prerelease.
Remove the .repo files we construct, since they're not necessary
once the container is built (no one should be dnf'ing anything
in the container).

Signed-off-by: Dan Mick <dan.mick@redhat.com>

container/Containerfile fix boolean IS_RELEASE usage

Existing syntax was just wrong.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

qa/tasks/nvme_loop: update task to work with new nvme list format

Specifically on some centos 9 tests, we've seen that a newer
version of some nvme related package is causing this task to fail
with "KeyError: 'DevicePath'" due to the format of the output
of the nvme list command changing. This patch adds handling for
the new format we've seen while also still supporting the old
format (necessary for the tests running on ubuntu).

Fixes: https://tracker.ceph.com/issues/69067
Signed-off-by: Adam King <adking@redhat.com>

qa/workunits/mon: ensure election strategy is "connectivity" for stretch mode

The election strategy is randomly chosen for this type of test. Sometimes,
the test passes if the "connectivity" election strategy happens to be picked.
But if a different strategy, i.e. "classic", is picked, then the test will fail.

We can ensure that the election strategy is "connectivity" by setting it in the
workunit with the ceph CLI command. Although connectivity was specified in
stretch-mode-5-mons-8-osds.yaml, that config ultimately gets overridden by
the "qa/mon_config" yaml.

Fixes: https://tracker.ceph.com/issues/69107
Signed-off-by: Laura Flores <lflores@ibm.com>

Merge pull request #59862 from jamiepryde/isa-xor-handle-misaligned-buffers

erasure-code/isa: Handle case where data and coding buffers are not 3…

Reviewed-by: Jose J Palacios-Perez <perezjos@uk.ibm.com>

Merge pull request #58473 from YiteGu/add-mon-scrub-lat

mon/scrub: add mon scrub latency

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #60131 from Shreyasapale/wip-fix-shreya-68009

osd/scrubber: Clean up unused function add_double

container/make-manifest-list.py: use LOCALMANIFEST throughout

Signed-off-by: Dan Mick <dan.mick@redhat.com>

containers/make-manifest.py: Add --promote

The promote function finds the latest image on the prerelease repo,
finds all its tags, and copies them all to the release repo.

Signed-off-by: Dan Mick <dan.mick@redhat.com>

container/Containerfile, build.sh: Add PRERELEASE_USERNAME/PASSWORD

Used for accessing download.ceph.com

Signed-off-by: Dan Mick <dan.mick@redhat.com>

container/build.sh:

- change repo path to prerelease
- remove image id arg for release push
- include 'v' in version name for release push

Signed-off-by: Dan Mick <dan.mick@redhat.com>

container/make-manifest-list.py

- don't print command failure in worker; let the caller print them
  if desired (allow silent failure)
- allow for empty tags list
- look for CEPH_SHA1.  GIT_COMMIT was the sha1 of the ceph-container.git
  commit
- change default paths to prerelease
- add --dry-run to avoid final push
- rename 'HOST' to 'CONTAINER_HOST'
- Use ARCH_SPECIFIC_HOST instead of CONTAINER_HOST (which is used by podman)

Signed-off-by: Dan Mick <dan.mick@redhat.com>

Merge pull request #60172 from smanjara/rgw-multisite-two-zonegroups

qa/multisite: enable two zonegroup yaml

Merge pull request #60591 from smanjara/wip-datalog-autotrim

qa/multisite: add extra checkpoints in datalog_autotrim testcase

labeler: add container labelers

Signed-off-by: Ken Dreyer <kdreyer@ibm.com>

container/build.sh: use UTC dates for tags

Do not rely on the builder host's local timezone setting.

Signed-off-by: Ken Dreyer <kdreyer@ibm.com>

doc: fix EC overhead table m-numbering

Signed-off-by: Danny Al-Gaaf <danny.al-gaaf@bisect.de>

Merge pull request #60918 from gbregman/main

python-common/ceph/cephadm: Change NVME image version to 1.4.

Merge pull request #60917 from zdover23/wip-doc-2024-12-03-rados-ops-health-checks

doc/rados: make sentences agree in health-checks.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: fix sentences in health-checks (2 of x)

Make sentences agree at the head of each section in
doc/rados/operations/health-checks.rst. The sentences were sometimes in
the imperative mood and sometimes in the declarative mood.

This commit edits the second third of
doc/rados/operations/health-checks.rst.

Zac: cf. 000228

Signed-off-by: Zac Dover <zac.dover@proton.me>

python-common/ceph/cephadm: Change NVME image version to 1.4.
Fixes https://tracker.ceph.com/issues/69099

Signed-off-by: Gil Bregman <gbregman@il.ibm.com>

Merge pull request #60767 from rhcs-dashboard/smb-cluster

mgr/dashboard: list smb

Reviewed-by: Afreen Misbah <afreen@ibm.com>

Merge pull request #60099 from MaxKellermann/json_spirit_thread

json_spirit: add missing dependency on Boost::thread

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rados: make sentences agree in health-checks.rst

Make sentences agree at the head of each section in
doc/rados/operations/health-checks.rst. The sentences were sometimes in
the imperative mood and sometimes in the declarative mood.

This commit edits the first third of
doc/rados/operations/health-checks.rst.

Zac: cf. 000228

Signed-off-by: Zac Dover <zac.dover@proton.me>

crimson/os/seastore: convert offset/length in seastore logs to hex format

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: convert offset/length in segment logs to hex format

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60151 from shreyanshjain7174/wip-skoduri-cloud-restore

rgw/restore: Fixed status codes and response headers
Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #59449 from tth-trang/wip-65772-quincy

rgw/lc: Fix lifecycle not working while bucket versioning is suspended

Reviewed-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #60910 from clwluvw/putacl-log

rgw: relax RGWPutACLs_ObjStore::get_params read data log

Merge pull request #60254 from clwluvw/zonegroups-createbucket

rgw: initialize createparams zone_placement to avoid garbage value

Merge pull request #60589 from clwluvw/requested-gen-sync

rgw/multisite: revert "rgw/multisite: avoid redundant error repo entry logging

qa/multisite: enable two zonegroup yaml

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

Merge pull request #60906 from zdover23/wip-doc-2024-12-02-cephadm-services-mon-capitalize-ceph

doc/cephadm: s/ceph/Ceph/ in mon.rst

rgw: relax RGWPutACLs_ObjStore::get_params read data log

the input can be logged with level 20 than zero.

Fixes: https://tracker.ceph.com/issues/69093
Signed-off-by: Seena Fallah <seenafallah@gmail.com>

Merge pull request #60412 from ShwetaBhosale1/fix_issue_68605_modify_structure_of_default_images

cephadm: Modify the structure of the default container images

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@ibm.com>

Merge pull request #60903 from nbalacha/wip-nbalacha-rbd-man

doc/rbd: Add mirror snapshot schedule information

Reviewed-by: Zac Dover <zac.dover@proton.me>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rbd: Add mirror snapshot schedule information

Signed-off-by: N Balachandran <nibalach@redhat.com>

mgr/dashboard: list smb
Tracker: https://tracker.ceph.com/issues/69080

Signed-off-by: Dnyaneshwari Talwekar <dtalweka@redhat.com>

Merge pull request #60862 from rhcs-dashboard/fix-multisite-e2e-failures

mgr/dashboard: fix multisite e2e failures

Reviewed-by: Afreen Misbah <afreen@ibm.com>

doc/cephadm: s/ceph/Ceph/ in mon.rst

s/ceph/Ceph/ in doc/cephadm/services/mon.rst where appropriate.

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #60759 from rhcs-dashboard/add-smb-endpoints

mgr/dashboard: add smb endpoints

Reviewed-by: Afreen Misbah <afreen@ibm.com>

crimson/os/seastore: minor cleanup

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore: drop unused file

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #60892 from zdover23/wip-doc-2024-11-30-glossary-dashboard-plugin

doc/glossary.rst: add "Dashboard Plugin"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>