rgw/notifications: add const to APIs when possible

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/notifications: avoid unnecessary heap allocations

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

rgw/pubsub: remove subscription data structures

Signed-off-by: Yuval Lifshitz <ylifshit@redhat.com>

Merge pull request #49760 from kalebskeithley/ceph-rgw-zipper

rgw: refactor selected files for better above- vs below-the-line

Merge pull request #48828 from yuvalif/wip-yuval-cython-fails-clang16

rgw/file: change function signature to match the one generated by cython

reviewed-by: cbodley, mattbenjamin

Merge pull request #49530 from yuvalif/wip-yuval-signed-comparisson

rgw: fix signed to unsigned comparisson warning

reviwed by: mattbenjamin

Merge pull request #49642 from cbodley/wip-58286

rgw/beast: fix interaction between keepalive and 100-continue

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

rgw: refactor selected files for better above- vs below-the-line

Move more files into driver/rados for better above- vs below-
the-line. Use #pragma once everywhere (versus fixing all the

Signed-off-by: Kaleb S. KEITHLEY <kkeithle@redhat.com>

Merge pull request #49820 from zdover23/wip-doc-2023-01-21-rados-config-pool-pg-config-ref

doc/rados: refine pool-pg-config-ref.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

Merge pull request #49778 from zdover23/wip-doc-2023-01-18-rados-config-ceph-conf-runtime-changes

doc/rados: refine ceph-conf.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rados: refine ceph-conf.rst

Correct grammar and usage in ceph-conf.rst.

https://tracker.ceph.com/issues/58485

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49735 from linuxkidd/wip-rgw-gap-list

rgw/rgw-gap-list: refactoring and adding more error checking

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #49315 from tengfWang/master

rgw: fix the problem of duplicate idx when bi list

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #47743 from cbodley/wip-rgw-replication-status

rgw: add replication status header to s3 GetObj response

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #48801 from adk3798/extra-entrypoint-args

mgr/cephadm: support for extra entrypoint args

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #48827 from rkachach/fix_issue_58001

mgr/cephadm: fix prometheus haproxy/ingress deps update

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge pull request #49556 from senax/haproxy

mgr/cephadm: increase ingress timeout values

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #46209 from phlogistonjohn/jjm-format-nfs-mod

mgr/nfs: use object_format decorators to simplify response handling

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #49189 from pcuzner/improve-orch-host-ls

orchestrator: improvements to the orch host ls command

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #49823 from zdover23/wip-doc-2023-01-22-install-recommended-methods-rewrite

doc/install: refine index.rst

doc/install: refine index.rst

Refine English sentences in doc/install/index.rst. Remove adverbial
phrases of time that refer to Nautilus-era features as "new", since that
was four years ago.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/rados: refine pool-pg-config-ref.rst

Remove pleonasm.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49817 from ivancich/wip-flight-copyright-license-info

rgw: add copyright & licensing to flight integration source files

Reviewed-By: Adam Emerson <aemerson@redhat.com>

rgw: add copyright & licensing to flight integration source files

Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #49791 from idryomov/wip-doc-exclusive-lock-transitions

doc/rbd/rbd-exclusive-locks: warn about automatic lock transitions

Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #49784 from SUSE/wip-fix-58501

ceph.spec.in: Replace %usrmerged macro with regular version check

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #49764 from zdover23/wip-doc-2023-01-17-ceph-volume-lvm-encryption-refine

doc/ceph-volume: refine encryption.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rbd/rbd-exclusive-locks: warn about automatic lock transitions

A lot of people aren't aware of automatic lock transitions and
wrongfully assume that exclusive lock means that the image remains
locked for as long as the client is running.  Redo the explanation
and add a warning.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

doc/rbd/rbd-exclusive-locks: don't mention "profile rbd" requirement twice

It's (much better) described in the Blocklisting section.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

rgw: fix the problem of duplicate idx when bi list

fix: https://tracker.ceph.com/issues/58213

Signed-off-by: wangtengfei <wangtengfei@inspur.com>

Merge pull request #49798 from adamemerson/wip-radosgw-cr-test-build-fix

build: Fix build of `radosgw-cr-test`

Reviewed-by: Casey Bodley <cbodley@redhat.com>

build: Fix build of `radosgw-cr-test`

Remove dependency on `libneorados_cls_fifo` removed in commit
`bf222906b01cbb9e8d55ae4685004a22cddcc138`.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/rgw-gap-list: refactor / add more err checks

- rename function `log_out` to `log` to prevent mental confusion with
  'logging out' of the shell, vs sending a log message.

- Centralize error checking to provide more consistent output and ease
  of error check implementation

- Add check that supplied pool name(s) exist.

- Relocate the awk script creation to just before it's needed.

- Provide usage help for specifying multiple pools on the command line

- Add hostname to log output lines.

- add checks to ensure intermediate files are not empty before
  attempting comparison.

- switch to getopt to allow  more option flexibility, removing
  dependency on positional arguments which could cause parsing / user
  confusion.
  This is a **BREAKING change** due to change in command line syntax.

- add option for temp directory specification on the command line.

- set -m option to be presence based, vs needing 0|1 param

- fix typo in usage

Signed-off-by: Michael J. Kidd <linuxkidd@gmail.com>

Merge pull request #48898 from ceph/wip-rgw-multisite-reshard-reef

rgw: multisite stabilization for reef

Reviewed-By: Adam Emerson <aemerson@redhat.com>

Merge pull request #48848 from stefan-chivu/rbd-wnbd-live-resize

rbd: live resize support for rbd-wnbd

Reviewed-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/ceph-volume: refine encryption.rst

Improve the word choice and grammar of
doc/ceph-volume/lvm/encryption.rst.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49280 from stefan-chivu/dokan-file-dir-access

dokan: Made file/dir access mode configurable

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49759 from rhcs-dashboard/fix-pg-metrics

mgr/prometheus: export zero valued pg state metrics

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

ceph.spec.in: Replace %usrmerged macro with regular version check

%usrmerged is deprecated in favour of %if 0%{?suse_version} < 1550
(see https://en.opensuse.org/openSUSE:Usr_merge for details)

Fixes: https://tracker.ceph.com/issues/58501
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #49780 from zdover23/wip-doc-2023-01-19-cephadm-install-link-fix

doc/install: link to "cephadm installing ceph"

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/install: link to "cephadm installing ceph"

Link to "Installing Ceph" in the cephadm documentation instead of (as
was the case before this commit) to the cephadm overview page. Anyone
who clicks on the "cephadm" link in the context of the
doc/install/index.rst page is more likely to expect installation
instructions than to expect an explanation of what cephadm is.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49734 from linuxkidd/wip-rgw-gap-orphan-tool-error-checks

rgw/rgw-orphan-list: refactor and add more checks to the tool

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

rgw/rgw-orphan-list: refactor / add error checking

standardize log output
add check that supplied pool[s] exist on the cluster
add return code checks for all commands that can affect output
add check for empty intermediate files
add usage output, including a -h command line option
add command line option to specify temporary directory to use
  ( instead of /tmp )
require command line supplied pools to be encased in double quotes "
  when multiple pools are provided
add warning for indexless buckets
move major segments to separate functions

Signed-off-by: Michael J. Kidd <linuxkidd@gmail.com>

Merge pull request #48632 from adamemerson/wip-57562

rgw: Fix race condition on FIFO client on new head creation

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Krunal Chheda <kchheda3@bloomberg.net>
Reviewed-by: Shilpa Jagannath <smanjara@redhat.com>

qa: add windows run-tests.ps1

We have a few Python rbd-wnbd tests that are invoked explicitly
by the ceph-build scripts [1].

There are a few issues with that:

* it's a separate repo that has to be updated whenever we add new
  tests
* new tests that reside in the ceph repo will not be executed by
  the PR check
* some tests may be missing in case of older branches

For this reason, we're adding a new script as part of the Ceph
repo that will take care of invoking the Windows rbd-wnbd tests.

The ceph-build script has already been updated accordingly [2].

[1] https://github.com/ceph/ceph-build/blob/main/scripts/ceph-windows/run_tests#L73-L80
[2] https://github.com/ceph/ceph-build/pull/2094

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>
Co-Authored-By: Ionut Balutoiu <ibalutoiu@cloudbasesolutions.com>

Merge pull request #47794 from rhcs-dashboard/auth-add

mgr/dashboard: create authx users

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #49765 from rhcs-dashboard/fix-bucket-encryption-checkbox

mgr/dashboard: fix bucket encryption checkbox

Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #48729 from ivancich/wip-flight-select-add-flight

rgw: initial commit adding Arrow Flight functionality

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #48509 from jliebl-git/jliebl-ldap-fix

rgw: LDAP fix resource leak with wrong credentials

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #49636 from liangmingyuanneo/wip-read-olh-log

rgw: remove guard_reshard in bucket_index_read_olh_log

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49463 from ivancich/wip-log-follow-bi-list

rgw: add logging to radosgw-admin bi list

Reviewed-by: Ali Maredia <amaredia@redhat.com>

Merge pull request #49648 from ivancich/wip-fix-fp-error

rgw: fix FP error when calculating enteries per bi shard

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49181 from ljflores/wip-envlibrados-rocksdb-fix

qa/workunits/rados: skip running envlibrados rocksdb tests on ubuntu

qa/workunits/rados: skip running envlibrados rocksdb tests on ubuntu

This test passes on centos and rhel, but fails on ubuntu from an
invalid pointer. Since the envlibrados rocksdb tests are experimental
and don't have any actual users, we can just run them on rhel and
centos.

At the moment, the actual bug is not fully understood, but it was
decided that fixing it is low priority, and removing the test from
problematic distros is okay for the time being. This commit
is considered a workaround to the actual issue.

Related tracker: https://tracker.ceph.com/issues/57632
Signed-off-by: Laura Flores <lflores@redhat.com>

mgr/dashboard: fix bucket encryption checkbox

Fixes: https://tracker.ceph.com/issues/58474
The encryption checkbox in the bucket creation form remains disabled after setting the vault authentication method as agent.

Signed-off-by: Aashish Sharma <aasharma@redhat.com>

rgw: fetch_remote_obj() adds x-amz-replication-status=REPLICA

implements the easy part of amazon's x-amz-replication-status feature:

https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-status.html#replication-status-overview

Signed-off-by: Casey Bodley <cbodley@redhat.com>

mgr/dashboard: create authx users

Signed-off-by: Pere Diaz Bou <pdiazbou@redhat.com>
Signed-off-by: Nizamudeen A <nia@redhat.com>
Co-authored-by: Nizamudeen A <nia@redhat.com>

Merge pull request #49533 from mkogan1/wip-fips-openssl3

rgw: under fips & openssl 3.x allow md5 iusage in select rgw ops

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #49710 from cyx1231st/wip-seastore-cleanup-overwrite

crimson/os/seastore/object_data_handler: cleanup the overwrite path

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #49743 from xxhdx1985126/wip-omap-fixes

crimson/os/seastore/omap_manager: minor bugs fixes

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

mgr/prometheus: export zero valued pg state metrics

Fixes: https://tracker.ceph.com/issues/58471
Signed-off-by: Avan Thakkar <athakkar@redhat.com>
As per the Prometheus documentation, omitting zero metrics is not a best practice. The metric value for all PG_STATES should be initialized to zero.

mgr/cephadm: fix prometheus haproxy/ingress deps update
Fixes: https://tracker.ceph.com/issues/58001
Signed-off-by: Redouane Kachach <rkachach@redhat.com>

qa: add test_rbd_wnbd resize test

We're adding a test for the newly introduced live resize feature.
It will simply extend/shrink the image, wait for the new size to
be picked up and then run FIO tests to validate the resized image.

While at it, we're fixing two unrelated linter warnings:

  E275 missing whitespace after keyword

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

test/crimson/seastore/test_omap_manager: keep test_omap_mappings consistent with "rm_key_range" config

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/omap_manager: fix a capture-by-reference related issue

Variables that are not guaranteed to be available as long as the continuations
live shouldn't be captured by reference

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #49725 from anthonyeleven/56521

doc/releases: Clarify Quincy CephFS MDS upgrade process

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49747 from zdover23/wip-doc-2023-01-16-luks-versions

doc/ceph-volume: update LUKS docs

crimson/os/seastore: add FIXME to consider pending extent with absolute paddr

During object data overwrite.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/os/seastore/object_data_handler: cleanup the overwrite path

* implement factory and type for extent_to_write_t.
* refactor overwrite_plan_t with explicit size calculations and
  defined operations.
* cleanup the overwrite path accordingly.

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #49716 from zdover23/wip-doc-2023-01-12-rbd-iscsi-initiator-linux-prompts

doc/rbd: refine iscsi-initiator-linux.rbd

Reviewed-by: Ilya Dryomov <idryomov@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/ceph-volume: update LUKS docs

Remove references that claim that Ceph uses only LUKS version 1.

https://tracker.ceph.com/issues/58354

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49745 from zdover23/wip-doc-2023-01-15-start-documenting-ceph-escaping-bold-characters-within-words

doc/start: add RST escape character rules for bold

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/rbd: format iscsi-initiator-linux.rbd better

Add prompts and clean up the lists in doc/rbd/iscsi-initiator-linux.rbd.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

doc/start: add RST escape character rules for bold

Explain how to escape the bold notation (**) within words in RST.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49736 from zdover23/wip-doc-2023-01-14-dev-basic-workflow-git-commands

doc/dev: add git branch management commands

Reviewed-by: Cole Mitchell <cole.mitchell@gmail.com>

doc/dev: add git branch management commands

Add git branch deleting and search commands to the "Basic Workflow" page
of the Developer Guide.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #48496 from adk3798/apply-spec-parsing

cephadm: only pull host info from applied spec, don't try to parse yaml

Reviewed-by: John Mulligan <jmulligan@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49024 from rkachach/fix_issue_58067

mgr/cephadm: fixing mgr port shown in ceph orch ps

Reviewed-by: Adam King <adking@redhat.com>

Merge PR #49731 into main

* refs/pull/49731/head:
tools/cephfs: fix traceback from missing envvar

Reviewed-by: Milind Changire <mchangir@redhat.com>

Merge pull request #48617 from adk3798/upgrade-status-help

mgr/orchestrator: fix upgrade status help message

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #48384 from jmolmo/Rook_PDFetcher

mgr/rook: Device inventory

Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Travis Nielsen <tnielsen@redhat.com>

Merge pull request #48592 from adk3798/offline-upgrade

mgr/cephadm: improve offline host handling, mostly around upgrade

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #48574 from adk3798/debug-log-spam

mgr/cephadm: make logging refresh metadata to debug logs configurable

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #47383 from rhcs-dashboard/ceph-exporter-cephadm-changes

exporter: cephadm changes

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49389 from adk3798/git-archive-fix

qa/tasks/cephadm.py: fix pulling cephadm from git.ceph.com

Reviewed-by: Redouane Kachach <rkachach@redhat.com>

Merge pull request #49726 from rhcs-dashboard/cephadm-e2e-fixes

mgr/dashboard: cephadm dashboard e2e fixes

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

rgw: Resolve sync stuck in previous gen

In case if any data log entries are missing for the older
generations, the sync server may not mark those shards as done
and can get stuck in that old gen for ever.

To avoid that, whenever a future gen entry is read, write the undone (shard,gen)
entry to error repo so that it can be processed and marked as done
and hence sync can progress eventually.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #49635 from rkachach/fix_issue_58378

mgr/mgr_util: fixing some tox issues

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: John Mulligan <jmulligan@redhat.com>

rgw/multisite: return error from RGWLastCallerWinsCR() to track marker update failures

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: Check for lost lease more often

Flush the marker tracker and abort if we don't still have it.

Resolves: rhbz#2129718
Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw/multisite: don't disable versioning on archive zone upon source bucket modifications

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: object should generate version id on archive zone, even when versioning
is disabled on primary

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: take the bucket lease before checking if bucket sync is enabled/disabled

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw/multisite: Add testcases for sync policy feature

Added test cases for the various use-cases of multisite sync
policy feature listed in
https://docs.ceph.com/en/latest/radosgw/multisite-sync-policy/

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw: fix get sync policy handler if bucket_id is empty

Fixes: https://tracker.ceph.com/issues/48416
bucket was passed in without bucket_id, now reading entrypoint
info if needed.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>

rgw: LatencyConcurrencyControl warns on very high latency

Limited to only warn every five minutes.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

mgr/dashboard: cephadm dashboard e2e fixes

Fixes: https://tracker.ceph.com/issues/58406
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #49417 from athanatos/sjust/backburner/wip-smp-option

crimson/osd/main: get full config before starting reactor

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

rgw: Reduce data sync parallelism in response to RADOS lock latency

Lock latency in RGWContinuousLeaseCR gets high enough under load that
the locks end up timing out, leading to incorrect behavior.

Monitor lock latency and cut concurrent operations in half if it goes
above ten seconds.

Cut currency to one if it goes about twenty seconds.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>

rgw: `SimpleRadosWriteAttrsCR` uses an async RADOS call

Don't go through the 'system object' cache. This also saves us the use
of the RADOS async completion processor.

Signed-off-by: Adam C. Emerson <aemerson@redhat.com>