rgw/cloud: Add custom headers for objects transitioned to cloud

Some of the cloud services, (like MCG Noobaa/Azure Namespace store),
may not be able to map and store the objects which contain s3 style
metadata keys.

To help such services determine if the objects being transitioned
are from RGW and whether or not ignore such attrs, added below two
headers for the objects being copied/transitioned from RGW -

1) x-rgw-cloud : true/false
(set to "true" if the object is being transitioned/synced from RGW)

2) x-rgw-cloud-keep-attrs : true/false
- if set to default "true" , the cloud service should store all the
x-amz-meta-* attrs. If cannot be mapped/stored, the operation
should fail
- if set to "false", the destination cloud can ignore such attrs and
just store the object data being sent.

Also fixed a bug in the cloudtier module wherein the user-defined attrs
were not being copied to the cloud endpoint as part of transition

Fixes: https://tracker.ceph.com/issues/57980
Fixes: https://tracker.ceph.com/issues/58796
Signed-off-by: Soumya Koduri <skoduri@redhat.com>

Merge pull request #50077 from zdover23/wip-doc-2023-02-12-multisite-failover

doc/rgw: refine "Failover and Disaster Recovery"

doc/rgw: refine "Failover and Disaster Recovery"

English grammar and syntax pass for "Failover and Disaster Recovery" in
doc/radosgw/multisite.rst.

https://tracker.ceph.com/issues/58632

Co-authored-by: Cole Mitchell <cole.mitchell@gmail.com>
Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50066 from zdover23/wip-doc-2023-02-10-multisite-configuring-secondary-zones

doc/rgw: refine "Configuring Secondary Zones"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49763 from Svelar/vstart4

vstart: check mgr status after starting mgr

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #50063 from zdover23/wip-doc-2023-02-10-cephfs-quota-extended-cephfs-attributes-and-getfattr

doc/cephfs: add note about CephFS extended attributes and getfattr

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: refine "Configuring Secondary Zones"

English grammar and syntax pass (and some pragmatics, too, where I had
enough understanding to be helpful that way) for "Configuring Secondary
Zones" in doc/radosgw/multisite.rst.

https://tracker.ceph.com/issues/58632

Co-authored-by: Cole Mitchell <cole.mitchell@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

vstart: check mgr status after starting mgr.

Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

doc/cephfs: add note about CephFS extended attributes and getfattr

Supersedes https://github.com/ceph/ceph/pull/50060

Add a note on the fact that CephFS extended attributes are not printed
by "getfattr /some/cephfs/dir/ -d -m -" even when the CephFS directory
has some extended attributes set on it.

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/6ENI42ZMHTTP2OONBRD7FDP7LQBC4P2E/

Primary Author: Rishabh Dave <ridave@redhat.com>
Co-author: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #49132 from aclamk/wip-aclamk-bs-excl-lock

blk/kernel: Add O_EXCL for block devices

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Mark Nelson <mnelson@redhat.com>

Merge pull request #50023 from rosinL/fix-vstart

vstart: correct condition for checking crimson

Reviewed-by: Matan Breizman <mbreizma@redhat.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #49744 from xxhdx1985126/wip-fix-seastar-mem-leak

seastar: fix mem leak caused by no maximum memory limit for cpu core

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #48703 from sseshasa/wip-fix-mclk-builtin-profile-mods

osd: Restore defaults of mClock built-in profiles upon modification

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #48591 from rzarzynski/wip-osd-snaptrimmer-waitscrub-fix-name

osd: fix name of WaitScrub state of SnapTrimmer

Merge pull request #47702 from ifed01/wip-ifed-more-pin-fix

os/bluestore: fix onode ref counting.

Merge pull request #49888 from phlogistonjohn/jjm-more-utils-coverage

cephadm: more test coverage for various utils functions

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #50030 from zdover23/wip-doc-2023-02-09-multisite-to-config-2ndary-zones

doc/rgw: refine multisite to "config 2ndary zones"

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: refine multisite to "config 2ndary zones"

Refine the English in doc/radosgw/multisite.rst up to (but not
including) "Configure Secondary Zones".

https://tracker.ceph.com/issues/58632

Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #50027 from joscollin/wip-drop-aitalic

cephfs-top: drop curses.A_ITALIC

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #50021 from rhcs-dashboard/rename-endpoints

mgr/rgw: rename endpoints to zone_endpoints to match rgw spec

Reviewed-by: rkachach <NOT@FOUND>

Merge pull request #48783 from rhcs-dashboard/fix-grafana-stat-panel

mgr/dashboard: Replace vonage-status-panel with native grafana stat panel

Reviewed-by: Nizamudeen A <nia@redhat.com>

cephfs-top: drop curses.A_ITALIC

Fixes: https://tracker.ceph.com/issues/58663
Signed-off-by: Jos Collin <jcollin@redhat.com>

Merge pull request #49627 from rhcs-dashboard/box-podman-osd

cephadm/box: create osds with podman.

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

vstart: correct condition for checking crimson

Signed-off-by: luo rixin <luorixin@huawei.com>

Merge pull request #50022 from zdover23/wip-doc-2023-02-08-multisite-maintenance

doc/radosgw: refine "Maintenance" in multisite.rst

Merge pull request #50011 from zmc/deps-broken-repos

install-deps.sh: Fail when downloading apt repos

doc/radosgw: refine "Maintenance" in multisite.rst

Refine the text in the "Maintenance" section of
doc/radosgw/multisite.rst.

https://tracker.ceph.com/issues/58632

Co-authored-by: Cole Mitchell <cole.mitchell@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

mgr/rgw: rename endpoints to zone_endpoints to match rgw spec

As per the docs (and the codes), the rgw_spec.endpoint should actually
be rgw_spec.zone_endpoint.

```
rgw_realm: myrealm
rgw_zonegroup: myzonegroup
rgw_zone: myzone
zone_endpoints: http://<rgw_host1>:<rgw_port1>, http://<rgw_host2>:<rgw_port2>
```
Refer: https://docs.ceph.com/en/latest/mgr/rgw/#rgw-realm-operations

Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #49723 from anthonyeleven/anthonyeleven/58123

ceph: Clarify usage message for admin-socket commands

install-deps.sh: Fail when downloading apt repos

More specifically, if we get e.g. a 504 while attempting to download a
.list file, fail the build and log the status code.

Signed-off-by: Zack Cerza <zack@redhat.com>

Merge pull request #49994 from zdover23/wip-doc-2023-02-05-rgw-multisite-refine-2

doc/rgw: refine ~50-~140 of multisite.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: refine ~50-~140 of multisite.rst

Remove pleonasm from approximately line 50 to line 140 of
doc/rgw/multisite.rst.

Follows https://github.com/ceph/ceph/pull/49993.

https://tracker.ceph.com/issues/58632

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49993 from zdover23/wip-doc-2023-02-04-rgw-multisite-top-matter-refine

doc/rgw: refine 1-50 of multisite.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

doc/rgw: refine 1-50 of multisite.rst

Refine the English in the first fifty lines of multisite.rst. This is
part of a project whose goal is the creation of a document that will
impart a good aerial view of RGW to a first-time reader in the minimum
number of words.

Co-authored-by: Anthony D'Atri <anthony.datri@gmail.com>
Signed-off-by: Zac Dover <zac.dover@proton.me>

Merge pull request #49981 from zdover23/wip-doc-2023-02-03-rgw-multisite-zones-refinement

doc/rgw: refine "Zones" in multisite.rst

doc/rgw: refine "Zones" in multisite.rst

Clean up the English under the section called "Zones". Part of a larger
project aimed at giving the reader more understanding earlier in the
documentation of how buckets and zones and zone groups and endpoints
work.

https://tracker.ceph.com/issues/58632

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49968 from zdover23/wip-doc-2023-02-02-jaegertracing-refinement

doc/jaegertracing: refine index.rst

Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>

Merge pull request #49848 from ljflores/wip-tracker-58545

qa/workunits: give telemetry opt-in check more time

doc/jaegertracing: refine index.rst

Make English in jaegertracing/index.rst more elegant.

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49935 from athanatos/sjust/wip-58486

crimson: fix incorrect interval check in PG::submit_transaction

Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/net/ProtocalV2: no need to define fmt::ptr() since seastar has done it

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

seastar: bump up seastar submodule to fix mem leaks

fix mem leak caused by no maximum memory limit for cpu core

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #49812 from petrutlucian94/poll_hup

msg: fix poll event driver, handling POLLHUP and POLLERR

ceph: Clarify usage message for admin-socket commands

Signed-off-by: Anthony D'Atri <anthonyeleven@users.noreply.github.com>

Merge pull request #48295 from NitzanMordhai/wip-nitzan-osd-start-osdmap-hang-adding-debug-msg

OSD: during test start, not all osds started due to consum map hang

Merge pull request #45247 from andriytk/update-ceph-mgw

rgw: update motr sal

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge PR #49842 into main

* refs/pull/49842/head:
Revert "cmake: do not link ceph-dencoder against global"

Reviewed-by: Tim Serong <tserong@suse.com>

Merge pull request #49946 from cityofships/pkg-docs

doc/cephadm: grammar / syntax in install.rst

Signed-off-by: Piotr Parczewski <piotr@stackhpc.com>

crimson/osd/pg: remove incorrect map epoch check in PG::submit_transasction

It's entirely fine for the map_epoch to change while the op is processed
as long as none of the intervening epochs caused an interval change.  In
general, the correct way to check for an interval change is with
has_reset_since.

We don't need to do this check here at all because IOInterruptCondition
will already have performed it against the captured epoch when the
continuation resumed.

Introduced: 31418020fd58
Fixes: https://tracker.ceph.com/issues/58486
Signed-off-by: Samuel Just <sjust@redhat.com>

Revert "cmake: do not link ceph-dencoder against global"

Reverting since this is causing the following ceph-dencoder exception:

        [src/tcmalloc.cc:332] Attempt to free invalid pointer 0x558f14e3a000

NOTE: I didn't debug this any more further since there is sizable backlog
of PRs that need testing/merging, but reverting the change makes it work.

Fixes: http://tracker.ceph.com/issues/58219
Signed-off-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #49913 from neesingh-rh/wip-51459

doc: add the damage types that scrub can repair

Reviewed-by: Zac Dover <zac.dover@proton.me>

Merge PR #49048 into main

* refs/pull/49048/head:
mds: fix and skip submitting invalid osd request

Reviewed-by: Dhairya Parmar <dparmar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Rishabh Dave <ridave@redhat.com>

doc: add the damage types that scrub can repair

Fixes: https://tracker.ceph.com/issues/51459
Signed-off-by: Neeraj Pratap Singh <neesingh@redhat.com>

Rollback submodules changes made by accident

Signed-off-by: Andriy Tkachuk <andriy.tkachuk@seagate.com>

Merge pull request #48541 from lxbsz/wip-qa-upgrade-fix

qa/cephfs: fix confusing config when upgrading mds nodes

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Dhairya Parmar <dparmar@redhat.com>

Merge pull request #48049 from dparmar18/wip-dparmar-qa-mixed-clients-57280

qa/fs/mixed-clients: specify distros for tests

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #48251 from dparmar18/wip-dparmar-MDS-57359

mds/Server: do not allow -ve reclaim flags to cause client eviction

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>

Merge pull request #47753 from mchangir/mon-disable-snap-id-allocation-for-fsmap-pools

mon: disable snap id allocation for fsmap pools

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Greg Farnum <gfarnum@redhat.com>
Reviewed-by: Kotresh HR <khiremat@redhat.com>
Reviewed-by: Anthony D'Atri <anthony.datri@gmail.com>
Reviewed-by: Ramana Raja <rraja@redhat.com>

Merge pull request #47114 from lxbsz/wip-56532

ceph_fuse: retry the test_dentry_handling if fails

Reviewed-by: Rishabh Dave <ridave@redhat.com>
Reviewed-by: Milind Changire <mchangir@redhat.com>
Reviewed-by: Venky Shankar <vshankar@redhat.com>

client: switch to old method to invalidate dcache if euid != 0

Force to use the old and slow method to invalidate the dcache
if the euid is non-root, or the remount may fail with return
code 1 or 32.

https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

ceph_fuse: make it to force invalidating dentries when kernel >=3.18

The remount will fail randomly for unknown reasons. And in certain
circumstance we can reprodce this very easy, which will block our
testing. Make it posible to force to old method to invalidate the
dcache when the "client_try_dentry_invalidate" option is enabled
even kernel version >= 3.18.0

Fixes: https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

ceph_fuse: retry the test_dentry_handling if fails

For some unknown reason the remount will fail, we need to retry it
for the test_dentry_handling.

Fixes: https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

client: move a client's option to mds-client.yaml

mds_max_retries_on_remount_failure option is used by Client.cc only.

Fixes: https://tracker.ceph.com/issues/56532
Signed-off-by: Xiubo Li <xiubli@redhat.com>

crimson/osd/osd_operations/client_request: clarify get_obc stage message

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #49906 from evershalik/main

Update README.md

Merge pull request #49905 from zdover23/wip-doc-2023-01-29-rados-config-osd-config-ref-osd-op-thread-suicide-timeout

doc/rados: document osd_op_thread_suicide_timeout

Merge pull request #49908 from Matan-B/wip-matanb-upgrade-doc

doc/dev/developer_guide: Add Upgrade Testing Documentation

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/dev/developer_guide/testing_integration_tests: Add Upgrade Testing Documentation

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

doc/rados: document osd_op_thread_suicide_timeout

Add default value for osd_op_thread_suicide_timeout configuration value
to the OSD Configuraton Reference.

https://tracker.ceph.com/issues/58590

Signed-off-by: Zac Dover <zac.dover@gmail.com>

Update README.md

Signed-off-by: SHANKAR <98207888+evershalik@users.noreply.github.com>

Merge branch 'master' into update-ceph-mgw

crimson/osd/object_context_loader: add logging for notify_on_change

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/object_context_loader: convert to log macros

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/object_context_loader: take backend by reference

It's required and we don't check for null.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_backend: remove stopping and peering

These two state variables duplicate checks that *should* already
be handled by the IOInterruptCondition.  None of the stopping
checks should ever trigger because the caller would be in an
interruptible future context which already performed that check.

Moreover, peering doesn't really work -- it relies on the callback
firing prior the call to on_activate_complete(), and there isn't any
guarantee that will happen.  Storing the epoch from when the
callback was created as we do in IOInterruptCondition
would be required.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/pg_interval_interrupt_condition: use new logging macros

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/osd/replicated_backend: use new logging macros

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/common: move log macros to log.h from os/seastore/logging.h

These have been handy in imposing uniformity on seastore logging,
let's start using them in PG/OSD code.

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #49891 from bluikko/bluikko-doc-host-management-prompts

doc/cephadm: Fix prompts in host-management.rst

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #49897 from bluikko/bluikko-doc-rados-operations-prompt

doc/rados/operations: Fix double prompt

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/cephadm: Fix example commands

Separate command output block from the command to avoid printing prompt on it.
Also remove double prompt.

Fix indentation of two example commands after prompt.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

doc/rados/operations: Fix double prompt

In monitoring.rst a double prompt was rendered, one non-selectable and one selectable. Remove the selectable prompt.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #49894 from bluikko/bluikko-doc-rados-operations-control-indentation

doc/rados/operations: Fix indentation

Reviewed-by: Zac Dover <zac.dover@gmail.com>

doc/rados/operations: Fix indentation

Fix invalid indentation that caused indentation to be rendered wrong in control.rst.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #49890 from bluikko/bluikko-doc-ceph-krb-auth

doc/dev: use underscores in config vars

Reviewed-by: Zac Dover <zac.dover@proton.me>

doc/dev: use underscores in config vars

Use underscores instead of spaces in config vars in ceph_krb_auth.rst.

Signed-off-by: Ville Ojamo <14869000+bluikko@users.noreply.github.com>

Merge pull request #49109 from NitzanMordhai/wip-nitzan-fixing-few-rados/test.sh

Wip nitzan fixing few rados/test.sh

Reviewed-by: Kefu Chai <tchaikov@gmail.com>
Reviewed-by: Brad Hubbard <bhubbard@redhat.com>

cephadm/tests: add test coverage of check_unit function

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #49887 from cbodley/wip-qa-rgw-run-s3tests-tox

qa/workunits/rgw: run-s3tests.sh uses tox

Reviewed-by: Ali Maredia <amaredia@redhat.com>

cephadm/tests: add test coverage for find_program

Signed-off-by: John Mulligan <jmulligan@redhat.com>

cephadm/tests: add test coverage for find_executable

Also sets no coverage pragma on a windows only line. I'm assuming
cephadm isn't going to be run on windows any time soon.

Signed-off-by: John Mulligan <jmulligan@redhat.com>

Merge pull request #49799 from phlogistonjohn/jjm-cephadm-utils-tests

cephadm/tests: add test coverage for copy_tree, copy_files, move_files, recursive_chown

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #48597 from ronen-fr/wip-rf-osd-bitfields

osd: change two ObjectContext flags from bitfields to full variables

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

qa/workunits/rgw: run-s3tests.sh uses tox

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #49624 from ronen-fr/wip-rf-unique-formatter

common: extend Formatter API with a unique_ptr-returning static function

Merge pull request #49537 from rzarzynski/wip-bug-57546

Revert "osd/PeeringState: proc_lease_ack break once found from OSD"

Merge pull request #48854 from ifed01/wip-ifed-small-chunk-bluefs

os/bluestore: enable 4K allocation unit for BlueFS

Merge pull request #49314 from SUSE/wip-tserong-qa-crash-chown

qa/workunits/rados/test_crash: chown crash files to ceph user

Merge pull request #48799 from DanWritesCode/add-exception-handling

mon: add exception handling to ceph health mute

Reviewed-by: Laura Flores <lflores@redhat.com>