mgr/cephadm/tests: remove `_deploy_cephadm_binary`

(not needed)

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/tox.ini: Add python 3.7 environment

Plus fixes.

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43905 from rhcs-dashboard/fix-53242-master

mgr/dashboard: dashboard does not show degraded objects if they are less than 0.5% under "Dashboard->Capacity->Objects block

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #44023 from rhcs-dashboard/kcli-expanded-monitoring

mgr/dashboard: cephadm e2e start script: --expanded: deploy monitoring stack

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44064 from MrFreezeex/fix-statusupdater-utest

rbd-mirror: make RemoveImmediateUpdate test synchronous

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #44045 from rhcs-dashboard/upgrade-cypress

mgr/dashboard: upgrade Cypress to the latest stable version

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #44019 from liu-chunmei/crimson-background-recovery

crimson/osd: add delay for background_recovery

reviewed-by:  Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43989 from cfsnyder/wip-53308

osd/OSDMap.cc: clean up pg_temp for nonexistent pgs

Reviewed-by: xie xingguo <xie.xingguo@zte.com.cn>

Merge PR #44036 into master

* refs/pull/44036/head:
.github/pull_request_template: drop teuthology reference
.github/pull_request_template: add cleanup option

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Patrick Donnelly <pdonnell@redhat.com>
Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #44061 from Matan-B/wip-matanb-doc-teuthology

doc/dev: adding Teuthology suggested resources

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43996 from rhcs-dashboard/predefined-labels

mgr/dashboard: Predefine labels in create host form

Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

rbd-mirror: make RemoveImmediateUpdate test synchronous

Try fixing sporadic failure linked in the tracker in
TestMockMirrorStatusUpdater.RemoveImmediateUpdate by making it
synchronous.

Fixes: https://tracker.ceph.com/issues/53375
Signed-off-by: Arthur Outhenin-Chalandre <arthur.outhenin-chalandre@cern.ch>

mgr/dashboard: upgrade Cypress to the latest stable version

- Remove unneeded dependency that was causing UI performance issues: zone.js
- Ignore 'ResizeObserver loop limit exceeded' error.
- run-frontend-e2e-tests.sh refactoring: create rgw dashboard user through
  'ceph dashboard set-rgw-credentials' and use it on rgw buckets' tests.

Fixes: https://tracker.ceph.com/issues/53357
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #43244 from ronen-fr/wip-rf-scrub-command

osd: make 'pg deep-scrub' command initiate a scrub

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43722 from lxbsz/caps_doc

doc: update the capabilities doc for cephfs

Reviewed-by: Venky Shankar <vshankar@redhat.com>

doc/dev: adding Teuthology suggested resources

Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Merge pull request #43982 from guits/refactor_cv_human_readable_func

ceph-volume: human_readable_size() refactor

Merge pull request #44007 from tchaikov/wip-cmake-python3.10

cmake: check for python(\d)\.(\d+) when building boost

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43992 from rhcs-dashboard/flaky-inventory-test-fix

mgr/dashboard: fix flaky inventory e2e test

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43958 from rhcs-dashboard/daemon-event-padding

mgr/dashboard: Daemon Events listing using bootstrap class

Reviewed-by: Aashish Sharma <aasharma@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43866 from rhcs-dashboard/add-hint-provisioned-images

mgr/dashboard: provisioned values is misleading in RBD image table

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43843 from cbodley/wip-test-cls-rgw-stats

test/cls/rgw: add index transaction simulator to model bucket stats

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #43888 from mgfritch/cephadm-expect-hostname

cephadm: fixup expect-hostname message

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43873 from guits/add_shared_folder_shell_cmd

cephadm: add --shared_ceph_folder to shell cmd

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43876 from sebastian-philipp/all-osd-at-once

mgr/cephadm: create osds at all hosts at once

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

mgr/dashboard: dashboard does not show degraded objects if they are less than 0.5% under "Dashboard->Capacity->Objects block

This PR is intended to fix this issue

Fixes: https://tracker.ceph.com/issues/53242
Signed-off-by: Aashish Sharma <aasharma@redhat.com>

Merge pull request #44033 from ljflores/wip-update-email-id

mailmap: add Laura Flores

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: ljflores <NOT@FOUND>
Reviewed-by: neha-ojha <NOT@FOUND>

Merge pull request #43795 from myoungwon/wip-paddr-split

seastore: generalize paddr_t

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/dashboard: Predfine labels in create host form

Also retains the previously created labels by user in the form

Fixes: https://tracker.ceph.com/issues/53315
Signed-off-by: Nizamudeen A <nia@redhat.com>

mgr/dashboard: fix flaky inventory e2e test

When `inventory.getTableCount('total').should('be.eq', totalDiskCount);`
this line is executed the table was not loaded properly and hence the
getTableCount returns 0 on the first try but on second try it passes
since the table is loaded. But in orch e2es the retries are set to 0. I
am not sure if it makes sense to set it to 1. Anyway I am adapting the
test a bit to expect the count to be equal to totalDiskCount so that the
test will wait a bit.

Fixes: https://tracker.ceph.com/issues/53353
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43524 from Rethan/feat-expiration-time

rbd: when trash mv, show expiration time if it's not now

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

Merge pull request #43852 from hualongfeng/show_feature

tools/rbd: make rbd info display dirty-cache feature

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>

Merge pull request #43907 from cybozu/rbd-correct-encoding-of-snap-protection-record-in-exporting

rbd: correct encoding of snap protection record in exporting image

Reviewed-by: Mykola Golub <mykola.golub@clyso.com>

.github/pull_request_template: drop teuthology reference

It is not clear what role this has relative to the needs-qa label.

Signed-off-by: Sage Weil <sage@newdream.net>

.github/pull_request_template: add cleanup option

Signed-off-by: Sage Weil <sage@newdream.net>

seastore: generalize paddr_t

Due to necessity to support both segment and rbm addrs,
paddr_t needs to be generlized.
To do so, this commits introduces seg_paddr_t---
seg_paddr_t inherits paddr_t, and deals with common_addr
according to the semantic of segment.

Signed-off-by: Myoungwon Oh <myoungwon.oh@samsung.com>

Merge pull request #43999 from kamoltat/wip-autoscale-profile-scale-up-default

pybind/mgr/pg_autoscale: revert to default profile scale-up

Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #43974 into master

* refs/pull/43974/head:
qa: disable metrics on kernel client during upgrade

Reviewed-by: Xiubo Li <xiubli@redhat.com>
Reviewed-by: Jeff Layton <jlayton@redhat.com>

mailmap: add Laura Flores

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #43987 from rhcs-dashboard/53123-dashboard-nfs-cleanup

mgr/dashboard: NFS non-existent files cleanup

Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: ljflores <NOT@FOUND>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #43983 from rhcs-dashboard/rgw-add-realm-column

mgr/dashboard: rgw daemon list: add realm column

Reviewed-by: Waad Alkhoury <walkhour@redhat.com>
Reviewed-by: Alfonso Martínez <almartin@redhat.com>
Reviewed-by: Avan Thakkar <athakkar@redhat.com>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Nizamudeen A <nia@redhat.com>
Reviewed-by: sebastian-philipp <NOT@FOUND>

Merge pull request #43824 from cbodley/wip-qa-rgw-upgrade-octopus-multisite-cv

qa/upgrade: rgw multisite upgrade test excludes ceph-volume

Reviewed-by: J. Eric Ivancich <ivancich@redhat.com>

Merge pull request #43764 from gregsfortytwo/wip-rbd-crash-consistency

doc: fix up rbd snapshot docs around crash consistency

Reviewed-by: Mykola Golub <mykola.golub@clyso.com>
Reviewed-by: Sunny Kumar <sunkumar@redhat.com>

pybind/mgr/pg_autoscale: revert to default profile scale-up

pg_autoscale module will now start out all the pools
with a scale-up profile by default.

Added tests in workunits/mon/pg_autoscaler.sh
to evaluate if the default pool creation is
a scale-up profile

Updated documentation and release notes to
reflect the change in the default behavior
of the pg_autoscale profile.

Fixes: https://tracker.ceph.com/issues/53309
Signed-off-by: Kamoltat <ksirivad@redhat.com>

qa: disable metrics on kernel client during upgrade

v16.2.4 MDS triggers an assert from these messages.

Also: add latest pacific for extra coverage.

Fixes: https://tracker.ceph.com/issues/53293
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa/rgw: run ceph_test_cls_rgw_stats in rgw/verify suite

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: model multipart uploads in ceph_test_cls_rgw_stats

Signed-off-by: Casey Bodley <cbodley@redhat.com>

test/rgw: add ceph_test_cls_rgw_stats simulator

a simulator that generates lots of create/delete operations against a
set of object names, and tracks the expected bucket stats for comparison

each operation has a 70% chance to complete successfully, a 10% chance to
cancel, 10% chance to suggest update, and 10% chance to suggest removal

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: add some missing equality comparisons

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: RGWObjCategory to string

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: RGWModifyOp <-> string conversions

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: move stream operator from rgw_common.h

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: input parameters are const

Signed-off-by: Casey Bodley <cbodley@redhat.com>

cls/rgw: add rgw_bucket_dir_stats alias

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #41557 from ifed01/wip-ifed-better-daemonperf

os/bluestore: improve usability for bluestore/bluefs perf counters

Signed-off-by: Igor Fedotov <igor.fedotov@croit.io>
Reviewed-by: Laura Flores lflores@redhat.com

mgr/dashboard: cephadm e2e start script: deploy monitoring stack when --expanded

Fixes: https://tracker.ceph.com/issues/53333
Signed-off-by: Alfonso Martínez <almartin@redhat.com>

Merge pull request #43886 from nmshelke/doc-fix-53054

doc: prerequisites fix for cephFS mount

Reviewed-by: Venky Shankar <vshankar@redhat.com>

Merge pull request #44008 from rzarzynski/wip-crimson-leaky-objectcontextregistry

crimson/osd: fix leaks of ObjectContext in the registry.

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

Merge pull request #43326 from pdvian/wip-doc-config-correction

doc/dev/config: Replace invalid config debug-pg

Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/osd: add delay for background_recovery

to avoid interuptor nested.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #43980 from liu-chunmei/crimson-peerevent-nested

crimson: add delay for peering_event start when nested

reviewed-by : Samuel Just <sjust@redhat.com>,
                       Kefu Chai <tchaikov@gmail.com>,
                       Xuehan Xu <xxhdx1985126@gmail.com>,
                       Radoslaw Zarzynski <rzarzyns@redhat.com>.

crimson/osd: fix leaks of ObjectContext in the registry.

The patch is supposed to fix the following problems (extra
debugs onboard):

```
NFO  2021-11-16 01:18:38,713 [shard 0] osd - ~OSD: OSD dtor called
INFO  2021-11-16 01:18:38,713 [shard 0] osd - Heartbeat::Peer: osd.6 removed
INFO  2021-11-16 01:18:38,714 [shard 0] osd - Heartbeat::Peer: osd.5 removed
INFO  2021-11-16 01:18:38,714 [shard 0] osd - Heartbeat::Peer: osd.2 removed
INFO  2021-11-16 01:18:38,714 [shard 0] osd - ~ShardServices: ShardServices dtor called
INFO  2021-11-16 01:18:38,714 [shard 0] osd - ~ObjectContextRegistry: ShardServices dtor called; unref_size=3, size=3
INFO  2021-11-16 01:18:38,714 [shard 0] osd - ~ObjectContextRegistry: unreferenced p=0x619000115380
INFO  2021-11-16 01:18:38,714 [shard 0] osd - ~ObjectContextRegistry: unreferenced p=0x619000114980
INFO  2021-11-16 01:18:38,714 [shard 0] osd - ~ObjectContextRegistry: unreferenced p=0x619000112680
INFO  2021-11-16 01:18:38,714 [shard 0] osd - ~ObjectContextRegistry: set p=0x619000114980
INFO  2021-11-16 01:18:38,714 [shard 0] osd - ~ObjectContextRegistry: set p=0x619000115380
INFO  2021-11-16 01:18:38,714 [shard 0] osd - ~ObjectContextRegistry: set p=0x619000112680
INFO  2021-11-16 01:18:38,738 [shard 0] osd - crimson shutdown complete

=================================================================
==33351==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 2808 byte(s) in 3 object(s) allocated from:
    #0 0x7fe10c0327b0 in operator new(unsigned long) (/lib64/libasan.so.5+0xf17b0)
    #1 0x55accbe8ffc4 in ceph::common::intrusive_lru<ceph::common::intrusive_lru_config<hobject_t, crimson::osd::ObjectContext, crimson::osd::obc_to_hoid<crimson::osd::ObjectContext> > >::get_or_create(hobject_t const&) (/usr/bin/ceph-osd+0x3b000fc4)

Objects leaked above:
0x619000112680 (936 bytes)
0x619000114980 (936 bytes)
0x619000115380 (936 bytes)
```

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

cmake: check for python(\d)\.(\d+) when building boost

now that python3.10 has two digits as its minor version, we should be
able to match it.

this change fixes the build with python3.10. without this change, we'd
have

error: wrong library name 'python310' in the --with-<library> option.

when building with python3.10

Signed-off-by: Kefu Chai <tchaikov@gmail.com>

mgr/cephadm: use asyncio.gather to parallelize things

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: make OSDService.create_from_spec_one async

And gather all results. Plus a lot of mechanical adjustments

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

mgr/cephadm: Add CephadmOrchestrator.wait_async()

less typing

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43409 from linuxbox2/wip-rgwadmin-logtest

qa/rgw: use local runner with cmdline radosgw_admin.py

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #35100 from soumyakoduri/cloudtiering

rgw/CloudTransition: Transition objects to cloud endpoint

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>
Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #42780 from ronen-fr/wip-ronenf-unique-scrub

osd/scrub: mark PG as being scrubbed, from scrub initiation to Inactive state

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge PR #43880 into master

* refs/pull/43880/head:
mgr/cephadm: turn off asyncssh debug output

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43093 from zdover23/wip-doc-2021-09-09-rados-bootstrap-options-2

doc/rados: update mon_host & friends options

Reviewed-by: Neha Ojha <nojha@redhat.com>

mgr/dashboard: Daemon Events listing using bootstrap class

Fixes: https://tracker.ceph.com/issues/53282
Signed-off-by: Nizamudeen A <nia@redhat.com>

Merge pull request #43969 from sebastian-philipp/core

doc/cephadm: core dumps

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Michael Fritch <mfritch@suse.com>

rgw/CloudTransition: Replace Coroutines with RGWRestConn APIs

To avoid the overhead of using coroutines during lifecycle transition,
RGWRESTStream* APIs are used to transition objects to remote cloud.

Also handled few optimizations and cleanup stated below:
* Store the list of cloud target buckets as part of LCWorker instead
  of making it global. This list is maintained for the duration of
  RGWLC::process(), post which discarded.
* Refactor code to remove coroutine based class definitions which are no
  longer needed and use direct function calls instead.
* Check for cloud transitioned objects using tier-type and return error if
  accessed in RGWGetObj, RGWCopyObj and RGWPutObj ops.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/RGWRESTConn: Define a wrapper to send PUT/POST stream request

Similar to "get_resource()", add an API "send_resource()" to send
PUT/POST/DELETE Stream request on RGWRestConn

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Include aws region name for remote endpoint

With commit#81ad226, aws auth v4 rquires region name for remote
endpoint connection. Include the same in the tier parameters.

& misc fixes

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Handle versioned objects

For versioned and locked objects, similar semantics as that of LifecycleExpiration are applied as stated below -

If the bucket versioning is enabled and the object transitioned to cloud is
 - current version, irrespective of what the config option "retain_object" value is, the object is not deleted but instead delete marker is created on the source rgw server.
 - noncurrent version, it is deleted or retained based on the config option "retain_object" value.

If the object is locked, and is
 - current version, it is transitioned to cloud post which it is made noncurrent with delete marker created.
 - noncurrent version, transition is skipped.

Also misc rebase fixes and cleanup -

* Rename config option to "retain_head_object"

to reflect its functionality to keep head object post transitioning
to cloud if enabled

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Skip transition to cloud if the object is locked

If an object is locked, skip its transition to cloud.

@todo: Do we need special checks for bucket versioning too?
If current, instead of deleting the data, do we need to create
a delete marker? What about the case if retain_object is set to true.

& misc rebase fixes

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Change tier-type to cloud-s3

Currently the transition is supported to cloud providers
that are compatible with AWS/S3. Hence change the tier-type to
cloud-s3 to configure the S3 style endpoint details.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: handle versioned objects

If the object is versioned, to avoid objects getting overwritten
post transition to cloud, append object versionID to the target
object name

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Add documentation

Also to avoid object name collisions across various buckets
post cloud transition, add bucket name to the object prefix.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Do not allow data pool for tier type storage classes

Tier type storage classes should not be allowed to have data
pools

& few other fixes/cleanup stated below -

* If the tier_targets are not configured, do not dump them in
the 'zonegroup get' command.

* If not configured, by default a bucket of below name convention -
"rgwx-$zonegroup-$storage_class-cloud-bucket"

is created in the remote cloud endpoint to transition objects to.

* Rename config option 'tier_storage_class' to 'target_storage_class'.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Fail GET on cloud tiered objects

As per https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
GET operation may fail with “InvalidObjectStateError” error if the
object is in GLACIER or DEEP_ARCHIVE storage class and not restored.
Same can apply for cloud tiered objects. However STAT/HEAD requests
shall return the metadata stored.

& misc fixes

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Verify if the object is already tiered

Add class to fetch headers from remote endpoint and verify if the object
is already tiered.

& Few other fixes stated below -

* Erase data in the head of cloud transitioned object
* 'placement rm' command should erase tier_config details
* A new option added in the object manifest to denote if the
  object is tiered in multiparts

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Store the status of multipart uploads

Store the status of multipart upload parts to verify if the object
hasn't changed during the transition and if yes, abort the upload.

Also avoid re-creating target buckets -

Its not ideal to try creating target bucket for every object
transition to cloud. To avoid it caching the bucket creations in
a map with an expiry period set to '2*lc_debug_interval' for each
entry.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Delete cloud tiered objects by default

Added a new option "retain_object" in tier_config which determines
whether a cloud tiered object is deleted or if its head object is
retained. By default the value is false i.e, the objects get
deleted.

XXX: verify that if Object is locked (ATTR_RETENTION), transition is
not processed. Also check if the transition takes place separately for
each version.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Update object metadata and bi post cloud tranistion

After transitioning the object to cloud, following updates are done
to the existing object.

* In bi entry, change object category to CloudTiered
* Update cloud-tier details (like endpoint, keys etc) in Object Manifest
* Mark the tail objects expired to be deleted by gc

TODO:
* Update all the cloud config details including multiparts
* Check if any other object metadata needs to be changed
* Optimize to avoid using read_op again to read attrs.
* Check for mtime to resolve conflicts when multiple zones try to transition obj

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Tier objects to remote cloud

If the storage class configured is of cloud, transition
the objects to remote endpoint configured.

In case the object size is >mulitpart size limit (say 5M),
upload the object into multiparts.

As part of transition, map rgw attributes to http attrs,
including ACLs.

A new attribute (x-amz-meta-source: rgw) is added to denote
that the object is transitioned from RGW source.

Added two new options to tier-config to configure multipart size -
* multipart_sync_threshold - determines the limit of object size,
when exceeded transitioned in multiparts
* multipart_min_part_size - the minimum size of the multipart upload part

Default values for both the options is 32M and minimum value supported
is 5M.

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

rgw/CloudTransition: Add new options to configure tier endpoint

As mentioned in https://docs.google.com/document/d/1IoeITPCF64A5W-UA-9Y3Vp2oSfz3xVQHu31GTu3u3Ug/edit,
the tier storage class will be configured at zonegroup level.

So the existing CLI "radosgw-admin zonegroup placement add  <id> --storage-class <class>" will be
used to add tier storage classes as well but with extra tier-config options mentioned below -

--tier-type : "cloud"
--tier-config : [<key,value>,]

These tier options are already defined to configure cloud sync module which are being reused here.

TODO:
* Add multipart options (if any , like part size, threshold)
* Document
* Test upgrade/downgrade

Signed-off-by: Soumya Koduri <skoduri@redhat.com>

crimson: add delay for peering_event start when nested

Delay the second (nested) peerevent::start to let the first finish.
Then avoid interruptor nesting which will cause local interrupt_cond
not equal global interrupt_cond.

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge PR #43929 into master

* refs/pull/43929/head:
qa/suites/orch/cephadm: verify /var/log/ceph/$fsid ownership
cephadm: only make_log_dir for ceph daemons

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge PR #43934 into master

* refs/pull/43934/head:
qa/suites/rados/dashboard: use single-container-host.yaml
qa/distros: add single-container-host.yaml
qa/suites: use distros/container-hosts/
qa/distros/container-hosts: add 8.stream + crun
qa/distros/container-hosts: add collection of container targets

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

qa/upgrade: rgw multisite upgrade test excludes ceph-volume

E: Unable to locate package ceph-volume

Signed-off-by: Casey Bodley <cbodley@redhat.com>

osd/OSDMap.cc: clean up pg_temp for nonexistent pgs

Fixes an issue where the OSDMap does not clear pg-temp entries for PGs that no longer exist.

Fixes: https://tracker.ceph.com/issues/53308
Signed-off-by: Cory Snyder <csnyder@iland.com>

Merge pull request #43433 from soumyakoduri/wip-skoduri-dbstore-lc

rgw/dbstore: handle lc related state

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>

Merge pull request #43921 from aclamk/wip-aclamk-fix-omap-upgrade-fix

 BlueStore: Omap upgrade to per-pg fix fix

Reviewed-by: Igor Fedotov <igor.fedotov@croit.io>

qa/rgw: use local runner with cmdline radosgw_admin.py

Restore ability to run radosgw_admin.py unit standalone--improved
to use vstart_runner hooks.

Local rgwadmin(...) wrapper suggested as a cleanup in review by Casey.

Fixes: https://tracker.ceph.com/issues/52837
Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

os/bluestore: Fix omap upgrade to per-pg scheme

This is fix to regression introduced by fix to omap upgrade: https://github.com/ceph/ceph/pull/43687
The problem was that we always skipped first omap entry.
This worked fine with objects having omap header key.
For objects without header key we skipped first actual omap key.

Fixes: https://tracker.ceph.com/issues/53260
Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

os/bluestore: Add more legacy -> per PG upgrade tests

Signed-off-by: Adam Kupczyk <akupczyk@redhat.com>

doc/cephadm: core dumps

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>