test/rgw: add --gen to radosgw-admin/help.t

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generations to error repo with binary format

adds a backward-compatible binary encoding for error repo keys that can
contain a generation number along with the bucket and shard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add generation support to BILog admin APIs

Signed-off-by: Shilpa Jagannath <smanjara@redhat.com>

rgw: remove rgw_bucket_shard_sync_info::full_marker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: disable backoff on data sync error_retry_time

this backoff is triggered often by the per-bucket lease for full sync,
and causes tests to fail with checkpoint timeouts

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: rgw_multisite_tests task loads default plugins

the ability to filter tests by attribute is provided by the
nose.plugins.attrib plugin, which wasn't being loaded by default

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: disable multisite tests for 'bucket sync disable'

Signed-off-by: Casey Bodley <cbodley@redhat.com>

qa/rgw: add missing meta checkpoint to test_multipart_object_sync

if metadata sync hasn't finished, the 'bucket checkpoint' commands may
not find its bucket info

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSyncBucketCR holds lease over Init state too

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: fix up BucketShardIncrementalSync log message

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: move full sync from SyncBucketShard to SyncBucket

renamed ListBucketShardCR to ListRemoteBucketCR and removed the shard-id
parameter

renamed BucketFullSyncShardMarkerTrack to BucketFullSyncMarkerTrack,
which now updates the bucket-level rgw_bucket_sync_status

renamed BucketShardFullSyncCR to BucketFullSyncCR

BucketSyncCR now takes a bucket-wide lease during full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync checkpoint' waits for full sync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

radosgw-admin: 'bucket sync status' displays new full sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add rgw_read_bucket_full_sync_status()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: InitBucketShardSyncStatus always sets state to Incremental

full sync happens as the bucket level, so the shards will always start
in StateIncrementalSync

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: split SyncBucket from SyncBucketShard

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add InitBucketFullSyncStatusCR

a coroutine to initialize a bucket for full sync using a new bucket-wide
sync status object

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosWriteAttrsCR supports exclusive create

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: system objects can set exclusive on set_attrs()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add exclusive flag to RGWSimpleRadosWriteCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: generalize error handling in RGWShardCollectCR

RGWShardCollectCR was hard-coded to ignore ENOENT errors and print a
'failed to fetch log status' error message. this moves that logic into a
handle_result() virtual function. it also exposes the member variables
'status' and 'max_concurrent' as protected, so they can be consulted or
modified by overrides of handle_result() and spawn_next()

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add sync_bucket_shard_cr() factory function

allows other code to spawn this coroutine without having the class
definition

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to RGWSyncBucketShardCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add full_status_oid() for buckets

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: use const for string constants in rgw_data_sync.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to inc_status_oid

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: add data structures for bucket sync status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: rename to rgw_read_bucket_inc_sync_status

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: remove unused RGWRunBucketsSyncBySourceCR

Signed-off-by: Casey Bodley <cbodley@redhat.com>

rgw: RGWSimpleRadosReadCR copies out objv_tracker

Signed-off-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #43141 from sebastian-philipp/doc-orch-rm

doc/cephadm: Removing a service

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #42998 into master

* refs/pull/42998/head:
qa/tasks/rook: add testing for host label add/rm
mgr/rook: host add/rm label in rook orchestrator

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #43147 from krunerge/fix_typo

doc/dev/crimson/poseidonstore: fix typo

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43063 from guits/guits-trailing-spaces

ceph-volume: remove trailing spaces in code

Merge pull request #42951 from ronen-fr/wip-ronenf-pg-dtor

osd/scrub: destruct the scrubber shortly before the PG is destructed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>-
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev/crimson/poseidonstore: fix typo

Signed-off-by: krunerge <krunerge@tencent.com>

Merge pull request #42555 from hualongfeng/retire_error_change

librbd/cache/pwl/ssd: remove correct m_blocks_to_log_entries entry

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43038 from majianpeng/librbd-pwl-exclusive-lock

librbd: require exclusive lock for reads if pwl cache is enabled

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd: remove correct m_blocks_to_log_entries entry

When retiring, m_blocks_to_log_entries doesn't remove
corresponding write_entry (should be `*it` not `entry`)
that will be retired. It leads to read error. And
there should also consider discard entries.

Fixes: https://tracker.ceph.com/issues/52579
Signed-off-by: Feng Hualong <hualong.feng@intel.com>

Merge pull request #43143 from sebastian-philipp/further-reading

doc/cephadm: Add lots of links to other chapters

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41869 from ronen-fr/wip-ronenf-fmt-scrub

common, osd: add fmtlib formatting of some OSD types

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43056 from Matan-B/wip-matanb-scrub-passkey

osd: Replacing friend declarations for passkey interface

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42684 from ronen-fr/wip-ronenf-scrub-token

osd/scrub: tag replica scrub messages to identify stale events

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

doc/cephadm: Add lots of links to other chapters

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: Removing a service

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43134 from ceph/debhelpermerged

debian/control: dh-systemd is part of debhelper now

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

debian/control: dh-systemd is part of debhelper now

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=822670

Signed-off-by: David Galloway <dgallowa@redhat.com>

qa/tasks/rook: add testing for host label add/rm

This commit adds simple tests that check if the orchestrator
can add and remove labels from nodes.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

Merge pull request #43104 from ceph/e2e-tests-documentation

doc/dev: specify location of e2e-tests script

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #43053 from a16bitsysop/fsevents

mgr/dashboard: use -f for npm ci to skip fsevents error

Reviewed-by: a16bitsysop <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #42984 from majianpeng/pwl-ssd-race-bug

librbd/cache/pwl/ssd: fix a race between get_cache_bl() and remove_cache_bl()

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42883 from majianpeng/pwl-ssd-calc-allocated-bug

librbd/cache/pwl: fix m_bytes_{allocated,cached} calculation on reopen

Reviewed-by: Yin Congmin <congmin.yin@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43124 from xxhdx1985126/wip-om-log-fix

crimson/os/seastore: fix some debug outputs in omap manager

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore: fix some debug outputs in omap manager

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43108 from xxhdx1985126/wip-epm-roll-leak

crimson/os/seastore: fix epm rolling_segment leak when the transaction is interrupted

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43118 from rzarzynski/wip-crimson-assert-v2-on-bind

crimson/net: assert the address is v2 on attempt to bind.

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/rook: host add/rm label in rook orchestrator

This commit adds the functionality for adding/removing labels
to hosts using the rook orchestrator and orch host label add/rm.
The labels are added to the kubernetes node object as kubernetes
labels prefixed by "ceph-label/".

This commit also changes ceph orch host ls to display only the ceph
labels.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

doc/dev: specify location of e2e-tests script

I was just going through these steps myself, and I found it difficult to locate and run `run-frontend-e2e-tests.sh`. Specifying the location of it might make it clearer to others where the correct script is located.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #42912 from dsavineau/ceph_volume_lvm_active_no_systemd

ceph-volume: fix lvm activate --all --no-systemd

Merge pull request #43013 from rzarzynski/wip-msg-const-features_supported

msg: make Policy::features_supported static and constant.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42853 from sseshasa/wip-fix-vstart-mon-permissions

mon/MonCap: Update osd profile to allow cmd to set iops capacity on mon db

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/net: assert the address is v2 on attempt to bind.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43014 from dsavineau/ceph_volume_lvm_active_id_fsid_mandatory

ceph-volume: fix lvm activate arguments

Merge pull request #43079 from petrutlucian94/hv_addr_doc

doc/rbd: describe Hyper-V disk addressing limitations

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43086 from idryomov/wip-rbd-validate-pool-async

librbd: fix pool validation lockup

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #43107 from SUSE/wip-fix-52553

pybind/rados: don't close watch in dealloc if already closed

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #43042 from dsavineau/ceph_volume_lvm_migrate_args

ceph-volume: fix lvm migrate without args

crimson/os/seastore: fix epm rolling_segment leak when the transaction is interrupted

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: better epm output

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

osd: Replacing friend declerations for passkey interface

removing friend decleration from PrimaryLogScrub and ReplicaReservations

Signed-off-by: Matan Breizman <Matan.Brz@gmail.com>

pybind/rados: don't close watch in dealloc if already closed

Fixes: https://tracker.ceph.com/issues/52553
Signed-off-by: Tim Serong <tserong@suse.com>

Merge pull request #41803 from xxhdx1985126/wip-extent-placement-manager-2

crimson/os/seastore: add extent placement manager

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43105 from liu-chunmei/clean-alien-in-store-nbd

crimson/tools: clean alien in store-nbd

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/tools: clean alien in store-nbd

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #42604 from sseshasa/wip-skip-osd-benchmark

osd: Add config option to skip running the osd benchmark during init and update documentation.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43052 from dsavineau/ceph_volume_lvm_deactive_remove_all_arg

ceph-volume: remove --all ref from deactivate help

Merge pull request #43045 from sebastian-philipp/qa-tox-import-yaml

qa: tox.ini: verify yaml syntax

Reviewed-by: Sage Weil <sage@newdream.net>

ceph-volume: fix lvm migrate without args

When running the `lvm migrate` subcommand without any args then the
ceph-volume command fails with a stack trace.

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 151, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/migrate.py", line 520, in main
    self.migrate_osd()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/migrate.py", line 403, in migrate_osd
    if self.args.osd_id:
AttributeError: 'Migrate' object has no attribute 'args'

That's because we're exiting the parse_argv function but we continue to
execute the migrate_osd function. We should instead exit from the main function.

This update the parsing argument to have the same code than new-db and
new-wal classes.
Now the parsing is done in the make_parser function but the argv testing is
done in the main function allowing to exit the program and displaying the
help message when no arguments are provided.

Fixes: https://tracker.ceph.com/issues/51811
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #42859 from mgfritch/cephadm-check-host-no-engine

mgr/cephadm: show unhandled exceptions during host add

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #42989 from sebastian-philipp/post-remove-osd-key-missing

mgr/cephadm: Add OSDService.post_remove()

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #43010 from mgfritch/cephadm-log-thread-ident

cephadm: add thread ident to log messages

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43048 from dsavineau/ceph_volume_lvm_migrate_no_systemd

ceph-volume: support no_systemd with lvm migrate

Merge pull request #43051 from dsavineau/ceph_volume_lvm_migrate_docs

doc/ceph-volume: add lvm migrate/new-db/new-wal

Merge pull request #43050 from dsavineau/fix_raw_list_with_lsblk

ceph-volume: fix raw list with logical partition

librbd: drop ValidatePoolRequest::m_op_work_queue

It is unused now.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: fix pool validation lockup

Concurrent rbd_pool_init() or rbd_create() operations on an unvalidated
(uninitialized) pool trigger a lockup in ValidatePoolRequest state
machine caused by blocking selfmanaged_snap_{create,remove}() calls.
There are two reactor threads by default (librados_thread_count) but we
effectively need N + 1 reactor threads for N concurrent pool validation
requests, especially for small N.

Switch to aio_selfmanaged_snap_{create,remove}().  At the time this
code was initially written, these aio variants weren't available.  The
workqueue offload introduced later worked prior to the move to asio in
pacific.

Fixes: https://tracker.ceph.com/issues/52537
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43006 from CongMinYin/fix-assert-in-handle_flushed_sync_point

librbd/cache/pwl: don't clear next_sync_point_entry prematurely

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43074 from wangshuaich/rbd-children-crash-fix

librbd: fix use-after-free on ictx in list_descendants()

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

doc/rbd: describe Hyper-V disk addressing limitations

Hyper-V identifies passthrough VM disks by number instead of SCSI ID, although
the disk number can change across host reboots. This means that the VMs can end
up using incorrect disks after rebooting the host, which is an important
security concern. This issue also affects iSCSI and Fibre Channel disks.

We're going to document this Hyper-V limitation along with possible
workarounds.

Signed-off-by: Lucian Petrut <lpetrut@cloudbasesolutions.com>

crimson/os/seastore: better debug outputs

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: allow getting empty delta from omap extents

there are chances that a transaction gets invalidated after some of
its extents have their delta prepared.

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: add ExtentIndex::clear_and_dispose

When a transaction is interrupted and needs to repeat, its reset_preserve_handle() method
is called to clear various extent set and list. The problem is the clear operation call
ExtentIndex::clear() instead of ExtentIndex::erase(), which would leave CachedExtent::parent_index
still pointing to the write_set/delayed_set while the CachedExtent is no longer linked to
those sets. This would make CachedExtent::~CachedExtent() to try to erase itself from
CachedExtent::parent_index even it's not linked, which would cause failures in the successive
operations

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/store_nbd: involve the extent placement manager and the scanner

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: make fresh extents go though the extent placement manager

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: deal with transaction conflicts when doing mkfs

Cleaner transactions can conflict with mkfs transactions, retry mkfs transactions
when that happens

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: set journal_tail_target during replay

This is a bug fix, otherwise if crimson-osd boot up multiple times without
filling up more than one segment, segments may be used up and can't be
reclaimed as they would have the same journal tail

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: set root block to clean at the initialization phase

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: exclude ool extents from the judgement is_in_journal

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: use extent placement manager to rewrite extents

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: refactor extent rewrite procedure

Move logical extents' rewrite from LBManager to TransactionManager

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>