rgw: remove user-caps from RGWOp_User_Modify

The adminOps API for Modify User checks for user-caps value in the
request but does handle that in later code path. Caps have seperate apis
add_cap and remove_cap already present to modify exisitng caps in user.
Hence removing unused cap variables in RGWOp_User_Modify::execute().

Signed-off-by: Jiffin Tony Thottan <jthottan@redhat.com>

Merge pull request #43115 from mgfritch/cephadm-rm-repo-no-podman

cephadm: skip podman check during `rm-repo`

Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43129 from cyx1231st/wip-seastore-optimize-onode

crimson/os/seastore: minimize the static size of onode_layout_t

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43106 from rzarzynski/wip-crimson-msgr-bind

crimson: add support for ms_bind_retry_{delay,count}

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

Merge pull request #43157 from liu-chunmei/seastore-binary-search

crimson/seastore:: add binary search for lba and omap iterator search

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: minimize the static size of onode_layout_t

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson: exclude oid when encode/decode object_info_t to/from object store

Signed-off-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore:: add binary search for lba and omap iterator search

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #43158 from athanatos/sjust/wip-seastore-omap

crimson/os/seastore: SeaStore::get_omap_iterator should seek_to_first

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43161 from rzarzynski/wip-crimson-fix-no-bs

crimson/os: allow to build crimson when WITH_BLUESTORE=OFF

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/os: add the preamble to futurized_store.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/os: allow to build crimson when WITH_BLUESTORE=OFF.

This lets to ensure nobody is accidentally linking with the alienized
version of `common`.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: switch try_bind() to bind().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: bind() respects now ms_bind_retry_{delay,count}.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: make try_bind() aware about address_not_available.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: decouple listen_ertr from bind_ertr.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: decouple listen_ertr from FixedCPUServerSocket.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43109 from guits/guits-cv-24044

ceph-volume: util/prepare fix osd_id_available()

crimson/os/seastore: SeaStore::get_omap_iterator should seek_to_first

Add debugging as well.

Signed-off-by: Samuel Just <sjust@redhat.com>

vstart.sh: default to msgr2 for crimson

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge pull request #43123 from liu-chunmei/lba_metrics_iter

crimson/seastore: add lba tree metrics for lba tree profiling

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>

crimson/seastore: add lba tree metrics for lba tree profiling

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #43152 from athanatos/sjust/wip-simplify-delayed

crimson/sesatore: simplify epm related transaction structures, fix some bugs

Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore/transaction: add trace debugging for write_set

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cached_extent: fix unused variable warning

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction_manager: fix delayed alloc addr in rewrite_logical_extent

Setting nlextent's to the addr of the original extent violates invariants
in the intrusive_set we're using for ExtentIndex and generally breaks the
assumption in cache that paddrs are unique until reuse.  This means we'll
do an extra update_mapping for now -- we can refactor SegmentCleaner to
avoid it later.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/scanner: fix budget_used ref in scan_valid_records

Fixes: https://tracker.ceph.com/issues/52582
Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/cache: more get_extent debugging

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/extent_placement_manager: clarify else branch delayed_alloc_or_ool_write

Also shorten long line.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore: replace fresh_block_list with inline/ool_block_list

With this change, Transaction::write_set is the union of mutually exclusive
lists
- delayed_alloc_list, inline_block_list, and mutated_block_list prior to
  calling ExtentPlacementManager::delayed_alloc_or_ool_write

and

- inline_block_list, ool_block_list, and mutated_block_list after
  calling ExtentPlacementManager::delayed_alloc_or_ool_write

Signed-off-by: Samuel Just <sjust@redhat.com>

Merge PR #43136 into master

* refs/pull/43136/head:
qa/tasks/kubeadm: change calico encap to IPIPCrossSubnet
qa/suites/orch/rook/smoke: add host networking to matrix
qa/tasks/rook: fix shadowing of config arg in rook_cluster()

Reviewed-by: Joseph Sawaya <jsawaya@redhat.com>

Merge pull request #42417 from rzarzynski/wip-bl-optimize-emptybptrs

common/bl, tests: optimize carriage handling in bufferlist::c_str()

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

qa/tasks/kubeadm: change calico encap to IPIPCrossSubnet

Signed-off-by: Sage Weil <sage@newdream.net>

crimson/os/seastore/cached_extent: assert !parent_index in ExtentIndex::insert

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/so/seastore: clarify ExtentIndex clear()

ExtentIndex::clear() needs to handle clearing parent_index.

Signed-off-by: Samuel Just <sjust@redhat.com>

crimson/os/seastore/transaction: remove delayed_set

delayed_set isn't necessary.  Delayed extents have distinct paddr segments
already, so it's simpler to have a single index for all pending extents.

Signed-off-by: Samuel Just <sjust@redhat.com>

tests/bl: verify the carriage handling in bufferlist::c_str().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/bl: abstract from _carriage checking in c_str().

This is intended to enchance readability and sligthly
generalize the previous, `_carriage`-aware optimization.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

common/bl: c_str() considers _carriage now.

It could be that `_carriage` points on empty bptr which is
also stored in the `_buffers` container -- this happens
when we'are trying to preserve the appendable space to not
waste memory and avoid new allocation.
Before the change this approach was imposing unncessary
rebuilding on `c_str()`. Now we take into consideration
the special case when a bufferlist has the extra, empty
bptr at its end.

Fixes: https://tracker.ceph.com/issues/51725
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

ceph-volume: util/prepare fix osd_id_available()

The current check only allows to request an OSD id that exists but
marked as 'destroyed'.
With this small fix, we can now use `--osd-id` with an id that doesn't
exist at all.

Fixes: https://tracker.ceph.com/issues/50880
Signed-off-by: Guillaume Abrioux <gabrioux@redhat.com>

Merge pull request #43141 from sebastian-philipp/doc-orch-rm

doc/cephadm: Removing a service

Reviewed-by: Michael Fritch <mfritch@suse.com>

Merge PR #42998 into master

* refs/pull/42998/head:
qa/tasks/rook: add testing for host label add/rm
mgr/rook: host add/rm label in rook orchestrator

Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

Merge pull request #43147 from krunerge/fix_typo

doc/dev/crimson/poseidonstore: fix typo

Reviewed-by: Kefu Chai <tchaikov@gmail.com>

Merge pull request #43063 from guits/guits-trailing-spaces

ceph-volume: remove trailing spaces in code

Merge pull request #42951 from ronen-fr/wip-ronenf-pg-dtor

osd/scrub: destruct the scrubber shortly before the PG is destructed

Reviewed-by: Kefu Chai <kchai@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>-
Reviewed-by: Neha Ojha <nojha@redhat.com>

doc/dev/crimson/poseidonstore: fix typo

Signed-off-by: krunerge <krunerge@tencent.com>

Merge pull request #42555 from hualongfeng/retire_error_change

librbd/cache/pwl/ssd: remove correct m_blocks_to_log_entries entry

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43038 from majianpeng/librbd-pwl-exclusive-lock

librbd: require exclusive lock for reads if pwl cache is enabled

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

librbd/cache/pwl/ssd: remove correct m_blocks_to_log_entries entry

When retiring, m_blocks_to_log_entries doesn't remove
corresponding write_entry (should be `*it` not `entry`)
that will be retired. It leads to read error. And
there should also consider discard entries.

Fixes: https://tracker.ceph.com/issues/52579
Signed-off-by: Feng Hualong <hualong.feng@intel.com>

Merge pull request #43143 from sebastian-philipp/further-reading

doc/cephadm: Add lots of links to other chapters

Reviewed-by: Zac Dover <zac.dover@gmail.com>

Merge pull request #41869 from ronen-fr/wip-ronenf-fmt-scrub

common, osd: add fmtlib formatting of some OSD types

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #43056 from Matan-B/wip-matanb-scrub-passkey

osd: Replacing friend declarations for passkey interface

Reviewed-by: Ronen Friedman <rfriedma@redhat.com>
Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #42684 from ronen-fr/wip-ronenf-scrub-token

osd/scrub: tag replica scrub messages to identify stale events

Reviewed-by: Neha Ojha <nojha@redhat.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

doc/cephadm: Add lots of links to other chapters

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

doc/cephadm: Removing a service

Signed-off-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43134 from ceph/debhelpermerged

debian/control: dh-systemd is part of debhelper now

Reviewed-by: Ken Dreyer <kdreyer@redhat.com>
Reviewed-by: Kefu Chai <tchaikov@gmail.com>

debian/control: dh-systemd is part of debhelper now

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=822670

Signed-off-by: David Galloway <dgallowa@redhat.com>

qa/tasks/rook: add testing for host label add/rm

This commit adds simple tests that check if the orchestrator
can add and remove labels from nodes.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

cephadm: skip podman check during `rm-repo`

allow the `rm-repo` command to succeed when podman is not installed

Signed-off-by: Michael Fritch <mfritch@suse.com>

Merge pull request #43104 from ceph/e2e-tests-documentation

doc/dev: specify location of e2e-tests script

Reviewed-by: Ernesto Puerta <epuertat@redhat.com>

Merge pull request #43053 from a16bitsysop/fsevents

mgr/dashboard: use -f for npm ci to skip fsevents error

Reviewed-by: a16bitsysop <NOT@FOUND>
Reviewed-by: Ernesto Puerta <epuertat@redhat.com>
Reviewed-by: Pere Diaz Bou <pdiazbou@redhat.com>

Merge pull request #42984 from majianpeng/pwl-ssd-race-bug

librbd/cache/pwl/ssd: fix a race between get_cache_bl() and remove_cache_bl()

Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #42883 from majianpeng/pwl-ssd-calc-allocated-bug

librbd/cache/pwl: fix m_bytes_{allocated,cached} calculation on reopen

Reviewed-by: Yin Congmin <congmin.yin@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43124 from xxhdx1985126/wip-om-log-fix

crimson/os/seastore: fix some debug outputs in omap manager

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>

crimson/os/seastore: fix some debug outputs in omap manager

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

Merge pull request #43108 from xxhdx1985126/wip-epm-roll-leak

crimson/os/seastore: fix epm rolling_segment leak when the transaction is interrupted

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43118 from rzarzynski/wip-crimson-assert-v2-on-bind

crimson/net: assert the address is v2 on attempt to bind.

Reviewed-by: Samuel Just <sjust@redhat.com>

mgr/rook: host add/rm label in rook orchestrator

This commit adds the functionality for adding/removing labels
to hosts using the rook orchestrator and orch host label add/rm.
The labels are added to the kubernetes node object as kubernetes
labels prefixed by "ceph-label/".

This commit also changes ceph orch host ls to display only the ceph
labels.

Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

doc/dev: specify location of e2e-tests script

I was just going through these steps myself, and I found it difficult to locate and run `run-frontend-e2e-tests.sh`. Specifying the location of it might make it clearer to others where the correct script is located.

Signed-off-by: Laura Flores <lflores@redhat.com>

Merge pull request #42912 from dsavineau/ceph_volume_lvm_active_no_systemd

ceph-volume: fix lvm activate --all --no-systemd

Merge pull request #43013 from rzarzynski/wip-msg-const-features_supported

msg: make Policy::features_supported static and constant.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #42853 from sseshasa/wip-fix-vstart-mon-permissions

mon/MonCap: Update osd profile to allow cmd to set iops capacity on mon db

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

crimson/net: assert the address is v2 on attempt to bind.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #43014 from dsavineau/ceph_volume_lvm_active_id_fsid_mandatory

ceph-volume: fix lvm activate arguments

Merge pull request #43079 from petrutlucian94/hv_addr_doc

doc/rbd: describe Hyper-V disk addressing limitations

Reviewed-by: Mykola Golub <mgolub@suse.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43086 from idryomov/wip-rbd-validate-pool-async

librbd: fix pool validation lockup

Reviewed-by: Deepika Upadhyay <dupadhya@redhat.com>
Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #43107 from SUSE/wip-fix-52553

pybind/rados: don't close watch in dealloc if already closed

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #43042 from dsavineau/ceph_volume_lvm_migrate_args

ceph-volume: fix lvm migrate without args

crimson/os/seastore: fix epm rolling_segment leak when the transaction is interrupted

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

crimson/os/seastore: better epm output

Signed-off-by: Xuehan Xu <xxhdx1985126@gmail.com>

osd: Replacing friend declerations for passkey interface

removing friend decleration from PrimaryLogScrub and ReplicaReservations

Signed-off-by: Matan Breizman <Matan.Brz@gmail.com>

pybind/rados: don't close watch in dealloc if already closed

Fixes: https://tracker.ceph.com/issues/52553
Signed-off-by: Tim Serong <tserong@suse.com>

crimson/net: s/do_bind/do_listen/ in SocketMessenger.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/net: simplify error handling in FixedCPUServerSocket::listen().

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #41803 from xxhdx1985126/wip-extent-placement-manager-2

crimson/os/seastore: add extent placement manager

Reviewed-by: Chunmei Liu <chunmei.liu@intel.com>
Reviewed-by: Yingxin Cheng <yingxin.cheng@intel.com>
Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #43105 from liu-chunmei/clean-alien-in-store-nbd

crimson/tools: clean alien in store-nbd

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/tools: clean alien in store-nbd

Signed-off-by: chunmei-liu <chunmei.liu@intel.com>

Merge pull request #42604 from sseshasa/wip-skip-osd-benchmark

osd: Add config option to skip running the osd benchmark during init and update documentation.

Reviewed-by: Josh Durgin <jdurgin@redhat.com>
Reviewed-by: Neha Ojha <nojha@redhat.com>

Merge pull request #43052 from dsavineau/ceph_volume_lvm_deactive_remove_all_arg

ceph-volume: remove --all ref from deactivate help

Merge pull request #43045 from sebastian-philipp/qa-tox-import-yaml

qa: tox.ini: verify yaml syntax

Reviewed-by: Sage Weil <sage@newdream.net>

ceph-volume: fix lvm migrate without args

When running the `lvm migrate` subcommand without any args then the
ceph-volume command fails with a stack trace.

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 151, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/migrate.py", line 520, in main
    self.migrate_osd()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/migrate.py", line 403, in migrate_osd
    if self.args.osd_id:
AttributeError: 'Migrate' object has no attribute 'args'

That's because we're exiting the parse_argv function but we continue to
execute the migrate_osd function. We should instead exit from the main function.

This update the parsing argument to have the same code than new-db and
new-wal classes.
Now the parsing is done in the make_parser function but the argv testing is
done in the main function allowing to exit the program and displaying the
help message when no arguments are provided.

Fixes: https://tracker.ceph.com/issues/51811
Signed-off-by: Dimitri Savineau <dsavinea@redhat.com>

Merge pull request #42859 from mgfritch/cephadm-check-host-no-engine

mgr/cephadm: show unhandled exceptions during host add

Reviewed-by: Adam King <adking@redhat.com>

Merge pull request #42989 from sebastian-philipp/post-remove-osd-key-missing

mgr/cephadm: Add OSDService.post_remove()

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Sage Weil <sage@newdream.net>

Merge pull request #43010 from mgfritch/cephadm-log-thread-ident

cephadm: add thread ident to log messages

Reviewed-by: Adam King <adking@redhat.com>
Reviewed-by: Juan Miguel Olmo Martínez <jolmomar@redhat.com>
Reviewed-by: Sebastian Wagner <sewagner@redhat.com>

Merge pull request #43048 from dsavineau/ceph_volume_lvm_migrate_no_systemd

ceph-volume: support no_systemd with lvm migrate

Merge pull request #43051 from dsavineau/ceph_volume_lvm_migrate_docs

doc/ceph-volume: add lvm migrate/new-db/new-wal

Merge pull request #43050 from dsavineau/fix_raw_list_with_lsblk

ceph-volume: fix raw list with logical partition

librbd: drop ValidatePoolRequest::m_op_work_queue

It is unused now.

Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

librbd: fix pool validation lockup

Concurrent rbd_pool_init() or rbd_create() operations on an unvalidated
(uninitialized) pool trigger a lockup in ValidatePoolRequest state
machine caused by blocking selfmanaged_snap_{create,remove}() calls.
There are two reactor threads by default (librados_thread_count) but we
effectively need N + 1 reactor threads for N concurrent pool validation
requests, especially for small N.

Switch to aio_selfmanaged_snap_{create,remove}().  At the time this
code was initially written, these aio variants weren't available.  The
workqueue offload introduced later worked prior to the move to asio in
pacific.

Fixes: https://tracker.ceph.com/issues/52537
Signed-off-by: Ilya Dryomov <idryomov@gmail.com>

Merge pull request #43006 from CongMinYin/fix-assert-in-handle_flushed_sync_point

librbd/cache/pwl: don't clear next_sync_point_entry prematurely

Reviewed-by: Jianpeng Ma <jianpeng.ma@intel.com>
Reviewed-by: Ilya Dryomov <idryomov@gmail.com>