mgr/orchestrator: "addr" is optional for constructing InventoryNode

this addresses a regression introduced by 5276871e15

Signed-off-by: Kefu Chai <kchai@redhat.com>

qa/tasks/mgr/orch: s/service ls/ps/

to fix the broken test of "test_load_data". it's a regression introduced
by aacc9a650f052fd5be543e9265ec94833b8e8bb3

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge PR #33343 into master

* refs/pull/33343/head:
qa/suites/rados/cephadm/upgrade: add simple upgrade test
qa/tasks/cephadm: improve shell subcommand

Reviewed-by: Michael Fritch <mfritch@suse.com>

qa/suites/rados/cephadm/upgrade: add simple upgrade test

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: improve shell subcommand

- pass teuth job params through environment
- run commands via bash -c

Signed-off-by: Sage Weil <sage@redhat.com>

Merge PR #33073 into master

* refs/pull/33073/head:
qa/suites/rados/cephadm: deploy prometheus.a
mgr/cephadm: implement prometheus add/update
mgr/cephadm: teach _create_daemon how to provision prometheus
mgr/orch: add prom hooks

Reviewed-by: Patrick Seidensal <pseidensal@suse.com>

Merge pull request #33332 from jan--f/c-v-filestore-zap-fix

ceph-volume: don't remove vg twice when zapping filestore

Merge pull request #33320 from jan--f/c-v-fix-filestore-journal-size

ceph-volume: pass journal_size as Size not string

ceph-volume: don't remove vg twice when zapping filestore

Signed-off-by: Jan Fajerski <jfajerski@suse.com>
Fixes: https://tracker.ceph.com/issues/44149

Merge pull request #32679 from rzarzynski/wip-crimson-watchnotify_part1

crimson: add support for watch / notify, part 1

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33296 from tchaikov/wip-crimson-cmake

cmake: compile crimson-auth with crimson::cflags

Reviewed-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge pull request #33097 from dillaman/wip-43933

librbd: tweak deep-copy to avoid creating last snapshot until sync is complete

Reviewed-by: Mykola Golub <mgolub@suse.com>

Merge pull request #33298 from sebastian-philipp/debian-rook-jsonpatch

debian: add python3-jsonpatch as dependency

Reviewed-by: Kefu Chai <kchai@redhat.com>

ceph-volume: pass journal_size as Size not string

Fixes: https://tracker.ceph.com/issues/44148
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #32174 from ronen-fr/admin_commands_3

common,crimson: supporting admin-socket commands

Reviewed-by: Samuel Just <sjust@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

debian: add python3-jsonpatch as dependency

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>

Merge pull request #33283 from jan--f/c-v-zap-on-vg-without-lv

ceph-volume: avoid calling zap_lv with a LV-less VG

Merge pull request #32690 from rosinL/wip-timeevent

msg/async/dpdk: destroy fd in do_request

Reviewed-by: Kefu Chai <kchai@redhat.com>

cmake: compile crimson-auth with crimson::cflags

* move auth related stuff into crimson/CMakeLists.txt, so we can
  link them against crimson::cflags, which populates the necessary
  definitions and other cxx flags when building these source files.
* expose crimson::cflags as a public library of crimson, as
  crimson-osd links against crimson. and the cflags should be populated
  to crimson-osd, otherwise they are compiled with different compiler
  options.

Signed-off-by: Kefu Chai <kchai@redhat.com>

Merge pull request #33084 from tchaikov/wip-crimson-pg-coll

crimson/os: init PG with pg coll not meta coll

Reviewed-by: Samuel Just <sjust@redhat.com>

Merge pull request #33241 from nagpavan-chilakam/patch-1

doc/architecture.rst: fix a typo in EC section

Reviewed-by: Kefu Chai <kchai@redhat.com>

Merge PR #33194 into master

* refs/pull/33194/head:
qa: add tests for mds_join_fs cluster affinity
qa: update cluster warning message for removed MDS
doc: add section on new mds_join_fs behavior
mon/MDSMonitor: enforce mds_join_fs cluster affinity
mon/MDSMonitor: use type of info.rank or mds_rank_t
qa: accept operation on current fs status
qa: add method to enable multifs
qa: fix nested generator use
qa: manage config changes through mons

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Kefu Chai <kchai@redhat.com>

crimson/osd: fix the lifetime management of OpsExecuter.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: dispatch MWatchNotify on watch removal.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: improve debugs in OpsExecuter::do_op_notify_ack.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: implement the complete phase of Notify propagation.

Q: How to test?
A:  * `${CEPHBIN:-bin}/rados -p rbd touch moj_watch`
    * `${CEPHBIN:-bin}/rados -p rbd watch moj_watch`
    * `${CEPHBIN:-bin}/rados -p rbd notify moj_watch test2`

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: dispatch MWatchNotify on CEPH_OSD_OP_NOTIFY.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

messages: MWatchNotify takes notifier_gid during construction.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: create_n_propagate

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: pass real connection in OpsExecuter::do_op_watch_subop_watch()

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: minor fix formatting in Watch.cc.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: clean-up raw connection checks.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

osd: unify connected() and is_connected() of Watch.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson/osd: drop the legacy path for undecodable notify acks.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: implement CEPH_OSD_OP_NOTIFY_ACK.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: implement CEPH_OSD_OP_NOTIFY.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: implement CEPH_OSD_WATCH_OP_PING.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: implement CEPH_OSD_WATCH_OP_RECONNECT.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: add execution units for WATCH and UNWATCH.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

crimson: introduce watchers to ObjectContext.

Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Merge PR #33244 into master

* refs/pull/33244/head:
doc/mgr/orchestrator_modules: update docs
mgr/orch: break out apply_$type() in orchestrator.py interface
qa/tasks/cephadm: adapt to new cli
mgr/orch: add --force arg to 'daemon rm'
mgr/orch: fix host set-addr prefix
mgr/orch: service action cleanup
mgr/orch: prevent rm mon|mgr
mgr/orch: service rm -> rm
mgr/cephadm: add mon
mgr/orch: add mon hook
mgr/cephadm: add mgr
mgr/orch: add 'daemon add mgr'
mgr/orch: orch * add -> orch daemon add *
mgr/volumes: use apply_mds
mgr/orch: move to service_apply; remove update_foo()
mgr/rook: move to remove_service()
mgr/orch: remove old remove_{osds,rgw,mds,nfs,...} methods
mgr/orch: test remove_daemons and remove_service

Reviewed-by: Sebastian Wagner <swagner@suse.com>

qa/suites/rados/cephadm: deploy prometheus.a

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: implement prometheus add/update

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: teach _create_daemon how to provision prometheus

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add prom hooks

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #30862 from zhangsw/fix-rgw-lc-repeated

rgw: set bucket attr twice when delete lifecycle config.

Reviewed-by: Daniel Gryniewicz <dang@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge PR #33248 into master

* refs/pull/33248/head:
cephadm: fix deploy crash when no `args.fsid`

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Sebastian Wagner <swagner@suse.com>

Merge pull request #31088 from joke-lee/rgw-storage-class-apend-obj

rgw: set correct storage class for append

Reviewed-By: Casey Bodley <cbodley@redhat.com>

Merge pull request #31661 from joke-lee/sts-args

rgw: sts: add all http args to req_info

Reviewed-by: Casey Bodley <cbodley@redhat.com>
Reviewed-By: Abhishek Lekshmanan <abhishek@suse.com>

Merge pull request #33053 from tpsilva/add_max_connections

rgw: make max_connections configurable in beast

Reviewed-by: Casey Bodley <cbodley@redhat.com>

Merge pull request #33165 from clwluvw/policy-subuser

rgw: Add support bucket policy for subuser

Reviewed-By: Adam Emersen <aemerson@redhat.com>

Merge pull request #32636 from ivancich/wip-avoid-0-length-ordered-list-result

rgw: address 0-length listing results when non-vis entries dominate

Reviewed-by: Matt Benjamin <mbenjamin@redhat.com>

Merge pull request #32805 from smithfarm/wip-cmake-logging

rpm: add cmake_verbose_logging switch

Reviewed-by: David Disseldorp <ddiss@suse.com>
Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge PR #29427 into master

* refs/pull/29427/head:
mgr/rook: Make use of rook-client-python when talking to Rook
cmake: Integrate Rook client generation
mgr/rook: Automatically generate Rook client interface
Add submodule to rook-client-python.git

Reviewed-by: Sage Weil <sage@redhat.com>
Reviewed-by: Juan Miguel Olmo <jolmomar@redhat.com>

ceph-volume: avoid calling zap_lv with a LV-less VG

Fixes: https://tracker.ceph.com/issues/44125
Signed-off-by: Jan Fajerski <jfajerski@suse.com>

crimson/common: add show_config(Formatter) to ConfigProxy API

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

common/cmdparse: pass cmdmap by const reference

Signed-off-by: Kefu Chai <kchai@redhat.com>

common,crimson: supporting admin-socket commands

- basic infrastructure:
  - the seastar::thread that handles commands arriving over the UNIX_domain socket
  - command dispatching
- the implementation of some commands.
- simplified locking scheme: access to the API blocks (the registration elements) is
  now under one course-grained RW-lock.
- OSD shutdown support;

Signed-off-by: Ronen Friedman <rfriedma@redhat.com>

qa: add tests for mds_join_fs cluster affinity

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: update cluster warning message for removed MDS

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc: add section on new mds_join_fs behavior

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/MDSMonitor: enforce mds_join_fs cluster affinity

Before this, "mds_join_fs" config enforced a preference for the standby
the monitors would select. Now the monitors actively enforce this
by purposefully removing an MDS wither lower "affinity". An MDS standby
has highest affinity if its mds_join_fs is the file system in question
or a vanilla standby (no mds_join_fs).

Fixes: https://tracker.ceph.com/issues/43392
Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

mon/MDSMonitor: use type of info.rank or mds_rank_t

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: accept operation on current fs status

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: add method to enable multifs

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: fix nested generator use

Apparently Python doesn't allow nested generator calls. Hurray.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

qa: manage config changes through mons

This provides a generic framework for modifying Ceph configuration
changes in tests through the monitors rather than the asok interface or
local ceph.conf changes. Any changes are reverted during test teardown.

A future patch will convert existing tests manipulating the local
ceph.conf or admin socket.

Signed-off-by: Patrick Donnelly <pdonnell@redhat.com>

doc/mgr/orchestrator_modules: update docs

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: break out apply_$type() in orchestrator.py interface

Also clean up the method descriptions.

Signed-off-by: Sage Weil <sage@redhat.com>

qa/tasks/cephadm: adapt to new cli

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add --force arg to 'daemon rm'

This is needed when removing mons.

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: fix host set-addr prefix

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: service action cleanup

- no 'service' prefix
- single name arg (type[.id...])

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: prevent rm mon|mgr

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: service rm -> rm

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: add mon

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add mon hook

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/cephadm: add mgr

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: add 'daemon add mgr'

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: orch * add -> orch daemon add *

Adjust help string too

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/volumes: use apply_mds

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #32609 from joke-lee/rgw-website-43563

rgw: ReplaceKeyPrefixWith and ReplaceKeyWith can not set at the same …

Merge pull request #33247 from tchaikov/wip-cmake-cleanup

cmake: do not link against unused libs

Reviewed-by: Casey Bodley <cbodley@redhat.com>

mgr/orch: move to service_apply; remove update_foo()

Signed-off-by: Sage Weil <sage@redhat.com>

Merge pull request #33219 from dillaman/wip-44066-2

rbd: fixed additional issues with CEPH_ARGS processing

Reviewed-by: Mykola Golub <mgolub@suse.com>

rpm: add cmake_verbose_logging switch

This switch sets CMAKE_VERBOSE_MAKEFILE and causes the CMakeOutput.log
and CMakeError.log files to be dumped after cmake execution finishes.

Signed-off-by: Nathan Cutler <ncutler@suse.com>

Merge pull request #33182 from smithfarm/wip-cbs-basic-auth

ceph-backport.sh: use Basic Authentication instead of access_token

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

mgr/rook: move to remove_service()

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: remove old remove_{osds,rgw,mds,nfs,...} methods

Signed-off-by: Sage Weil <sage@redhat.com>

mgr/orch: test remove_daemons and remove_service

Signed-off-by: Sage Weil <sage@redhat.com>

librbd: deep-copy objects from specified start position

Only read data from from after the specified start position and copy it
to the specified starting write position in the destination.

Fixes: https://tracker.ceph.com/issues/43933
Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: ensure deep-copy snapshot map includes all destination snap ids

When deep-copying from an arbitrary start snapshot id, the snap sequence
will be missing all older snapshots. Additionally, snapshot types that
are not deep-copied still need to be included in the destination snap
map.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: deep-copy snapshots from a specified start/end position

Allow the snapshots to be arbitrarily copied from any source image
start/end snapshot ids. If the end snapshot is not a user-snapshot,
it will associate to the destination image HEAD revision.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: deep-copy should accept a lower-bound for the destination snap_id

For snapshot-based mirroring, we will want to prevent the modification of
snapshots below the last sync snapshot and to prevent the copying of data
below that lower-bound as well. This commit just adds the new parameter and
future commits will update the snapshot and object copy behavior.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

cls/rbd: store primary to non-primary snap mapping in mirror snapshot

The mirror non-primary snapshot namespace already stores the copy
progress so might as well store the required snapshot mapping
table.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: pass mirror image mode from create image to enable mirror

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

librbd: expose mirror uuid via the API

The mirror snapshot primary vs non-primary state can be
determined by looking at the primary_mirror_uuid within
the mirror snapshot namespace.

Signed-off-by: Jason Dillaman <dillaman@redhat.com>

Merge pull request #33173 from smithfarm/wip-bri-basic-auth

backport-resolve-issue: use Basic Authentication instead of "access_token"

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

Merge pull request #33105 from smithfarm/wip-44014

backport-resolve-issue: populate tracker_description method

Reviewed-by: Jan Fajerski <jfajerski@suse.com>

doc/architecture.rst: fix a typo in EC section

In erasure coding section under Architecture, there is a mention of k = 2 + M =1 for
number of data copies and redundancy copies respectively, which is a bit ambiguous.
The proposal is to change to k = 2, M = 1 as the + sign is not needed here.

Signed-off-by: Nag Pavan Chilakam <nagpavan.chilakam@gmail.com>